Documentation System - Technical Implementation Presentation

Audience: Development Team
Duration: 30-45 minutes
Date: 2025-11-28

---

Agenda

1. Overview (5 min) - What we built and why
2. Technical Architecture (10 min) - How it works
3. Implementation Flow (10 min) - From code to published docs
4. Live Demonstration (10 min) - Azure Static Web App & Elasticsearch
5. Developer Experience (5 min) - How to use it
6. Q&A (5-10 min)

---

1. Overview: What We Built

The Problem

• Documentation scattered across repos
• No central knowledge base
• Hard to find information
• Documentation often outdated
• No IDE integration

The Solution: Distributed Authoring, Centralized Publishing (Docusaurus)

┌────────────────────────────────────────────────────────────────────────────┐
│                    DOCUMENTATION LAYERS                                   │
├────────────────────────────────────────────────────────────────────────────┤
│  1. SOURCE LAYER      → Code Comments + Markdown Articles (in each repo)   │
│  2. EXPORT LAYER      → Each source repo generates its own Markdown docs   │
│  3. AGGREGATION LAYER → Central repo imports docs from all source repos    │
│  4. BUILD LAYER       → Docusaurus builds unified static site              │
│  5. DISTRIBUTION      → Azure Static Web Apps + Central Repo + Elasticsearch│
│  6. ACCESS LAYER      → Web Browser + MCP + Search API                    │
└────────────────────────────────────────────────────────────────────────────┘

Key Change: Each source repository is responsible for generating its own documentation (API and articles) as Markdown. The central documentation repository then imports these generated docs from all source repos and builds the unified Docusaurus site for publishing and search.

Migration Note

We have migrated from DocFX to Docusaurus for static site generation. This enables a more modern, React-based documentation experience, easier integration with JavaScript/React repos, and improved extensibility.

Multi-Repo, Multi-Language Support

The system now supports both C#/.NET and JavaScript/Node/React repositories, and aggregates documentation from two integrated repositories:

Language	Comment Style	Extraction Tool	Output Format
C#	XML (///)	Custom scripts/tools	Markdown
JavaScript	JSDoc (/** */)	jsdoc-to-markdown	Markdown
Both	Markdown articles	Docusaurus build	HTML

Key Difference: JavaScript repos use per-file documentation generation, creating one markdown file per source file for better organization. All documentation is now unified and published via Docusaurus.

Current Status

✅ 2 repositories integrated:

• Jupiter.Market.Elastic.Comparables
• Jupiter.Market.Elastic.Titles (or update with actual second repo name)

✅ Fully operational:

• Central documentation repository (Docusaurus-based)
• Elasticsearch index (jupiter-documentation)
• Azure Static Web App deployment
• MCP server integration

---

2. Technical Architecture

Component Breakdown

2.1 Source Layer: Where Documentation Lives

XML Comments in C# Code

/// <summary>
/// Orchestrates the end-to-end indexing pipeline.
/// </summary>
/// <remarks>
/// See detailed article: <a href="~/articles/indexer-service-logic.md">
/// IndexerService Logic</a>.
/// </remarks>
/// <param name="dealId">The deal ID to index.</param>
/// <returns>Success status and error message if applicable.</returns>
public async Task<(bool, string)> IndexDealByIdAsync(
    Guid dealId, 
    CancellationToken cancellationToken)

JSDoc Comments in JavaScript Code

/**
 * Orchestrates the end-to-end indexing pipeline.
 * @param {string} dealId - The deal ID to index
 * @param {Object} options - Processing options
 * @returns {Promise<{success: boolean, error: string}>} Result object
 * @async
 * @example
 * const result = await indexDealById('123', { validate: true });
 * if (result.success) console.log('Indexed successfully');
 */
async function indexDealById(dealId, options = {}) {
    // Implementation
}

Markdown Articles in /docs

docs/
├── introduction.md
├── getting-started.md
├── deployment-guide.md
└── articles/
    ├── indexer-service-logic.md
    ├── data-service-logic.md
    └── elastic-index-service-logic.md

2.2 Export & Aggregation: Source Repos → Central Repo

What Happens:

1. Each source repo (C# or JS) generates its own Markdown documentation (API and articles) using its own pipeline or scripts.
2. The central documentation repo pulls or receives these generated docs (e.g., via submodules, CI artifact download, or direct copy).
3. The central repo aggregates all imported docs into a single docs/ structure for Docusaurus.

2.3 Build Layer: Docusaurus

What Docusaurus Does:

1. Builds a unified static site from the aggregated Markdown docs
2. Supports both API reference and conceptual docs from all repos
3. Generates a modern, React-based static website
4. Supports custom themes, plugins, and search

Configuration (docusaurus.config.js):

module.exports = {
  title: 'Nimbus Docs',
  url: 'https://[your-site].azurestaticapps.net',
  baseUrl: '/',
  presets: [
    [
      '@docusaurus/preset-classic',
      {
        docs: {
          path: 'docs',
          routeBasePath: '/docs',
        },
        // ...
      },
    ],
  ],
  // ...
};

2.3 Distribution Layer: Azure DevOps Pipeline

Pipeline Stages (azure-pipelines-docusaurus.yml):

# In each source repo:
Stage 1: GenerateDocs
  ├─ Build project
  ├─ Extract comments to Markdown (API + articles)
  └─ Publish generated docs as pipeline artifact or push to shared location

# In central docs repo:
Stage 2: ImportDocs
  ├─ Download/copy generated docs from all source repos
  ├─ Aggregate into /docs structure
  └─ Validate import

Stage 3: BuildDocumentation
  ├─ Build Docusaurus site (`npm run build`)
  └─ Publish artifact

Stage 4: IndexToElastic (main only)
  ├─ Download documentation artifact
  ├─ Run indexing script
  └─ Index all HTML/Markdown to Elasticsearch

Stage 5: PublishToCentral (main only)
  ├─ Push to Azure Static Web Apps
  └─ Optionally push to central repo

2.4 Access Layer: Multiple Consumption Points

1. Azure Static Web App (Docusaurus)

• URL: https://[your-static-web-app].azurestaticapps.net
• Hosts complete documentation site (Docusaurus-powered)
• Built-in search functionality
• Responsive, modern UI

2. Elasticsearch Index

• Index: jupiter-documentation
• Full-text search across all repos
• Metadata-rich documents
• Fast query performance

3. MCP Server Integration

• Server: elasticsearch-jupiter
• IDE integration (VS Code, Cursor, etc.)
• Natural language queries
• Context-aware results

---

3. Implementation Flow: From Code to Published Docs

The Complete Journey (Distributed Authoring, Central Import)

┌────────────────────────────────────────────────────────────────────────────┐
│ STEP 1: Developer Writes Code (in Source Repo)                             │
├────────────────────────────────────────────────────────────────────────────┤
│  - Add code comments (XML/JSDoc) and Markdown articles in source repo      │
│                                                                            │
│  - Run local/pipeline doc generation (e.g., `npm run docs:generate` or     │
│    custom script) to produce Markdown docs in a known output folder        │
└────────────────────────────────────────────────────────────────────────────┘
                              ↓
┌────────────────────────────────────────────────────────────────────────────┐
│ STEP 2: Commit & Push to Source Repo                                       │
├────────────────────────────────────────────────────────────────────────────┤
│  - Commit code and generated docs to source repo or publish as artifact    │
└────────────────────────────────────────────────────────────────────────────┘
                              ↓
┌────────────────────────────────────────────────────────────────────────────┐
│ STEP 3: Central Repo Imports Docs from All Source Repos                    │
├────────────────────────────────────────────────────────────────────────────┤
│  - Central repo pipeline downloads/copies generated docs from each repo    │
│  - Aggregates all docs into unified /docs structure                        │
└────────────────────────────────────────────────────────────────────────────┘
                              ↓
┌────────────────────────────────────────────────────────────────────────────┐
│ STEP 4: Central Repo Builds Unified Docusaurus Site                        │
├────────────────────────────────────────────────────────────────────────────┤
│  - Runs `npm run build` to generate static site from all imported docs     │
└────────────────────────────────────────────────────────────────────────────┘
                              ↓
┌────────────────────────────────────────────────────────────────────────────┐
│ STEP 5: Full Pipeline Deployment                                           │
├────────────────────────────────────────────────────────────────────────────┤
│  - Indexes docs to Elasticsearch                                          │
│  - Publishes site to Azure Static Web Apps                                │
└────────────────────────────────────────────────────────────────────────────┘
                              ↓
┌────────────────────────────────────────────────────────────────────────────┐
│ STEP 6: Documentation Available Everywhere                                 │
├────────────────────────────────────────────────────────────────────────────┤
│  ✓ Azure Static Web App (Docusaurus)                                      │
│  ✓ Elasticsearch                                                          │
│  ✓ Central Repository                                                     │
└────────────────────────────────────────────────────────────────────────────┘

Elasticsearch Indexing Details

Document Structure:

{
  "id": "jupiter-market-elastic-comparables-api-indexerservice",
  "repository": "Jupiter.Market.Elastic.Comparables",
  "title": "IndexerService Class",
  "content": "Orchestrates the end-to-end indexing pipeline...",
  "url": "/api/Jupiter.Market.Elastic.Services.IndexerService.html",
  "type": "api",
  "namespace": "Jupiter.Market.Elastic.Services",
  "className": "IndexerService",
  "tags": ["indexing", "service", "pipeline"],
  "lastModified": "2025-11-28T14:30:00Z"
}

Indexing Script (scripts/index-docs-to-elastic.ps1):

1. Scans _site/ directory recursively
2. Parses HTML files
3. Extracts text content (strips HTML tags)
4. Categorizes as "api" or "article"
5. Extracts metadata (namespace, class name, etc.)
6. Bulk indexes to Elasticsearch

---

4. Live Demonstration

Demo 1: Azure Static Web App

URL: https://[your-app-name].azurestaticapps.net

What to Show:

1. Homepage - Clean, professional landing page
2. Navigation - Main menu with API Reference and Articles
3. API Reference - Browse generated API documentation
   • Show namespace hierarchy
   • Click into a class (e.g., IndexerService)
   • Demonstrate XML comment rendering
   • Show cross-references and links
4. Articles Section - Conceptual documentation
   • Open an article (e.g., "IndexerService Logic")
   • Show code examples
   • Demonstrate article-to-API links
5. Search Functionality - Built-in search
   • Search for "IndexerService"
   • Show instant results
   • Click through to documentation

Demo 2: Elasticsearch Index

Tool: Kibana Dev Tools or MCP Server

Query 1: Find all documents from a repository

GET jupiter-documentation/_search
{
  "query": {
    "term": {
      "repository.keyword": "Jupiter.Market.Elastic.Comparables"
    }
  },
  "size": 5
}

Query 2: Full-text search

GET jupiter-documentation/_search
{
  "query": {
    "multi_match": {
      "query": "IndexerService pipeline",
      "fields": ["title^2", "content", "className"]
    }
  },
  "highlight": {
    "fields": {
      "content": {}
    }
  }
}

Query 3: Filter by document type

GET jupiter-documentation/_search
{
  "query": {
    "bool": {
      "must": [
        { "term": { "type": "api" } },
        { "match": { "namespace": "Jupiter.Market.Elastic.Services" } }
      ]
    }
  }
}

Demo 3: MCP Server Integration

In VS Code/Cursor:

1. Open Copilot chat
2. Use MCP tool to query documentation:

@elasticsearch-jupiter search for "How does IndexerService work?"

3. Show results with highlighted matches
4. Demonstrate context-aware responses
5. Show how it links to full documentation

Example Queries:

• "What does DataService do?"
• "Show me examples of comparable mapping"
• "How is deal data enriched?"
• "What are the main services in this project?"

---

5. Developer Experience

For Documentation Authors

Writing XML Comments:

/// <summary>
/// Brief description of what this does
/// </summary>
/// <remarks>
/// Detailed explanation with links to articles:
/// <a href="~/articles/my-article.md">My Article</a>
/// </remarks>
/// <param name="paramName">Parameter description</param>
/// <returns>Return value description</returns>
/// <exception cref="ExceptionType">When this is thrown</exception>
/// <seealso cref="RelatedClass"/>

Writing Articles:

# Article Title

Brief introduction.

## Overview
High-level explanation.

## Implementation Details
Code examples and technical details.

## See Also
- [Related Article](related-article.md)
- API Reference: [ClassName](xref:Namespace.ClassName)

For Documentation Consumers

1. Browse Documentation

• Visit Azure Static Web App
• Navigate through API reference
• Read conceptual articles
• Use built-in search

2. Search from IDE

• Use MCP server integration
• Natural language queries
• Get instant answers
• Jump to full documentation

3. Direct API Access

• Query Elasticsearch directly
• Build custom search interfaces
• Integrate with other tools

Local Development

Build docs locally (in source repo):

# Build the project
dotnet build

# Generate Markdown docs (API + articles)
npm run docs:generate   # or custom script for C#

Build and preview unified docs (in central repo):

# Import/copy generated docs from all source repos into /docs
npm install
npm run build
npm run serve

View at: http://localhost:3000

---

6. Key Benefits

For Individual Developers

✅ Always up-to-date - Docs generated from code
✅ Easy to find - Search + IDE integration
✅ Clear examples - Code samples and explanations
✅ Faster onboarding - Comprehensive guides

For Teams

✅ Consistent - Same system across all repos
✅ Centralized - One place for all knowledge
✅ Collaborative - Easy to contribute
✅ Discoverable - Search across all projects

For Organization

✅ Automated - No manual doc generation
✅ Scalable - Works for any number of repos
✅ Low maintenance - Docs update with code
✅ Professional - High-quality output

---

7. Next Steps

Immediate (This Sprint)

1. ✅ Complete integration of 2 repositories
2. 🔄 Train team on writing XML comments
3. 🔄 Establish documentation standards
4. 📋 Identify next 3 repositories to integrate

Short-term (Next Month)

1. Integrate 3-5 more repositories
2. Set up documentation review process
3. Create documentation templates
4. Gather feedback and iterate

Long-term (Next Quarter)

1. Integrate all active repositories
2. Build custom search UI
3. Add analytics and metrics
4. Explore AI-powered documentation assistance

---

8. Technical Details for Implementation

Prerequisites

• .NET 9.0 SDK (for C# projects)
• Node.js & npm (for Docusaurus and JS projects)
• Docusaurus CLI tool
• Azure DevOps access
• Elasticsearch cluster access
• Central repository access

Pipeline Variables Required

# Elasticsearch
ELASTIC_CLOUD_ID: [secret]
ELASTIC_API_KEY: [secret]

# Central Repository
CENTRAL_DOCS_REPO: https://dev.azure.com/org/project/_git/docs
CENTRAL_DOCS_PAT: [secret]

# Azure Storage (alternative)
AZURE_STORAGE_ACCOUNT: [account-name]
AZURE_SERVICE_CONNECTION: [connection-name]

File Checklist for New Source Repos

• Doc generation script (e.g., npm run docs:generate or PowerShell)
• Output folder for generated Markdown docs (e.g., /generated-docs)
• Pipeline step to publish generated docs as artifact or push to shared location
• .csproj - Enable XML generation (for C#)

File Checklist for Central Docs Repo

• docusaurus.config.js - Docusaurus configuration
• index.md - Documentation homepage
• sidebars.js - Main navigation
• docs/ - Aggregated documentation folder (imported from all repos)
• scripts/index-docs-to-elastic.ps1 - Indexing script
• azure-pipelines-docusaurus.yml - Pipeline definition

---

9. Q&A Preparation

Common Questions

Q: How long does it take to integrate a new repository?
A: 2-3 hours for initial setup, then ongoing as you write docs.

Q: Do we need to document everything?
A: Focus on public APIs and key concepts. Internal implementation details are optional.

Q: What if documentation gets out of sync?
A: It can't - docs are generated from code and Markdown on every merge to main.

Q: Can we customize the look and feel?
A: Yes, Docusaurus supports custom themes, React components, and plugins.

Q: How do we handle breaking changes?
A: Update code comments and articles as part of the code change.

Q: What about versioning?
A: Currently single version. Docusaurus supports versioning if needed.

Q: How much does this cost?
A: Minimal - Azure Static Web Apps free tier, Elasticsearch existing cluster.

---

10. Resources

Documentation

• DocFX Documentation
• XML Documentation Comments
• Implementation Guide

Internal Resources

• Central Documentation Repository: [Link]
• Azure Static Web App: [Link]
• Elasticsearch Cluster: [Link]
• MCP Server Configuration: [Link]

Support

• Platform Team: [Contact]
• Documentation Channel: [Slack/Teams]
• Issue Tracker: [Link]

---

Appendix: Architecture Diagrams

System Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         REPOSITORIES                             │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │   Repo 1     │  │   Repo 2     │  │   Repo N     │          │
│  │              │  │              │  │              │          │
│  │ Code Comments│  │ Code Comments│  │ Code Comments│          │
│  │ + Articles   │  │ + Articles   │  │ + Articles   │          │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘          │
│         │                  │                  │                  │
│         └──────────────────┴──────────────────┘                  │
│                            ↓                                     │
│                    ┌───────────────┐                             │
│                    │ Docusaurus Build│                            │
│                    │  (Pipeline)   │                             │
│                    └───────┬───────┘                             │
│                            ↓                                     │
│              ┌─────────────┴─────────────┐                       │
│              ↓                           ↓                       │
│    ┌──────────────────┐       ┌──────────────────┐              │
│    │  Elasticsearch   │       │  Central Repo    │              │
│    │  (Search Index)  │       │  (Aggregation)   │              │
│    └──────────────────┘       └────────┬─────────┘              │
│                                         ↓                        │
│                              ┌──────────────────┐                │
│                              │  Azure Static    │                │
│                              │  Web App         │                │
│                              │  (Hosting)       │                │
│                              └──────────────────┘                │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Data Flow

Source Code (.cs)

## 11. JavaScript/React/Node Integration

### Overview

**New Capability**: Documentation system now supports JavaScript repositories using JSDoc and jsdoc-to-markdown.

### Key Differences from C#

| Aspect | C# | JavaScript |
|--------|-----|-----------|
| Comments | XML (`///`) | JSDoc (`/** */`) |
| Tool | DocFX metadata | jsdoc-to-markdown |
| Output | YAML files | Markdown per file |
| Organization | Namespace-based | File-based |

### JSDoc Example

```javascript
/**
 * User authentication service
 * @class
 * @example
 * const auth = new AuthService({ apiKey: 'xxx' });
 * await auth.login('user@example.com', 'password');
 */
class AuthService {
  /**
   * Authenticates a user
   * @param {string} email - User email
   * @param {string} password - User password
   * @returns {Promise<AuthToken>} Authentication token
   * @throws {AuthenticationError} When credentials invalid
   */
  async login(email, password) {
    // Implementation
  }
}

Per-File Documentation

Benefits:

• ✅ Mirrors source code structure
• ✅ Easier navigation
• ✅ Better for large codebases
• ✅ Faster incremental builds

Example Structure:

api/
├── services/
│   ├── AuthService.md
│   ├── UserService.md
│   └── DataService.md
├── components/
│   ├── UserCard.md
│   └── Dashboard.md
└── utils/
    ├── formatters.md
    └── validators.md

Build Process

JavaScript Source (.js/.jsx)
    ↓ [jsdoc-to-markdown]
Markdown Files (per source file)
    ↓ [DocFX build]
Static HTML Site
    ↓ [Same distribution as C#]
Elasticsearch + Central Repo + Azure

Implementation Steps

1. Install Dependencies

   npm install --save-dev jsdoc-to-markdown glob fs-extra

2. Create Generation Script

   • scripts/generate-docs.js
   • Processes all .js/.jsx files
   • Generates markdown per file
   • Creates table of contents

3. Update Pipeline

   • Add Node.js installation
   • Run npm run docs:generate
   • Continue with DocFX build

4. Write JSDoc Comments

   • Document all public functions
   • Document all classes
   • Document React components
   • Include examples

React Component Example

/**
 * User profile card component
 * @component
 * @param {Object} props - Component props
 * @param {User} props.user - User object
 * @param {Function} [props.onEdit] - Edit callback
 * @returns {React.Element} Rendered component
 * @example
 * <UserCard 
 *   user={userData} 
 *   onEdit={(user) => handleEdit(user)}
 * />
 */
function UserCard({ user, onEdit }) {
  return (
    <div className="user-card">
      <h2>{user.name}</h2>
      {onEdit && <button onClick={() => onEdit(user)}>Edit</button>}
    </div>
  );
}

Resources

• Full Guide: docs/JAVASCRIPT_INTEGRATION_GUIDE.md

• JSDoc Documentation: https://jsdoc.app/

• jsdoc-to-markdown: https://github.com/jsdoc2md/jsdoc-to-markdown

  ↓ [Compilation] XML Documentation (.xml) ↓ [DocFX Metadata] YAML Metadata (.yml) ↓ [DocFX Build] Static HTML Site (/_site) ↓ [Pipeline Distribution] ├─→ Elasticsearch (indexed) ├─→ Central Repo (aggregated) └─→ Azure (hosted)

---

**End of Presentation**

**Questions?**

Contact: Platform Team  
Documentation: [Link to this guide]  
Last Updated: 2025-11-28