MageTech AI Forge — Technical Architecture 🛠️ & Engineering Documentation

01System Overview & Goals LIVE

MageTech AI Forge is a full-stack AI application and agent platform that combines retrieval-augmented generation (RAG), autonomous agents and the Model Context Protocol (MCP) into one product surface. The platform is deliberately local-first: the default runtime is a fully self-hosted stack with Ollama serving open-weight models, so customers keep data on hardware they control. The architecture is provider-agnostic, so managed providers (OpenAI, Google Gemini, Anthropic, Azure OpenAI) can be added later without redesigning the core.

Product pillars

System context

flowchart LR
  subgraph USER["Users & Integrations"]
    U1[("Web browser")]
    U2[("External integrations\n(post-MVP)")]
  end

  subgraph WEB["Client Tier — Next.js 16 (App Router)"]
    W1["UI components\n(shadcn-style, Tailwind 4)"]
    W2["TanStack Query cache"]
    W3["Zod + React Hook Form"]
    W4["SSE stream reader"]
  end

  subgraph API["API Tier — FastAPI (Python 3.12)"]
    A1["v1 routers: auth, chat, agents,\nknowledge, mcp, models, usage"]
    A2["Services: LLMClient, document_parser,\nknowledge, agents, chat, mcp"]
    A3["Async SQLAlchemy + Redis"]
    A4["Rate limiting (slowapi)"]
  end

  subgraph AI["AI Tier"]
    AI1["Ollama — qwen2.5:7b + nomic-embed-text\n(Live)"]
    AI2["LangGraph + LlamaIndex\n(Planned)"]
    AI3["Managed providers: OpenAI,\nGemini, Anthropic (Future)"]
  end

  subgraph DATA["Data Tier"]
    D1[("PostgreSQL\n(pgvector-ready)")]
    D2[("Redis\n(cache, queues, rate limits)")]
  end

  W2 --> W1
  W3 --> W1
  W4 --> W1
  W1 -- "HTTPS / JSON / SSE" --> A1
  A1 --> A2
  A2 --> A3
  A3 --> D1
  A3 --> D2
  A2 -- "HTTP (Ollama API)" --> AI1
  AI1 -.-> AI2
  AI2 -.-> AI3
  A1 -- "audit + telemetry" --> D2
  A2 -- "token accounting" --> D1

live components ship in the current MVP, dashed edges indicate planned/future capability.

Goals

• Run the entire AI stack locally by default — no mandatory third-party API keys.
• Keep the product open source and self-hostable with one docker compose up command.
• Provide grounded answers over customer knowledge bases with citations and source links.
• Expose every capability through a documented, versioned HTTP API (OpenAPI 3.1).
• Guard business value with security, rate limiting, audit logging and RBAC-ready tenancy from day one.

Non-goals (current release)

• No hosted/cloud control plane — multi-instance fleet management is future work.
• No public plugin marketplace yet (PLANNED).
• No fine-tuning or custom model training within the platform.
• No native mobile clients — responsive web only for now.

02Design Principles LIVE

The architecture is governed by a small set of principles that guide every decision. Each principle maps to concrete, verifiable behavior in the codebase.

Requirements classification

Throughout this document every component, endpoint and decision carries one of four status badges that separate shipped, in-flight and aspirational work:

03Technology Stack LIVE

Versions below are the pinned truth from the workspace lockfiles (apps/web/package.json, apps/api/pyproject.toml) at the time of writing. Bumping a dependency must start from these references and re-run the full quality gate (§16).

Frontend

Backend

AI, Data & Infrastructure

Selection rationale

04AI Architecture LIVE

The AI tier is built around a single provider gateway (services/llm_client) so that the rest of the platform never talks to a specific model vendor. Today the gateway speaks to Ollama; tomorrow it can route to OpenAI, Gemini or Anthropic with configuration only.

LLMClient gateway

flowchart TB
  APP["Application services\n(chat, agents, knowledge, models)"]
  GATEWAY["LLMClient\n(llm_client.py)"]
  RES["Provider resolution\nLLM_PROVIDER = auto | ollama | mock"]
  PROBE["Probe Ollama /api/tags\n(30s availability cache)"]
  OL["Ollama provider\nHTTP client (httpx)"]
  OKC["Ollama Models\ngeneration + embed"]
  MO["Mock provider\n(deterministic dev output)"]
  FUT["Managed providers\nOpenAI / Gemini / Anthropic"]
  TOKEN["Token accounting\nstream → eval_count / len//4"]

  APP --> GATEWAY
  GATEWAY --> RES
  RES --> PROBE
  PROBE --> OL
  OL --> OKC
  RES --> MO
  OL -. provider add .-> FUT
  GATEWAY --> TOKEN
  PROBE -.->|unavailable| MO

• Provider selection — LLM_PROVIDER=auto probes the local Ollama tag endpoint; a 30-second availability cache prevents repeated failed probes. If no provider is reachable the gateway degrades to the mock provider so tests and demos stay deterministic.
• Generation — chat and chat_stream map to Ollama’s /api/chat (non-streaming and streaming). Stream consumers receive SSE events meta → delta → done followed by data: [DONE].
• Embeddings — /api/embed produces 384-dimensional vectors from nomic-embed-text for document chunks.
• Models API — the models router lists, registers, synchronizes (/api/tags) and pulls models into Ollama (/api/pull).
• Token accounting — per message, per run and per model. Streaming responses report eval_count from Ollama; when the source omits it, tokens are estimated with len(text)//4 (≈4 chars/token). This feeds the usage analytics and per-model cost view.

Ollama integration

Model registry defaults

Model entries are persisted in the models table with provider metadata so usage analytics can attribute cost per model even as the registry grows.

Adding a managed provider FUTURE

# Conceptual provider registration (target design)
class OpenAIProvider(BaseProvider):   # future
    async def chat(self, messages, **kw): ...
    async def embed(self, text): ...

# llm_client.py: provider_map[d] = OpenAIProvider() when configured

05Knowledge & Retrieval (RAG) LIVE

A RAG pipeline turns uploaded documents into answerable knowledge. The platform ingests, parses, chunks, embeds and stores documents per knowledge base, then retrieves the top-k most similar chunks for grounded question answering.

Ingestion pipeline

flowchart LR
  U["Upload\n(multipart PDF/text)"]
  P["Parse\npypdf + preview\n(PDF/text/MD/CSV)"]
  C["Chunk\nsize 800 / overlap 160"]
  E["Embed\nnomic-embed-text\n(384-dim)"]
  S[("Store\nPostgreSQL\nJSON embeddings")]
  R["Chat / Q&A runtime"]
  REP["Async reprocess\n(thumbs up / edit)"]

  U --> P --> C --> E --> S --> R
  S -. job .-> REP

Processing details

• Upload & validation — PDF, TXT, MD, CSV; documents are truncated to MAX_DOCUMENT_CHARS = 2,000,000 before parsing to bound cost and latency.
• Parse — document_parser extracts text and a preview; the parsed text becomes the unit of chunking.
• Chunk — overlapping chunks at CHUNK_SIZE = 800 characters with CHUNK_OVERLAP = 160 to preserve context across chunk boundaries.
• Embed & store — each chunk is embedded and persisted in document_chunks with source pointers (document id, index, char range).
• Async option — with USE_CELERY=true, ingestion is dispatched to the worker as the documents.process task; otherwise it runs in-process on the API node.
• Reprocess — documents can be re-chunked/re-embedded after edits or model changes.

Retrieval & answering

sequenceDiagram
    participant U as User / Web
    participant C as Chat Service
    participant Q as Knowledge Service
    participant DB as PostgreSQL
    participant LLM as Ollama LLM

    U->>C: POST /api/v1/chat {question}
    C->>Q: /knowledge/search {query, k}
    Q->>DB: embed query (384-dim)
    Q->>DB: cosine top-k chunks (workspace-scoped)
    DB-->>Q: top-k chunks + source docs
    Q-->>C: ranked context
    C->>LLM: grounded prompt (system + chunks)
    LLM-->>C: answer
    C-->>U: answer + citation sources

• Query embedding — the question is embedded with the same model family as the chunks so vectors are comparable.
• Top-k retrieval — cosine similarity on stored vectors (JSON-array math) scoped to the tenant workspace; the number of chunks k is configurable per request.
• Grounded generation — the system prompt instructs the model to answer from the supplied chunks only and to cite sources; chunks carry document references so answers can link back to source documents.
• Search endpoint — POST /api/v1/knowledge/search?k=… returns ranked chunks for inspection and debugging.

Vector storage evolution

06Agents & LangGraph LIVE

Agents in the MVP are persistent, tool-capable runtimes: create an agent, attach a system prompt and model, then run it with input. Runs produce a stream of steps (tool calls and text) that are recorded for audit and replay. The orchestration layer is designed to be replaced by LangGraph state graphs without changing the domain model or API contract.

Agent run lifecycle

1. Create agent — name, description, system prompt, model, temperature; persisted in agents.
2. Run — POST /api/v1/agents/{id}/run with an input message creates an agent_run and executes the loop.
3. Execute — the runtime generates a response; if the model requests tools, each selected MCP tool is invoked through the MCP service and its result is fed back until the loop terminates.
4. Record — every step, token count, tool call and final output lands in agent_runs (and via executions view) for replay and audit.
5. List — the executions dashboard lists runs (latest 200) with status, duration and token usage; detail view shows the full step trace.

stateDiagram-v2
    [*] --> Pending
    Pending --> Running: executor acquires
    Running --> ToolCall: model requests tool
    ToolCall --> Running: tool result appended
    ToolCall --> Completed: max iterations reached
    Running --> Completed: final answer
    Running --> Failed: exception / provider error
    Completed --> [*]
    Failed --> [*]

Run statuses

Chat layer

Conversation endpoints (/api/v1/chat) provide the conversational surface on top of the same AI gateway: create conversations, post messages, and stream answers (SSE). Messages reference the owning conversation and workspace, preserving tenant isolation and enabling context continuation. Streaming uses the event sequence meta → delta* → done followed by data: [DONE].

LangGraph evolution PLANNED

flowchart LR
  subgraph LG["LangGraph State Machine (planned)"]
    START[Start] --> COND{guardrails pass?}
    COND -->|no| REJ[Reject]
    COND -->|yes| RET[Retrieve\nLlamaIndex]
    RET --> GEN[Generate / plan]
    GEN --> CALL{needs tool?}
    CALL -->|yes| TOOL[Execute MCP tool]
    TOOL --> GEN
    CALL -->|no| CIT[Cite + persist]
    CIT --> END[End]
  end

• Stateful graphs — replace the linear loop with a declarative state machine: retrieval, planning, tool execution, memory and citations become typed nodes with checkpoints.
• Reusable nodes — each node wraps an existing service (knowledge, MCP, LLMClient), so the current feature set survives the migration.
• Checkpoints — LangGraph checkpoints give resume-from-step and richer executions dashboards.
• Memory — memories table (planned) plugged in as a summarization node.

07MCP Integration LIVE

The Model Context Protocol (MCP) layer is the platform’s tool surface. It connects servers, enumerates their tools, lets operators enable/disable each tool, and invokes tools through a standardized, audited path. It is also the seam where the platform itself can be consumed as MCP tooling — the MCP SDK route is planned.

Server taxonomy & security gates

Built-in tools

Tool call flow

flowchart LR
  AGENT["Agent runtime\n(asks for tool)"]
  CHK{"tool enabled?"}
  GATE["MCP service\nworkspace-scoped lookup"]
  COOK["Arguments validated\n+ calculator AST whitelist"]
  SSE["SSE JSON-RPC invoke\n(remote server)"]
  LOG["Audit log + call_count"]
  RES["Result → agent context"]
  REJ["Rejected\n430/403 error"]

  AGENT --> GATE
  GATE --> CHK
  CHK -->|no| REJ
  CHK -->|yes| COOK
  COOK --> SSE
  SSE --> LOG
  LOG --> RES

Management surface

• Servers — register MCP servers (name + connection details), list and delete; each server is workspace-scoped.
• Tools discovery — GET /api/v1/mcp/tools lists discovered tools with status (enabled/disabled) and call counts.
• Enable / disable — PATCH /api/v1/mcp/tools/{id} flips gates; agents only see enabled tools.
• Invoke — POST /api/v1/mcp/tools/call executes a tool call end-to-end with audit.

MCP SDK PLANNED

Serving the platform itself as an MCP server, plus an mcp-sdk-python companion, is planned. This lets external agents call MageTech knowledge tools over the same protocol — turning the platform into a transitive tool network while keeping the same enable/disable and audit gates.

08Data & Storage Architecture LIVE

PostgreSQL is the system of record for every business entity; Redis holds ephemeral state (rate-limit counters, cache, and the Celery broker) and pgvector is provisioned as an extension for the planned native vector search. All migrations are Alembic-managed (0001_initial → f84138ce8b31).

Live schema inventory

Planned tables PLANNED

roles permissions projects agent_versions agent_tools agent_execution_steps memories token_blacklist subscriptions usage_limits

• token_blacklist — accelerates revocation of leaked/rotated refresh tokens instead of relying on short expiry only.
• subscriptions + usage_limits — meter usage against plan limits (feeds metrics collection).
• agent_execution_steps — per-step trace storage replacing in-blob JSON on runs for queryability.

Entity relationships

erDiagram
  users ||--o{ user_workspace_roles : "member of"
  workspaces ||--o{ user_workspace_roles : "grants"
  workspaces ||--o{ knowledge_bases : "owns"
  knowledge_bases ||--o{ documents : "contains"
  documents ||--o{ document_chunks : "split into"
  workspaces ||--o{ agents : "owns"
  agents ||--o{ agent_runs : "executes"
  workspaces ||--o{ conversations : "owns"
  conversations ||--o{ messages : "contains"
  workspaces ||--o{ mcp_servers : "owns"
  mcp_servers ||--o{ mcp_tools : "exposes"
  agents }o--o{ mcp_tools : "may call (enabled)"
  workspaces ||--o{ models : "registers"

Caching & ephemeral state

Migrations

# Apply schema migrations
alembic upgrade head

# New migration
alembic revision --autogenerate -m "describe change"
alembic upgrade head

All schema changes land via Alembic, are reviewed, and the document_chunks storage change (§05) is the flagship planned migration.

09API Design & Integration LIVE

The API is versioned REST (JSON) with OpenAPI 3.1 documentation served by FastAPI at /docs. All endpoints are async, tenant-scoped where applicable, and uniform in error shape, so SDK and UI clients share one integration contract.

Conventions

Endpoint inventory

OpenAPI 3.1 JWT Bearer SSE REST x-workspace-id

Example: authenticated chat

curl -X POST http://localhost:8000/api/v1/chat \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -H "X-Workspace-Id: ws_123" \
  -d '{"conversation_id": null, "content": "Summarize our Q3 plan"}'
# → {"id": "msg_…", "role": "assistant", "content": "…", "sources": [...]}

Example: streaming response

data: {"type":"meta","model":"qwen2.5:7b"}
data: {"type":"delta","content":"Hell"}
data: {"type":"delta","content":"o there"}
data: {"type":"done","usage":{"prompt_tokens":12,"completion_tokens":34}}
data: [DONE]

10Security Architecture LIVE

Security is enforced in layers: identity → session → authorization → tenant isolation → rate limiting → audit. The design is fail-closed: unexpected states reject rather than degrade silently.

Authentication flow

sequenceDiagram
    participant C as Client (web/API)
    participant A as Auth service
    participant DB as PostgreSQL
    participant RL as Redis rate-limiter

    C->>A: POST /auth/login (email + password)
    A->>RL: rate-limit check (10/min/key)
    A->>DB: fetch user by email
    A->>A: verify Argon2id hash
    A-->>C: access JWT (15 min, HS256) + refresh JWT
    A-->>C: Set-Cookie: refresh (HttpOnly, SameSite=Lax, Path=/api/v1/auth/refresh)
    Note over C,A: Access JWT is short-lived, refresh rotates on refresh
    C->>A: POST /auth/refresh (cookie)
    A->>A: verify refresh JWT, rotate
    A-->>C: new access + refresh pair

Measures

OWASP-oriented checklist

• CSRF — mitigated for state-changing calls by Bearer tokens + SameSite cookie; refresh route is POST-only.
• Injection — SQLAlchemy ORM parameterization everywhere; no raw SQL in services.
• SSRF — MCP remote servers are operator-registered URLs; stdio (local exec) transport rejected.
• Secrets — never logged (structlog redacts headers/secrets); .env never committed; distribution via .env.example.
• Input validation — Pydantic v2 schemas on every route; multipart uploads validated for type/size.
• Rate-limit bypass — limits keyed per client identity + route, applied before auth work.

11Multi-tenancy LIVE

The platform is multi-tenant by construction. A workspace is the tenant boundary: knowledge bases, documents, agents, conversations, MCP servers and model entries all belong to exactly one workspace.

Model

flowchart TB
  U["User"]
  M["memberships\n(user_workspace_roles)"]
  W["Workspace A"]
  W2["Workspace B"]
  KB["knowledge_bases A"]
  DB["documents A"]
  AG["agents A"]
  CV["conversations A"]
  MC["mcp A"]
  MD["models A"]

  U --> M
  M --> W
  M --> W2
  W --> KB
  KB --> DB
  W --> AG
  W --> CV
  W --> MC
  W --> MD
  W2 -.->|same tables, different workspace_id| X

• Row-level tenancy — every business table carries workspace_id; service queries always filter or join through the workspace.
• Selection — apps send X-Workspace-Id; the API validates membership and falls back to the user’s first workspace when omitted.
• Membership roles — user_workspace_roles already encodes member/owner semantics, ready for granular RBAC enforcement.
• Isolation proof points — knowledge search, agent tool calls, chat history and usage aggregation all scope by workspace; tests assert cross-workspace leakage is impossible (IDOR cases).

Defense in depth

tenant boundary = workspace x-workspace-id rbac-ready idor-tested

12Frontend Architecture LIVE

The web application is a Next.js 16 App Router project (TypeScript, Tailwind 4). It is intentionally component-driven, typed, and treats the backend OpenAPI schema as a contract.

Application routes

Key patterns

Folder conventions

apps/web/src
  app/        # App Router pages + layouts
  components/ # ui/ primitives + feature components
  lib/        # api client, queries, utils
  hooks/      # shared hooks (e.g. SSE reader)
  schemas/    # Zod mirrors of API contracts
  styles/     # global CSS + design tokens

Accessibility baseline

• Semantic landmarks, skip-link, keyboard-focusable controls, aria-label on icon buttons.
• Contrast meets WCAG AA in both themes; focus-visible ring states.
• Reduced-motion respected via prefers-reduced-motion.

13Backend Architecture LIVE

The API is a layered async FastAPI application. Dependencies flow one way: routers → services → repositories → domain, with infrastructure (db, redis, rate limits) and core (config, security, errors, logging) available to any layer.

Layers

flowchart LR
  R["api/v1 routers\n(auth, chat, agents, knowledge,\nmcp, models, usage, dashboard…)"]
  DEP["deps — auth, tenant, pagination"]
  S["services\n(llm_client, document_parser, knowledge,\nchat, agents, mcp, models, usage…)"]
  REP["repositories\ndata access (SQLAlchemy)"]
  DOM["domain\nschemas + entities (Pydantic/SQLAlchemy)"]
  INFRA["infrastructure\ndb engine, redis, rate_limit"]
  CORE["core\nconfig, security, errors, logging, constants"]
  W["worker\ncelery_app + tasks"]

  R --> DEP --> S --> REP --> INFRA
  S --> DOM
  CORE --> S
  W --> S
  INFRA --> DB[(PostgreSQL / Redis)]

Module responsibilities

Execution & resilience

• Async everywhere — async SQLAlchemy (asyncpg), async Redis, httpx.AsyncClient for Ollama/MCP; pool size=10, max_overflow=20, pool_pre_ping to dodge dead connections.
• Uniform errors — AppError → status + {"error":{...}}; unexpected exceptions are caught, logged with stack, and mapped to 500 without leaking internals.
• Streaming — StreamingResponse yields SSE frames with meta → delta → done → [DONE]; clients treat connection/llm errors as structured error events.
• Graceful degradation — rate limiting falls back to in-memory when Redis is down; providers fall back to mock when Ollama is unreachable; health endpoint reports each dependency’s state independently.
• Background work — long ingestion can move to Celery (USE_CELERY=true) without changing the API contract.

14Codebase & Conventions LIVE

The repository is a workspace monorepo that separates the web client, the API, infrastructure definitions and documentation so each can evolve with its own toolchain.

Repository layout

Magetech-aiforge/
  apps/
    web/       # Next.js 16 client
    api/       # FastAPI service
  infrastructure/
    docker/    # compose, env template
    postgres/  # init.sql (extensions)
  document/    # how-it-works.html, this doc
  docs/        # ARCHITECTURE.md, ROADMAP.md, DEVELOPMENT.md
  tests/e2e/   # Playwright specs
  package.json # root scripts (lint, test, e2e, build)
  AGENTS.md    # coding-agent rules

Quality gates

Style rules

• TypeScript strict; types never substituted with any in app code.
• Backend: ruff-format default, no unused imports, docstrings on public modules.
• Migrations always paired with their schema test or at least an idempotence check.
• New API surface ships with OpenAPI-visible examples and a Zod mirror in the web app.
• No secrets in code; environment via .env.example only.

15Infrastructure & DevOps LIVE

The distribution story is one command: docker compose up brings up the full local-first stack. Environments are configured through typed env vars with sensible defaults and production guards.

Compose topology

flowchart LR
  N["nginx/proxy (edge)"]
  FE["frontend\nNext.js (port 3000)"]
  API["backend\nuvicorn (port 8000)"]
  W["worker\ncelery (optional)"]
  PG[("postgres\npgvector:pg16 / :5433")]
  RD[("redis\n7-alpine / :6380")]
  OL["ollama\n:11434"]

  N --> FE
  N --> API
  API --> PG
  API --> RD
  API --> OL
  API --> W
  W --> RD
  W --> OL
  PG --> OL

Services

init.sql runs on first boot and provisions extensions: vector pgcrypto uuid-ossp. All data lives in named Docker volumes (or the host for local native dev).

Environment configuration

APP_ENV=dev
DATABASE_URL=postgresql+asyncpg://…
REDIS_URL=redis://…
CELERY_BROKER_URL=redis://…
JWT_SECRET=change-me-in-prod
JWT_REFRESH_SECRET=change-me-in-prod
JWT_ACCESS_TOKEN_EXPIRE_MINUTES=15
JWT_REFRESH_TOKEN_EXPIRE_DAYS=7
CORS_ORIGINS=["http://localhost:3000"]
RATE_LIMIT_AUTH=10/min
RATE_LIMIT_DEFAULT=200/min
SEED_ADMIN_EMAIL=admin@magetech.com
SEED_ADMIN_PASSWORD=ChangeMe_Admin_123!
LLM_PROVIDER=auto
OLLAMA_BASE_URL=http://localhost:11434
COOKIE_SECURE=false
USE_CELERY=false
MAX_DOCUMENT_CHARS=2000000
CHUNK_SIZE=800
CHUNK_OVERLAP=160
DEMO_DASHBOARD_DATA=true

Operations & future FUTURE

• Backups — daily pg_dump + WAL archiving; restore drill documented in runbook.
• Upgrades — rolling compose pull + alembic upgrade head under maintenance window.
• Progressive rollout — manual npm/pypi publish path today; CI/CD with build pipeline, image registry and canary deploys (FUTURE).
• Orchestration — Kubernetes manifests for multi-node scale with HA Postgres and Redis (FUTURE).

16Testing & Quality LIVE

Testing is layered: fast unit tests exercise pure logic, integration tests exercise the API against real infrastructure, and E2E tests prove journeys in a browser. Every layer is part of the merge gate.

Test pyramid

flowchart TB
  E2E["E2E — Playwright (root tests/e2e)\nauth → chat → knowledge → MCP"]
  INT["Integration — pytest + pytest-asyncio\nAPI + async SQLAlchemy + Redis\n(workspace/tenant isolation, IDOR cases)"]
  UNIT["Unit — pytest (api) + Vitest (web)\nschemas, security, parser, hooks, components"]

  E2E --> INT --> UNIT

Coverage & quality targets

• API business services ≥ 85% line coverage; routers bound by integration contract tests.
• Web: every Zod schema has a positive + negative parse test; every query hook has a mocked-data test.
• Security: explicit tests for login brute-force (429), IDOR cross-workspace access, stdio MCP rejection, default-secret prod guard.
• Deterministic AI tests: mock provider guarantees stable output — no flaky model-dependent assertions.
• Contract: OpenAPI schema + Zod mirrors checked in CI for drift.

17Observability & AgentOps LIVE

Every request, LLM call and agent step is observable. The design makes debugging, cost attribution and audit straightforward from a single log stream plus the audit and usage stores.

Structured logging

{"event":"chat.completed","level":"info","request_id":"r8f1…",
 "workspace_id":"ws_123","model":"qwen2.5:7b",
 "prompt_tokens":12,"completion_tokens":34,"duration_ms":812}

• structlog emits JSON in production, readable console in dev; secrets/headers redacted.
• Every request tagged with request_id so logs, audit rows and traces join.
• LLM calls logged with model, token counts and latency — the raw material for AgentOps dashboards.

Telemetry surfaces

AgentOps practice

Agent runs persist the full step trace, status transitions and token use. This is the debugging foundation for “why did the agent do that?” and feeds evaluation datasets for prompt/safety regression testing — bridging §16 and §19.

18Performance & Scalability LIVE

Performance budgets apply end-to-end: the LLM call dominates latency, so everything else is minimized and cached.

Techniques in play

Latency budget (p95, local single node)

Scaling guidance

• Phase 1 — tune pool size, enable USE_CELERY for ingestion offload.
• Phase 2 — migrate chunks to native pgvector + HNSW index (§05).
• Phase 3 — horizontal API replicas behind the proxy with Redis-backed limiting; separate model nodes (§21).

19AI Safety, Ethics & Governance PLANNED

Safety is designed in at the architecture level — not bolted on at prompt time. The current release already contains several hard safety properties; the governance surface grows with the roadmap.

Hard safety properties (live)

• No arbitrary execution — the only code-executing tool (calculator) uses an AST whitelist; MCP stdio transport is rejected; no tool can spawn shell/OS processes.
• Grounded answers — RAG system prompt constrains the model to supplied chunks with source citations; hallucinations are structurally reduced, and retrieval sourcing makes answers checkable.
• Tenant isolation — an agent in workspace A cannot retrieve from workspace B; verified by IDOR tests.
• Loop bounds — agent tool loops are capped; runaway cost/behavior is impossible by construction.
• Audit trail — every tool invocation and privileged action is recorded in audit_logs.

Governance roadmap

Operational safeguards

• Evaluation before rollout — new models/providers pass the eval harness and a safety checklist before being suggested to tenants.
• Incident runbook — prompt-injection/sensitive-data incidents follow the audit trail from request_id to responsible workspace.
• Documented limits — per-tenant usage caps (planned usage_limits) stop unbounded spend before it begins.

20Integrations & Extensibility PLANNED

MageTech AI Forge is built to interoperate. Today MCP is the extensibility spine (§07); the next layers are webhooks, domain APIs and an SDK/plugin model.

Integration layers

flowchart LR
  APP["MageTech AI Forge"]
  WC["Webhooks (planned)\nevents: document.processed, agent.completed,\nrun.failed, usage.threshold"]
  OAU["OAuth2 2-legged (planned)\nservice-to-service, scoped tokens"]
  MCP["MCP SDK (planned)\nserve platform tools to external agents"]
  SF["ServiceFlow / CRM\n(Typeform, Make, Pipedrive, Jotform…)"]
  EXT["Plugin registry (future)"]

  APP --> WC
  APP --> OAU
  APP --> MCP
  APP --> SF
  APP --> EXT

Planned capabilities

Extensibility principles

• Every new connector lands behind the existing seams: LLMClient (AI), MCP (tools), webhooks (events), API (data).
• External calls are always tenant-scoped, authenticated and audit-logged.
• No connector shall require modifying core services — they compose, never fork.

21Roadmap & Milestones PLANNED

The roadmap is phased so each milestone is shippable and independently valuable. Status badges in this section reflect the docs/ROADMAP.md plan: the MVP (Phases 1–2) is largely live; Phase 3 (LlamaIndex) and Phase 4 (LangGraph) are the next planning targets; Phase 5 (MCP SDK) and Phase 7 (managed providers + ai-gateway) prepare for hosted scale.

Milestones

Timeline view

timeline
  title MageTech AI Forge Roadmap
  2025 : Research & architecture prototype
         : Provider gateway + RAG spike
  2026 : MVP-1 — auth, workspaces, RAG chat (LIVE)
         : MVP-2 — agents, MCP, usage analytics (LIVE)
         : MVP-3 — RBAC, blacklist, E2E expansion (PLANNED)
         : MVP-4 — pgvector, webhooks, OAuth2 (PLANNED)
  2027 : Phase 4 — LangGraph + LlamaIndex (PLANNED)
         : Phase 5 — MCP SDK (PLANNED)
         : Phase 7 — managed providers, ai-gateway (FUTURE)

Dependencies

• pgvector migration (MVP-4) depends on the extension already provisioned in init.sql.
• LangGraph (Phase 4) reuses current agent domain models and MCP tool layer — no data migration.
• Webhooks (MVP-4) precede OAuth2 connectors so external systems can subscribe first.
• Managed providers (Phase 7) require eval harness (§19) to be in place.

22Production Readiness Checklist PLANNED

Use this checklist as the release gate for any environment labelled production. Items marked LIVE are already enforced; others are required before go-live.

Deploy & config

• Rotate SEED_ADMIN_PASSWORD and set unique JWT_SECRET/JWT_REFRESH_SECRET (enforced by validate_production()).
• Set COOKIE_SECURE=true and a locked-down CORS_ORIGINS allowlist.
• Enable TLS termination at the proxy; redirect HTTP → HTTPS; HSTS headers.
• Pin service image versions and alembic upgrade head as part of the release step.
• Configure daily pg_dump backups + WAL archiving; test a restore.

Security

• Run the §10 checklist (rate limits, CORS, credentials hygiene, Argon2id, Scrypt verification path).
• Verify MCP servers have no stdio servers and only intended SSE endpoints enabled.
• Limit MAX_DOCUMENT_CHARS/upload sizes to bound ingestion cost.
• Set per-tenant usage limits and monitor the usage dashboard for anomalies.
• Pen-test: authentication, IDOR across workspaces, SSRF via MCP URLs, prompt-injection paths.

AI / RAG quality

• Baseline eval over a golden question set (retrieval hit-rate, answer faithfulness).
• Validate citation rendering end-to-end (chunk → answer → source link).
• Confirm provider fallback behaves (mock) so outages degrade loudly, not silently.

Operations

• Alert on /api/v1/health non-200, 429s, worker stagnation, and Ollama restarts.
• Drill the incident runbook using request_id → logs → audit → executions.
• Capacity plan: shred CPU/GPU headroom, Redis memory, and pgvector scale limits (§18).

23Glossary LIVE