AI

Letting every module embed its own LLM SDK creates three problems at once. (a) Cost-tracking turns into per-module spreadsheets that never agree. (b) PII redaction becomes a decentralized policy that drifts. (c) Provider failover requires per-module change windows whenever Anthropic / OpenAI degrades. The AI Gateway pattern is the standard answer: pay the cost of one integration once, let every other module call a single typed RPC, and centralise the policy.

The bet is the same bet AUTH and BRAIN make: pay the cost once at the substrate. Without AI Gateway, each module re-implements PII redaction (and at least one of them gets it wrong), each module embeds its own SDK (and the OpenAI Python client and the Anthropic Python client disagree on streaming chunks), and the regulator's "show me every prompt that touched personal data" question becomes a forensic project. With AI Gateway, that question is a SQL query.

PRD §8.5 + §9.7 + §11.2.1 give the full picture; this table is the working summary.

Six internal pipelines between the gRPC ingress and the provider egress: tenant policy resolution, persona stamping, PII redaction, cache lookup, router selection, and accounting. The diagram below shows the full request path for a chat.complete call.

Internal components

AI Gateway is mostly stateless — its source of truth is BRAIN (provider config, tenant policy, persona prompts) and Redis (cache, idempotency keys). DuckDB holds the cost-roll-up for fast dashboard queries. The entities below show the read + write surfaces.

Provider + model matrix (P0)

AI Gateway speaks gRPC internally and exposes a thin REST surface for non-internal callers. A federated GraphQL subgraph publishes the read-side (usage, model catalogue). No public MCP tools — the gateway is infrastructure, not directly agent-callable.

gRPC API (canonical)

syntax = "proto3";
package cyberos.ai.v1;

service AIGateway {
  // Streaming chat completion (SSE end-to-end).
  rpc ChatComplete(ChatRequest) returns (stream ChatChunk);
  // Non-streaming variant for batch jobs.
  rpc ChatCompleteSync(ChatRequest) returns (ChatResponse);
  // Embedding (single or batch).
  rpc Embed(EmbedRequest) returns (EmbedResponse);
  // Reranking.
  rpc Rerank(RerankRequest) returns (RerankResponse);
  // Cost lookup for the calling tenant.
  rpc UsageMTD(TenantRef) returns (UsageReport);
  // Model catalogue.
  rpc ListModels(Empty) returns (ModelList);
}

message ChatRequest {
  repeated Message messages = 1;
  string persona = 2;                 // "cuo" | "genie" | "hr-assistant"
  string route_class = 3;             // "chat" | "reasoning"
  string idempotency_key = 4;
  ModelHint hint = 5;                 // optional; tenant policy may override
  bool stream = 6;
  map<string, string> metadata = 7;   // free-form, recorded in audit
}

message Message {
  string role = 1;        // "user" | "assistant" | "system" (system reserved)
  string content = 2;
  repeated Tool tool_calls = 3;
}

message ChatChunk {
  string content_delta = 1;
  bool   done = 2;
  Usage  usage = 3;       // emitted on final chunk
}

message Usage {
  int32 tokens_in = 1;
  int32 tokens_out = 2;
  string model_id = 3;
  string cache_state = 4;
  double usd_cost = 5;
  bool   redaction_applied = 6;
  string persona_version = 7;
  string brain_chain = 8;
}

message EmbedRequest {
  repeated string inputs = 1;
  string model = 2;        // default: "bge-m3"
}

message EmbedResponse {
  repeated Embedding embeddings = 1;
  Usage usage = 2;
}

message Embedding { repeated float vector = 1; }

REST + SSE surface (planned, edge-only)

GraphQL subgraph (read-only)

extend schema
  @link(url: "https://specs.apollo.dev/federation/v2.5", import: ["@key", "@requiresScopes"])

type AIInvocation @key(fields: "id") @requiresScopes(scopes: [["ai.usage_read"]]) {
  id: ID!
  tenantId: ID!
  actor: String!
  routeClass: RouteClass!
  modelId: String!
  personaVersion: String!
  tokensIn: Int!
  tokensOut: Int!
  usdCost: Float!
  latencyMs: Int!
  cacheState: CacheState!
  redactionApplied: Boolean!
  failoverPath: String!
  ts: DateTime!
}

type UsageReport @key(fields: "tenantId month") {
  tenantId: ID!
  month: String!                    # "2026-05"
  totalCalls: Int!
  totalTokens: Int!
  totalUsdCost: Float!
  capUsdCost: Float!
  percentUsed: Float!
  byModel: [ModelUsage!]!
}

type ModelUsage {
  modelId: String!
  calls: Int!
  tokensIn: Int!
  tokensOut: Int!
  usdCost: Float!
}

enum RouteClass { CHAT REASONING EMBED RERANK IMAGE }
enum CacheState { MISS HIT BYPASS }

type Query {
  aiUsageMTD(tenantId: ID): UsageReport!
  aiInvocations(since: DateTime, limit: Int = 50): [AIInvocation!]!
    @requiresScopes(scopes: [["ai.usage_read"]])
  aiModels: [Model!]!
}

Flow 1 — Streaming chat completion (cache miss)

Flow 2 — Cache hit (deterministic replay)

Flow 3 — Provider failover (primary degraded)

Flow 4 — Per-tenant cost cap enforcement

Flow 5 — PII redaction (Vietnamese CCCD)

A single AI invocation traverses ten states between caller and audit row. Most of the time is in Routing (provider RTT); cache-hit paths skip from CacheCheck straight to Streaming.

Latency budget per route class

Performance NFRs from PRD §11.2.1 + reliability from §11.2.2. Cross-referenced at nfr-catalog.html#ai.

AI Gateway depends on three internal services and four external providers (P0). It is depended on by every CyberOS module that calls an LLM.

AI Gateway is the chokepoint for "what did the AI see, and at what cost?" — making it the regulator's first call for AI Act + PDPL questions.

AI Gateway-specific risks tracked in the risk register.

9 KPIs covering latency, cost, redaction quality, and reliability.

Two CLIs: cyberos-ai for operators (tenant policy, cost reports, model catalogue) and the standard OpenAI-compatible curl path for ad-hoc testing.

1. Quick chat call

$ curl https://ai.cyberos.com/v1/chat/completions \
    -H "Authorization: Bearer $CYBEROS_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"model":"claude-3.5-sonnet","messages":[{"role":"user","content":"summarise Q1 OKRs"}],"stream":false}'

{
  "id": "ai_01HZJ8…XK",
  "model": "bedrock:anthropic.claude-3.5-sonnet",
  "persona_version": "genie-v1.0.2",
  "choices": [{"message":{"role":"assistant","content":"Q1 OKRs are …"}}],
  "usage": {"prompt_tokens":120,"completion_tokens":450,"usd_cost":0.0075,
            "cache_state":"miss","redaction_applied":false,"failover_path":"primary"}
}

2. Operator — view MTD usage

$ cyberos-ai usage mtd --tenant stephen-personal

Tenant:       stephen-personal
Month:        2026-05
─────────────────────────────
Total calls:  14,823
Tokens-in:    8.2 M
Tokens-out:   3.1 M
USD spent:    $97.42 / $150.00 (64.9%)

By model:
  bedrock:claude-3.5-sonnet    11,420 calls   $74.20  (76%)
  bedrock:claude-3-haiku        3,210 calls    $4.80   (5%)
  bge-m3 (embed)                  193 batches  free

By cache state:
  hit   4,612   (31.1%)  ← above target
  miss 10,211   (68.9%)

3. Operator — update tenant policy

$ cyberos-ai policy set --tenant acme-corp \
    --primary bedrock --fallback anthropic \
    --require-zdr true --cap-usd-monthly 500

[policy updated]
  tenant:    acme-corp
  primary:   bedrock
  fallback:  anthropic
  zdr:       required
  cap:       $500/month
[audit]    brain seq=14841

4. Operator — list models

$ cyberos-ai models list

ID                                                  ROUTE     ZDR   PRICE (in/out per 1k)
bedrock:anthropic.claude-3.5-sonnet                 chat      ✓     $0.003 / $0.015
bedrock:anthropic.claude-3-haiku                    chat      ✓     $0.00025 / $0.00125
anthropic:claude-sonnet-4.5                         chat      ✓     $0.003 / $0.015
openai:gpt-4o                                       chat      ✓     $0.0025 / $0.01
openai:o1-mini                                      reason    ✓     $0.003 / $0.012
self-hosted:bge-m3                                  embed     —     free
self-hosted:bge-rerank-v2-m3                        rerank    —     free

5. Operator — failover drill

$ cyberos-ai chaos failover --provider bedrock --duration 60s

[chaos]   injected 100% error rate on bedrock for 60 s
[detect]  primary failure recognised @ +6.2 s
[failover] secondary (anthropic) active @ +6.4 s
[recovery] bedrock errors cleared @ +60 s
[breaker] half-open @ +90 s ; closed @ +91 s
[result]   (FR pending) PASSED (failover ≤ 30 s)

6. Operator — export monthly invoice

$ cyberos-ai invoice export --tenant acme-corp --month 2026-05 --output invoice.csv

[invoice] tenant=acme-corp  month=2026-05  rows=14,823  written invoice.csv (1.2 MB)
[lines]   by_model · by_route · by_persona · by_date