MController + NVIDIA Dynamo
AI inference architecture — shared & dedicated namespaces, AI Gateway and admin control plane
1. Admin Portal
Ops control plane
Routing policy editor
model routes, shared vs dedicated upstreams, weight / canary
GPU health dashboard
DCGM / nvidia-smi via Prometheus, MIG + GPU status
LLM usage metrics
tokens, TTFT, ITL, RPS, error rate, showback
Tenant & quota ops
tenant-*, ResourceQuota view
Endpoint catalog
ModelEndpoint Ready + endpointURL
2. AI Gateway
Client front door
AuthN & AuthZ
API keys, JIT, tenant identity
Rate limiting
per-tenant / per-model
Metering & usage capture
tokens, requests → billing hooks
Smart model routing
route to shared / dedicated by tenant & SLO
OpenAI-compatible API
clients stay compatible, MController manages topology
3. MController
Control plane (INFaaS) · mcontroller-system
REST API
/v1/tenants, /v1/model-registries, /v1/model-deploys
Custom resources
Tenant → ModelRegistry → ModelEndpoint
Serving providers
NVIDIA Dynamo · AMD · Intel
Reconcile loop
API → CRs → Operator → DynamoGraph
4. Kubernetes + NVIDIA platform
Unified substrate
Kubernetes control plane
API server, etcd, scheduler, controller manager
NVIDIA GPU Operator
device plugin, drivers, MIG manager
Dynamo platform
dynamo-system, Grove + KAI scheduler
Gateway API / Istio
shared gateway in dynamo-gateway namespace
5. Inference namespaces — shared vs dedicated
Isolation = namespace + quota + NetworkPolicy
SHARED · namespace dynamo-shared
multi-tenant pool- 1 · Model registry PVC (RWX) — dynamo-model-registry
- 2 · Shared Dynamo frontend — port 8000, DYN_ROUTER_MODE=kv
- 3 · HTTPRoute — path /models/<model>
- 4 · Model ID — shared/<model>
DEDICATED · namespace tenant-<name>
isolated per tenant- 1 · NS + SA + ResourceQuota + LimitRange + NetworkPolicy
- 2 · Dedicated DynamoGraphDeployment — frontend + vLLM / SGLang / TRT-LLM worker
- 3 · ClusterIP service — port 8000
- 4 · Model ID — <tenant>/<model> → endpointURL = ClusterIP
6. Dynamo workload shape
Request path inside a namespace
7. GPU nodes, network & observability
| Cluster | GPUs | Utilisation | Fabric | Throughput | Health |
|---|---|---|---|---|---|
| Cluster-A | 8 × B300 | RDMA / RoCE | 612K t/s | Healthy | |
| Cluster-B | 8 × B300 | RDMA / RoCE | 548K t/s | Healthy | |
| Cluster-C | 8 × B300 | RDMA / RoCE | 587K t/s | Healthy | |
| Cluster-D | 8 × B300 | RDMA / RoCE | 521K t/s | Healthy |
GPU nodes
B300 / H100-class
Frontend network
RDMA / RoCE
Storage
NVMe / PVC model cache
Observability
DCGM telemetry + Dynamo metrics
Engineered workflow
Provision to decommission
- 1Discover GPUs
- 2Bootstrap GPU Operator + Dynamo Platform
- 3Install MController
- 4Create tenant
- 5Register model (artifact → PVC)
- 6Deploy shared or dedicated model
- 7Ready + endpointURL
- 8Configure AI Gateway upstream → policies
- 9Clients call gateway
- 10Admin portal watches GPU health & LLM usage
- 11Scale / re-route
- 12Decommission & reclaim
Namespace map: mcontroller-system · dynamo-system · gpu-operator · dynamo-shared · tenant-*