Route inference under explicit budgets and capacity

Route inference under explicit budgets and capacity An architecture diagram generated by Archify. AI Application · versioned request · Architecture component AI Application versioned request Model Gateway · reserve quota first · Architecture component Model Gateway reserve quota first Budget Ledger · atomic reservation · Architecture component Budget Ledger atomic reservation Provider API · timeout + usage · Architecture component Provider API timeout + usage Local Queue · bounded token backlog · Architecture component Local Queue bounded token backlog Model Server · vLLM / accelerator · Architecture component Model Server vLLM / accelerator GPU Resources · memory + utilization · Architecture component GPU Resources memory + utilization request reserve remote route local route dispatch inference Legend Backend Database Cloud Security Message bus External