SELF_HOSTED: TRUECLOUD_PROVIDER: AWSAPI_STYLE: OPENAI
A self-hosted LLM gateway for routing, caching, guardrails, and observability across model providers.
01 // UNIFIED_API
One Interface
A stable chat completions endpoint that sits in front of multiple model targets. Switch providers without changing a single line of client code.
OPENAICLAUDE+MORE
02 // INFRASTRUCTURE
AWS-Native
Built for the AWS ecosystem. Deploys via Terraform to ECS, using ElastiCache for fast caching and Bedrock for foundational models.
ECSCLOUDWATCHBEDROCK
03 // LOGIC_ENGINE
Policy Driven
Routing, retry, and fallback logic. Define timeouts, cooldown periods, and cache behavior directly in your config.
ROUTINGRETRYCACHING
// Capabilities
01
Configurable Routing
Route requests by metadata, model aliases, weighted targets, and fallback chains.
multi-providerweighted routingfallback chains
02
Gateway Orchestration
Coordinate validation, cache checks, cooldowns, provider calls, retries, and terminal responses.
lifecycle controlshort circuitretry logic
03
OpenAI-style API Surface
Send chat completion requests through a single gateway endpoint with optional routing metadata.
streaming supportunified schemasingle endpoint
04
Python SDK
Call Mantis from application code without manually constructing each HTTP request.
pip installableasync clientrouting metadata
// Team
// Next Steps