Stop overspending on AI.
Flux is a self-hosted LLM gateway for AI agents and AI-powered apps. It enforces one budget across an entire agent run, including retries and subagents, and routes each call to the cheapest model that meets a quality floor. As a run uses up its budget, Flux shifts to cheaper models before it ever hard-stops.
Flux routes every wrapper request and agent step to the best-fit model, with up to 80% estimated savings in our offline benchmarks and millisecond routing decisions.
Self-hosted · OpenAI-compatible · Five providers · 1,400+ automated tests
Last updated:
estimated cost savings
routing decisions under 10 ms
major AI providers
automated tests
catalog models
Illustrative estimates from offline benchmarks; your results depend on your workload. The test count comes from our own test suite. How these figures were measured.
Change the base URL. Keep your agents, tools and keys.
Flux is the self-hosted control plane for AI wrappers and agents. Setup is one base-URL change: the OpenAI-compatible wrapper guide walks through it, and the benchmark methodology explains every number on this page.
from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="your-flux-token", ) response = client.chat.completions.create( model="flux-auto", messages=[ {"role": "user", "content": "Summarize this document"} ], )
# Same OpenAI-compatible call, plus two headers POST /v1/chat/completions X-Flux-Run-Id: run-123 X-Flux-Step-Type: plan # Flux tracks spend against run-123 and keeps a # higher quality floor for "plan" steps than for # mechanical steps like tool_result.
Budget guardrail
Reserves estimated cost before dispatch, blocks what would exceed the cap
Step-aware routing
Plan, tool-select, and final steps keep a quality floor
Self-hosted
Routing, budgets, and telemetry stay in your infrastructure
Cut model costs without downgrading every request.
Most AI systems send every request to one expensive model. Flux evaluates each request and agent step, then selects the least expensive available model that satisfies the configured capability and quality requirements.
estimated savings in Flux's offline evaluations on synthetic requests, not live traffic.
per-baseline results, offline synthetic estimates
estimated savings versus GPT-4o
estimated savings versus Claude Sonnet 4.6
estimated savings versus Gemini 2.5 Pro
Illustrative estimate from offline benchmarks; your results depend on your workload. Measured with registry pricing and catalog-rated model quality. Results vary by workload, routing policy, pricing and model availability. Read the full benchmark methodology →
Flux makes routing decisions in milliseconds.
Routing is pure heuristic logic. No LLM call sits in the decision path, so the decision itself is effectively free next to inference.
local routing decisions completed under 10 ms
median routing-decision latency
P95 routing-decision latency
Illustrative figures from offline benchmarks; your results depend on your workload. Routing-decision measurement only. Provider inference, network latency and complete HTTP gateway overhead are measured separately. See how routing latency was measured →
Two jobs, deliberately separate.
Flux routes every call to the best-fit model on cost, capability, latency and reliability. Separately, it applies an enforced estimated spend guardrail on what an agent run can consume.
01 · Capability-aware, cost-conscious routing
Every request is classified by task type, filtered down to models that meet configured capability and quality-floor requirements, then scored on cost against quality and dispatched. The logic is pure heuristics, with no LLM call in the routing path.
Request
Incoming call
Classify
Task type + complexity
Constrain
Capability + quality floor
Score
Cost × quality
Route
Dispatch to provider
Reconcile
Log actual cost
02 · Estimated per-run agent budget guardrails
A budget is set per agent run. Flux reserves the estimated cost of each step before dispatch, blocks any request predicted to exceed the configured budget, and reconciles the reservation against actual reported usage once the response comes back.
Below 70%: normal model tier
70 to 90%: degrade to cheaper models
90%: warning raised. 100%: hard stop with a 429
This is a guardrail based on token and price estimates, not a guarantee against every provider-side billing discrepancy. Pooling one shared budget across multiple concurrent agents over a time window is on the roadmap and not yet built.
Route every step of an AI agent intelligently.
A planning step and a tool-result summary do not require the same model. Flux applies step-aware routing so agents can reserve premium models for the steps that need them, across OpenAI-compatible wrapper traffic and multi-step agent workflows alike.
X-Flux-Run-Id: run-123 X-Flux-Step-Type: plan
- planPlanning
- tool_selectTool selection
- tool_resultTool-result interpretation
- reflectReflection
- finalFinal-answer generation
Your AI control plane. Your infrastructure.
Deploy Flux beside your application and keep routing policy, budgets and telemetry under your control. Model requests still go to the providers you configure; Flux itself does not require another managed routing service.
Automatic fallback
Typed fallback chains for rate limits, timeouts, and content filters, handled separately per failure mode.
Circuit breakers
A provider that starts failing is tripped out of the candidate pool until it recovers.
Model-availability filtering
Only models marked available in the catalog enter scoring, so routing doesn't pick something you can't call.
Per-tenant budgets
Tag requests with a customer or tenant ID for per-customer cost tracking and daily ceilings.
Estimated run-spend guardrails
Reserves estimated cost before dispatch and reconciles it against actual reported usage after the call.
Streaming and non-streaming
Both request modes are supported through the OpenAI-compatible wrapper.
Structured output
Routes structured-output requests only to models that support the required response format.
Tool calling
Routes tool-calling requests only to models in the catalog marked as supporting function/tool calling.
Local dashboard and telemetry
Prometheus-format routing and budget metrics, plus a local dashboard. Telemetry stays in your infrastructure.
Capability-aware routing
Filters candidates to models that support the request's required capabilities before scoring.
Prompt-cache-aware selection
Avoids switching providers mid-conversation when doing so would break a warm prompt cache.
Redis-backed multi-worker enforcement
Required for correct shared run-budget state when Flux is deployed across multiple worker processes.
Spend control is an enforced estimated spend guardrail based on token and price estimates, not an absolute billing guarantee. Quality floors are a configured quality threshold based on catalog ratings, not a guarantee of objective answer quality. Flux is not a medical, legal, or safety decision system. High-stakes classification is rule-based and should not replace application-level safety controls.
What we can back up, and what we can't yet
We'd rather tell you the boundary than let you find it in production.
Verified
- 1,400+ automated tests
- Offline routing benchmark passes all nine benchmark checks
- 500/500 local routing decisions completed under 10 ms
- Security scanning reports no medium/high Bandit findings
- Wrapper and agent-routing code paths have automated test coverage
Not yet proven
- Savings on real customer production traffic
- Live answer-quality superiority across every provider/model
- Empirical agent-step-specific quality ratings for every model
- Native incremental streaming for every provider
- Universal medical/legal intent detection
Verification status as of Aug 10, 2026. Test and latency figures come from our own offline runs; your results depend on your workload.
The self-hosted control plane purpose-built for cost-aware wrappers and step-aware AI agents.
LiteLLM and Portkey can also be self-hosted (see LiteLLM's proxy deployment docs and Portkey's documentation). Flux puts the points below in one layer: routing that decides per request and per agent step, with cost governance next to it. For a row-by-row breakdown, see Flux vs OpenRouter vs LiteLLM.
- 01Per-request automatic model selection. Sub-millisecond, pure-Python heuristic routing based on task type and complexity scoring. You don't maintain the routing config by hand.
- 02Step-aware agent routing. Planning, tool selection, tool-result interpretation, reflection and final answers each get their own routing treatment.
- 03Cost governance in the routing layer. Per-tenant budgets and estimated run-spend guardrails live where the routing decision is made, not in a separate billing system.
- 04Adaptive learning from response quality. Per-(model, task) exponential moving average, optionally per customer, so routing adjusts as it accumulates outcomes.
- 05Local telemetry and policy ownership. Routing policy, budgets and telemetry stay in your infrastructure. Model requests go only to providers you configure.
See what intelligent routing could save
Pick the model you'd otherwise send everything to. The estimate uses the offline synthetic benchmark result for that baseline, not live traffic, so it can differ from the headline figure.
current baseline model
estimated savings based on the selected Flux benchmark scenario
per month
That's $20.4k/year versus GPT-4o at 84.8%.
Illustrative estimate. Actual savings depend on workload, model mix, pricing and routing configuration. See methodology.
View benchmark →Every number above, and exactly how we got it
Offline synthetic evaluation. Reproducible from the committed benchmark artifact.
- sample count
- 500 requests, fixed set, same set across all baselines
- workload composition
- Mixed task types: summarization, classification, extraction, code, reasoning, and multi-step agent traces
- baselines
- Every request sent to a single model: GPT-4o (84.8%), Claude Sonnet 4.6 (89.3%), Gemini 2.5 Pro (82.2%)
- model catalog & pricing snapshot
- 30 catalog models across five providers, priced from the committed registry snapshot, Aug 2026. Current list prices: OpenAI API pricing, Anthropic API pricing, Gemini API pricing, Groq pricing, Mistral pricing
- test date
- August 2026
- environment
- Single local machine, no network calls to providers; routing evaluated offline
- routing latency definition
- Time to produce a routing decision only. Excludes provider inference, network latency and full HTTP gateway overhead
- quality basis
- Configured quality threshold from catalog ratings, not graded answer quality on live traffic
- benchmark artifact
- Benchmark scripts and results are available to early-access users on request. Contact the Flux team to review them.
Savings figures are estimates derived from registry pricing and catalog-rated model quality on a synthetic request set. They are not measurements of live production traffic. Actual savings depend on workload, model mix, provider pricing and routing configuration. Competitor capability statements elsewhere on this site are drawn from each project's public documentation; source links and access dates accompany the Flux vs OpenRouter vs LiteLLM comparison.
Benchmark data as of Aug 25, 2026
Built for teams already worried about the bill
Already had a runaway agent bill
One bad loop, one missing guardrail, and a single agent run outspent a week of normal traffic. Flux's per-run budget reservation is built for exactly that failure mode.
Margin-sensitive, fixed-price contracts
If an agent's cost overruns eat into your margin, an estimated per-run budget cap is the guardrail your pricing model assumes exists.
Regulated buyers: fintech, healthtech, GCCs
Routing logic, budgets, and telemetry stay in your infrastructure, which matters when your policies rule out a hosted gateway.
If one of these sounds familiar, talk to the Flux team about your agent workloads or compare the four ways to work with Flux. Custom plans are priced per governed run and tenant; see how Flux pricing works.
Four ways to work with Flux
Flux OS is free and you install it yourself. The other three are sold under a signed agreement. There is no self-serve checkout.
- 01Flux OS. A free, open-source router under the MIT license. It sends each request to the cheapest model that clears a quality bar and reroutes to another model when a call fails. No budgets, dashboard, spend tracking or telemetry. See real Flux OS output or Flux OS on GitHub.
- 02Flux Custom Plans. The full, proprietary Flux product, self-hosted: run-level budget governance, graceful degradation, hierarchical caps and the dashboard, set up for your infrastructure. How Flux Custom Plans work.
- 03Enterprise Solutions. Flux for larger deployments that need custom quality floors, dedicated support or a setup tailored to their workflow. What Enterprise Solutions cover.
- 04Managed SLM Deployment. We fine-tune an open-license small language model on your own data and deploy it on your own server, for a setup fee plus ongoing maintenance. How Managed SLM Deployment works.
You still pay your chosen model providers directly when you use Flux. To hear when early access opens, send a note through the Flux contact form.
Install, run, point your SDK
Start with flux-os. It is free, MIT licensed and installs from GitHub. It sends each request to the cheapest model that clears a quality bar and reroutes to another model when a call fails. Run budgets, spend tracking and the dashboard are part of full Flux, which we set up with you on a custom plan.
- Works with an unmodified OpenAI SDK client
- Runs on your machine and calls your providers with your keys
- Also works as a Python library, with no server
flux-os, free and MIT licensed · github.com/vbc1406/flux-os
pip install "git+https://github.com/vbc1406/flux-os" # See what it would do. No API keys needed for this: flux-os route --all "Prove that there are infinitely many primes" export OPENAI_API_KEY=sk-... ANTHROPIC_API_KEY=sk-ant-... GROQ_API_KEY=gsk_... flux-os serve # http://127.0.0.1:8000 from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") resp = client.chat.completions.create( model="auto", # or flux-cheap / flux-fast / a concrete model id messages=[{"role": "user", "content": "Explain backpropagation in two sentences."}], ) print(resp.model) # the model that served it
prefer no server?
from flux_os import FluxOS flux = FluxOS() resp = flux.chat.completions.create(model="auto", messages=[{"role": "user", "content": "hi"}]) print(resp.choices[0].message.content, resp.flux["model"])
Stop overspending
on AI
Self-hosted routing, cost governance and budget guardrails. Now in early access.
Frequently asked questions
What is Flux?
Flux is a self-hosted LLM gateway that controls AI agent cost. It sits between your app or agent and your model providers, tracks spend across a whole agent run, and routes each call to the cheapest capable model.
What is agent cost governance?
Agent cost governance means controlling what an entire agent run is allowed to spend, including retries and subagents, instead of only limiting individual requests. Flux gives each run one shared budget.
How does Flux degrade models as the budget runs out?
Degradation to cheaper models starts at 70% of the run budget, a warning is raised at 90%, and Flux only hard-stops with a 429 once the budget is fully used. Degradation is the primary mechanism and the hard stop is the last resort.
How is Flux different from OpenRouter or LiteLLM?
To our knowledge, gateways like OpenRouter and LiteLLM route or proxy individual requests, and their budget limits are typically scoped to API keys, teams or users. Flux tracks cumulative spend across an entire agent run using a shared run ID, so routing gets more cost-sensitive as the run consumes its budget. Flux is also fully self-hosted.
Do I have to change my code to use Flux?
Very little. Flux exposes an OpenAI-compatible endpoint, so you point your OpenAI SDK base URL at Flux and pass an X-Flux-Run-Id header to group calls into a run.
Where does my data go?
In the self-hosted deployment, Flux runs in your own infrastructure. Requests go from your deployment to the model providers you configure with your own API keys.
How much can Flux save?
Our offline benchmark on synthetic requests shows up to 80% estimated savings. It is an estimate, not a measurement of live traffic. Results depend on your task mix, so see the Benchmark Methodology section for how the numbers were measured and their limits.
Which model providers does Flux support?
Flux supports five model providers: OpenAI, Anthropic, Google, Groq and Mistral.
Is Flux open source?
Flux OS, our lightweight self-hosted router, is free and open source under the MIT license. It does routing and rerouting only, with no budgets, dashboard, spend tracking or telemetry. The full Flux product, with run-level budget governance and hierarchical caps, is proprietary and offered on custom plans.
Can I try Flux for free?
Yes. Flux OS is free and open source, so you can self-host it and get familiar with the approach before moving to the full product or a custom plan.
Do you offer on-prem or private small language models?
Yes. We fine-tune an open-license small language model, for example Qwen, on your data and deploy it on your own servers. The base model's own license applies, you need the rights to the training data, and performance is only guaranteed as far as a signed statement of work says. Pricing is a setup fee plus ongoing maintenance. Contact us for a quote.