Flux routes every wrapper request and agent step to the best-fit model — delivering up to 89% estimated savings in our benchmarks with millisecond routing.
from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="your-flux-token", ) response = client.chat.completions.create( model="flux-auto", messages=[ {"role": "user", "content": "Summarize this document"} ], )
# Same OpenAI-compatible call — add two headers POST /v1/chat/completions X-Flux-Run-Id: run-123 X-Flux-Step-Type: plan # Flux tracks spend against run-123 and keeps a # higher quality floor for "plan" steps than for # mechanical steps like tool_result.
Most AI systems send every request to one expensive model. Flux evaluates each request and agent step, then selects the least expensive available model that satisfies the configured capability and quality requirements.
in Flux's synthetic evaluations, measured across three baseline scenarios.
Offline synthetic evaluation using registry pricing and catalog-rated model quality. Results vary by workload, routing policy, pricing and model availability. Full methodology →
Routing is pure heuristic logic — no LLM call sits in the decision path, so the decision itself is effectively free next to inference.
Routing-decision measurement only. Provider inference, network latency and complete HTTP gateway overhead are measured separately. Methodology →
Flux routes every call to the best-fit model on cost, capability, latency and reliability — and separately, applies an enforced estimated spend guardrail on what an agent run can consume.
Every request is classified by task type, filtered down to models that meet configured capability and quality-floor requirements, then scored on cost against quality and dispatched — pure heuristic logic, no LLM call in the routing path.
A budget is set per agent run. Flux reserves the estimated cost of each step before dispatch, blocks any request predicted to exceed the configured budget, and reconciles the reservation against actual reported usage once the response comes back.
This is a guardrail based on token and price estimates, not a guarantee against every provider-side billing discrepancy. Pooling one shared budget across multiple concurrent agents over a time window is on the roadmap — not yet built.
A planning step and a tool-result summary do not require the same model. Flux applies step-aware routing so agents can reserve premium models for the work that genuinely needs them — across OpenAI-compatible wrapper traffic and multi-step agent workflows alike.
Deploy Flux beside your application and keep routing policy, budgets and telemetry under your control. Model requests still go to the providers you configure; Flux itself does not require another managed routing service.
Typed fallback chains for rate limits, timeouts, and content filters, handled separately per failure mode.
A provider that starts failing is tripped out of the candidate pool until it recovers.
Only models marked available in the catalog enter scoring, so routing doesn't pick something you can't call.
Tag requests with a customer or tenant ID for per-customer cost tracking and daily ceilings.
Reserves estimated cost before dispatch and reconciles it against actual reported usage after the call.
Both request modes are supported through the OpenAI-compatible wrapper.
Routes structured-output requests only to models that support the required response format.
Routes tool-calling requests only to models in the catalog marked as supporting function/tool calling.
Prometheus-format routing and budget metrics, plus a local dashboard — no data leaves your infrastructure.
Filters candidates to models that support the request's required capabilities before scoring.
Avoids switching providers mid-conversation when doing so would break a warm prompt cache.
Required for correct shared run-budget state when Flux is deployed across multiple worker processes.
Spend control is an enforced estimated spend guardrail based on token and price estimates, not an absolute billing guarantee. Quality floors are a configured quality threshold based on catalog ratings, not a guarantee of objective answer quality. Flux is not a medical, legal, or safety decision system — high-stakes classification is rule-based and should not replace application-level safety controls.
We'd rather tell you the boundary than let you find it in production.
LiteLLM and Portkey can also be self-hosted — self-hosting isn't the differentiator. What's rare is the combination below: routing that decides per request and per agent step, with cost governance in the same layer. Sources and access dates in the methodology.
Sub-millisecond, pure-Python heuristic routing based on task type and complexity scoring — not a config you maintain by hand.
Planning, tool selection, tool-result interpretation, reflection and final answers each get their own routing treatment.
Per-tenant budgets and estimated run-spend guardrails live where the routing decision is made, not in a separate billing system.
Per-(model, task) exponential moving average, optionally per customer — routing sharpens as it accumulates outcomes.
Routing policy, budgets and telemetry stay in your infrastructure. Model requests go only to providers you configure.
Pick the model you'd otherwise send everything to. The estimate uses the measured Flux benchmark scenario for that baseline, not one blanket percentage.
Offline synthetic evaluation. Reproducible from the committed benchmark artifact in the repo.
Savings figures are estimates derived from registry pricing and catalog-rated model quality on a synthetic request set — not measurements of live production traffic. Actual savings depend on workload, model mix, provider pricing and routing configuration. Competitor capability statements elsewhere on this site are drawn from each project's public documentation; source links and access dates accompany the comparison in the repo README.
One bad loop, one missing guardrail, and a single agent run outspent a week of normal traffic. Flux's per-run budget reservation is built for exactly that failure mode.
If an agent's cost overruns eat into your margin, an estimated per-run budget cap is the guardrail your pricing model assumes exists.
Routing logic, budgets, and telemetry stay in your infrastructure — a requirement hosted gateways can't meet by policy.
Flux is free to self-host under AGPL-3.0. You still pay your chosen model providers (OpenAI, Anthropic, Google, Groq, Mistral) directly — Flux only controls routing and spend on top of that. Embedding or reselling Flux needs a separate commercial license. Exact managed-tier pricing isn't finalized; join the waitlist for early access and founding pricing.
We're an early-stage startup building Flux in the open. For any queries, reach us at fluxllmdev@gmail.com
Flux isn't published to PyPI yet — install from a local clone in editable mode, then point your existing OpenAI SDK client at the server.
# Clone and install (editable, no PyPI package yet) git clone https://github.com/vbc1406/flux-router.git cd flux-router pip install -e . flux serve # Point the OpenAI SDK at it — nothing else changes from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1") resp = client.chat.completions.create( model="flux-auto", messages=[{"role": "user", "content": "Summarize this document..."}] )
from router.flux import make_flux flux = make_flux() # reads provider keys from env resp = asyncio.run(flux.complete( "Summarize this document...", user_id="u123", ))
Self-hosted routing, cost governance and budget guardrails. Free and open source today.