Inferenz sits in front of OpenAI and Anthropic, reads every request as it comes in, and sends it to the cheapest model that can actually handle it — automatically, in milliseconds, with no changes to your product.
Inferenz is a real sequence, not a feature list: every request passes through the same four stages, in order, before a single token is generated.
Swap your base URL from api.openai.com to your Inferenz endpoint. Every request your product sends now passes through Inferenz before it reaches any model provider.
baseURL change onlyBefore anything else, the request is compared against recent traffic. Exact or near-duplicate prompts are served instantly from cache — zero model calls, zero cost.
< 1ms · $0A cache miss gets scored for real cognitive difficulty — not word count. Simple requests route to a cheap model immediately. Genuinely hard ones go straight to your premium model. Anything in between gets a second check before it's allowed to spend more.
< 5ms · no model callWhat you paid and what you would have paid on your old setup are both recorded, per request. That log is what your dashboard shows you — and it's the only basis for what we ever invoice.
Full audit trailEvery request passes through all three in order — each one resolving as many requests as possible before the next runs. By the time a premium model is involved, it genuinely needs to be there.
Before the request reaches any classifier or model, it is compared against recent traffic. The Cache layer runs two checks in sequence.
First, an exact match — byte-for-byte identical to a prior request. Second, a similarity check that catches the same question phrased differently: "summarise this email" and "can you summarise this email for me?" are semantically identical and should cost the same — zero.
The similarity method is lightweight by design. It does not use an embedding model (which would itself cost money and time). Instead it uses a combination of token-level overlap and character-sequence matching — two techniques that together catch the variation patterns that actually appear in production traffic: word reordering, minor additions, and paraphrasing.
Real AI product traffic is not random — it clusters around a defined set of use cases. A customer support product does not receive a random sample of all possible human questions. It receives the same 50–200 underlying queries, rephrased millions of times. The cache hit rate on repetitive workloads is high by the nature of the product, not by luck.
A cache miss is scored for genuine cognitive difficulty before any model sees it. The key insight: you do not need a model to decide whether a prompt needs a model. Difficulty is a property of the request itself, and it is measurable.
The engine analyses every prompt across multiple independent signals simultaneously. The primary signal is based on Bloom's Taxonomy — a framework from educational psychology, developed at the University of Chicago and validated across decades of research, that classifies cognitive tasks from simple recall through to creative synthesis. The verbs and structure of a request reliably indicate which level of reasoning it requires.
This is combined with information density measurement — based on Claude Shannon's foundational 1948 information theory work — which quantifies how much meaningful content is packed into the request. Structural signals (code, conditional logic, multi-step instructions) and domain vocabulary contribute further signals. Each is independent. Together they produce a routing score that is deterministic: the same prompt produces the same routing decision every time, without drift or randomness.
2025 research applying Bloom's-based classification to LLM routing achieves classification accuracy above 93% on held-out prompt sets. Importantly, the failure modes are predictable — when the Complexity engine is uncertain, it routes to the Cascade layer rather than guessing. The system is designed to be conservative: it never sends a complex request to a cheap model based on a borderline score.
Some requests genuinely sit between simple and complex. For these, the Cascade layer takes a different approach: rather than guessing, it attempts the cheaper model first, then independently assesses whether the response actually holds up before delivering it to your user.
The verification step checks the response across multiple confidence signals: whether the response is complete relative to what was asked, whether the model hedged or expressed uncertainty, whether it refused or partially refused, and whether it followed any specified format. These failure modes are reliable and cheap to detect.
If the cheap response passes, the saving is logged and delivered. If it fails, the request is automatically escalated to the premium model and answered properly. Your user experiences no degradation in either case — the verification happens before delivery, not after.
This approach is validated by AutoMix, published at NeurIPS 2024, which demonstrated that attempting cheap models first and verifying responses before delivery achieves significant cost reduction with statistically negligible quality degradation at the population level. Cheap model failures are almost always detectable through the response itself — the model hedges, refuses, gives a suspiciously short answer, or fails to follow format instructions. These signals are reliable gatekeepers.
No monthly fee. No setup cost. No lock-in. We take a percentage of the savings we generate — and that percentage decreases as your savings grow. Our incentive is always aligned with yours.
Savings are verified per-request via shadow accounting · Invoiced monthly, based on that record
A scan isn't a sales pitch — it's your own usage data, run through our routing engine on the next page, so you see the exact saving before you touch a line of code.
A 30-day CSV export from your OpenAI or Anthropic dashboard. Token counts and model names only — no prompt content required.
Every row is classified using the same Cache, Complexity, and Cascade logic that runs in production — not a rough estimate. You upload the CSV and the scan runs on the page.
A one-page result: your current spend, your optimised spend, and the monthly saving. If the number's small, we'll tell you that too.
No pressure, no auto sign-up. If it's worth doing, switching over is a one-line change whenever you're ready.
Runs on the live engine · If the saving is small, we say so
No pressure, no script — just tell us what you're building and what you're spending, and we'll tell you honestly if this is worth your time.
hello@inferenz.com.au