Langfuse put its AI Gateway into public beta on November 4. You point your app or coding agent at Langfuse instead of OpenAI or Anthropic, then change two values: the base URL and the API key. Request and response bodies stay exactly as they are.
Two problems go away at once. Nobody shares a provider key anymore, and every call that used to be invisible becomes a trace.
What shipped
An admin adds OpenAI and Anthropic credentials under Organization settings, picks which models gateway keys can call, and picks the project that receives the traces. Everyone else issues their own key and copies setup instructions from the key dialog.
The gateway serves the provider APIs as they are, streaming included: OpenAI Responses, OpenAI Chat Completions, Anthropic Messages. It doesn't translate between them. An OpenAI client can only reach OpenAI models. That constraint is what keeps setup to two lines. Request and response bodies are whatever your client already sends.
Day one support covers Claude Code, Claude Desktop, Codex, the GitHub Copilot CLI and app, OpenCode, and pi. Each one is configured with a base URL and a key, so there is no plugin or wrapper library to install.
The keys do the actual work
A proxy that logs traffic is a weekend project. What makes this usable inside a company is the key model.
- User keys and service account keys. A user key belongs to the person who created it. It dies when they lose gateway access or leave. Service account keys belong to the org, so CI and shared services keep working when people move.
- Model allowlists. Admins enable or disable models per credential. A new model doesn't quietly become available to everyone.
- Expiry and revocation. Set an expiry on any key. Revocation is immediate. Creating and revoking keys both land in the audit log.
- Key metadata. Stamp a key with
team: platformorcost_center: 4711. The gateway attaches it to every generation recorded with that key, and the client can't override it. - Gateway-only scope. A gateway key can call the gateway. It can't read or write anything else in Langfuse.
The provider key never leaves Langfuse, which kills the shared .env file that six people can read and nobody wants to rotate.
Cost per person, not per invoice
Owners and admins get an Analytics tab with total cost, requests, and tokens, plus cost over time and the top users, keys, and models by cost. Filter by user or key to find where a spike came from. Then open the trace behind it.
One detail to settle before you turn it on. Capture mode is per organization and the default is USAGE: model, tokens, cost, latency, and model parameters, with no prompts or outputs. If you want input and output in the trace, choose FULL. Treat it as a security decision as much as a debugging one, and make it deliberately.
Key metadata is what makes the spend view match your org chart. The traces are ordinary Langfuse traces, so you can build dashboards on those fields later, including cost by team and cost by cost center.
Why the ClickHouse ownership matters
Every trace, observation, and score in Langfuse lives in ClickHouse. That has been true since v3 moved tracing data off Postgres in December 2024, and it is true on Cloud and self-hosted. Self-hosting still requires ClickHouse, with no alternative OLAP database supported.
In January 2026, ClickHouse acquired Langfuse. The core stays open source and self-hostable, though some add-on features need a license key. Cloud keeps running as a separate service.
The tracing layer and the gateway now sit on a database your data team may already operate, owned by a company with every commercial reason to keep that workload fast. If your objection to hosted observability has been "we are not sending prompt data to another startup," the self-hosted gateway runs next to your own ClickHouse. Few vendors can shorten that conversation the same way.
It is also one less vendor in the AI data path. Gateway, traces, evals, and prompts in one system, instead of a gateway vendor plus an observability vendor plus a spreadsheet that reconciles them.
Where this beats the warehouse gateways
Snowflake and Databricks both shipped AI gateways this year, and we compared them in another post. Those make sense when your center of gravity is the warehouse. Governed data, identities, and usage accounting already live there.
Langfuse starts from a different assumption. AI traffic is already spread across dozens of apps and agents, most of which you will never instrument on a useful timeline, and the people generating the spend are engineers running coding agents. Swapping a base URL is the only enforcement mechanism that survives contact with that.
Keep one limitation straight. The gateway sees model calls, not what your app does around them. Retrieval, tool execution, and your own logic stay invisible unless you also instrument the app with the SDK. You can merge the two by passing a traceparent header, and the gateway records its generation inside your existing trace. Merging them takes work, because the gateway cannot infer what your app did around the call.
What is still broken
It is a beta, and Langfuse documents the rough edges. Read them before you rely on gateway traces for audits or chargeback.
- Large inputs are not recorded. Bodies over 5 MiB produce a generation with no input. Long Codex and Claude Code sessions hit this. Output over 1 MiB is truncated in the trace even though the client gets the full response.
- Some costs are wrong right now. Failed and cancelled OpenAI calls get estimated usage and cost. OpenAI cache write tokens are priced at zero. Anthropic one-hour cache writes are billed at the five-minute rate, about 37.5% too low. Use this for trends, not invoicing.
- Trace granularity varies by agent. Claude Code, OpenCode, and pi record one trace per model call by default. Codex groups a turn into one trace.
- No reliability features yet. There are no retries, fallbacks, or caching. Cloud applies rate limits, plus a 16 MiB body cap and 600 seconds per call.
- Spend caps are roadmap. Hard caps, PII redaction, SCIM provisioning, and more providers are all listed as future work.
Two commercial details. Your provider still bills model usage to your own account, and each generation the gateway records counts as a billable Langfuse unit. On the HIPAA region you need a BAA with Langfuse and with your model provider before you send PHI.
How I would roll it out
Week one. Pick one team already burning real money on Claude Code or Codex. Add the credentials, enable only the models you have approved, set 90-day key expiry, attach cost center metadata.
Week two. Reconcile. Compare the gateway cost dashboard against the provider invoice on a sample of calls. Write down the known gaps above so nobody argues about a 3% difference later.
Week three. Move one production agent you cannot instrument. Change the base URL, confirm traces arrive with usage, cost, and latency, and let it run long enough to see a real spike.
Week four. Decide capture mode and retention, document the audit trail, then talk about broader rollout.
What I wouldn't do is put the gateway in the request path for anything that cannot survive a beta provider outage. There are no fallbacks yet. Treat it as a control and observability layer with a single point of failure, and scope it that way.
The short version
Observability and gateway stop being separate purchases: every call that walks through the door is already a trace, priced, attributed to a key, and attached to a person. Most AI cost conversations are missing exactly that, which is why a pilot is worth running.
Pick one team and run it for a month. Then compare the cost dashboard to the provider invoice. If they match, you have chargeback for coding agents. If they do not, you learned that before you scaled. If you want a second pair of eyes on the rollout, this is what we do.
Sources
- Langfuse, "Launching the Langfuse AI Gateway" (November 4, 2026)
- Langfuse documentation, "AI Gateway" overview and roadmap
- Langfuse documentation, "Send your first request"
- Langfuse FAQ, "What are the known limitations of the AI Gateway beta?"
- Langfuse engineering, "How Langfuse runs ClickHouse at agent scale"
- Langfuse, "Langfuse joins ClickHouse" (January 16, 2026)
- ClickHouse, "ClickHouse welcomes Langfuse: The future of open-source LLM observability"
- Langfuse documentation, "Self-host Langfuse"
