Most of my work runs through AI now, and Pi is basically always open in my terminal.
It just feels good to use: fast, no waffling, you say one thing and it moves, rarely spinning in circles. It’s noticeably smoother than the native agents. But feel is hard to pin down. I couldn’t tell you where exactly it’s better, or by how much.
Then I saw this Databricks benchmark. They paired the major models with the major CLI tools, two by two, and ran the whole matrix against their own multi-million-line production codebase. For the same completed tasks, Pi burned noticeably fewer tokens.
The moment the numbers landed, my vague hunch finally had something solid under it.
This piece covers three things:
- what the benchmark actually shows;
- why Pi is so cheap for what you get;
- how to pick the right agent for yourself.
1. What Pi Is
First, what Pi is. It’s an open-source command-line coding tool, a project started in August 2025, now sitting at 70k+ stars.
It does roughly what Claude Code and Codex do. But take the same model and wrap it in a different agent, and the results can swing a lot, because that outer shell decides how much gets sent to the model each turn, which tools the model gets, how it judges whether a task is done, and so on.
And as an independent open-source project, not being locked to any one model is a real advantage too.
2. The Verdict: Pi > Claude Code/Codex
On July 8, 2026, Databricks published an internal benchmark report. Among the authors is Matei Zaharia, the creator of Spark and the company’s current CTO. They paired the major models with the major CLI tools, two by two, ran the whole thing against their own multi-million-line production codebase, and plotted an agent-by-model cost-to-quality chart.
Of the 7 top-scoring model-agent combos on the chart, 4 are Pi. The single highest pass rate, 90%, is also Pi.
In other words, on real production code, an open-source project beat the official tools that Anthropic and OpenAI each built specifically for their own models.
And it’s much cheaper. On the same Opus 4.8 at high effort, Claude Code averages $1.94 per task and Pi runs about $0.95, more than half off, with the pass rate differing by just 2 points.
The benchmark also makes another thing clear: a cheaper model isn’t automatically the better deal.
Sonnet 5 has a lower per-token price than Opus 4.8, yet its total cost per task is higher, burning 1.9x the tokens Opus does.
What you should actually compare is “how much it costs to finish one task,” not “the sticker price per million tokens.”
3. How the Data Was Collected
Before you trust a score, it’s worth seeing how it was produced.
No more public test sets
The usual references for coding ability today are public benchmarks like SWE-Bench and Terminal-Bench. Databricks says outright that it didn’t use them, and the reasoning is solid: the problems are public, the answers are online, and every lab sweeps that content into training.
The result is that a chunk of a model’s score on those problems comes from having seen them, not from being able to solve them. It’s the same as getting the exam questions before the exam.
There’s a second issue: tasks in public benchmarks don’t look much like what engineers actually write day to day.
The former is more like doing exercises. The latter is changing things inside a codebase that’s been running for years, carrying its own history and its own internal conventions.
Using their own merged PRs as the test set
Databricks engineers merge thousands of code changes a day. They pulled from that pool with a filter: a change had to be self-contained, not sprawling across too many modules, and it had to come with a solid test suite.
Then they threw out three categories: anything submitted by bots, anything from service accounts, and anything that was itself AI-generated.
Every surviving change got a human pass, boiling down what it was trying to do into a one-line task description, while stripping out any part of the original description that gave away the solution — leave it in, and the problem hands you the answer.
The codebase spans more than a dozen languages: Scala backend services, Rust at the systems layer, React plus TypeScript on the frontend, protobuf and gRPC interface definitions, Bazel build configs, all of it.
Grading uses the test suite the PR originally shipped with. First, does the code compile; if it compiles, run the tests; all tests pass, it counts as passed. No second model was used as a judge.
That deserves its own sentence. Plenty of benchmarks these days take the easy road and let GPT or Claude decide whether another model did well, and that scoring method is shaky, because the judge misreads things and has its own preferences. A test suite is binary: it passed or it didn’t.
They patched the hole where agents peeked at git history
The report admits something: early in the runs, some models scored “unrealistically high.”
Digging through the execution logs, they found these models were going straight into the git commit history and pulling out the correct answer — every problem came from an already-merged commit, the git history in the workspace wasn’t isolated cleanly, so the answer was left sitting in the exam room.
They later plugged that hole, and I read this as a good sign.
A benchmark report willing to write “we got it wrong at first” is more credible than one that only reports wins. It also tells you the test set was built and fixed as they went.
4. Why Pi Is So Cheap for What You Get
A lean system prompt
Before Claude Code does anything, before it’s even your turn to speak, it has already sent the model 20,000-plus tokens.
A token is the smallest unit a model uses to bill and process text. A Chinese character is roughly one to two tokens; a normal line of code, a dozen or so. Twenty-thousand-plus tokens is about a 30,000-to-40,000-word booklet.
This number was reverse-engineered by the community, not published officially — someone intercepted the requests and analyzed the full payload Claude Code sends each time. As of the February 2026 version, the system prompt itself is about 23,500 tokens, plus definitions for 27 built-in tools, and those definitions run close to 100,000 characters on their own. Together they form the fixed cost of every conversation.
Pi’s side: the system prompt is under 1,000 tokens, with 4 built-in tools — read file, write file, edit file, run command.
23,500 vs 1,000. 27 tools vs 4.
Someone tested an extreme case: say “hello” through Claude Code, and those two words are worth a handful of tokens, but the actual request burned over 40,000.
Even with caching, the lean system still wins outright
At this point someone will say it’s not a problem — everyone has prompt caching now, and within a session everything after the second turn bills at a tenth.
True, but only half true.
Caching solves the billing problem of “resending identical content.” What it doesn’t solve is this: those 20,000-plus tokens genuinely occupy the model’s attention on every single turn.
Databricks’ key finding is this: Pi sends the model about a third of the content the other tools do, per turn. That gap persists across the whole task, and the longer the task, the more files you touch, the denser the back-and-forth, the more it compounds.
There’s another spot that’s easy to miss: subtasks.
Claude Code lets you hand work off to a separate subagent, but each subagent, on every one of its own turns, has to load the full system prompt and tool definitions again from scratch.
They measured one task: done directly, 121,000 tokens; split across two subagents, 513,000 tokens, more than 4x.
A leaner prompt makes the model smarter
This is the point I most want to nail down.
Longer context makes the model dumber. There’s evidence for this.
The 2023 Stanford and Berkeley paper Lost in the Middle tested six model families and found that the same piece of information, placed at the start or end of the input, gets used accurately; placed in the middle, accuracy drops by more than 30%. The cause ties back to the model’s internal positional encoding — stretch the distance and attention naturally decays.
In 2025 Chroma ran a more real-world test across 18 models. The conclusion: as input grows, performance drops 20% to 50%. They specifically noted that a model advertising a 200k-token window already shows clear degradation by 50k tokens. And the decay isn’t linear: the more a task needs synthesis and the fuzzier the semantics, the faster it falls off.
Put the two together: those extra 20,000 tokens you send every turn both charge you on the bill and dilute the model’s attention on the part that actually matters.
Saving tokens and raising quality are, here, the same thing.
As models get smarter, the thinner the harness the better
Pi’s author, Mario Zechner, said something to this effect: in a harness, how much is actually load-bearing, and how much is only there because everyone else does it that way? Pi answers that question by deleting.
He also said the tool should be small enough that you can hold every one of its tokens in your head.
That’s the opposite of how big companies think about products. Over the past year, Claude Code and Codex have kept piling on features — task management, team collaboration, all sorts of subsystems.
The system prompt I mentioned earlier grew 28% within two weeks in February 2026, and the growth was mostly the newly added task-management and collaboration modules.
More features isn’t a bad thing. But every feature has to claim a chunk of the system prompt to tell the model “you can also do this,” and that chunk gets paid for and takes up attention on every single turn — even when this particular task of yours has no use for it.
The conspiracy angle
On Hacker News, someone put it more bluntly.
A top comment: Anthropic can’t shake the suspicion of “inflating token consumption with its own tool,” because the data is right there, and Pi is more than twice as efficient at managing tokens.
Someone below added: the incentives simply don’t line up. The bigger the context Claude Code carries, the more Anthropic makes.
I don’t want to state this as fact — the main reason big-company system prompts are long is genuinely that they have more features and more scenarios to cover, not necessarily pure profit motive.
But one thing is objectively true: neither Anthropic nor OpenAI has any incentive to “save users tokens.”
Which is probably why an independent open-source project does this better.
5. So Does Pi Just Crush Claude Code/Codex Across the Board?
Up to here it reads like a Pi landslide. But we should look at the full data.
That Databricks comparison chart has six groups, each one the same model at the same effort level, only the shell swapped. Here’s the complete data:
| Model + effort | Pi | Native tool | Delta | Cost |
|---|---|---|---|---|
| Opus 4.8 high | 85% | 87% | −2 | Pi 2.08x cheaper |
| Opus 4.8 xhigh | 90% | 88% | +2 | Pi 1.46x cheaper |
| Opus 4.8 max | 82% | 89% | not labeled | Pi 1.20x cheaper |
| GPT-5.5 medium | 83% | 80% | +3 | Pi 1.54x cheaper |
| GPT-5.5 high | 81% | 83% | −2 | Pi 1.22x cheaper |
| GPT-5.5 xhigh | 78% | 80% | −1 | Pi 1.44x cheaper |
Across the six, Pi wins two and loses four. But three of those four losses are within 2 points, while the cost advantage holds steady between 1.2x and 2.08x.
Databricks’ summary of the chart: “significant cost difference, quality held constant.”
Still, I see a few problems in it.
At high effort, Pi’s advantage vanishes
Look at the third row: Opus 4.8 max, 82% vs 89%.
Same model, only the effort dialed from xhigh to max, and Pi drops from the field-leading 90% to 82%, now losing to the native tool by 7 points.
That directly contradicts “quality held constant.”
Seven points is a big gap in this data — enough to knock this combo from the top tier (82% to 90%) down to the second tier (71% to 82%).
And this is the one row of the six with no delta labeled. The other five dutifully print “−2 pt” or “+3 pt”; only this one is blank. The body text doesn’t mention the data point either.
I don’t think this is deliberate concealment — the chart and the numbers are all right there for anyone to count. But a data point that contradicts the core conclusion is sitting there without a word of explanation.
As for why, the plausible read is: at max effort, the model’s own reasoning gets very long, and Pi’s “send as little as possible” strategy may no longer be enough to keep up.
And the feature gaps that come with minimalism
Beyond the benchmark, Pi has some real gaps.
No permission system.
This is Pi’s own project docs: it builds in nothing to restrict file access, processes, network, or secrets, and by default runs with the full permissions of whoever launched it.
Want boundaries, you wrap it in a container or a sandbox yourself; the docs give three ways to do it.
By contrast, Claude Code stops to ask you every time it edits a file or runs a command.
For a beginner, that confirmation has value.
Genuinely fewer features.
Subagents, MCP extensions, plan mode — most of what Claude Code has, Pi mostly doesn’t. The speed and cost you save come at the cost of fewer features.
6. How to Choose
Bottom line first: this isn’t a single-choice question. It depends on who you are.
If you write clear requirements, use Pi
The criteria are concrete. If these describe you, switching to Pi likely pays off immediately:
- You spell out which file to change, what behavior you want, and how edge cases should be handled
- Your project has a written spec doc (
AGENTS.mdorCLAUDE.md) laying out structure, commands, and conventions - You’re on pay-as-you-go API billing, not a subscription
- Your task volume is high enough that cost already makes you anxious
For this kind of person, the stuff those 20,000-plus tokens in Claude Code provide, you’ve already provided yourself. You’re paying twice for a manual you don’t need.
The Databricks data applies directly to this scenario: same work, bill cut by half or more, pass rate wobbling within a few points.
Two practical notes: don’t mindlessly crank effort to the max — that 82% vs 89% happened at the max tier, and in the Databricks data Pi does best at xhigh; also, Pi has no permission limits, so before running unfamiliar code or letting it execute anything destructive, wrap it in a container first.
If you’re new to AI, stay on Claude Code
Same, with concrete criteria:
- You’re not yet sure how to describe requirements to a tool like this, and often fire off a one-liner like “fix this problem for me”
- Your project has no docs written for the AI to read
- Claude Code is covered by the official subscription
- You can’t do without Subagent/PlanMode/Permission/MCP and the like
For this kind of person, those 20,000-plus tokens are insurance, not waste.
They hold a large set of “here’s what to do in this situation” fallback rules — which files to look at when a request is vague, which commands to run before editing code, when to stop and ask the user. The terser your prompt, the more these rules fill in for you.
Pi’s minimalism has a prerequisite: it assumes you know what you want. That assumption doesn’t hold for beginners. In community feedback, “unreliable with low-quality prompts” is one of the most common complaints about Pi.
And subscription users switching over have to pay separately for the API. The half-off only holds if you were paying by usage to begin with.
The middle path
If you sit on both sides, split by task:
- Well-defined mechanical work — change an interface, batch-rename, fill in a module following an existing pattern — goes to Pi. This work is the cheapest and needs the least hand-holding.
- Fuzzy exploratory work — don’t know where the bug is, hunting for something in an unfamiliar codebase, need to understand before touching anything — stays with Claude Code, letting its fallback rules help you sharpen the question.
Running both side by side doesn’t conflict.
The Last Word
Will Pi stay ahead? I don’t know. Anthropic and OpenAI can trim their system prompts anytime; that’s an engineering problem, not a capability one.
But the thing this data reveals will keep holding:
Half of what you pay has nothing to do with the model itself. It has to do with how the agent wrapped around it manages context.
Before this benchmark came out, almost everyone’s attention was on the model — which one is smarter, which one scores higher.
What the Databricks data shows is that the same model, on the same task, with a different agent around it, can cost twice as much.