Should You Delete Your CLAUDE.md? We Benchmarked 4 Claude Code Setups on Opus 5.5

Yes, mostly. On Claude Opus 5.5, a CLAUDE.md written for older models bought us no extra correctness and cost a lot. We ran the same 10 coding tasks in Claude Code under four setups, 90 graded runs in all. A legacy CLAUDE.md with four "always use me" skills passed the same hidden tests as an empty repo, at 2.1 times the cost and 1.8 times the turns. In its original form it never got started: one line asking the model to write out its full reasoning made Opus 5.5 refuse all 10 tasks. A three-line CLAUDE.md that kept only the project facts was the cheapest setup we tested.

The question is live because Boris Cherny, who leads Claude Code at Anthropic, told a Y Combinator audience in July to delete your CLAUDE.md, skills and hooks every six months. Since then a wave of "delete your setup" posts has gone around, and none of them measured anything. This post gives you the numbers, the exact setups, what broke, and what to keep.

What we found: Same quality across every setup that ran. The difference was cost and time: $3.05 for 20 runs with no CLAUDE.md, $2.77 with a three-line one, $4.60 with our own real 1,462-word framework file, and $6.28 with a legacy rules file. Run on 2026-10-28 with Claude Code 2.1.312 and claude-opus-5-5.

What did Boris Cherny actually say about deleting CLAUDE.md?

He said it on stage, and the quote is checkable. In Y Combinator's video Boris Cherny: We Cut 80% of Claude Code's Prompt (uploaded 2026-07-27), he says, per YouTube's English captions:

"For people that aren't building agentic products, but you're using Claude Code, every 6 months delete your Claude MD. Delete your skills. Delete your hooks. See what the model does and it might surprise you. And actually for Opus 5, this is something we really do recommend is just try deleting all of these things because the model might really just not need all those instructions that you needed for past models."

In the same talk he says the Claude Code team deleted 80% of the product's own system prompt for Opus 5, because much of it corrected behaviour the new model no longer gets wrong. Each time a new model ships, the team runs what he calls an ablation: delete the whole system prompt, then bring it back line by line to see what each line actually does.

Two caveats. The captions write "Claude MD", which is CLAUDE.md spoken aloud (we only fixed capitalisation in the quote). And he framed it as an experiment ("see what the model does"), not a rule to delete everything for good. The LinkedIn version of the advice, for example Ruben Hassid's "One Claude" post, turns it into a stronger claim: that long instructions and skills cause "context bloat." That's his argument, and the benchmark below is the first test of it we know of.

Does Anthropic say old prompts hurt newer models?

Partly, and more narrowly than the social posts suggest. We couldn't find any Anthropic page that says skills written for older models "degrade output quality" in general. What the docs do say:

So the official claim is about cost and refusals, not lower quality. Our results line up with that exactly.

How we ran the benchmark

We wrote 10 small but real Python tasks, each with a hidden test suite the agent never saw. Every hidden suite fails on the starting code and passes against a reference solution we wrote first, so a failure means the agent got something wrong, not the test.

Each task ran in a fresh git repo under four setups:

The legacy setup is a composite we wrote from the patterns Anthropic's docs now flag: all-caps CRITICAL and MUST rules, "if in doubt, use a tool," mandatory verification subagents, a PLAN.md and CHANGELOG.md on every change. It isn't any one team's file, but each of those patterns is easy to find in public CLAUDE.md files. The AY file is a setup we actually use.

Every run used Claude Code 2.1.312 in headless mode (claude -p) with --model claude-opus-5-5 at default effort. User-level settings, skills and MCP servers were switched off (--setting-sources project, an empty --strict-mcp-config), so each setup saw only its own files. Permissions were acceptEdits plus the same tool allowlist for every setup (read, edit, write, search, python3, ls, cat, subagents, skills). We ran each task twice per setup and graded with the hidden tests after restoring the original visible tests. Cost is the total_cost_usd that Claude Code reports, at list price.

Results: same quality, very different bills

Every miss in the table is the same assertion on task 8. Our hidden test expected "Straße" to become strae. Agents produced stra-e or strasse, which are defensible readings of a prompt that never said what to do with ß. Count that as a spec gap on our side, and every setup that ran solved every task.

The legacy rules file doubled cost and bought nothing

With the reasoning line removed (more on that below), the legacy setup passed exactly what the bare setup passed. It cost more on all 10 tasks, took more turns on 9 of them, and ran 2.4 times longer on average. It also did what it was told: 9 of its 20 runs wrote a PLAN.md or CHANGELOG.md nobody asked for, and its runs added a median of 126 lines against 43.5 for bare, mostly extra tests and docstrings. It was the only setup that spawned subagents (4 in total), because a skill said to. The worst single run was the SemVer task: 24 turns and $1.00, against $0.27 for the most expensive bare run of any task.

That's the over-verification Anthropic describes, measured. The model already checks its own work. Telling it to check twice, with subagents, a checklist and a changelog, is pure overhead.

Our real framework file cost 51% more, through context alone

Our own CLAUDE.md didn't change behaviour much: 6.5 turns on average against 6.1, and no stray files. It still cost more than bare on all 10 tasks, because it's big. The smallest first-request cache write was 15,342 tokens with our file and skills loaded, against 7,786 for bare. Every turn then reads that larger context back. Most of the file (lock protocol, board, handoffs) has nothing to do with fixing a pagination bug, and the 17 skills are about blog posts and SEO, yet their descriptions ride along on every request.

The three-line file beat nothing at all

The trimmed setup was the cheapest overall ($2.77, 9% under bare), had the lowest median wall time (20.6 s against 25.0 s), and wrote the least output. It was cheaper than bare on 7 of the 10 tasks. Telling the model up front how to run the tests and what not to touch saved it some exploring. That's the "delete the procedure, keep the invariants" result in miniature.

One old line got every run refused

The legacy file as first written contained a line that was common advice for older models: think step by step, and write your full reasoning in the response before any code. On Opus 5.5, all 10 runs stopped on the first request with an API error, refusal category [reasoning_extraction], after about 3 seconds and $0.11 each. No code was written.

We deleted that one line and changed nothing else. The next 20 runs had zero refusals. That matches the Opus 5 prompting guide word for word: instructions that ask the model to write out its reasoning can be declined. If you want the reasoning, keep thinking on and read the summarised thinking blocks. If you want an explanation, ask for a short summary of what was done and why.

Watch out: In headless mode this fails loudly, but inside a busy session it can read like a flaky API. If Claude Code started refusing ordinary coding requests after you moved to Opus 5 or 5.5, search your CLAUDE.md, skills and custom commands for "reasoning", "think step by step" and "show your work" first.

What to keep in your CLAUDE.md

Going by these runs and Anthropic's own guidance, here's what we'd keep and what we'd cut:

Keep

• Exact commands the model can't guess: test, build, lint, how to start the dev server.
• Hard constraints: "stdlib only," "never push," "don't edit tests to make them pass," which directories are off limits.
• Repo facts that aren't in the code: the gotcha that burns a session, a deprecated module that still looks live.

Cut

• Verification rituals: "verify twice," "use a subagent to check your work," checklists to fill in after every edit.
• Anything asking for written-out reasoning or a fixed thinking format.
• All-caps CRITICAL/MUST/NEVER language and "if in doubt, use X."
• Process files nobody reads (PLAN.md, CHANGELOG.md on every change) unless your team really uses them.
• Skills that don't match the repo's work. Their descriptions load on every request even when they never trigger.

A fast way to do the cut: copy the file aside, run three or four real tasks from your backlog with no CLAUDE.md, note what went wrong, and add back one line per failure. That's the ablation Cherny describes, at repo scale. Claude Code's /doctor prompt-audit can also flag stale or conflicting rules first; we covered what it found in our setup in our Claude Code prompt audit. For good short files to copy from, see these CLAUDE.md examples from real repos.

If you'd rather have someone do this across a team's repos, skills and hooks, that's part of our Claude Code training for engineering teams, and our Claude Code development agency builds and maintains these setups for clients.

Limits of this test

• Small tasks. These are 10 single-file Python tasks. A large monorepo is where real project facts in CLAUDE.md pay off most, so the trimmed setup's edge could grow there, and the cost of a bloated file could too.
• Two runs per task per setup. That's enough to see a 2x cost gap repeat on every task, not enough to rank setups a few percent apart. Treat bare against trimmed as a tie with a lean.
• One model, default effort. Everything ran on claude-opus-5-5. A setup that's dead weight here may still help a smaller model.
• Restricted shell. The same allowlist applied everywhere, so some compound shell commands were denied in every setup (9 to 22 per setup) and the agent took another route. It affected all four, but it's not identical to a fully open session.
• The legacy file is a composite. We built it from documented anti-patterns, so it's a realistic worst case, not an average. Our own file is the real-world data point.

FAQ

Should I delete my CLAUDE.md for Opus 5.5?

Delete the procedure, keep the facts. In our benchmark a three-line CLAUDE.md with test commands and constraints was the cheapest setup, cheaper than having no file at all, while a long rules file doubled cost with no quality gain. Start from empty and add back only lines that fix a failure you actually saw.

Did Boris Cherny really say to delete CLAUDE.md every six months?

Yes. In Y Combinator's video "Boris Cherny: We Cut 80% of Claude Code's Prompt" (July 2026), he says to delete your CLAUDE.md, skills and hooks every six months and see what the model does, and that for Opus 5 the team really does recommend trying it. He framed it as an experiment, not a permanent rule.

Do Claude Code skills still work on Opus 5.5?

Yes, they load and trigger. Our legacy skills ran exactly as written. That was the problem: TDD rituals, review checklists and verification subagents doubled cost on tasks the model already solved alone. Keep skills that encode knowledge the model lacks, and drop ones that only repeat process. Our Claude Code skills workflow covers which kind is worth writing.

Why does Claude Code refuse with reasoning_extraction?

Your prompt, CLAUDE.md or a skill is asking the model to write out its internal reasoning. Anthropic's Opus 5 guide says such requests may be declined with the reasoning_extraction refusal category. In our test one such line got 10 of 10 runs refused, and removing just that line fixed all of them.

Does a long CLAUDE.md cost more even if the model ignores it?

Yes. Our 1,462-word framework file barely changed the model's behaviour, but it cost 51% more than an empty repo across 20 runs, because every request carries the larger context. Unrelated skills add to it too, since their descriptions load each turn.

How do I test my own Claude Code setup?

Pick three or four real tasks with a clear pass check and run them headless with claude -p --output-format json, once with your setup and once without. Compare total_cost_usd, num_turns and whether the checks pass. If you want help running this across a team, our Claude Code agencies guide lists who does it, us included.

---

Sources: Y Combinator, Boris Cherny: We Cut 80% of Claude Code's Prompt, Anthropic, Prompting Claude Opus 5, Anthropic, Prompting Claude Opus 5.5, Anthropic, Prompting best practices, Ruben Hassid, One Claude (LinkedIn), AY Automate benchmark runs, 2026-10-28.