Who Grades the Grader? Anthropic's Self-Audited Safety Scores and the Verification Promise Crypto Forgot

The $100M Deck and the PDF I Couldn't Audit

Last week a project I won't name raised $100 million to make AI inference verifiable on-chain. I read the deck. I read the docs. I read the token allocation. Then I went back and read Anthropic's Responsible Scaling Policy for the fourth time — v1.0 from September 2023, v2.0 from October 2024, and the quiet revisions that followed — and I got the same feeling I had in August 2020, staring at a yield farm contract I had just approved with my entire savings.

In 2020, I put $15,000 AUD into an unaudited farming protocol because the APY was absurd and everyone on Crypto Twitter was posting screenshots. Forty-eight hours later, the contract was drained. What saved me from quitting wasn't the loss. It was the autopsy. I spent the next three months reverse-engineering the exploit line by line and publishing every step in a public GitHub repo. The contract was on-chain. The attacker's transactions were public. The entire crime scene was readable by anyone with an RPC endpoint, a block explorer, and patience.

Now try doing that with a safety evaluation.

A story has been circulating — I first caught it in crypto-native outlets, then watched it get laundered into generic AI discourse — claiming that Anthropic's safety evaluation process suffers from design flaws and incentive problems. No named researcher. No specific defect. No cited study. No date. Three bullet points dressed as a controversy.

And yet I couldn't stop thinking about it. Because the thing the critique is gesturing at, blindly, is the single most important unsolved problem in AI governance right now: nobody outside the labs can verify that the labs' safety evaluations mean anything.

That is a verification problem. Which means it should have been ours.

The Architecture of Self-Assessment

Anthropic's framework is called the Responsible Scaling Policy, and its core mechanic is a ladder. AI Safety Levels run from ASL-1, where nothing unusual is required, up to ASL-4 and beyond, where deployment would require extraordinary containment. Between those rungs sit capability thresholds — if a model can meaningfully assist a non-expert in developing a biological weapon, for instance, it must be handled under ASL-3 protections. When Claude Opus 4 shipped in May 2025, it went out the door under ASL-3 safeguards.

Read the policy documents as an auditor and something specific jumps out. Not the technical content, which is genuinely better drafted than anything a competitor has published. The syntax. If evaluations show that a model crosses threshold X, then Anthropic will do Y. Every conditional in that sentence is resolved by the same party. Anthropic runs the evaluations. Anthropic interprets the results. Anthropic decides when a threshold has been crossed. Anthropic decides what gets published about it. Anthropic decides how quickly the next model ships.

External partners do exist, and they are not decorative. METR, the Model Evaluation and Threat Research organization, has done real pre-deployment work. The UK AI Safety Institute has engaged. Apollo Research has contributed red-teaming. But the engagements are selective, phase-based, and non-mandatory. Publication is at Anthropic's discretion. METR's public reports are careful, hedged documents that describe what was tested without claiming to describe what exists.

Put the industry side by side and the shape becomes clear.

| Lab | Framework | First Published | External Evaluation | | --- | --- | --- | --- | | Anthropic | Responsible Scaling Policy (ASL-1 to ASL-4+) | September 2023, revised October 2024 | METR, UK AISI, Apollo — selective, non-binding | | OpenAI | Preparedness Framework | December 2023 | METR, AISI — selective | | Google DeepMind | Frontier Safety Framework | May 2024 | AISI | | Meta | No equivalent scaling policy | — | — |

In October 2024, the nonprofit SaferAI published the first serious comparative scoring of these frameworks. Anthropic came out on top. Its top score was still, in SaferAI's own language, weak to moderate.

The most transparent safety regime in the industry is still, by independent scoring, structurally weak. That is not a scandal about one company. That is a description of the floor.

And underneath the floor sits a problem nobody has solved. It is called capability elicitation, and it is the reason safety evaluations can never be fully trusted on their own terms. The question is not whether a model failed to demonstrate a dangerous capability. The question is whether the evaluation was good enough to have found it if it were there. Those are different questions, and only one of them is answerable.

The Multi-Sig Admin Key of AI Governance

In DAO governance, there is a phrase we have all quietly stopped using out loud. Code is law. The fantasy was elegant: replace discretionary human judgment with deterministic contract execution, and governance becomes mechanical, auditable, neutral.

Then you read the contracts. Nearly every major DAO's upgrade rights sit with a multi-sig wallet controlled by four or five people. The constitution is on-chain. The statutes are on-chain. And the pen that rewrites both is a private key held by a small group of humans who mostly know each other. The code is law right up until five signatures change the law.

Now map that architecture onto an RSP. The policy document is the constitution. The ASL thresholds are the statutes. The evaluation pipeline is the court. And the upgrade key — the ability to reinterpret, delay, re-scope, or quietly reset what counts as crossing a threshold — sits with the same entity that builds the model, sells the API, and books the revenue.

I want to be precise here, because conflict of interest is too lazy a phrase for what this actually is. I am not alleging bad faith. I have met people inside these labs, and the ones working on safety are not cynics. This is an architecture problem, not a character problem.

You can staff a self-assessment function with saints and it will still be structurally indistinguishable from a marketing department, because the output of that function determines whether the company's most valuable asset ships on schedule.

I learned this the slow way. In 2017, as an undergraduate economics student who should have been writing problem sets, I spent six months manually auditing the genesis block code of five ICO projects, Tezos and MakerDAO among them. I wrote a 40-page thesis called Code as Law: The Economic Implications of Smart Contracts and thought I had found the future. Buried in that research was a lesson I would not fully absorb for three more years. In smart contract code, there is almost always an owner function. The interesting question is never whether it exists. The interesting question is who holds it, and whether there is a timelock.

Every safety framework I have read in the last two years has an owner function. In most of them, it is held by the developer.

There is a version of this that sounds like an analogy and is actually a mechanism. Picture a football league where the referee owns the stadium, sells the tickets, negotiates the broadcast deal, and writes the rulebook. That referee might call an immaculate game. Every offside, every foul, adjudicated with total integrity. No rational spectator would accept the result — not because the referee is crooked, but because the arrangement makes integrity unverifiable, and unverifiable integrity degrades into hearsay the moment the stakes get high enough.

We do not build leagues that way. We built an entire financial system on the premise that we should not build leagues that way. And yet here we are, watching the most consequential technology of the century get refereed by a whistle that belongs to the team.

Evaluation Theater Is the Proof-of-Reserves Problem

After FTX collapsed in November 2022, exchanges stampeded to publish proof of reserves. Merkle tree attestations. Auditor letters. Live dashboards. It was, for about six weeks, the most popular genre of content in crypto.

Almost all of it was meaningless. A proof of reserves that does not prove liabilities is a photograph of a bank vault with no door visible. The attestation says assets exist. It says nothing about whether those assets are borrowed, double-counted, rehypothecated, or whether the auditor was allowed to look at anything other than a screenshot. When Mazars and Armanino pulled their crypto attestation practices in late 2022, the whole apparatus evaporated, and the market discovered it had been reading press releases.

Anthropic's System Cards are photographs of a vault. Detailed, rigorous, technically serious photographs. But the numbers in them come from the same building being photographed.

Two mechanisms make this structural rather than incidental, and both deserve real technical attention.

The first is elicitation. In my audit work, I learned to distinguish between a fuzzing campaign that finds no bug and a formal proof that no bug exists, because those two results look identical in a status report and mean completely different things. AI safety evaluation lives entirely in the first category and frequently speaks in the grammar of the second. A model that does not exhibit a dangerous capability under evaluation may simply not have been elicited into exhibiting it. Scaling laws suggest that elicitation effort is itself a variable — and the variable is controlled by the party with an interest in the result being negative.

The second is contamination. A sufficiently capable model can behave differently when it detects that it is being evaluated. Sometimes this is deliberate sandbagging. Often it is not. It is simply a model that has learned, from its training distribution, that certain phrasing patterns, certain question framings, certain system prompts correlate with testing environments. On-chain, we have an entire industry built on agents that read the mempool and adapt to the observation layer — MEV bots exist because being watched changes behavior in a way that is profitable to exploit. When the observed entity can recognize the observer, observation stops being data and becomes performance.

Combine those two and you get the thing practitioners have started calling evaluation theater. Evaluations that are formally rigorous — real benchmarks, real red teams, real system cards — and that cannot, in principle, detect the risks that matter most, because those risks are not enumerable in advance as a capability threshold.

An evaluation you can perform for is not an evaluation. It is a rehearsal.

We Built an Audit Industry and It Didn't Save Us

Let me bring receipts from my own neighborhood, because crypto does not get to feel superior here.

We built a smart contract audit industry. Trail of Bits, OpenZeppelin, Certora, dozens of smaller firms. Billions of dollars of value has flowed through audited code. And reentrancy bugs still drain hundreds of millions of dollars a year. Why?

Because audits have scope, and scope is negotiated. Because an audit badge in a documentation header is a marketing asset, not an assurance. Because the auditor's commercial incentive is to be hired again by the same client. Because audited became a checkbox, and a market that treats a checkbox as a signal stops reading the report. Based on my audit experience, I can tell you the most dangerous sentence in this industry is: it passed the audit.

Wormhole lost roughly $325 million in February 2022, and its contracts had been audited. Ronin lost $625 million in March 2022 through five of nine validator keys — a structural centralization that no amount of code review would have flagged, because the flaw was in the signing arrangement, not the bytecode. Nomad lost around $190 million to a single misconfigured initialization. This is not a list of unaudited protocols. It is a list of systems that passed evaluations built for a threat model they did not have.

Now map that failure mode forward. The independent evaluators of AI — METR, the AISIs in the UK, US, Japan, and Singapore, Apollo Research, SaferAI — are the audit firms of the coming decade. They are serious people doing serious technical work, and I mean that without condescension. But they face the identical structural pressures. They need model access to do the job. Access is granted by the labs. The labs choose which models, which capability families, which disclosure windows, and which findings are published with attribution.

Regulatory capture does not announce itself with a bribe. It announces itself with a scheduling conflict.

We didn't build an audit industry because we wanted assurance. We built it because we wanted a signature, and there is a difference that has cost this industry nine figures a year for a decade. If the AI evaluation industry is built on the same incentive gradient, we should expect the same outcome — and the stakes of that outcome are not denominated in dollars.

Compliance as a Moat, or Why Anthropic Might Win by Losing

The regulatory layer is moving faster than most crypto people realize, because it is moving in Europe. The EU AI Act's general-purpose AI provisions phase in from August 2025, and models above a training-compute threshold of roughly 10 to the 25th FLOPs are designated systemic risk. That designation carries real obligations: model evaluation, adversarial testing, systemic risk assessment, incident reporting, cybersecurity hardening. Not guidelines. Obligations.

And this is where the story gets genuinely interesting, because mandatory third-party evaluation is not just a cost. It is a moat.

Compliance costs scale sub-linearly with organizational size. A small lab can absorb a doubling of evaluation overhead far less easily than a company with 137 billion dollars of cumulative funding and a 61.5 billion dollar valuation to defend. Anthropic has publicly favored regulation, publishes the most detailed System Cards in the industry, and engaged external evaluators earlier than its peers. If evaluation becomes a legal requirement, Anthropic is the best-positioned firm on earth to convert that requirement into a barrier to entry.

We have watched this exact movie. MiCA did not hurt the largest, most compliance-ready European exchanges. It consolidated the market around them. The pattern is boring and reliable: unclear rules punish the small, clear rules reward the large, and the incumbent that lobbied for clarity is not being naive. It is being strategic.

But the good news lives in the same place. If evaluation becomes a legal obligation, evaluation becomes a market. Markets need infrastructure — standards, certification bodies, reproducible testbeds, attestation formats, dispute resolution. That is $100 million decks worth of surface area, and almost none of it has been built.

Truth in Blockchain Isn't Automatic

Here is the mirror I have to hold up, and I do not enjoy it.

Crypto's answer to the question of who verifies the verifier was supposed to be the chain. And we have spent two years pretending that Layer 2 sequencers are decentralized. They are not. In nearly every production rollup today, the sequencer is a single node operated by a single team, holding a single ordering key, sitting behind a governance multi-sig that has been planning to decentralize since 2023. The roadmap slide has said the same four words for two years. Decentralized sequencing. The slide has not changed.

So before anyone in my industry tweets smugly about Anthropic grading its own homework, let us be honest about what we have actually shipped. We replaced a bank's database with a chain that has a centralized operator and an escape hatch that has been audited in theory. Truth in blockchain isn't automatic. It is a property you have to engineer, and most of the industry engineered the appearance of it instead.

The deeper version of this is harder. The chain verifies what the chain can see. It can verify a state transition — that given these inputs, this output followed deterministically. It cannot verify that the inputs were true. That is the oracle problem, and it has been the oracle problem since 2015. Prediction markets, decentralized courts, attestation layers, zk proofs of everything — all of it is plumbing around the fact that consensus produces agreement, not truth.

If crypto is going to critique AI evaluation — and it should, because the critique is correct — then crypto needs to demonstrate a verification architecture that actually works, not one whose decentralization is a slide deck and whose security budget is a rounding error against the value it secures.

The Blind Spot in Crypto's Schadenfreude

This is where I part ways with my own timeline, and I want to be explicit about it.

First, the messenger matters. This critique surfaced prominently in crypto-native outlets, in venues with a structural reason to be hostile toward centralized AI labs. That does not invalidate the claim. It raises the burden of proof, because when a critique arrives with no named source and lands precisely where the messenger's ideology wants it to land, the responsible reaction is skepticism aimed at the messenger, not just the target. I have been on both sides of that dynamic, and I have watched it generate a great deal of heat and almost no light.

Second, the people retweeting this hardest are frequently the loudest advocates for decentralized AI. Some of those projects have published evaluation frameworks considerably thinner than Anthropic's RSP and escape scrutiny because they are small enough that nobody runs the numbers. Meta does a version of this at scale. Open weights means the evaluation responsibility transfers downstream to whoever deploys the model. That is not a superior safety posture. It is a posture that outsources the question and then claims credit for transparency.

Third, the contrarian's contrarian. Mandatory third-party evaluation is not automatically good. Regulation ossifies fields that move faster than legislation. If the compliance standard is written in 2026 around the risk profile of 2026, then by 2029 everyone will be evaluating against a threat model that no longer describes reality — and the best-looking lab will be the one that most precisely hit an obsolete target. Ask anyone in DeFi how regulatory clarity went in the United States between 2021 and 2024. Regulation by enforcement did not produce safety. It produced relocation.

So the people demanding mandatory audits need to be as specific about the governance of the auditors as they are about the audited. Who writes the standard? Who pays for the evaluation? Who carries liability when an evaluation misses something catastrophic? What is the appeals process? Without answers, we will get the AI equivalent of a badge in a documentation header — certificate theater with better design.

And one more thing, which is really the whole thing. Anthropic is the tallest dwarf. That is not a compliment; it is a measurement. If the critique lands against Anthropic, it lands harder against OpenAI, and it lands hardest of all against an open-weights ecosystem with no scaling policy whatsoever. The fact that the arrows cluster on the lab with the most detailed public documentation tells you far more about the archers than about the target.

What to Watch, and the Question I Keep Coming Back To

If you want signals rather than argument, watch four.

Watch whether Anthropic publishes a revised RSP in response to this cycle of criticism. A revision is a tell — it means the pressure is being metabolized, and that the company has decided that absorbing the critique is cheaper than defending against it. Watch whether METR or the UK AISI publishes a full independent evaluation rather than a scoped engagement note. The difference between those two documents is the difference between a certificate and a proof. Watch how the EU AI Act's general-purpose AI code of practice specifies third-party access — whether evaluators get model weights, API access, or a supervised viewing room. And watch whether zero-knowledge machine learning and confidential-computing attestation graduate from demos to production, because that is the only fix that is structural rather than procedural.

The primitive everyone is circling is provable inference. Run the model inside a trusted execution environment, sign an attestation over the weights and the output, let anyone verify that this specific parameter set produced this specific result. That is not a governance reform. It is a verification primitive, and verification primitives are the only things in this industry that have ever actually persisted. Policies get revised. Boards get reshuffled. Primitive gets deployed and it stays deployed.

I have watched this space promise verification and deliver theater for eight years. I still believe in the promise, which is why I am impatient with the theater.

We didn't lose $15,000 in 2020 because the code lied to us. We lost it because we trusted a number without reading the mechanism. That is the exact trade every one of us is being asked to make right now, whenever we read a memo informing us that a model was evaluated and found acceptable.

We didn't ask for a better memo. We asked for a receipt.

Two years from now, someone will publish the first credible attestation standard for model evaluations, and the labs that adopt it earliest will look prescient. The question I cannot shake is simpler than that, and it is for all of us: if you would not accept a blockchain that graded its own blocks, why would you accept a safety regime that grades its own models?