AI Policy

13,524 Answers Show Where AI Goes Quiet

The Oversight Board tested 10 models from 6 providers across 10 jurisdictions. Refusal rates were 34% in restrictive countries versus 14% in permissive ones. The method is now a procurement instrument.

AI PolicyBy Servola Tech Desk2026-07-175 min read

AI-assisted, edited by humans. | Editor-in-Chief: Leon Soliman | Editorial standards

The Test That Was Actually Run

The most useful thing published on 16 July 2026 was not an argument. It was a procedure. The Oversight Board, the independent body that Meta funds to review its content decisions, took 7 prompt templates, pointed them at 10 jurisdictions, aimed each at 4 targets, and ran every combination 5 times across 10 models from 6 providers. Anthropic, DeepSeek, Google, Meta, OpenAI and xAI were all in the set. The run produced 13,524 responses.

This is the Board's first evaluation of large language models, and the framing it chose was whether those models are stifling political speech. Strip away the framing and what remains is a grid. Ask the same kind of question about a different country, hold everything else constant, and record what comes back. That design is what makes the result something other than an anecdote.

The Asymmetry Runs the Wrong Way

The models refused most often where speech was least free. In jurisdictions the Board classed as restrictive, requests for material critical of a government were declined 34% of the time. In permissive jurisdictions, the same class of request was declined 14% of the time. The models were more than twice as likely to decline to criticise a repressive government as an accountable one.

That direction matters more than the raw rate. A uniform refusal rate would be a policy choice you could argue about. A refusal rate that rises with the political risk profile of the jurisdiction being asked about is something else: a safety behaviour designed to be neutral has, in aggregate, ended up tracking the very thing it was meant to ignore. The practical effect is that the protection lands on the governments that are least accountable.

None of this requires a claim about intent. Nobody needs to have chosen it. It is what 13,524 answers, sorted by jurisdiction, show.

The Method Is the Product

Until this week, "our model is appropriately cautious" was a vendor assertion that no buyer could test. It sat in the same category as a claim about culture or intent: agreeable, unfalsifiable, worth nothing at a negotiating table. The Board's publication changes the category, because the design is written down and the grading is validated. 7 templates, 10 jurisdictions, 4 targets, 5 repetitions, with a classifier that agreed with human raters 97% of the time.

Reproducibility is the whole contribution. A buyer with a modest engineering budget can build that grid, point it at any model under consideration, and read a number off the other end. The refusal rate joins latency and price as a measurable property of the thing being bought.

That reframing is what an owner should take from this. The interesting move is not the finding about political speech. It is that an argument which was ideological last month is a spec line this month.

What a Buyer Should Now Test

Any European business running an LLM in a workflow that touches government should now measure this rather than assume it. That covers more ground than it first sounds: public affairs, regulatory research, journalism, compliance work, country risk, and any market-entry analysis of a jurisdiction you are considering. If your research tool quietly declines to analyse a government, the blind spot correlates with the countries where you most need the analysis, and nothing in the interface tells you it happened.

The test is per jurisdiction, not global. A model that answers cleanly about the country you are sitting in tells you nothing about how it will handle the country you are entering. Run the grid against the markets you actually operate in, before signing, and keep the output with the rest of your evaluation evidence.

On the regulatory read-across, our reading is this: the EU AI Act's transparency obligations for general-purpose AI are the natural place this pressure lands, because a documented and reproducible refusal-measurement method is exactly the kind of evidence a buyer or a regulator can use. The Board's study creates no legal obligation on anyone. It creates an instrument, and instruments tend to get picked up.

The Limits, Stated Plainly

Three qualifications sit on this finding and all three are load-bearing. The first is the publisher. The Oversight Board is funded by Meta and operates independently of it, and Meta is one of the six providers evaluated. That cuts in both directions: it is a reason to read the methodology rather than the summary, and it is also the reason the methodology was published in enough detail to be checked.

The second is that a refusal is not automatically an error. There are requests a model should decline, and no serious buyer wants a tool with no limits at all. The measurement tells you where the line sits and whether it moves with the jurisdiction. It does not tell you the line is in the wrong place in any individual case.

The third is scope. The study measures refusal, not accuracy. A model that answers every question about every government is not thereby correct about any of them. Those are two different tests, and this is the first one.

Frequently asked questions

What did the Oversight Board actually measure?

Whether large language models decline to produce material critical of a government, and whether that rate changes with the jurisdiction being asked about. The Board collected 13,524 responses from 10 models across 6 providers, using 7 prompt templates, 10 jurisdictions, 4 targets and 5 repetitions of each combination. The headline result was a 34% refusal rate in restrictive jurisdictions against 14% in permissive ones.

Does the fact that Meta funds the Board undermine the finding?

It is a reason to read the method rather than the press summary, not a reason to dismiss the result. The Board operates independently of Meta, Meta is itself one of the six providers evaluated, and the design was published in enough detail that a third party can rerun it. A study you can reproduce is judged on its procedure, not on who paid for it.

What should we do differently in procurement?

Add refusal behaviour to your model evaluation checklist and test it per jurisdiction you operate in, rather than accepting a vendor statement about guardrails. Build the grid the Board described, run it against each model under consideration before you sign, and keep the output alongside your other evaluation evidence.

The finding will be argued about. The instrument will outlast the argument. From this week, any owner who accepts a vendor's word about how cautious a model is has chosen not to check something that takes an afternoon to measure.

Oversight BoardMetaLLM evaluationAI procurementmodel refusalAI Act

Microsoft Trained Its Sellers to Call Claude Slower

Microsoft trained its sellers to call Claude slower and less accurate inside Office. A separate report says it is swapping the model out to cut cost. Read the ranking accordingly.

4 min read

Artificial Intelligence

Moonshot's Biggest Model Has No Thinking Dial

Moonshot launched Kimi K3 on 16 July as the world's first open 3T-class model. The weights arrive on 27 July, and the reasoning effort control currently has one setting: max. Two facts that decide what it costs you.

8 min read

Digital Sovereignty

The EU Council Revived the Rule That Scans Private Messages

The EU Council revived the lapsed rule that lets providers scan private messages with AI. For owners, confidential communication is now a governance choice.

2 min read

← All articles