A meeting on 28 July, and a number nobody wanted
On 28 July, senior Amazon engineers briefed colleagues on what had happened to a set of internal AI projects. The Financial Times reported the contents. One project had been using Anthropic's Claude Sonnet to match author details against e-commerce listings. It spent 1.8 million dollars. It ran 860 percent over budget. It went five months before anyone caught it, and it never shipped. A team described the pattern as catastrophically expensive.
Two smaller cases came with it. A financial auditing tool overshot by roughly 541,000 dollars. A logistics project came in about 134,000 dollars above plan. A senior Amazon employee summarised the underlying problem to the FT in one sentence: it is difficult to figure out how much anything AI related costs. Amazon has responded that it is experimenting, learning and improving how it uses the technology, has called the examples cherry-picked, and is introducing mandatory peer reviews and spending caps on AI projects.
Start with the arithmetic, because nobody published it. If 1.8 million dollars represents an 860 percent overrun, the approved budget was around 190,000 dollars. That is the number a control should have defended.
Two months earlier, the same mistake wore a different costume
On 29 May, Amazon shut down KiroRank, an internal leaderboard on its Kiro developer platform that ranked staff by how many AI tokens they consumed. It had been set up to drive adoption. What it drove was a practice staff named tokenmaxxing: pointing agents at work that did not need doing, so the ranking would move. Dave Treadwell, a senior vice president, told staff in a memo that the board had been built with good intentions and asked people not to use AI just for the sake of using AI.
Read the two episodes as one. In May, Amazon rewarded consumption and received consumption. In July, it discovered consumption nobody had authorised and nobody had read. The leaderboard and the runaway job are not separate failures of discipline; they are the same gap, which is that the organisation had a meter running and no habit of looking at it.
The instruction sits in the replacement, not the shutdown. Amazon did not simply delete the leaderboard. It replaced the metric with normalised deployments, which counts AI-assisted code that actually ships.
Why a metered failure stays invisible
Conventional software announces its own failures. A bad loop exhausts memory and the process dies, a page times out, an alert fires and somebody is paged at three in the morning. A misconfigured model call does none of that. It succeeds. It returns a response, charges for the tokens, and goes round again. There is no crash to detect because from the platform's point of view nothing went wrong, and that is precisely the condition under which five months can pass.
Three things then stretch the detection window. Billing is monthly, so the finance function sees the shape of a problem weeks after the spend, and sees it aggregated. Pricing is per token, so cost scales with volume rather than with instance count, which is the axis most infrastructure alerting was built to watch. And agents multiply tokens, because an agentic loop makes many model calls where a chat interface makes one.
The consequence for a European operator is the same regardless of region: the meter runs in the cloud account, the invoice arrives on a monthly cycle, and nothing in between tells you the job stopped being useful in March.
The controls existed and were not switched on
This is the uncomfortable part of the story, and it is why it belongs in a decisions column rather than a technology one. The mitigations were available on the platform the whole time. AWS publishes batch inference at half the on-demand token price for work that does not need an immediate answer, which is the exact profile of a five-month background matching job. It offers prompt caching so a repeated context is not paid for at full rate on every call. It offers cheaper model tiers and routing between them. None of that is exotic and none of it required a negotiation.
A job that ran for five months without an answer being urgently needed is the textbook batch workload. Running it on demand instead is not a technical failure, it is a decision nobody was asked to make, because the person who configured the job was not the person who saw the invoice.
Design against that separation. The engineer who can start a metered job should see its running cost in the same interface where they start it.
Four controls to put in before your next agent ships
Set a hard ceiling per job, not per account. An account-level budget alert tells you the total moved; it does not tell you which job moved it, and by the time an account budget trips, the five months have already happened. Second, require an explicit answer to one question at approval: does this need to be answered now? If the answer is no, it is a batch workload and it should be priced as one.
Third, put an expiry on every long-running job. A matching task that has been running since March should have needed a human to renew it in April. Fourth, measure the output. If a project's score is the AI-assisted work that shipped, then a deployment that never shipped scores zero no matter how many tokens it consumed, and it would have surfaced in the first monthly review rather than the fifth.
Read next: Amazon Cut Its AGI Team in a $200bn AI Year | Your Human-Checked Data Was Already Part Machine
