We've all gotten used to thinking of AI in terms of how much we can get done for the $200 a month we're spending on the subscription, and that's by design. Big labs have introduced these heavily subsidized subscriptions to foster dependence on their models and tools, knowing that eventually they'll have to start passing on the real cost of inference on to customers.
I've been warning about this for months, and I keep getting shut down with claims that "inference will get cheaper and the real cost will settle somewhere around what we're paying now". And maybe that will happen, but I think what we saw with yesterday's release of the Claude PR Review Tool points to a different reality.
The new Claude Code PR tool differs from the Claude Code GitHub Action PR review in that it spawns multiple agents to examine the changes, each focused on a specific area of concern. It's not a new concept, of course. Many of us have been doing the same thing for a long time with custom runners and local pre-flight checks. What is new, however, is that this tool is only available via the API. You can't power it with your Claude subscription, so you're paying API prices for the tokens consumed. The $15 to $25 per PR estimate that Anthropic published with their announcemnt was met with exactly the response you'd expect from a fan base that's hooked on the fixed price model.
The first thing I'll say about the tool is that it's not really that good yet. So far, nobody who has used it has been blown away by the results. It's better than the Claude Code GitHub Actions tool, and it's certainly better than pushing code into production without any review, but in my limited testing and from conversations with others who have also used it, the signal to noise ratio is still not great. It still misses real bugs and tends focus on pedantic, low-value criticism. The hand-rolled tooling that we're using in our custom runners does a better job, at least for now. But for teams that don't have the resources to create and iterate on custom tooling, it could be a valuable addition to their workflow.
More importantly, however, its release might be a signal that we're nearing the end of the AI Trial Period. That they didn't limit this release to enterprise plans that already pay API pricing tells me that they're looking for ways to migrate their retail customers from a strictly subscription model to the more sustainable (for them) pay for use model.
Maybe there's nothing in the tea leaves here. Maybe we'll still be getting $5,000 of inference for $200 a month a year from now. But maybe we won't. Which leave us with the question we should have started asking ourselves a long time ago. Would this thing that we're using every day still be worth it if it cost 10x as much? I think for almost everyone who uses this technology regularly, the answer to that is complicated.
As for me, my approach is going to remain largely unchanged. I'll continue to limit my use of LLMs to pieces of work where the value I get from it is high enough that I'd still do it if the subscriptions went away tomorrow. I'll continue to ensure that I'm not relying on LLMs to do any work that could be done with deterministic tooling. I'll continue to replace frontier models with lower cost, open-weight, distilled models where the output is of sufficiently high quality. And I'll continue to ensure that there is no part of my work that I couldn't do without AI.
I continue to land right back where I started on the journey of learning to integrate AI into my workflow. It's a really valuable tool that, at pretty much any price, will have real utility in the application development lifecycle. And it will never be more than a tool.