I Failed to Ship Today

So far in 2026, we've launched two B2C and four B2B SaaS products along with nearly a dozen open-source projects. Our Built for Cloud series of tools is made up of fully baked, fork and deploy SaaS apps. There are three in the catalog now and we're dogfooding another three right now internally and with our clients. We're a tiny team. Only three developers, and we do all of our own shipping between client work. We can move fast because we don't have very many days like I had today.

Over the past few weeks, we've been rolling out a tool called Ballast. I've written about it here before. It was originally envisioned as a replacement for PHPScore, putting a numerical value on technical debt and helping teams identify the high-impact debt that's slowing them down. I'll do a fuller writeup on the evolution of PHPScore to Ballast in due time, because there are lots of good lessons to be learned from that experience. Suffice it to say, Ballast has started getting a bit of traction in recent days, and over the weekend I was getting intermittent messages about high CPU usage from Laravel Cloud. It was the weekend, and last week was a hard week for my family, so I decided to ignore it and just spend time with them. I figured I'd log in Monday, increase the compute, and maybe turn on autoscaling.

When I got to the office this morning, I set off to do my normal Monday work and I fired off a Claude agent to investigate the high CPU. The agent crunched away for about 15 minutes and what it came back with robbed me of the rest of my day. It was 100% my fault, and I really should have caught it sooner.

Ballast has a directory of open-source projects that we've scored for durability. Many of them have been submitted by curious users and maintainers, so the list is growing pretty steadily. Because Ballast is completely stack-agnostic, we have repos in our directory written in a bunch of different languages. When I built out the directory table, I thought it might be intresting for visitors to be able to filter by language, so I made it a multi-select filter with the selected options persisted to the query string. I also, crucially, failed to do any validation to ensure that the values being added to the filter in the URL were actually in the Languages Enum. This turned out to be a catastrophic mistake.

The results are paginated and the URLs are unbounded because any string could be added to the array of languages on which we're filtering. This means that this directory exposes a theoretically infinte number of different URLs. In a human-driven world, this wouldn't be a big deal. But most of the internet these days is bots. Bots will misbehave and do it at a scale that humans could never aspire to. What Claude found in my logs was hundreds of requests coming every minute from hundreds of IPs, all going to different directory-rooted URIs, many of of which included things we don't even have in our langauge filters. None of the requests came in to the same URL twice, so the caching I thought I had in place wasn't doing anything.

I briefly employed Cloud's "Under Attack" mode, putting an interstitial between us and all traffic, and I immediately changed the filtering to a single select and added validation. That was enough to turn the site back on and start working on optimizing. I added a custom middleware to redirect any requests that had an invalid language string to the main directory without creating a session. I improved some caching and queries internally. I added a handful of Cloudflare caching rules to keep the majority of these requests off the server entirely. It worked. The site is back to normal and much more robust than it was this morning. But it took most of my attention for most of today, which means I didn't ship anything new today.

The reason this is unforgivabe isn't that I should have known better when I built the multi-select filtering with the unvalidated filter values on the query string. I should have, but we all miss things, particularly when we're moving fast. The reason it's unforgiveable is that we have a tool that we plug in for all of our clients that would have immediately identified the problem, surfaced it to me, and suggested a fix. The bot traffic started picking up two weeks ago, and our tooling would have caught it then. I didn't notice it until it got so bad that the site started to bog down. I simply didn't plug our own tool into our own product.

So my next day off from client work is going to be a little different to most. I'm going to make Artisan Build my client for the day. I'm going to plug in all of the tools that we've deployed to keep our clients shipping fast, and I'm going to make sure that our launch playbook includes those integrations so they land on day one, instead of waiting until it feels like we need them.

The key to shipping fast this year is to significantly reduce the number of days you don't spend shipping. Observability is the key here. Whether your daily active users are measured in tens or millions, knowing where you're throwing exceptions, where your queries are slow, what's vulnerable to attack, where your routes are dragging, and where you're missing the cache you think you're hitting will keep your days from being snatched away from you like mine was today. And the great thing is, you don't need pretty charts or graphs. You don't need to keep a close eye on any of it yourself. All you need is a cron job that runs every morning and asks an MCP-connected AI to look at the measurements for the last 24 hours and make sure everything that seems to be working really is. You can keep your focus on what will move the needle for your business and ship confidently.