The week the site was quietly broken

For about a week, from the tenth to the sixteenth of June, the live site was broken and I did not know it. Every page that read from the database returned a 500. The home page looked fine, so when I glanced at it I saw a working site. The pages that actually mattered, the dentist profiles and the neighborhood lists, were dead.

The cause was small and stupid in the way these things usually are. A couple of runtime environment variables, the ones that point the app at the database, were missing in production. The build still passed. The site still deployed. It just could not reach its own data once it was live.

The real damage was that nothing was watching. I had health checks, but they checked the wrong thing. A check that loads the home page will tell you the server is up. It will not tell you that the page a real visitor wants is returning an error. Google noticed before I did. Search impressions fell off a cliff between the twelfth and the fifteenth while I was looking at a home page that loaded fine.

The fix took an afternoon. The lesson took longer. I added a probe that does what a visitor does: it curls the actual database-backed pages, on a schedule, and alerts me if any of them stop returning real content. A box cron runs it every five minutes. If a profile page goes blank again, my phone tells me, not Google three days later.

This is the part of building in public that does not make a good screenshot. The launch week numbers were already humbling, 33 visitors across 12 days, no conversions, an 85 percent bounce rate, and exactly one of 41 neighborhood hubs indexed. Finding out that a chunk of that flat week was a silent outage did not feel good. But a directory is only worth anything if the pages load, so a monitor that watches the right pages is not overhead. It is the product.