HubSpot Engineering

Latest posts

Writing lately

Monitoring Driven DevOps

Last week, I attended the Velocity Conference on web performance and operations out in California. The conference itself was really good; a ton of good talks on topics like post-mortems, building systems for failure, failures that people have hit, how to successfully embrace developer ownership of product operations. Basically a lot of the things that we embrace and encounter every day at HubSpot.

On Tuesday night, there was a set of Ignite talks. If you've never been to an Ignite talk, the idea is that you have 5 minutes to give a talk. Pressure, right? Well, to make it more fun, you also have 20 slides and they auto-advance every 15 seconds. Okay, lots of pressure. And somehow I talked myself into submitting a proposal which was accepted so I then had to write a talk.

Just Too Late View more presentations from katzj.

As with many of the Velocity talks, mine was a story. About six months ago, I shifted to running a team that is responsible for automating, standardizing and improving the infrastructure that powers HubSpot. Part of this has involved a bit of a mindshare shift from that of just being a developer to caring a bit more about the operations side of things. In doing so, I've noticed some differences that have made me feel a bit more like the work I'm doing is reactive and just a little bit too late. Part of this is because the goal of operations work is that nothing goes wrong... this means that you build something and nothing happens! So you don't get to some of the preventative work you'd like.

But then, inevitably, the bad thing occurs. And then you do some sort of post-mortem. One of the things we've been finding in a lot of ours is that we could have found the problem faster instead of having our customers report the problem. So a lot of our corrective actions end up being things which would allow us to do so: alerts in our monitoring system, automated testing and similar things to find out sooner.

From here, I talked a bit about how we fared with the Amazon EBS problems a couple of months ago. Overall, we fared pretty well. But there were some areas where we had problems. One of these was that we had a database which was not being backed up according to our standards. We were lucky and it wasn't a lasting problem. But as part of the post-mortem, I went ahead and backed up the server. I then put a monitor in place to ensure the server was backed up. And then I generalized it out so that it applied to all of our servers. And this is when we discovered that there were a few other servers on the brink of disaster. So then I backed them up. Coincidentally, my team was working on the automated build process for our database servers of the future, so we went ahead and built it into that process so that servers would get it automatically in the future.

But I had a little bit of an inspiration here. As a developer, I would do test driven development; I'd write a unit test and then get my code to work to make the test pass. As an infrastructure developer, I was doing monitor driven infrastructure development; I wrote a monitor and then made the infrastructure match that state. Pretty powerful; I could do it with our old stuff or with new things we were building. And adding a monitor to start with is cheap; from there you can make easier value judgements on what the cost is of not making preventative improvements.

You do have to be careful that your monitors aren't noisy in the process, but it's working. I finally don't feel like I'm always behind and getting to things just too late. Instead, I'm getting to things before they explode which in the end means a more stable system, happier developers, and happier customers using our product.

Newsletter

For Agents

View Markdown

Share

More posts

Incident Report: August 11, 2026

A change to a service's team-ownership metadata cut its new instances off from a cache, and a rollback restarted every instance, so CRM record pages failed for 18 minutes on August 11. What happened, and what we're changing.

Hedging Erasure-Coded Reads in HDFS

Erasure coding halves the disk our HBase data needs, but every byte lives on one DataNode. Hedging slow reads by rebuilding the slow cell from its stripe's parity cut slow-read host-minutes on our case-study cluster by 98%.

HubSpot Incident Report: July 22, 2026

Customers in Europe faced CRM access issues on July 22 due to a configuration change. Learn about the incident, its impact, and our commitment to improved reliability.