Ever have that feeling you are forgetting something right as you leave work? You are probably thinking about your keys or your lunch box but I am talking about your SSL certificate. They don’t last forever, we know this when we setup SSL but that doesn’t stop it from sneaking up on us. It has happened to the big guys like Instagram and Google, at Teem recently, and of course for myself with my own home server.
At Teem, we aim for zero down-time deploys; so, one of the most important things we must validate is that things will not break mid-deploy!
The most sensitive step of the deploy process is the changes to our database. Prior to the automation I am about to describe, validation of the database migrations required specialized knowledge about Postgres, the changes to the application model, load on the database for that model, and a bit of general experience. This obviously slows down reviews and subsequently deploys. Worse, it was simply too easy to miss problem migrations when depending on only peer reviews. To make our lives easier we created a series of validation checks to ensure that each database migration will be backwards compatible.