Today at 10:11am we experienced a 12-minute outage, which caused HTTP vals to return 503 errors and other types of vals to fail. In the end, the root cause was a deployment timing issue where database migrations were deployed successfully, but our application code deployment hung for several minutes. The new database migrations were incompatible with the old application code and crashed the process.

We aim to make all database migrations maintain backward compatibility, but in this case, we only discovered through the delayed deployment feedback that the new migrations were not compatible with previous versions.