Sporadically delivered thoughts on Continuous Delivery

Monitor Your Development Infrastructure as if It Were Business Critical

| Comments

A development team’s infrastructure - development and QA environments, CI servers, SCM servers, etc. - are indisputably business critical, but rarely given the kind of monitoring attention that production environments are. This is a missed opportunity, not only to ensure the continuity of development work, but also to gain valuable insight.

Reasons to monitor your application in every environment it’s deployed to:

1 Keep your development team moving

This is the obvious one. You need to know before you run out of disk space, RAM, etc.

2 Optimize your CI / deployment pipeline

Do you know what the limiting factors are on the time it takes your automated tests to run? The shorter you make your dev/test/fix feedback loop, the more productive your team will be, so why not analyze and optimize it as you would any other key software system? If checkout from SCM takes 20% of the time to test results, what can you do to reduce it? Are your unit tests constrained by CPU, RAM, or disk I/O?

3 Understand your applications

We’re conditioned to think that measuring performance and resource consumption is only useful in an environment that mirrors our production hardware. But if we build up an awareness of how our application uses memory and other resources as a part of every execution and every environment, we’ll have a deep and intuitive understanding of what makes it tick.

4 Develop and test your monitoring

By having monitoring running against applications while they are still in development, you will find ways to improve how you monitor (“let’s measure the activity on our queues”), catch holes in your monitoring (“why didn’t our monitoring tell us when the dev server queue broker went down?”), and test changes to your monitoring.

Once you put monitoring in place in development and testing and make a habit of using it, it becomes a ubiquitous and indispensable part of your team’s working processes, similar to the shift to using CI well.

Don’t leave it as a low priority task, something to get around to at some point after you get around to setting up a perfect performance testing environment. Put it at the center of your team’s toolset for understanding your work.