Sporadically delivered thoughts on Continuous Delivery

Six Weeks to Provision a VM?!? #ChangeManagementFail

| Comments

A while back when I was working in a global financial institution we requested two virtual machines for development and testing. Our requirements were for a standard operating system and application server from the IT organisation’s service catalog, nothing special.

The VMs were delivered six weeks later.

They were configured differently from one another.

One of them was broken - the application server failed to start due to permission errors.

Sad fact: this is the norm in our industry

Why did it take so long to deliver virtual machines, and why could they not deliver consistent, working machines? The IT department had the most expensive virtualisation platform money could buy, and used two different big-brand, top-dollar enterprise automation products for provisioning and configuring systems and software.

The organisation also had the most comprehensive, SOX certified, ITIL compliant processes you could have for change management and IT service delivery. That’s why it took six weeks; there were at least five different departments involved in creating the VM and configuring the operating system, user accounts, application server, and networking. The process involves CAB meetings, detailed service request documents, handover documents and security audits.

This is not an unusual story. I’ve worked in and spoken with dozens of enterprise IT organisations over the past few years, and this kind of thing is painfully common. In fact, it’s the norm. People in large organisations take this for granted. “Of course things take a while, and there are communication issues. We’re big and complex!”

When I suggest things don’t have to be this way, that there are some (admittedly a minority) large organisations which handle this stuff effectively by taking a different approach, they recoil. They say:

“That kind of stuff might work for websites and startups, but we’re too big and complex.”

This reaction is puzzling at first glance. Do people really think that more rigorous change management practices are not relevant to larger, older organisations?

I suspect the real root of the rejection of agile practices in large organisations is a belief that traditional change management practices work. Or at least, that they would work if properly applied. It certainly sounds like it should work. Spend more time planning and checking changes, get more people to reviewing and approve them, document everything more thoroughly, and evaluate each deviation from plan even more thoroughly, then, surely, the outcome will be of higher quality.


But in practice, things almost never work out this way. Each handover to another person or group is an opportunity to introduce errors and deviation. More misunderstanding, more loss of context, and less ownership for the end result. Very few organisations using these practices actually achieve high levels of quality (Hint: Your company is not NASA), instead things get done through heroics and workarounds to the process, with plenty of mistakes and disasters. (No, that’s not the organisation from my story.)

A better way

An effective IT organisation should be able to deliver a virtual machine in a matter of hours or minutes, rather than weeks. And it should be able to deliver this more reliably and consistently than a traditional process does, ensuring that the VM is fully compliant with corporate standards for security, quality, and conformity.

How could the organisation with the six week delivery time for VM’s achieve this?

  1. Create standard VM templates with the commonly required packages baked into them. When someone requests a VM, spin up an instance from the relevant template, rather than having a series of steps to configure different aspects of the system.
  2. Automate the process for updating these templates. Each time a change is needed or a new version of packages are released, apply these to the template, conduct quality checks (with as much automation as possible), and make the template available.
  3. Involve the experts from the various functional specialties (OS, etc.) in creating the automation for updating templates, and for implementing the automated validation, to ensure any change complies with corporate standards.
  4. Create a self-service application, so that groups requiring VM’s can use a dashboard to spin them up, entering cost centre and budget approval information as necessary.

There is more to it than this, of course, including how to ensure that changes to the standard template are applied to existing VMs created from earlier templates. But this is a start.