Immutable Infrastructure: Development Environments and Team Collaboration

This article is part of a series on Immutable Infrastructure and discusses the scenario of Development Environments and Team Collaboration. It may be valuable to review Immutable Infrastructure Basics before reading this article.

Development environments are intended to be flexible and in motion by design. At the same time, teams need to be able to work together and eventually deliver their work to other environments. As teams form and change over time, these objectives are often at odds with each other. The illustration below shows this dynamic at play over time.

The above illustration shows that when Joe joined the team, the Language/Runtime, Operating System, Libraries, Data and IDE were at specific versions. When he setup his development environment (probably a laptop), he would likely install the current version of each of these. After some when Sara joined, these were at later versions and she probably installed what was current at that time. So it continues for each member of the team.

Works for me

This leads to a common issue on many teams, which is that a change that works for one person fails to work for others, or fails in some specific environment. This results in exchanges like this:

• Joe: Your last commit fails to compile
• Sara: It works for me

What follows is a flurry of effort from both Joe and Sara to see if they have the same code, the same versions of libraries, the same data, etc. In the best case scenario, the difference is identified, additional instructions are added to a team playbook and other team members all take action to bring their own environments up to date with the new standard.

Production Issues

The worst case of “Works for me” is when code fails in production and the team is unable to reproduce it. The same flurry of activity ensues as the team tries to identify how their development and test environments differ from production.

One common difference that further complicates this is that most developer laptops are Windows or Mac, and the production environment is Linux. In this case, it’s not possible to address the divergence, which can result in hard to fix bugs.

Side Effects

The above issues lead to a number of side effects, most of which try to strike a balance between productivity and availability. In the interest of uninterrupted productivity, there is often a proliferation of environments so each team or contributor has control over their own environment. In the interest of availability, operations teams move to restrict access and force even small changes be exhaustively vetted and to go through approvals. This results in an organization that has one foot on the gas and the other foot on the brakes.

Solutions

Immutable infrastructure alleviates many of the problems above and can allow for both improved productivity and higher availability.

Containers

Containers are the most flexible and compelling solution. They encapsulate all the variables shown in the illustration above. They enjoy strong native support in Linux. They have also been widely adopted by infrastructure providers (Cloud) and have strong orchestration solutions (Kubernetes).

Machine Images

If containers aren’t available, consider machine images. There are several ways to design the machine image, including automation using a tool like Packer. Similar to a container image, a machine image is a file system designed for a particular purpose.

Alternatives

Some alternatives that aren’t fully immutable, but provide some similar benefits to consistency and productivity include Scripted Environments and VDI.

Scripted Environments

Some languages and platforms offer scripted mechanisms to normalize development environments. One example of a language based mechanism is Python Virtual Environments. A more general example is a tool like Vagrant, which allows a predefined environment to be reproduced at will (see this example for building a Hadoop Cluster locally). While this isn’t immutable, it goes a long way toward providing consistency, but watch for drift and divergence.

Virtual Desktop Infrastructure (VDI)

VDI makes it possible to centrally manage many aspects of the Operating System and the environment setup so that it remains consistent. While not immutable, it is a more controlled environment that can ensure all developers have a consistent experience. If something needs to change, those changes can be more easily tested and propagated.

Demonstration

An example I designed to show the real power of this approach makes use of PC BASIC. I always ask people how long they think it would take to setup their laptop or a new server to run the following BASIC program:

10  CLS
20  FOR B = 2 TO 40 STEP 2
30  PRINT B
40  NEXT  B
50  END

The quickest I’ve ever heard someone claim to be able to get started is an hour. Using containers, this can be solved in minutes (depending on download speeds). Not only is it fast, but for everyone that works on developing the BASIC program, the results will be identical. Complete the following steps:

1. Pull this container image https://hub.docker.com/r/dwatrous/pcbasic
2. Create a file, EVENS.BAS, and paste the code above into it
3. From the directory where that file is, run the following docker command

docker run --rm -ti -v $(pwd):/basicsource dwatrous/pcbasic /basicsource/EVENS.BAS

It may be necessary to change the local path reference depending on your host OS. What you should see is all even numbers from 2-40. This didn’t require any modifications to the development environment beyond having Docker available. This can be reproduced on any host system with Docker, including in CI/CD pipelines, kubernetes, dev, test, prod, etc.

Navigation

• Immutable Infrastructure Basics
• Immutable Infrastructure: Development Environments and Team Collaboration (you are here)
• Immutable Infrastructure: CI/CD (test/stage/audit/etc.)
• Immutable Infrastructure: Production Release