Daniel Watrous on Software Engineering

A Collection of Software Problems and Solutions

Posts tagged deployment

Software Engineering

Developer Productivity and Vertical vs Horizontal Deployments

I’ve recently had many conversations related to developer productivity. In order for a developer to be productive, he must have control over enough of the application lifecycle to complete his work. When a developer gets stuck at any point in the application lifecycle, his productivity drops, which can often reduce morale too.

One question I’ve been asking is: how much of the application lifecycle needs to fall under the scope of the developer? In other words, how broad is the scope of the application lifecycle that needs to be available to a developer in order to keep him productive. Does the developer need to be able to create and configure his own server? If there is an application stack, should he also be empowered to deploy other applications and services in the stack on which his component depends? As development efforts increase, should capacity be increased to accommodate the individual development environments for each developer?

Vertical vs Horizontal deployments

As I was working through these questions with some colleagues, I began to make a distinction between a vertical and a horizontal deployment.

A vertical deployment is one that requires deploying all tiers and components in order to test any one of them. While this can create a less volatile development environment, it also increases the complexity and resource footprint required to develop an application. It also complicates integration, since any work done on other components or tiers in the stack are not available until the vertical development deployment is refreshed.

A horizontal deployment is one that focuses only on one component or tier. It is assumed that other application dependencies are provided elsewhere. This decreases development overhead and resource needs. It also speeds up integration, since changes made to other horizontal components become available more quickly. This can also increase developer productivity, since a developer is only required to understand his application, not the full stack.

In the above diagram I illustrate that many applications now have dependencies on other applications. This is especially true for microservices. However, it should not be necessary to deploy all related applications in order to develop one of them. I propose instead a horizontal deployment where all related applications are moving toward an integration deployment and that all development, QA and other validation work operate against the integration layer. For a team following the github flow, the initial branch, the pull request and finally the merge should represent stages in the horizontal progress toward production ready code. This also has the advantage of catching most integration problems in the development and QA stages, because production ready code can make it more quickly into the integration tier and is immediately available to any integrating applications.

Capacity benefits

One of the most obvious benefits to the horizontal approach is a reduced strain on compute and storage capacity. Sharing more of the vertical stack leaves available infrastructure resources free for other application teams. Naturally containers would accentuate this benefit even more.

When to go vertical

There are times when a developer will need to deploy other elements in the vertical stack. These may include database changes that would interfere with other development teams or coordinated modifications to interdependent applications. Even in these scenarios, it may be beneficial to develop against another team’s development deployment rather than their integration deployment.

Software Engineering

A Review of Docker

The most strikingly different characteristic of Docker, when compared to other deployment platforms, is the single responsibility per container Design (although some see it differently). One reason this looks so different is that many application developers view the complete software stack on which they deploy as a collection of components on a single logical server. For developers of larger applications, who already have experience deploying distributed stacks, the security and configuration complexity of Docker may feel more familiar. Docker brings a fresh approach to distributed stacks; one that may seem overly complex for developers of smaller applications to enjoy the convenience of Deploying their full stack to a single logical server.

Link to create applications

Docker does mitigate some of the complexity of a distributed stack by way of Linking. Linking is a way to connect multiple containers so that they have access to each other’s resources. Communication between linked containers happens over a private network between the two containers. Each container has a unique IP address on the private network. We’ll see later on that share volumes are a special case in Linking containers.

Statelessness and Persistence

One core concept behind Docker containers is that they are transient. They are fast and easy to start, stop and destroy. Once stopped, any resources associated with the running container are immediately returned to the system. This stateless approach can be a good fit for modern web applications, where statelessness simplifies scaling and concurrency. However, the question remains about what to do with truly persistent data, like records in a database.

Docker answers this question of persistence with Volumes. At first glance this appears to only provide persistence between running containers, but it can be configured to share data between host and container that will survive after the container exits.

It’s important to note that storing data outside the container breaches one isolation barrier and could be an attack vector back into any container that uses that data. It’s also important to understand that any data stored outside the container may require infrastructure to manage, backup, sync, etc., since Docker only manages containers.

Infrastructure Revision Control

Docker elevates infrastructure dependencies one level above system administration by encapsulating application dependencies inside a single container. This encapsulation makes it possible to maintain versioned deployment artifacts, either as Docker Buildfile or in binary form. This enables some interesting possibilities, such as testing a new server configuration or redeploying an old server configuration in minutes.

Two Ways to Build Docker Container Images

Docker provides two ways to create a new container.

  1. Buildfile
  2. Modify an existing container


A Buildfile is similar to a Vagrantfile. It references a base image (starting point) and a number of tasks to execute on that base image to arrive at a desired end state. For example, one could start with the Ubuntu base image and run a series of apt-get commands to install the Nginx web server and copy the default configuration. After the image is created, it can be used to create new containers that have those dependencies ready to go.

Images are containers in embryo, similar to how a class is an object in embryo.

A Buildfile can also be added to a git repository and builds can be automatically triggered whenever a change is committed against the Buildfile.

Modify an existing container

Unlike the Buildfile, which is a textfile containing commands, it is also possible to build a container from an existing image and run ‘/bin/bash’. From the bash prompt any desired changes can be made. These commands modify the actual image, which can then be committed into the DockerHub repository or stored elsewhere for later use.

In either case, the result is a binary image that can be used to create a container providing a specific dependency profile.

Scaling Docker

Docker alone doesn’t answer the question about how to scale out containers, although there are a lot of projects trying to answer that question. It’s important to know that containerizing an application doesn’t automatically make it easier to scale. It is necessary to create logic to build, monitor, link, distribute, secure, update and otherwise manage containers.

Not Small VMs

It should be obvious by this point that Docker containers are not intended to be small Virtual Machines. They are isolated, single function containers that should be responsible for a single task and linked together to provide a complete software stack. This is similar to the Single Responsibility Principle. Each container should have a single responsibility, which increases the likelihood of reuse and decreases the complexity of ongoing management.

Application Considerations

I would characterize most of the discussion above as infrastructure considerations. There are several application specific considerations to review.

PaaS Infection of Application Code

Many PaaS solutions infect application code. This may be in the form of requiring use of certain libraries, specific language versions or adhering to specific resource structures. The trade-off promise is that in exchange for the rigid application requirements, the developer enjoys greater ease and reliability when deploying and scaling an application and can largely ignore system level concerns.

The container approach that Docker takes doesn’t infect application code, but it drastically changes deployment. Docker is so flexible in fact, that it becomes possible to run different application components with different dependencies, such as differing versions of the same programming language. Application developers are free to use any dependencies that suit their needs and develop in any environment that they like, including a full stack on a single logical server. No special libraries are required.

While this sounds great, it also increases application complexity in several ways, some of which are unexpected. One is that the traditional role of system administrator must change to be more involved in application development. The management of security, patching, etc. need to happen across an undefined number of containers rather than a fixed number of servers. A related complexity is that application developers need to be more aware of system level software, security, conflicts management, etc.

While it is true that Docker containers don’t infect application code, they drastically change the application development process and blur traditional lines between application development and system administration.

Security is Complicated

Security considerations for application developers must expand to include understanding of how containers are managed and what level of system access they have. This includes understanding how Linking containers works so that communication between containers and from container to host or from container to internet can be properly secured. Management of persistent data that must survive beyond the container life cycle needs to enforce the same isolation and security that the container promises. This can become tricky in a shared environment.

Configuration is complicated

Application configuration is also complicated, especially communication between containers that are not running on a single logical server, but instead are distributed among multiple servers or even multiple datacenters. Connectivity to shared resources, such as a database or set of files becomes tricky if those are also running in containers. In order to accommodate dynamic life cycle management of containers across server and datacenter boundaries, some configuration will need to be handled outside the container. This too will require careful attention to ensure isolation and protection.


Docker and related containerization tools appear to be a fantastic step in the direction of providing greater developer flexibility and increased hardware utilization. The ability to version infrastructure and deploy variants in minutes is a big positive.

While the impacts on application development don’t directly impact the lines of code written, they challenge conventional roles, such as developer and system administrator. Increased complexity is introduced by creating a linked software stack where connectivity and security between containers need to be addressed, even for small applications.

Software Engineering

MongoDB Secure Mode

Security in MongoDB is relatively young in terms of features and granularity. Interestingly, they indicate that a typical use case would be to use Mongo on a trusted network “much like how one would use, say, memcached.

MongoDB does NOT run in secure mode by default.

As it is, the features that are available are standard, proven and probably sufficient for most use cases. Here’s a quick summary of pros and cons.

  • Pros
    • Nonce-based digest for authentication
    • Security applies across replica set nodes and shard members
  • Cons
    • Few recent replies on security wiki page
    • Course grained access control

User access levels

Course grained access control allows for users to be defined per database and given either read only or read/write access. Since there is no rigid schema in MongoDB, it’s not possible to limit access to a subset of collections or documents.

Limit to expected IPs

Along the lines of the ‘trusted network’ mentioned above, it’s recommended to configure each mongo instance to accept connections from specific ports. For example, you could limit access to the loopback address, or to an IP for a local private network.

Disable http interface

By default, a useful HTTP based interface provides information about the mongodb instance on a machine and links to similar interfaces on related machines in the replica set. This can be disabled by providing –nohttpinterface when starting mongod.

SSL ready

In cases where SSL security is required, Mongo can be compiled to include support for it. The standard downloads do not include this feature. A standard SSL key can be produced in the usual way, using openssl for example.

Software Engineering

WordPress plugin licensing: Google App Engine vs. Amazon EC2

In the introduction to this series, I outlined some of the requirements for the WordPress plugin licensing platform: Speed, reliability and scalability. These are critical. Just imagine what would happen if any of those were missing.

Requirements Justification

A slow platform might result in significantly fewer sales. One of our use cases is to provide a free, limited time trial, and poor performance when installing or using a plugin would almost certainly decrease sales conversions. Reliability issues would, at a minimum, reduce developer confidence when coupling a new plugin to the licensing platform. Finally, if the speed and reliability don’t scale then the market of potential consumers is limited to smaller plugins.

Possible solutions

Fortunately the problem of speed, reliability and scalability have already been solved. I know that it’s possible to build out servers, load balance them and otherwise build out systems to achieve these three aims, but I have something much simpler in mind. The two most compelling options available today both allow a developer to leverage the infrastructure of very large companies that exist solely on the internet: Amazon and Google.

The business model of both Amazon and Google require them to build out their own internal infrastructure to accommodate peak volume. The big downside to this is that the majority of the time, some or most of that infrastructure is sitting idle. A somewhat interesting upside to the scale of their infrastructure is that they have had to develop internal processes that enable them to expand supply in step with demand. In other words, they have to be able to add additional resources on the fly in the event of a new record peak. That may not sound as impressive as it is 🙂

At some point, each of these companies realized that they could leverage their unused infrastructure to increase their revenue. They more or less sub-lease existing resources to third parties. As this product model developed they may have isolated the resources they sell from the resources that power their main websites, but there is still a great deal of play between them. The two offerings are Google App Engine (GAE) and Amazon Web Services (AWS).

App Engine vs. Amazon Web Services


There are many more differences between these two platforms than I have time to get into here. However, one distinction between the two is helpful. Amazon offers a wide range of services (they add new services often) that provide the developer with a great deal of flexibility. However, the burden of choosing the right platform components and interconnecting them is also on the developer.

Google on the other hand has a more narrowly defined and inclusive platform. Rather than separating content distribution, processing, messaging, etc., Google keeps it all under the same hood. This reduces complexity for the developer at the cost of some flexibility.

This distinction is rather natural when you consider the diversity of products and engagement channels employed by Amazon and compare that to the more narrow range of services and engagement channels employed by Google.

The winner?

For the licensing project that I’m developing in this series, the scope is well defined and not overly complex. Google App Engine is the most appealing due to the ease of working in a local development environment and the ability to deploy and test on the live platform under the free quota limits (no initial cost or setup). It’s important to note that choosing Google’s platform instead of Amazon’s doesn’t make Amazon the loser and it doesn’t have to mean that I need to exclusively run on Google’s platform forever.

GAE provides both Python and Java enviornments. If I choose Java and approach the design carefully (e.g. good datastore abstraction…), it may not require too much effort to deploy on an Amazon EC2 instance if that becomes more appealing down the road.


The WordPress plugin licensing system will target the Google App Engine platform initially. Special attention will be given to abstracting the datastore so that I can take advantage of Google’s fast and scalable datastore and leave myself flexibility to move to an alternate if I deploy on Amazon’s platform in the future. Java is a first class citizen on both platforms. This provides some assurance that mainstream, mature frameworks will run smoothly. It also typically means that there will be plenty of documentation and support to accelerate development and deployment along.