Daniel Watrous on Software Engineering

A Collection of Software Problems and Solutions

Posts tagged abstraction

Software Engineering

Infrastructure as Code

One of the most significant enablers of IT and software automation has been the shift away from fixed infrastructure to flexible infrastructure. Virtualization, process isolation, resource sharing and other forms of flexible infrastructure have been in use for many decades in IT systems. It can be seen in early Unix systems, Java application servers and even in common tools such as Apache and IIS in the form of virtual hosts. If flexible infrastructure has been a part of technology practice for so long, why is it getting so much buzz now?

Infrastructure as Code

In the last decade, virtualization has become more accessible and transparent, in part due to text based abstractions that describe infrastructure systems. There are many such abstractions that span IaaS, PaaS, CaaS (containers) and other platforms, but I see four major categories of tool that have emerged.

  • Infrastructure Definition. This is closest to defining actual server, network and storage.
  • Runtime or system configuration. This operates on compute resources to overlay system libraries, policies, access control, etc.
  • Image definition. This produces an image or template of a system or application that can then be instantiated.
  • Application description. This is often a composite representation of infrastructure resources and relationships that together deliver a functional system.

Right tool for the right job

I have observed a trend among these toolsets to expand their scope beyond one of these categories to encompass all of them. For example, rather than use a chain of tools such as Packer to define an image, HEAT to define the infrastructure and Ansible to configure the resources and deploy the application, someone will try to use Ansible to to all three. Why is that bad?

A tool like HEAT is directly tied to the OpenStack charter. It endeavors to adhere to the native APIs as they evolve. The tools is accessible, reportable and integrated into the OpenStack environment where the managed resources are also visible. This can simplify troubleshooting and decrease development time. In my experience, a tool like Ansible generally lags behind in features, API support and lacks the native interface integration. Some argue that using a tool like Ansible makes the automation more portable between cloud providers. Given the different interfaces and underlying APIs, I haven’t seen this actually work. There is always a frustrating translation when changing providers, and in many cases there is additional frustration due to idiosyncrasies of the tool, which could have been avoided if using more native interfaces.

The point I’m driving at is that when a native, supported and integrated tool exists for a given stage of automation, it’s worth exploring, even if it represents another skill set for those who develop the automation. The insight gained can often lead to a more robust and appropriate implementation. In the end, a tool can call a combination of HEAT and Ansible as easily as just Ansible.

Containers vs. Platforms

Another lively discussion over the past few years revolves around where automation efforts should focus. AWS made popular the idea that automation at the IaaS layer was the way to go. A lot of companies have benefitted from that, but many more have found the learning curve too steep and the cost of fixed resources too high. Along came Heroku and promised to abstract away all the complexity of IaaS but still deliver all the benefits. The cost of that benefit came in either reduced flexibility or a steep learning curve to create new deployment contexts (called buildpacks). When Docker came along and provided a very easy way to produce a single function image that could be quickly instantiated, this spawned discussion related to how the container lifecycle should be orchestrated.

Containers moved the concept of image creation away from general purpose compute, which had been the focus of IaaS, and toward specialized compute, such as a single application executable. Start time and resource efficiency made containers more appealing than virtual servers, but questions about how to handle networking and storage remained. The docker best practice of single function containers drove up the number of instances when compared to more complex virtual servers that filled multiple roles and had longer life cycles. Orchestration became the key to reliable container based deployments.

The descriptive approaches that evolved to accommodate containers, such as kubernetes, provide more ease and speed than IaaS, while providing more transparency and control than PaaS. Containers make it possible to define their application deployment scenario, including images, networking, storage, configuration, routing, etc., in plain text and trust the Container as a Service (CaaS) to orchestrate it all.


Up to this point, infrastructure as code has evolved from shell and bash scripts, to infrastructure definitions for IaaS tools, to configuration and image creation tools for what those environments look like to full application deployment descriptions. What remains to mature are the configuration, secret management and regional distribution of compute locality for performance and edge data processing.

Software Engineering

Do Humans or Machines consume IaaS (REST API)

Infrastructure as a Service, like OpenStack and AWS, have made it possible to consume infrastructure on demand. It’s important to understand the ways in which both humans and machines interact with IaaS offerings in order to design optimal systems that leverage all possible automation opportunities. I drew the diagram below to help illustrate.


Everything is an API

At the heart of IaaS are REST APIs that provide granular access to every resource type, such as compute, storage and network. These APIs provide clarity about which resources are being managed and accommodate the type of independent evolution of each resource offering that keeps responsibilities focused. APIs include things like Nova for compute on the OpenStack side and EC2 for compute on the AWS side. Other IaaS providers have similar APIs with similar delineations between resource types.

Consume the API RAW

Since REST today is typically done over HTTP and makes use of the HTTP methods and response codes, it is relatively straight forward to use a tool such as curl, fiddler or a REST client to craft individual calls to APIs. This works fine for humans who want to understand the APIs better, but it is slow and doesn’t scale well. Machines (by way of software, obviously) can make those same calls using HTTP request libraries (e.g. python requests, Go http package, Java URLConnection, etc.).


After a couple of hours sending raw requests against the IaaS REST APIs, you’ll be thrilled to know that there are well designed abstractions that provide a more friendly interface to IaaS. The openstack Command Line Interface (CLI) allows you to operate against openstack APIs by typing concise commands at a shell prompt. The Horizon web console provides a web based user experience to manage those same resources. In both cases, the CLI and Horizon console are simply translating your commands or clicks into sequences of REST API calls. This means the ultimate interaction with IaaS is still based on the REST API, but you, the human, have a more human like interface.

Since machines don’t like to type on a keyboard or click on web pages, it makes sense to have abstractions for them too. The machines I’m talking about are the ones we ask to automatically test our code, deploy our software, monitor and scale and heal our applications. For them, we have other abscrations, such including HEAT and terraform. This allows us to provide the machine with a plain text description (often in YAML or similar markup), of what we want our infrastructure to look like. These orchestration tools then analyze the current state of our infrastructure and decide whether any actions are necessary. They then translate those actions into REST API calls, just like the CLI and Horizon panel did for the human. So again, all interaction with IaaS happens by way of the REST API, but the machine too gets an abstraction that humans can craft and read.

Automation is the end goal

Whichever method you pursue, the end goal is to automate as many of the IaaS interactions as possible. The more the machine can do without human interaction, the more time humans have to design the software running on top of IaaS and the more resilient the end system will be to failures and fluctuations. If you find yourself using the human abstractions more than telling machines what to do by way of infrastructure definitions, you have a great opportunity to change and gain some efficiency.

Software Engineering

Microservices are not as new as you think

All around me architects and business managers are beginning to mandate that internal software applications be built (and sometimes rebuilt) as microservices so that we can reuse them and compose applications more quickly. I do admit that the idea of a curated catalog of well designed microservices is attractive. Contrary to all the buzz that this is a new approach and will produce huge efficiencies, there is a lot of history that points to serious barriers to a microservice economy.

Microservices are Not New

What is being referred to as microservices today is just another name for abstraction and isolation, or code reuse. The idea is that a service should be designed to solve a single problem well in a way that is general (or abstract) enough to be broadly useful. This service is then isolated from its consumers which simplifies development and releases. Concepts around code reuse have date back at least 45 years.

The same goals of abstraction, isolation and reuse were the focus of Enterprise JavaBeans (EJB), which released its original specification in 1997. Much like microservices, EJBs were adopted by large enterprises who wanted all the benefits of code reuse and rapid application development through a type of composition. Unfortunately the complexity of the EJB APIs, and the apparent difficulty in designing easy to use interfaces reduced the value to the developer. Developers considered EJBs as adding complexity without delivering tangible benefit.

Similar efforts toward reuse have emerged as language libraries, drivers, patterns and so on. Many SaaS solutions today deliver on this same promise at a higher level making it possible to compose Enterprises from a collection of SaaS offerings.

But microservices are language independent…

Most discussion of microservices focuses on a RESTful interface (HTTP based) rather than a language specific interface (EJB). While this can make a microservice programming language agnostic, it requires an application developer to translate between the HTTP interaction with the service and their application code. Microservices must also deliver on the promise of simplifying, securing and scaling a service that is sufficiently abstract to be broadly useful, yet specific enough to solve a non-trivial problem.

Start with real problems, then abstract

It’s uncommon to see the most appropriate and useful abstraction for broadly useful services at the beginning of a project. The most useful abstractions often emerge over time as similar problems/solutions find their way into many different applications. Identifying these opportunities for reuse should start with existing solutions to the problems in your enterprise, but that alone isn’t enough. When recurring problems/solutions are identified, they still need to be useful when generalized. Excessive parameterization and configuration can limit the usefulness of the service. In other words, a microservice needs to be at least as easy to use (probably easier), and deliver at least as much value (probably more) than directly implementing the same functionality.

Some examples of microservices that are probably a bad idea:

  • spell checking
  • string formatting
  • email validation

Efficient language specific solutions already exist and have low latency for the cases listed above. They are also likely to have application specific implementations that interfere with making them generally useful. These functions are also unlikely to have associated enterprise wide standards.

Some examples that might make good microservices:

  • Fraud detection
  • Software licensing
  • Authentication and Authorization

In these cases, there are likely to be business specific rules that are common to the whole enterprise, which is benefited by centralizing a service. These types of services do not complete with well established norms or existing language specific libraries. Similar services are also offered through vendors and as SaaS, which further supports the possibility that it could be broadly useful and non-trivial to implement directly.

Start with a monolithic application

One of my favorite authors, Martin Fowler, recently published a follow up article to his previously published discussion of microservices architecture, which suggests the best way to develop microservices is to start by developing monolithic applications.

As he points out, if you already have a clear view of the boundaries for your service and are convinced that it can stand on it’s own and be broadly useful, there’s no reason not to start with microservices. What should be avoided is pre-mature optimization which drives the solution independent of the problem being solved.

Treat everything as a real product

One technique to make consistent decisions is to have a product owner who can align possible solutions with the actual product goals. Creating a microservice is a type of segmentation. The product is being split into two products and the would be product owner for the new microservice should be able to identify his consumer’s needs and where his product fits.

Follow the old rules

If I’ve convinced you that microservices are just an exercise in abstraction, isolation and code reuse, then you might find a lot of value in going back to basics for tips on how to design applications. Some of my favorite books on the subject are


Software Engineering

WordPress plugin licensing: Google App Engine vs. Amazon EC2

In the introduction to this series, I outlined some of the requirements for the WordPress plugin licensing platform: Speed, reliability and scalability. These are critical. Just imagine what would happen if any of those were missing.

Requirements Justification

A slow platform might result in significantly fewer sales. One of our use cases is to provide a free, limited time trial, and poor performance when installing or using a plugin would almost certainly decrease sales conversions. Reliability issues would, at a minimum, reduce developer confidence when coupling a new plugin to the licensing platform. Finally, if the speed and reliability don’t scale then the market of potential consumers is limited to smaller plugins.

Possible solutions

Fortunately the problem of speed, reliability and scalability have already been solved. I know that it’s possible to build out servers, load balance them and otherwise build out systems to achieve these three aims, but I have something much simpler in mind. The two most compelling options available today both allow a developer to leverage the infrastructure of very large companies that exist solely on the internet: Amazon and Google.

The business model of both Amazon and Google require them to build out their own internal infrastructure to accommodate peak volume. The big downside to this is that the majority of the time, some or most of that infrastructure is sitting idle. A somewhat interesting upside to the scale of their infrastructure is that they have had to develop internal processes that enable them to expand supply in step with demand. In other words, they have to be able to add additional resources on the fly in the event of a new record peak. That may not sound as impressive as it is 🙂

At some point, each of these companies realized that they could leverage their unused infrastructure to increase their revenue. They more or less sub-lease existing resources to third parties. As this product model developed they may have isolated the resources they sell from the resources that power their main websites, but there is still a great deal of play between them. The two offerings are Google App Engine (GAE) and Amazon Web Services (AWS).

App Engine vs. Amazon Web Services


There are many more differences between these two platforms than I have time to get into here. However, one distinction between the two is helpful. Amazon offers a wide range of services (they add new services often) that provide the developer with a great deal of flexibility. However, the burden of choosing the right platform components and interconnecting them is also on the developer.

Google on the other hand has a more narrowly defined and inclusive platform. Rather than separating content distribution, processing, messaging, etc., Google keeps it all under the same hood. This reduces complexity for the developer at the cost of some flexibility.

This distinction is rather natural when you consider the diversity of products and engagement channels employed by Amazon and compare that to the more narrow range of services and engagement channels employed by Google.

The winner?

For the licensing project that I’m developing in this series, the scope is well defined and not overly complex. Google App Engine is the most appealing due to the ease of working in a local development environment and the ability to deploy and test on the live platform under the free quota limits (no initial cost or setup). It’s important to note that choosing Google’s platform instead of Amazon’s doesn’t make Amazon the loser and it doesn’t have to mean that I need to exclusively run on Google’s platform forever.

GAE provides both Python and Java enviornments. If I choose Java and approach the design carefully (e.g. good datastore abstraction…), it may not require too much effort to deploy on an Amazon EC2 instance if that becomes more appealing down the road.


The WordPress plugin licensing system will target the Google App Engine platform initially. Special attention will be given to abstracting the datastore so that I can take advantage of Google’s fast and scalable datastore and leave myself flexibility to move to an alternate if I deploy on Amazon’s platform in the future. Java is a first class citizen on both platforms. This provides some assurance that mainstream, mature frameworks will run smoothly. It also typically means that there will be plenty of documentation and support to accelerate development and deployment along.