Today: May 9, 2021 5:25 pm
A collection of Software and Cloud patterns with a focus on the Enterprise

Tag: automation


In spite of the popularity of Let’s Encrypt for generating free SSL/TLS certificates, I think their getting started page stinks. In case you feel the same way, I’ll try to give you a real quick start that will get you to your first certificate. Certificate Authority Let me point out that anyone with Linux (or Docker for that matter) can create a strong SSL/TLS certificate and encrypt their data. No third party is necessary. Where third parties like Comodo, GeoTrust, Verisign and now Let’s Encrypt come in is as Certificate Authorities. Publishers of......

Continue Reading


IT General Controls: Infrastructure vs Routing

IT general controls are important for various reasons, such as business continuity and regulatory compliance. Traditionally, controls have focused on the infrastructure itself. In the context of long running servers in fixed locations, this was often an effective approach. As virtualization and container technologies become more prevalent, especially in public cloud, infrastructure focused IT controls can start to get in the way of realizing the following benefits: Just in time provisioning Workload migration Network isolation Tight capacity management DevOps Automated deployments Automated remediation One way to maintain strong IT controls can still get......

Continue Reading


I’ve recently had many conversations related to developer productivity. In order for a developer to be productive, he must have control over enough of the application lifecycle to complete his work. When a developer gets stuck at any point in the application lifecycle, his productivity drops, which can often reduce morale too. One question I’ve been asking is: how much of the application lifecycle needs to fall under the scope of the developer? In other words, how broad is the scope of the application lifecycle that needs to be available to a developer......

Continue Reading


Infrastructure as a Service, like OpenStack and AWS, have made it possible to consume infrastructure on demand. It’s important to understand the ways in which both humans and machines interact with IaaS offerings in order to design optimal systems that leverage all possible automation opportunities. I drew the diagram below to help illustrate. Everything is an API At the heart of IaaS are REST APIs that provide granular access to every resource type, such as compute, storage and network. These APIs provide clarity about which resources are being managed and accommodate the type......

Continue Reading


What is Cloud Native?

I hear a lot of people talking about cloud native applications these days. This includes technologists and business managers. I have found that there really is a spectrum of meaning for the term cloud native and that two people rarely mean the same thing when they say cloud native. At one end of the spectrum would be running a traditional workload on a virtual machine. In this scenario the virtual host may have been manually provisioned, manually configured, manually deployed, etc. It’s cloudiness comes from the fact that it’s a virtual machine running......

Continue Reading


In a previous post I demonstrated a method to deploy a multi-node Hadoop cluster using Vagrant and Ansible. This post builds on that and shows how to deploy a Hadoop cluster with an arbitrary number of slave nodes in minutes on OpenStack. This process makes use of the OpenStack orchestration layer HEAT to provision the resources, after which Ansible use used to configure those resources. All the scripts to do this yourself is available on github to clone and fork: https://github.com/dwatrous/hadoop-multi-server-ansible I have recorded a video demonstrating the entire process, including scaling the......

Continue Reading


I’ve recently been involved with several groups interested in using Hadoop to process large sets of data, including use of higher level abstractions on top of Hadoop like Pig and Hive. What has surprised me most is that no one is automating their installation of Hadoop. In each case that I’ve observed they start by manually provisioning some servers and then follow a series of tutorials to manually install and configure a cluster. The typical experience seems to take about a week to setup a cluster. There is often a lot of wasted......

Continue Reading


Last week I wrote about Vagrant, a fantastic tool to spin up virtual development environments. Today I’m exploring Ansible. Ansible is an open source tool which streamlines certain system administration activities. Unlike Vagrant, which provisions new machines, Ansible takes an already provisioned machine and configures it. This can include installing and configuring software, managing services, and even running simple commands. Ansible doesn’t require any agent software to be installed on the system being managed. Everything is executed over SSH. Ansible only runs on Linux (though I’ve heard of people running it in cygwin......

Continue Reading


I have a RESTful SaaS service I created which uses MongoDB. Each REST call creates a new record in a statistics collection. In order to implement quotas and provide user analytics, I need to process the statistics collection periodically and generate meaningful analytics specific to each user. This is just the type of problem map reduce was meant to solve. In order to accomplish this I’ll need to do the following: Map all statistics records over a time range Reduce the number of calls, both authenticated and anonymous Finalize to get the sum......

Continue Reading