One of the most significant enablers of IT and software automation has been the shift away from fixed infrastructure to flexible infrastructure. Virtualization, process isolation, resource sharing and other forms of flexible infrastructure have been in use for many decades in IT systems. It can be seen in early Unix systems, Java application servers and even in common tools such as Apache and IIS in the form of virtual hosts. If flexible infrastructure has been a part of technology practice for so long, why is it getting so much buzz now? Infrastructure as......
Continue Reading
Someone asked me today whether he should use HEAT or Ansible to automate his OpenStack deployment. My answer is that he should use both! It’s helpful to understand the original design decisions for each tool in order to use each effectively. OpenStack HEAT and Ansible were designed to do different things, although in the opensource tradition, they have been extended to accommodate some overlapping functionalities. Cloud Native In my post on What is Cloud Native, I show the five elements of application life cycle that can be automated in the cloud (image shown......
Continue Reading
I’ve recently had some people ask how I deploy MongoDB. For a while I used their excellent online tool to deploy and monitor my clusters. Unfortunately they changed direction and I couldn’t afford their new tools, so I turned to Ansible. In order more easily share the process, I posted a simple example that you can run locally using Vagrant to deploy MongoDB using Ansible. https://github.com/dwatrous/ansible-mongodb As soon as you finish running the Ansible script, you can immediately connect to MongoDB and start working with Data. If you’re looking to learn more about......
Continue Reading
I hear a lot of people talking about cloud native applications these days. This includes technologists and business managers. I have found that there really is a spectrum of meaning for the term cloud native and that two people rarely mean the same thing when they say cloud native. At one end of the spectrum would be running a traditional workload on a virtual machine. In this scenario the virtual host may have been manually provisioned, manually configured, manually deployed, etc. It’s cloudiness comes from the fact that it’s a virtual machine running......
Continue Reading
In a previous post I demonstrated a method to deploy a multi-node Hadoop cluster using Vagrant and Ansible. This post builds on that and shows how to deploy a Hadoop cluster with an arbitrary number of slave nodes in minutes on OpenStack. This process makes use of the OpenStack orchestration layer HEAT to provision the resources, after which Ansible use used to configure those resources. All the scripts to do this yourself is available on github to clone and fork: https://github.com/dwatrous/hadoop-multi-server-ansible I have recorded a video demonstrating the entire process, including scaling the......
Continue Reading
I’ve recently been involved with several groups interested in using Hadoop to process large sets of data, including use of higher level abstractions on top of Hadoop like Pig and Hive. What has surprised me most is that no one is automating their installation of Hadoop. In each case that I’ve observed they start by manually provisioning some servers and then follow a series of tutorials to manually install and configure a cluster. The typical experience seems to take about a week to setup a cluster. There is often a lot of wasted......
Continue Reading
The more I automate, the more I have to answer the question of how to manage my secrets. Secrets that frequently come up include: SSH key pairs SSL private keys Credentials for external resources, such as databases and SaaS integrations Before cloud, when server resources were not ephemeral, these could be managed manually when the server was created. In cloud environments, servers are created and destroyed automatically and from minute to minute, which leaves the question about how to manage secrets. The OpenStack community is working on one solution called Barbican. I’ve been......
Continue Reading
I have observed that discussions about CloudFoundry often lack accurate context. Some questions I get that indicate context is missing include: What Java version does CloudFoundry support? What database products/versions are available How can I access the server directly? There are a few reasons that the questions above are not relevant for CloudFoundry (or any modern PaaS environment). To understand why, it’s important to understand how we got to PaaS and where we came from. Landscape When computers were first becoming a common requirement for the enterprise, most applications were monolithic. All applicaiton......
Continue Reading
My objective in this post is to explore the use of Ansible to configure a multi-server LEMP stack. This builds on the preliminary work I did demonstrating how to use Vagrant to create an environment to run Ansible. You can follow this entire example on any Windows (or Linux) host. Ansible only runs on Linux hosts, not Windows. As a result, I needed to provision one Linux host to act as Ansible controller. One aspect of Ansible that I wanted to explore is the ability to manage multiple hosts with different configurations. For......
Continue Reading
Last week I wrote about Vagrant, a fantastic tool to spin up virtual development environments. Today I’m exploring Ansible. Ansible is an open source tool which streamlines certain system administration activities. Unlike Vagrant, which provisions new machines, Ansible takes an already provisioned machine and configures it. This can include installing and configuring software, managing services, and even running simple commands. Ansible doesn’t require any agent software to be installed on the system being managed. Everything is executed over SSH. Ansible only runs on Linux (though I’ve heard of people running it in cygwin......
Continue Reading