Daniel Watrous on Software Engineering

A Collection of Software Problems and Solutions

Posts tagged cloud

Software Engineering

HEAT or Ansible in OpenStack? Both!

Someone asked me today whether he should use HEAT or Ansible to automate his OpenStack deployment. My answer is that he should use both! It’s helpful to understand the original design decisions for each tool in order to use each effectively. OpenStack HEAT and Ansible were designed to do different things, although in the opensource tradition, they have been extended to accommodate some overlapping functionalities.

Cloud Native

In my post on What is Cloud Native, I show the five elements of application life cycle that can be automated in the cloud (image shown below). The two life cycle elements in blue, provision and configure, correspond to the most effective use of HEAT and Ansible.


OpenStack HEAT for Provisioning

HEAT is designed to capture details related to infrastructure and accommodate provisioning of that infrastructure on OpenStack. CloudFormation does the same thing in AWS and Terraform is an abstraction that has providers for both OpenStack and AWS (and many others).

HEAT provides vocabulary to define compute, storage, network and other infrastructure related resources. This includes the interrelationships between infrastructure resources, such as associating floating IPs with compute resources or binding a compute resource to a specific network. This also includes some bookkeeping items, like assigning key pairs for authentication and naming resources.

The end result of executing a heat template is a collection of one or more infrastructure resources based on existing images (VM, or volume).

Ansible for Configuration

Ansible, on the other hand, is designed to configure infrastructure after it has been provisioned. This includes activities like installing libraries and setting up a specific run time environment. System details like firewalls and log management, as well as application stack, databases, etc. are easily managed from Ansible.

Ansible can also easily accommodate application deployment. Activities such as moving application artifacts into specific places, managing users/groups and file permissions, tweaking configuration files, etc. are all easily done in Ansible.

The end result of executing an Ansible playbook is ready-to-use infrastructure.

Where is the Overlap?

Ansible can provision resources in openstack. HEAT can send a cloud-init script to a new server to perform configuration of the server. In the case of Ansible for provisioning, it is not nearly as articulate or granular for the purpose of defining infrastructure as HEAT. In the case of HEAT configuring infrastructure through cloud-init, you still need to find some way to dynamically manage the cloud-init scripts to configure each compute resource to fit into your larger system. I do use cloud-init with HEAT, but I generally find more value in leaving the bulk of configuration to Ansible.

Ansible inventory from HEAT

When using HEAT and Ansible together, it is necessary to generate the ansible inventory file from HEAT output. To accomplish this, you want to make sure HEAT outputs necessary information, like IP addresses. You can use your favorite scripting language to query HEAT and write the inventory file.

Example using both HEAT and Ansible

A while ago I published two articles that showed how I develop the Ansible configuration, and then extend that to work with HEAT for deploying complex, multi-server environments.

Install and configure a Multi-node Hadoop cluster using Ansible

Bulid a multi-server Hadoop cluster in OpenStack in minutes

The first article lays the foundation for deploying a complex system with Ansible. The second article builds on this by introducing HEAT to provision the infrastructure. The Ansible inventory file is dynamically generated using a python script and the OpenStack CLI.


While there is some ambiguity around the term provision in cloud parlance, I consider provision to be the process of creating infrastructure resources that are not generally configured. I refer to configuration as the process of operating against those provisioned resources to prepare them for a specific use case, such as running an application or a database. HEAT is a powerful tool for provisioning resources in OpenStack and Ansible is a great fit for configuring existing infrastructure resources.

Software Engineering

What is Cloud Native?

I hear a lot of people talking about cloud native applications these days. This includes technologists and business managers. I have found that there really is a spectrum of meaning for the term cloud native and that two people rarely mean the same thing when they say cloud native.

At one end of the spectrum would be running a traditional workload on a virtual machine. In this scenario the virtual host may have been manually provisioned, manually configured, manually deployed, etc. It’s cloudiness comes from the fact that it’s a virtual machine running in the cloud.

I tend to think of cloud native at the other end and propose the following definition:

The ability to provision and configure infrastructure, stage and deploy an application and address the scale and health needs of the application in an automated and deterministic way without human interaction

The activities necessary to accomplish the above are:

  • Provision
  • Configure
  • Build and Test
  • Deploy
  • Scale and Heal


Provision and Configure

The following diagram illustrates some of the workflow involved in provisioning and configuring resources for a cloud native application.

You’ll notice that there are some abstractions listed, including HEAT for openstack, CloudFormation for AWS and even Terraform, which can provision against both openstack and AWS. You’ll also notice that I include a provision flow that produces an image rather than an actual running resource. This can be helpful when using IaaS directly, but becomes essential when using containers. The management of that image creation process should include a CI/CD pipeline and a versioned image registry (more about that another time).

Build, Test, Deploy

With provisioning defined it’s time to look at the application Build, Test and Deploy steps. These are depicted in the following figure:

The color of the “Prepare Infrastructure” activity should hint that in this process it represents the workflow shown above under Provision and Configure. For clarity, various steps have been grouped under the heading “Application Staging Process”. While these can occur independently (and unfortunately sometimes testing never happens), it’s helpful to think of those four steps as necessary to validate any potential release. It should be possible to fully automate the staging of an application.


The discovery step is often still done in a manual way using configuration files or even manual edits after deploy. Discovery could include making sure application components know how to reach a database or how a load balancer knows to which application servers it should direct traffic. In a cloud native application, this discovery should be fully automated. When using containers it will be essential and very fluid. Some mechanisms that accommodate discovery include system level tools like etcd and DNS based tools like consul.

Monitor and Heal or Scale

There are loads of monitoring tools available today. A cloud native application requires monitoring to be close to real time and needs to be able to act on monitoring outputs. This may involve creating new resources, destroying unhealthy resources and even shifting workloads around based on latency or other metrics.

Tools and Patterns

There are many tools to establish the workflows shown above. The provision step will almost always be provider specific and based on their API. Some tools, such as terraform, attempt to abstract this away from the provider with mixed results. The configure step might include Ansible or a similar tool. The build, test and deploy process will likely use a tool like Jenkins to accomplish automation. In some cases the above process may include multiple providers, all integrated by your application.

Regardless of the tools you choose, the most important characteristic of a cloud native application is that all of the activities listed are automated and deterministic.

Software Engineering

OpenStack Development using DevStack

I’ve been sneaking up on CloudFoundry for a few weeks now. Each time I try to get a CloudFoundry test environment going I find a couple more technologies that serve either as foundation or support to CloudFoundry. For example, Vagrant, Ansible and Docker all feed into CloudFoundry. Today I come to OpenStack, by way of DevStack (see resources below for more links).


My objectives for this post are get OpenStack up and running by way of DevStack so that I can begin to explore the OpenStack API. CloudFoundry uses the OpenStack API to provision new VMs, so understanding the API is essential to troubleshooting CloudFoundry issues.

Some deviations from out of the box DevStack include:

  • Enable higher overcommit allocation ratios
  • Add Ubuntu image for new VMs
  • Remove quotas


For these experiments I am using a physical server with 8 physical cores, 16GB RAM and 1TB disk. The host operating system is Ubuntu 14.04 LTS. I’m following the DevStack instructions for a single machine. It may be possible to duplicate these experiments in a Vagrant environment, but the nested networking and resource constraints for spinning up multiple guests would muddle the effort.


DevStack looks for a local.conf in the same directory as the stack.sh script. This makes it possible to override default settings for any of the OpenStack components that will be started as part of the DevStack environment.

Passwords for Quick Resets

I found that I was frequently unstack.shing and clean.shing and stack.shing. To speed up this process I added the following lines to my local.conf to set passwords with default values. This prevented the prompts for these values so the script would run without any input.



I also included some networking defaults. NOTE: Even though I show [[local|localrc]] again in the snippet below, there is only one such section, so the values can be added to the above if you decide to use them.


As always, you may have to handle proxy environment variables for your network. Some features of OpenStack may require local non-proxy access, so remember to set the no_proxy parameter for local traffic.

The line about GIT is to accommodate some environments where git:// protocols are blocked. This allows GIT to operate over HTTPS, which can leverage proxy settings.

The last line about the VOLUME_BACKING_FILE_SIZE is to allocate more space to cinder to create volumes (we’ll need this to work with CloudFoundry in the future).

Add Ubuntu Image

The IMAGE_URLS parameter accepts a comma separated list of images to pre-load into OpenStack. Ubuntu publishes many cloud images for ease of inclusion into OpenStack.


Logging in DevStack

The Linux screen command allows multiple shell commands to be running at the same time and to survive a terminal exit. Adding the command below collects all log output into a single directory. This can be extremely helpful when troubleshooting.


The default location for $DEST is /opt/stack.


OpenStack makes it easy to overcommit physical resources. This can be a valuable tool to increase hardware utilization, especially when loads are unevenly distributed. Setting the allocation ratios for CPU, RAM and DISK affects the nova scheduler. CPU is the most safely overcommitted, which RAM coming up next and DISK last. A post-config on the nova.conf file can be accomplished by adding the snippet below to local.conf.

cpu_allocation_ratio = 16.0
ram_allocation_ratio = 4.0
disk_allocation_ratio = 1.0

Running DevStack

With a local.conf all set, OpenStack can be started and reset using the following DevStack commands from within the devstack directory.


Access OpenStack

At this point you should have a running OpenStack server. Depending on the IP address or DNS settings, you should be able to access it with a URL similar to the following:


Which should look something like this:


Remember that the credentials are whatever you set in your local.conf file (see above).


On a couple of occasions, unstack.sh and clean.sh were unable to properly reset the system and a subsequent stack.sh failed. In those cases it was necessary to manually remove some services. I accomplished this by manually calling apt-get remove package followed by apt-get autoremove. On one occasion, I finally tried a reboot, which corrected the issue (a first for me on Linux).

It’s very possible that I could have cycled through the running screen sessions or spent more time in the logs to discover the root cause.

Create and Use VMs

With a running OpenStack environment, we can now create some VMs and start testing the API. Security groups and key pairs can make it easy to work with newly created instances. You can manage both by navigating to Project -> Compute -> Access & Security -> Security Groups.

Create a Security Group

The first thing to do is add a security group. I called mine “standard”. After adding the security group, there is a short delay and then a new line item appears. Click “Manage Rules” for that line item.

I typically open up ICMP (for pings), SSH(22), HTTP(80) and HTTPS(443). This is what the rules look like.


Key Pairs

Key pairs can be managed by navigating to Project -> Compute -> Access & Security -> Key Pairs. All you have to do is click “Create Key Pair”. Choose a name that represents where you’ll use that key to gain access (such as ‘laptop’ or ‘workstation’). It will then trigger a download of the resulting pem file.

If you have a range of IP addresses that will allow off box access to these hosts, they you can use puttygen to get a version of the key that you can import into putty, similar to what I did here.

Otherwise, the typical use case will require uploading the pem file to the host where you are running OpenStack. I store the file in ~/.ssh/onserver.pem. Set the permissions to 400 on the pem file.

chmod 400 ~/.ssh/onserver.pem

Remove Quotas (optional)

You can optionally disable quotas so that an unlimited number of VMs of any size can be created (or at least up to the allocation ratio limits). This is done from the command line using the nova client. First you set the environment so you can fun the nova client as admin. Then you identify the tenant and finally call quota-update to remove the quotas. From the devstack folder, you can use these commands.

stack@watrous:~/devstack$ source openrc admin admin
stack@watrous:~/devstack$ keystone tenant-list
|                id                |        name        | enabled |
| 14f42a9bc2e3479a91fb163807767dbc |       admin        |   True  |
| 88d670a4dc714e8b982b3ee8f7a95554 |      alt_demo      |   True  |
| 6c4be67d187c4dcab0fba70d6a004021 |        demo        |   True  |
| 5d1f69d120514caab75bcf27a202d358 | invisible_to_admin |   True  |
| 422e13e213e44f08a79f64440b56ee5c |      service       |   True  |
stack@watrous:~/devstack$ nova quota-update --instances -1 --cores -1 --ram -1 --fixed-ips -1 --floating-ips -1 6c4be67d187c4dcab0fba70d6a004021

NOTE: Quotas can also be managed through the web interface as part of a project. When logged in as admin, navigate to “Admin -> Identity -> Projects”. Then for the project you want to update, click “More -> Modify Quotas”. Quotas will have to be modified for each project individually.

Launch a New Instance

Finally we can launch a new instance by navigating to Project -> Compute -> Instances and clicking the “Launch Instance” button. There are four tabs on the Launch Instance panel. The “Details” tab provides options related to the size of the VM and the image used to create it. The Access & Security tab makes it possible to choose a key pair to be automatically installed as well as any security groups that should apply.



You can see any running instances from the Instances view, including an assigned IP address to access the server. This should be on a private network and is based on the networking parameters you provided in local.conf above. If you have public IP addresses, you can assign those and perform other maintenance on your instance using the “More” menu for that instance line item, as shown here.


Connect to the Instance

Connect to the new instance from the host system by using the “-i” option to ssh to connect without needing a password.

ssh -i ~/.ssh/onserver.pem ubuntu@


Connect to the Internet from Guests

Networking is complicated in OpenStack. The process I have outlined in this post sets up DevStack to use nova-network. In order to route traffic from guests in OpenStack and the internet, iptables needs to be updated to handle masquerading. You can typically do this by running the command below on the OpenStack host (not the guest).

sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Depending on your setup, you may need to set this on eth1 instead of eth0 as shown.

NOTE: This iptables command is transient and will be required on each boot. It is possible to make iptables changes persistent on Ubuntu.


Networking remains the most complicated aspect of OpenStack to both manage and simulate for a development environment. It can feel awkward to have to connect to all instances by way of the host machine, but it greatly simplifies the setup. Everything is set now to explore the API.

Resources and related:


Software Engineering

Using Vagrant to Explore Ansible

Last week I wrote about Vagrant, a fantastic tool to spin up virtual development environments. Today I’m exploring Ansible. Ansible is an open source tool which streamlines certain system administration activities. Unlike Vagrant, which provisions new machines, Ansible takes an already provisioned machine and configures it. This can include installing and configuring software, managing services, and even running simple commands. Ansible doesn’t require any agent software to be installed on the system being managed. Everything is executed over SSH.

Ansible only runs on Linux (though I’ve heard of people running it in cygwin with some difficulty). In order to play with Ansible, I used Vagrant to spin up a control box and a subject box that are connected in a way that I can easily run Ansible commands. Here’s my Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  # define ansible subject (web in this case) box
  config.vm.define "subject" do |subject|
    subject.vm.box = "ubuntu/trusty64"
    subject.vm.network "public_network"
    subject.vm.network "private_network", ip: ""
    subject.vm.provider "virtualbox" do |v|
      v.name = "Ansible subject"
      v.cpus = 2
      v.memory = 768
    # copy private key so hosts can ssh using key authentication (the script below sets permissions to 600)
    subject.vm.provision :file do |file|
      file.source      = 'C:\Users\watrous\.vagrant.d\insecure_private_key'
      file.destination = '/home/vagrant/.ssh/id_rsa'
    subject.vm.provision :shell, path: "subject.sh"
    subject.vm.network "forwarded_port", guest: 80, host: 8080
  # define ansible control box (provision this last so it can add other hosts to known_hosts for ssh authentication)
  config.vm.define "control" do |control|
    control.vm.box = "ubuntu/trusty64"
    control.vm.network "public_network"
    control.vm.network "private_network", ip: ""
    control.vm.provider "virtualbox" do |v|
      v.name = "Ansible control"
      v.cpus = 1
      v.memory = 512
    # copy private key so hosts can ssh using key authentication (the script below sets permissions to 600)
    control.vm.provision :file do |file|
      file.source      = 'C:\Users\watrous\.vagrant.d\insecure_private_key'
      file.destination = '/home/vagrant/.ssh/id_rsa'
    control.vm.provision :shell, path: "control.sh"
  # consider using agent forwarding instead of manually copying the private key as I did above
  # config.ssh.forward_agent = true

Notice that I created a public network to get a DHCP external address. I also created a private network with assigned addresses in the open address space. This is so I can indicate to Ansible in the hosts file where to locate all of the inventory.

I had trouble getting SSH agent forwarding to work on Windows through PuTTY, so for now I’m manually placing the private key and updating known_hosts with the ‘ssh-keyscan’ command. You can see part of this in the Vagrantfile above. The remaining work is done in two scripts, one for the control and one of the subject.


#!/usr/bin/env bash
# set proxy variables
#export http_proxy=http://myproxy.com:8080
#export https_proxy=https://myproxy.com:8080
# install pip, then use pip to install ansible
apt-get -y install python-dev python-pip
pip install ansible
# fix permissions on private key file
chmod 600 /home/vagrant/.ssh/id_rsa
# add subject host to known_hosts (IP is defined in Vagrantfile)
ssh-keyscan -H >> /home/vagrant/.ssh/known_hosts
chown vagrant:vagrant /home/vagrant/.ssh/known_hosts
# create ansible hosts (inventory) file
mkdir -p /etc/ansible/
cat /vagrant/hosts >> /etc/ansible/hosts


#!/usr/bin/env bash
# fix permissions on private key file
chmod 600 /home/vagrant/.ssh/id_rsa

I also provide copy this hosts file into place on the control system so it knows against which inventory it should operate.


localhost   ansible_connection=local    ansible_connection=ssh

After running ‘vagrant up‘, I can verify that the control box is able to access the subject box using the ping module in ansible.



This post doesn’t demonstrate the use of Ansible, aside from the ping command. What it does do is provide an environment where I can build and run Ansible playbooks, which is exactly what I plan to do next.