OpenStack Development using DevStack

I’ve been sneaking up on CloudFoundry for a few weeks now. Each time I try to get a CloudFoundry test environment going I find a couple more technologies that serve either as foundation or support to CloudFoundry. For example, Vagrant, Ansible and Docker all feed into CloudFoundry. Today I come to OpenStack, by way of DevStack (see resources below for more links).

Objectives

My objectives for this post are get OpenStack up and running by way of DevStack so that I can begin to explore the OpenStack API. CloudFoundry uses the OpenStack API to provision new VMs, so understanding the API is essential to troubleshooting CloudFoundry issues.

Some deviations from out of the box DevStack include:

• Enable higher overcommit allocation ratios
• Add Ubuntu image for new VMs
• Remove quotas

Environment

For these experiments I am using a physical server with 8 physical cores, 16GB RAM and 1TB disk. The host operating system is Ubuntu 14.04 LTS. I’m following the DevStack instructions for a single machine. It may be possible to duplicate these experiments in a Vagrant environment, but the nested networking and resource constraints for spinning up multiple guests would muddle the effort.

Configuration

DevStack looks for a local.conf in the same directory as the stack.sh script. This makes it possible to override default settings for any of the OpenStack components that will be started as part of the DevStack environment.

Passwords for Quick Resets

I found that I was frequently unstack.shing and clean.shing and stack.shing. To speed up this process I added the following lines to my local.conf to set passwords with default values. This prevented the prompts for these values so the script would run without any input.

[[local|localrc]]
ADMIN_PASSWORD=secret
MYSQL_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=$ADMIN_PASSWORD

Networking

I also included some networking defaults. NOTE: Even though I show [[local|localrc]] again in the snippet below, there is only one such section, so the values can be added to the above if you decide to use them.

[[local|localrc]]
FLOATING_RANGE=192.168.1.224/27
FIXED_RANGE=10.11.12.0/24
FIXED_NETWORK_SIZE=256
FLAT_INTERFACE=eth0
GIT_BASE=${GIT_BASE:-https://git.openstack.org}
VOLUME_BACKING_FILE_SIZE=200G

As always, you may have to handle proxy environment variables for your network. Some features of OpenStack may require local non-proxy access, so remember to set the no_proxy parameter for local traffic.

The line about GIT is to accommodate some environments where git:// protocols are blocked. This allows GIT to operate over HTTPS, which can leverage proxy settings.

The last line about the VOLUME_BACKING_FILE_SIZE is to allocate more space to cinder to create volumes (we’ll need this to work with CloudFoundry in the future).

Add Ubuntu Image

The IMAGE_URLS parameter accepts a comma separated list of images to pre-load into OpenStack. Ubuntu publishes many cloud images for ease of inclusion into OpenStack.

[[local|localrc]]
IMAGE_URLS="https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img,http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-disk.img,http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-uec.tar.gz"

Logging in DevStack

The Linux screen command allows multiple shell commands to be running at the same time and to survive a terminal exit. Adding the command below collects all log output into a single directory. This can be extremely helpful when troubleshooting.

[[local|localrc]]
SCREEN_LOGDIR=$DEST/logs/screen

The default location for $DEST is /opt/stack.

Overcommitting

OpenStack makes it easy to overcommit physical resources. This can be a valuable tool to increase hardware utilization, especially when loads are unevenly distributed. Setting the allocation ratios for CPU, RAM and DISK affects the nova scheduler. CPU is the most safely overcommitted, which RAM coming up next and DISK last. A post-config on the nova.conf file can be accomplished by adding the snippet below to local.conf.

[[post-config|$NOVA_CONF]]
[DEFAULT]
cpu_allocation_ratio = 16.0
ram_allocation_ratio = 4.0
disk_allocation_ratio = 1.0

Running DevStack

With a local.conf all set, OpenStack can be started and reset using the following DevStack commands from within the devstack directory.

./stack.sh
./unstack.sh
./clean.sh

Access OpenStack

At this point you should have a running OpenStack server. Depending on the IP address or DNS settings, you should be able to access it with a URL similar to the following:

http://openstack.yourdomain.com/auth/login/

Which should look something like this:

Remember that the credentials are whatever you set in your local.conf file (see above).

Troubleshooting

On a couple of occasions, unstack.sh and clean.sh were unable to properly reset the system and a subsequent stack.sh failed. In those cases it was necessary to manually remove some services. I accomplished this by manually calling apt-get remove package followed by apt-get autoremove. On one occasion, I finally tried a reboot, which corrected the issue (a first for me on Linux).

It’s very possible that I could have cycled through the running screen sessions or spent more time in the logs to discover the root cause.

Create and Use VMs

With a running OpenStack environment, we can now create some VMs and start testing the API. Security groups and key pairs can make it easy to work with newly created instances. You can manage both by navigating to Project -> Compute -> Access & Security -> Security Groups.

Create a Security Group

The first thing to do is add a security group. I called mine “standard”. After adding the security group, there is a short delay and then a new line item appears. Click “Manage Rules” for that line item.

I typically open up ICMP (for pings), SSH(22), HTTP(80) and HTTPS(443). This is what the rules look like.

Key Pairs

Key pairs can be managed by navigating to Project -> Compute -> Access & Security -> Key Pairs. All you have to do is click “Create Key Pair”. Choose a name that represents where you’ll use that key to gain access (such as ‘laptop’ or ‘workstation’). It will then trigger a download of the resulting pem file.

If you have a range of IP addresses that will allow off box access to these hosts, they you can use puttygen to get a version of the key that you can import into putty, similar to what I did here.

Otherwise, the typical use case will require uploading the pem file to the host where you are running OpenStack. I store the file in ~/.ssh/onserver.pem. Set the permissions to 400 on the pem file.

chmod 400 ~/.ssh/onserver.pem

Remove Quotas (optional)

You can optionally disable quotas so that an unlimited number of VMs of any size can be created (or at least up to the allocation ratio limits). This is done from the command line using the nova client. First you set the environment so you can fun the nova client as admin. Then you identify the tenant and finally call quota-update to remove the quotas. From the devstack folder, you can use these commands.

stack@watrous:~/devstack$ source openrc admin admin
stack@watrous:~/devstack$ keystone tenant-list
+----------------------------------+--------------------+---------+
|                id                |        name        | enabled |
+----------------------------------+--------------------+---------+
| 14f42a9bc2e3479a91fb163807767dbc |       admin        |   True  |
| 88d670a4dc714e8b982b3ee8f7a95554 |      alt_demo      |   True  |
| 6c4be67d187c4dcab0fba70d6a004021 |        demo        |   True  |
| 5d1f69d120514caab75bcf27a202d358 | invisible_to_admin |   True  |
| 422e13e213e44f08a79f64440b56ee5c |      service       |   True  |
+----------------------------------+--------------------+---------+
stack@watrous:~/devstack$ nova quota-update --instances -1 --cores -1 --ram -1 --fixed-ips -1 --floating-ips -1 6c4be67d187c4dcab0fba70d6a004021

NOTE: Quotas can also be managed through the web interface as part of a project. When logged in as admin, navigate to “Admin -> Identity -> Projects”. Then for the project you want to update, click “More -> Modify Quotas”. Quotas will have to be modified for each project individually.

Launch a New Instance

Finally we can launch a new instance by navigating to Project -> Compute -> Instances and clicking the “Launch Instance” button. There are four tabs on the Launch Instance panel. The “Details” tab provides options related to the size of the VM and the image used to create it. The Access & Security tab makes it possible to choose a key pair to be automatically installed as well as any security groups that should apply.

You can see any running instances from the Instances view, including an assigned IP address to access the server. This should be on a private network and is based on the networking parameters you provided in local.conf above. If you have public IP addresses, you can assign those and perform other maintenance on your instance using the “More” menu for that instance line item, as shown here.

Connect to the Instance

Connect to the new instance from the host system by using the “-i” option to ssh to connect without needing a password.

ssh -i ~/.ssh/onserver.pem ubuntu@10.11.12.2

Connect to the Internet from Guests

Networking is complicated in OpenStack. The process I have outlined in this post sets up DevStack to use nova-network. In order to route traffic from guests in OpenStack and the internet, iptables needs to be updated to handle masquerading. You can typically do this by running the command below on the OpenStack host (not the guest).

sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Depending on your setup, you may need to set this on eth1 instead of eth0 as shown.

NOTE: This iptables command is transient and will be required on each boot. It is possible to make iptables changes persistent on Ubuntu.

Comments

Networking remains the most complicated aspect of OpenStack to both manage and simulate for a development environment. It can feel awkward to have to connect to all instances by way of the host machine, but it greatly simplifies the setup. Everything is set now to explore the API.

Resources and related:

http://www.stackgeek.com/blog/kordless/post/taking-openstack-for-a-spin
http://docs.openstack.org/grizzly/openstack-image/content/ch_obtaining_images.html
http://docs.openstack.org/user-guide/content/dashboard_create_images.html
http://askubuntu.com/questions/471154/how-to-import-and-install-a-uec-image-on-openstack