I’ve recently been involved with several groups interested in using Hadoop to process large sets of data, including use of higher level abstractions on top of Hadoop like Pig and Hive. What has surprised me most is that no one is automating their installation of Hadoop. In each case that I’ve observed they start by manually provisioning some servers and then follow a series of tutorials to manually install and configure a cluster. The typical experience seems to take about a week to setup a cluster. There is often a lot of wasted......
Continue Reading
It seems like most of the development around CloudFoundry and bosh happen on Linux or Mac. Getting things up and running in Windows was a real challenge. Below is how I worked things out. **Make sure you have a modern processor that supports all virtualization technologies, such as VTx and extended paging. Aside from the deviations mentioned below, I’m following the steps documented at https://github.com/cloudfoundry/bosh-lite Changes to Vagrantfile I’m using VirtualBox on Windows 7. To begin with, I modified the Vagrantfile to create two VMs rather than a single VM. The first is......
Continue Reading
Stackato, which is released by ActiveState, extends out of the box CloudFoundry. It adds a web interface and a command line client (‘stackato’), although the existing ‘cf’ command line client still works (as long as versions match up). Stackato includes some autoscale features and a very well done set of documentation. ActiveState publishes various VM images that can be used to quickly spin up a development environment. These include images for VMWare, KVM and VirtualBox, among others. In this post I’ll walk through getting a Stackato environment running on Windows using VirtualBox. Install......
Continue Reading
I may have just fallen in love with the tool Vagrant. Vagrant makes it possible to quickly create a virtual environment for development. It is different than cloning or snapshots in that it uses minimal base OSes and provides a provisioning mechanism to setup and configure the environment exactly the way you want for development. I love this for a few reasons: All developers work in the exact same environment Developers can get a new environment up in minutes Developers don’t need to be experts at setting up the environment. System details can......
Continue Reading