Daniel Watrous on Software Engineering

A Collection of Software Problems and Solutions

Software Engineering

nginx buildpack – realtime

CloudFoundry accommodates buildpacks which define a deployment environment. A buildpack is distinct from an application and provides everything the application needs to run, including web server, language runtime, libraries, etc. The most basic structure for a buildpack requires three files inside a directory named bin.

buildpack-files

The buildpack files discussed in this post can be cloned or forked at https://github.com/dwatrous/buildpack-nginx

Some quick points about these buildpack files

  • All three files must be executable via bash
  • Can be shell scripts or any language that can be invoked using bash
  • Explicit use of paths recommended

detect

The detect script is intended to examine application files to determine whether it can accommodate the application. If it finds evidence that it can provide an environment to run an application, it should echo a message and exit with a status of zero. Otherwise it should exit with a non-zero status. In this example, the detect script looks for an index file with the extension html or htm.

#!/usr/bin/env bash
 
if [[ ( -f $1/index.html || -f $1/index.htm ) ]]
then
  echo "Static" && exit 0
else
  exit 1
fi

compile

The compile script is responsible to gather, build and position everything exactly as it needs to be in order to create the droplet that will be used to deploy the application. The second line of the script below shows that cloudfoundry calls the compile script and passes in paths to build-dir and cache-dir.

Develop and test the compile script

All of the commands in this file will run on a docker instance created specifically to perform the staging operation. It’s possible to develop the compile script inside the same docker container that will be used to stage your applicationi.

#!/usr/bin/env bash
# bin/compile <build-dir> <cache-dir>
 
shopt -s dotglob    # enables commands like 'mv *' to see hidden files
set -e              # exit immediately if any command fails (non-zero status)
 
# create local variables pointing to key paths
app_files_dir=$1
cache_dir=$2
buildpack_dir=$(cd $(dirname $0) && cd .. && pwd)
 
# download and build nginx
mkdir -p $cache_dir
cd $cache_dir
wget http://nginx.org/download/nginx-1.6.2.tar.gz
tar xzf nginx-1.6.2.tar.gz
cd nginx-1.6.2
./configure
make
 
# create hierarchy with only needed files
mkdir -p $cache_dir/nginx/bin
mkdir -p $cache_dir/nginx/conf
mkdir -p $cache_dir/nginx/logs
cp $cache_dir/nginx-1.6.2/objs/nginx $cache_dir/nginx/bin/nginx
cp $cache_dir/nginx-1.6.2/conf/nginx.conf $cache_dir/nginx/conf/nginx.conf
cp $cache_dir/nginx-1.6.2/conf/mime.types $cache_dir/nginx/conf/mime.types
 
# move applicaiton files into public directory
mkdir -p $cache_dir/public
mv $app_files_dir/* $cache_dir/public/
# copy nginx error template
cp $cache_dir/nginx-1.6.2/html/50x.html $cache_dir/public/50x.html
 
# put everything in place for droplet creation
mv $buildpack_dir/bin/launch.sh $app_files_dir/
mv $cache_dir/public $app_files_dir/
mv $cache_dir/nginx $app_files_dir/
 
# ensure manifest not in public directory
if [ -f $cache_dir/public/manifest.yml ]; then rm $cache_dir/public/manifest.yml; fi
if [ -f $cache_dir/public/stackato.yml ]; then rm $cache_dir/public/stackato.yml; fi

Notice that after nginx has been compiled, the desired files, such as the nginx binary and configuration file, must be explicitly copied into place where they will be included when the droplet is packaged up. In this case I put the application files in a sub-directory called public and the nginx binary, conf and logs in a sub-directory called nginx. A shell script, launch.sh, is also copied into the root of the application directory, which will be explained in a minute.

release

The output of the release script is a YAML file, but it’s important to understand that the release file itself must be an executable script. The key detail in the release file is the line that indicates how to start the web server process. In some cases that may require several commands, which would require them to be encapsulated in another script, such as the launch.sh script shown below.

#!/usr/bin/env bash
 
cat <<YAML
---
default_process_types:
  web: sh launch.sh
YAML

launch.sh

The launch.sh script creates a configuration file for nginx that includes the PORT and HOME directory for this specific docker instance. It then starts nginx as the local user.

#!/usr/bin/env bash
 
# create nginx conf file with PORT and HOME directory from cloudfoundry environment variables
mv $HOME/nginx/conf/nginx.conf $HOME/nginx/conf/nginx.conf.original
sed "s|\(^\s*listen\s*\)80|\1$PORT|" $HOME/nginx/conf/nginx.conf.original > $HOME/nginx/conf/nginx.conf
sed -i "s|\(^\s*root\s*\)html|\1$HOME/public|" $HOME/nginx/conf/nginx.conf
 
# start nginx web server
$HOME/nginx/bin/nginx -c $HOME/nginx/conf/nginx.conf -p $HOME/nginx

Using a buildpack

There are two ways to use a buildpack: either install it on the cloud controller node and let the detect script select it or store it in a GIT repository and provide the GIT URL in manifest.yml. The GIT repository must be publicly readable and accessible from the docker instance where staging will occur (proxy issues may interfere with this).

To use this buildpack, in an empty directory, create two files:

manifest.yml

---
applications:
- name: static-test
  memory: 40M
  disk: 200M
  instances: 1
  buildpack: https://github.com/dwatrous/buildpack-nginx.git

index.html

<h1>Hello World!</h1>

From within that directory, target, login and push your app (can be done locally or in the cloud).

helion-static-nginx-app-deployed

Resources

https://developer.ibm.com/answers/questions/15623/buildpack-compilation-step-failed/
http://blog.lesc.se/2011/11/how-to-change-file-premissions-in-git.html

5 Comments nginx buildpack – realtime

  1. Suryaveer

    Thanks for nice posts on custom buildpacks. These are really helpful. Can you please suggest some way to install some dependencies during the compile step? For my application dependencies are installed using apt-get.

    Thanks.

    Reply
      1. Tommy

        When using cf push with nginx buildpack in manifest.yml file:
        buildpack: https://github.com/dwatrous/buildpack-nginx.git
        Compile and droplet seems success, but nginx still not start successfully.

        2018-04-25T13:45:50.443+08:00 [API/20] [OUT] Process has crashed with type: “web”
        2018-04-25T13:45:50.454+08:00 [API/20] [OUT] App instance exited with guid 4511766d-245f-44ab-9c53-fad76370bb51 payload: {“instance”=>”6b645f72-61c9-41de-707b-2555”, “index”=>0, “reason”=>”CRASHED”, “exit_description”=>”Codependent step exited”, “crash_count”=>3, “crash_timestamp”=>1524635150405840366, “version”=>”40c781b3-b852-4552-9798-e77567ba53f3”}
        2018-04-25T13:45:50.901+08:00 [CELL/0] [OUT] Cell f237c36d-0a62-4579-a51b-b11ee1d58145 successfully destroyed container for instance 6b645f72-61c9-41de-707b-2555

        Do you know how to start nginx up in cf with this buildpack? Thanks in advance!

        Reply
        1. Daniel Watrous

          Tommy,

          This tutorial is almost four years old. I’m not in a position to test this with you. It’s possible you will find some hints here http://software.danielwatrous.com/nginx-in-docker-for-stackato-buildpacks/ that will get you past this point.

          Cloudfoundry was working on a way to deploy containers directly. Maybe this can help https://docs.cloudfoundry.org/adminguide/docker.html. I prefer kubernetes these days, but you may be able to do something similar in cloudfounry, in which case you don’t need to worry about the buildpack. You can just create a container image.

          Reply
          1. Tommy

            Thanks Daniel!
            it’s related the nginx config issue, replace with the correct nginx config and it works with the buildpack.

Leave A Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.