Daniel Watrous on Software Engineering

A Collection of Software Problems and Solutions

Posts tagged bottle

Software Engineering

JWT based authentication in Python bottle

May applications require authentication to secure protected resources. While standards like oAuth accommodate sharing resources between applications, more variance exists in implementations of securing the app in the first place. A recent standard, JWT, provides a mechanism for creating tokens with embedded data, signing these tokens and even encrypting them when warranted.

This post explores how individual resource functions can be protected using JWT. The solution involves first creating a function decorator to perform the authentication step. Each protected resource call is then decorated with the authentication function and subsequent authorization can be performed against the data in the JWT. Let’s first look at the decorator.

jwtsecret = config.authentication.get('jwtsecret')
 
class AuthorizationError(Exception):
    """ A base class for exceptions used by bottle. """
    pass
 
def jwt_token_from_header():
    auth = bottle.request.headers.get('Authorization', None)
    if not auth:
        raise AuthorizationError({'code': 'authorization_header_missing', 'description': 'Authorization header is expected'})
 
    parts = auth.split()
 
    if parts[0].lower() != 'bearer':
        raise AuthorizationError({'code': 'invalid_header', 'description': 'Authorization header must start with Bearer'})
    elif len(parts) == 1:
        raise AuthorizationError({'code': 'invalid_header', 'description': 'Token not found'})
    elif len(parts) > 2:
        raise AuthorizationError({'code': 'invalid_header', 'description': 'Authorization header must be Bearer + \s + token'})
 
    return parts[1]
 
def requires_auth(f):
    """Provides JWT based authentication for any decorated function assuming credentials available in an "Authorization" header"""
    def decorated(*args, **kwargs):
        try:
            token = jwt_token_from_header()
        except AuthorizationError, reason:
            bottle.abort(400, reason.message)
 
        try:
            token_decoded = jwt.decode(token, jwtsecret)    # throw away value
        except jwt.ExpiredSignature:
            bottle.abort(401, {'code': 'token_expired', 'description': 'token is expired'})
        except jwt.DecodeError, message:
            bottle.abort(401, {'code': 'token_invalid', 'description': message.message})
 
        return f(*args, **kwargs)
 
    return decorated

In the above code the requires_auth(f) function makes use of a helper function to verify that there is an Authorization header and that it appears to contain the expected token. A custom exception is used to indicate a failure to identify a token in the header.

The requires_auth function then uses the python JWT library to decode the key based on a secret value jwtsecret. The secret is obtained from a config object. Assuming the JWT decodes and is not expired, the decorated function will then be called.

Authenticate

The following function can be use to generate a new JWT.

jwtexpireoffset = config.authentication.get('jwtexpireoffset')
jwtalgorithm = config.authentication.get('jwtalgorithm')
 
def build_profile(credentials):
    return {'user': credentials['user'],
            'role1': credentials['role1'],
            'role2': credentials['role2'],
            'exp': time.time()+jwtexpireoffset}
 
bottle.post('/authenticate')
def authenticate():
    # extract credentials from the request
    credentials = bottle.request.json
    if not credentials or 'user' not in credentials or 'password' not in credentials:
        bottle.abort(400, 'Missing or bad credentials')
 
    # authenticate against some identity source, such as LDAP or a database
    try:
        # query database for username and confirm password
        # or send a query to LDAP or oAuth
    except Exception, error_message:
        logging.exception("Authentication failure")
        bottle.abort(403, 'Authentication failed for %s: %s' % (credentials['user'], error_message))
 
    credentials['role1'] = is_authorized_role1(credentials['user'])
    credentials['role2'] = is_authorized_role2(credentials['user'])
    token = jwt.encode(build_profile(credentials), jwtsecret, algorithm=jwtalgorithm)
 
    logging.info('Authentication successful for %s' % (credentials['user']))
    return {'token': token}

Notice that two additional values are stored in the global configuration, jwtalgorithm and jwtexpireoffset. These are used along with jwtsecret to encode the JWT token. The actual verification of user credentials can happen in many ways, including direct access to a datastore, LDAP, oAuth, etc. After authenticating credentials, it’s easy to authorize a user based on roles. These could be implemented as separate functions and could confirm role based access based on LDAP group membership, database records, oAuth scopes, etc. While the role level access shown above looks binary, it could easily be more granular. Since a JWT is based on JSON, the JWT payload is represented as a JSON serializable python dictionary. Finally the token is returned.

Protected resources

At this point, any protected resource can be decorated, as shown below.

def get_jwt_credentials():
    # get and decode the current token
    token = jwt_token_from_header()
    credentials = jwt.decode(token, jwtsecret)
    return credentials
 
@appv1.get('/protected/resource')
@requires_auth
def get_protected_resource():
    # get user details from JWT
    authenticated_user = get_jwt_credentials()
 
    # get protected resource
    try:
        return {'resource': somedao.find_protected_resource_by_username(authenticated_user['username'])}
    except Exception, e:
        logging.exception("Resource not found")
        bottle.abort(404, 'No resource for username %s was found.' % authenticated_user['username'])

The function get_protected_resource will only be executed if requires_auth successfully validates a JWT in the header of the request. The function get_jwt_credentials will actually retrieve the JWT payload to be used in the function. While I don’t show an implementation of somedao, it is simply an encapsulation point to facilitate access to resources.

Since the JWT expires (optionally, but a good idea), it’s necessary to build in some way to extend the ‘session’. For this a refresh endpoint can be provided as follows.

bottle.post('/authenticate/refresh')
@requires_auth
def refresh_token():
    """refresh the current JWT"""
    # get and decode the current token
    token = jwt_token_from_header()
    payload = jwt.decode(token, jwtsecret)
    # create a new token with a new exp time
    token = jwt.encode(build_profile(payload), jwtsecret, algorithm=jwtalgorithm)
 
    return {'token': token}

This simply repackages the same payload with a new expiration time.

Improvements

The need to explicitly refresh the JWT increases (possibly double) the number of requests made to an API only for the purpose of extending session life. This is inefficient and can lead to awkward UI design. If possible, it would be convenient to refactor requires_auth to perform the JWT refresh and add the new JWT to the header of the request that is about to be processed. The UI could then grab the updated JWT that is produced with each request to use for the subsequent request.

The design above will actually decode the JWT twice for any resource function that requires access to the JWT payload. If possible, it would be better to find some way to inject the JWT payload into the decorated function. Ideally this would be done in a way that functions which don’t need the JWT payload aren’t required to add it to their contract.

The authenticate function could be modified to return the JWT as a header, rather than in the body of the request. This may decrease the chances of a JWT being cached or logged. It would also simplify the UI if the authenticate and refresh functions both return the JWT in the same manner.

Extending

This same implementation could be reproduced in any language or framework. The basic steps are

  1. Perform (role based) authentication and authorization against some identity resource
  2. Generate a token (like JWT) indicating success and optionally containing information about the authenticated user
  3. Transmit the token and refreshed tokens in HTTP Authorization headers, both for authenticate and resource requests

Security and Risks

At the heart of JWT security is the secret used to sign the JWT. If this secret is too simple, or if it is leaked, it would be possible for a third party to craft a JWT with any desired payload, and trick an application into delivering protected resources to an attacker. It is important to choose strong secrets and to rotate them frequently. It would also be wise to perform additional validity steps. These might include tracking how many sessions a user has, where those session have originated and the nature and frequency/speed of requests. These additional measures could prevent attacks in the event that a JWT secret was discovered and may indicate a need to rotate a secret.

Microservices

In microservice environments, it is appealing to authenticate once and access multiple microservice endpoints using the same JWT. Since a JWT is stateless, each microservice only needs the JWT secret in order to validate the signature. This potentially increases the attach surface for a hacker who wants the JWT secret.

Software Engineering

Configuration of python web applications

Hopefully it’s obvious that separating configuration from application code is always a good idea. One simple and effective way I’ve found to do this in my python (think bottle, flask, etc.) apps is with a simple JSON configuration file. Choosing JSON makes sense for a few reasons:

  • Easy to read (for humans)
  • Easy to consume (by your application
  • Can be version alongside application code
  • Can be turned into a configuration REST service

Here’s a short example of how to do this for a simple python application that uses MongoDB. First the configuration file.

{
    "use_ssl": false,
    "authentication": {
        "enabled": false,
        "username": "myuser",
        "password": "secret"},
    "mongodb_port": 27017,
    "mongodb_host": "localhost",
    "mongodb_dbname": "appname"
}

Now in the python application you would load the simple JSON file above in a single line:

# load in configuration
configuration = json.load(open('softwarelicense.conf.json', 'r'))

Now in your connection code, you would reference the configuration object.

# establish connection pool and a handle to the database
mongo_client = MongoClient(configuration['mongodb_host'], configuration['mongodb_port'], ssl=configuration['use_ssl'])
db = mongo_client[configuration['mongodb_dbname']]
if configuration['authentication']['enabled']:
    db.authenticate(configuration['authentication']['username'], configuration['authentication']['password'])
 
# get handle to collections in MongoDB
mycollection = db.mycollection

This keeps the magic out of your application and makes it easy to change your configuration without direct changes to your application code.

Configuration as a web service

Building a web service to return configuration data is more complicated for a few reasons. One is that you want to make sure it’s secure, since most configuration data involves credentials, secrets and details about deployment details (e.g. paths). If a hacker managed to get your configuration data that could significantly increase his attack surface. Another reason is that it’s another potential point of failure. If the service was unavailable for any reason, your app would still need to know how to startup and run.

If you did create a secure, highly available configuration service, the only change to your application code would be to replace the open call with a call to urllib.urlopen or something similar.

Update, October 31, 2016

The below implementation extends the above ideas.

{
   "common": {
      "authentication": {
         "jwtsecret": "badsecret",
         "jwtexpireoffset": 1800,
         "jwtalgorithm": "HS256",
         "authorize": {
            "role1": "query or group to confirm role1",
            "role2": "query or group to confirm role2"
         },
         "message":{
            "unauthorized":"Access to this system is granted by request. Contact PersonName to get access."
         }
      }
   }
}

The following class encapsulates the management of the configuration file.

import os
import json
 
class authentication_config:
    common = {}
 
    def __init__(self, configuration):
        self.common = configuration['common']['authentication']
 
    def get_config(self):
        return self.common
 
class configuration:
    authentication = None
 
    def __init__(self, config_directory, config_file):
        # load in configuration (directory is assumed relative to this file)
        config_full_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), config_directory, config_file)
        with open(config_full_path, 'r') as myfile:
            configuration_raw = json.load(myfile)
        # prepare authentication
        self.authentication = authentication_config(configuration_raw)
 
 
def main():
    # get path to configuration
    config_directory = 'conf'
    config_file = 'myapp.conf.json'
 
    config = configuration(config_directory, config_file)
    print config.authentication.get_config['jwtsecret']
 
 
if __name__ == '__main__':
    main()
Software Engineering

Explore CloudFoundry using Stackato and VirtualBox

Stackato, which is released by ActiveState, extends out of the box CloudFoundry. It adds a web interface and a command line client (‘stackato’), although the existing ‘cf’ command line client still works (as long as versions match up). Stackato includes some autoscale features and a very well done set of documentation.

ActiveState publishes various VM images that can be used to quickly spin up a development environment. These include images for VMWare, KVM and VirtualBox, among others. In this post I’ll walk through getting a Stackato environment running on Windows using VirtualBox.

Install and Configure Stackato

The obvious first step is to download and install VirtualBox. The steps shown below should work on any system that runs VirtualBox.

I follow these steps to install and configure the Stackato VM.
http://docs.stackato.com/admin/setup/microcloud.html

After downloading the VirtualBox images for Stackato, from VirtualBox I click “File->Import Appliance” and navigating to the unzipped contents of the VM download. I select the OVF file and click open.

import-stackato-vm-virtualbox

This process can take several minutes depending on your system.

import-stackato-vm-virtualbox-progress

After clicking next, you can configure various settings. Click the checkbox to Reinitialize the MAC address on all network cards.

import-stackato-vm-virtualbox-reinitialize-mac

Once the import completes, right click on the VM in VirtualBox and choose settings.

stackato-vm-virtualbox-settings

Navigate to the network settings and make sure it’s set to Bridged. The default is NAT. Bridged will allow the VM to obtain it’s own IP address on the network. This will facilitate access later on.

stackato-vm-virtualbox-settings-network

Depending on your system resources, you may also want to go into the System settings and increase the Base Memory and number of Processors. You can also ensure that all virtualization accelerators are enabled.

Launch the Stackato VM

After you click OK, the VM is ready to launch. When you launch the VM, there is an initial message asking if you want to boot into recovery mode. Choose regular boot and you will then see a message about the initial setup.

launch-stackato-virtualbox-initial-setup

Eventually the screen will show system status. You’ll notice that it bounces around until it finally settles on a green READY status.

launch-stackato-virtualbox-ready

At this point you have a running instance of Stackato.

Configure Stackato

You can see in the above screenshot that Stackato displays a URL for the management console and the IP address of the system. In order to complete the configuration of the system (and before you can SSH into the server), you need to access the web console using that URL. This may require that you edit your local hosts file (/etc/hosts on Linux or C:\Windows\System32\drivers\etc\hosts on Windows). The entries in your hosts file should look something like this:

16.85.146.131	stackato-7kgb.local
16.85.146.131	api.stackato-7kgb.local

You can now access the console using https://api.stackato-7kgb.local. Don’t forget to put “http://” in front so the browser knows to request that URL rather than search for it. The server is also using a self-signed certificate, so you can expect your browser to complain. It’s OK to tell your browser to load the page despite the self-signed certificate.

On the page that loads, you need to provide a username, email address, and some other details to configure this Stakcato installation. Provide the requested details and click to setup the initial admin user.

stackato-cloudfoundry-configuration

A couple of things just happened. First off, a user was created with the username you provide. The password you chose will also become the password for the system user ‘stackato‘. This is important because it allows you to SSH into your instance.

Wildcard DNS

Widlcard DNS, using a service like xip.io, will make it easier to access Stackato and any applications that you deploy there. First we log in to our VM over SSH and we use the node rename command to enable wildcard DNS.

kato node rename 16.85.146.131.xip.io

Stopping and starting related roles takes a few minutes after running the command above. Once the server returns to READY state, the console is available using the xip.io address, https://api.16.85.146.131.xip.io. This also applies to any applications that are deployed.

The entries in the local hosts file are no longer needed and can be removed.

Proxy Configuration

Many enterprise environments route internet access through a proxy. If this is the case for you, it’s possible to identify the upstream proxy for all Stackato related services. Run the following commands on the Stackato VM to enable proxy access.

kato op upstream_proxy set proxy.domain.com:8080
sudo /etc/init.d/polipo restart

It may also be necessary to set the http_proxy and https_proxy environment variables by way of the .bashrc for the stackato system user.

At this point you should be able to easily deploy a new app from the app store using the Stackato console. Let’s turn our attention now to using a client to deploy a simple app.

Use the stackato CLI to Deploy an App

The same virtual host that is running Stackato also includes the command line client. That meas it can be used to deploy a simple application and verify that Stackato is working properly. To do this, first connect to the VM using SSH. Once connected, the following steps will prepare the command line client to deploy a simple application.

  1. set the target
  2. login/authenticate
  3. push the app

To set the target and login, we use these commands

stackato target api.16.85.146.131.xip.io
stackato login watrous

The output of the commands can be seen in the image below:

stackato-cli-target-login

Example Python App

There is a simple python bottle application we can use to confirm our Stackato deployment. To deploy we first clone, then use the stackato client to push, as shown below.

stackato-cli-deploy-push-python

Here are those commands in plain text:

git clone https://github.com/Stackato-Apps/bottle-py3.git
cd bottle-py3/
stackato push -n

Using the URL provided in the output from ‘stackato push’ we can view the new app.

stackato-python-bottle-app

You can now scale up instances and manage other aspects of the app using the web console or the stackato client.

Software Engineering

Detecting Credit Card Fraud – Frequency Algorithm

About 13 years ago I created my first integration with Authorize.net for a client who wanted to accept credit card payments directly on his website. The internet has changed a lot since then and the frequency of fraud attempts has increased.

One credit card fraud signature I identified while reviewing my server logs for one of my e-commerce websites was consistent. I refer to this is a shotgun attack, since the hacker sends through hundreds of credit card attempts. Here’s how it works and what to look for.

  1. All requests from a single throw away IP address
  2. Each request uses a different card
  3. Throwaway details are often used, including a generic email with some numbers in it

Frequency Algorithm

On the other hand, the overwhelming majority of other transactions were performed using a single card. Even if there were multiple attempts, they generally used one or two cards, but not more. I guessed I could use an algorithm that worked as follows for each transaction.

  1. Create a hash based on the last four digits on the card, the expiration date. This could use MD5, SHA or any other algorithm.
  2. Create a counter for the IP address that submitted that combination of values as represented by the hash and initialize to one
  3. For each transaction attempt, repeat step 1. If the hash matches what was stored in step 2 then don’t increment. If it doesn’t match, then increment the counter to two.

This process is repeated for every transaction attempt. Notice that a customer is free to continue submitting different addresses or CCV values for a single card without incrementing the counter. If the counter reaches a threshold, all transactions submitted from that IP address can be dropped. In my implementation I provided for an hour retention of data on a given IP address. The hour retention is reset every time a transaction is attempted from the IP address, which could keep it blocked indefinitely.

This credit card fraud prevention algorithm was implemented as a RESTful service using python bottle and memcached and provides less 100ms response times under heavy load and high concurrency.