Daniel Watrous on Software Engineering

A Collection of Software Problems and Solutions

Posts tagged rest

Software Engineering

Do Humans or Machines consume IaaS (REST API)

Infrastructure as a Service, like OpenStack and AWS, have made it possible to consume infrastructure on demand. It’s important to understand the ways in which both humans and machines interact with IaaS offerings in order to design optimal systems that leverage all possible automation opportunities. I drew the diagram below to help illustrate.

terraform-stackato-automation0001

Everything is an API

At the heart of IaaS are REST APIs that provide granular access to every resource type, such as compute, storage and network. These APIs provide clarity about which resources are being managed and accommodate the type of independent evolution of each resource offering that keeps responsibilities focused. APIs include things like Nova for compute on the OpenStack side and EC2 for compute on the AWS side. Other IaaS providers have similar APIs with similar delineations between resource types.

Consume the API RAW

Since REST today is typically done over HTTP and makes use of the HTTP methods and response codes, it is relatively straight forward to use a tool such as curl, fiddler or a REST client to craft individual calls to APIs. This works fine for humans who want to understand the APIs better, but it is slow and doesn’t scale well. Machines (by way of software, obviously) can make those same calls using HTTP request libraries (e.g. python requests, Go http package, Java URLConnection, etc.).

Abstractions

After a couple of hours sending raw requests against the IaaS REST APIs, you’ll be thrilled to know that there are well designed abstractions that provide a more friendly interface to IaaS. The openstack Command Line Interface (CLI) allows you to operate against openstack APIs by typing concise commands at a shell prompt. The Horizon web console provides a web based user experience to manage those same resources. In both cases, the CLI and Horizon console are simply translating your commands or clicks into sequences of REST API calls. This means the ultimate interaction with IaaS is still based on the REST API, but you, the human, have a more human like interface.

Since machines don’t like to type on a keyboard or click on web pages, it makes sense to have abstractions for them too. The machines I’m talking about are the ones we ask to automatically test our code, deploy our software, monitor and scale and heal our applications. For them, we have other abscrations, such including HEAT and terraform. This allows us to provide the machine with a plain text description (often in YAML or similar markup), of what we want our infrastructure to look like. These orchestration tools then analyze the current state of our infrastructure and decide whether any actions are necessary. They then translate those actions into REST API calls, just like the CLI and Horizon panel did for the human. So again, all interaction with IaaS happens by way of the REST API, but the machine too gets an abstraction that humans can craft and read.

Automation is the end goal

Whichever method you pursue, the end goal is to automate as many of the IaaS interactions as possible. The more the machine can do without human interaction, the more time humans have to design the software running on top of IaaS and the more resilient the end system will be to failures and fluctuations. If you find yourself using the human abstractions more than telling machines what to do by way of infrastructure definitions, you have a great opportunity to change and gain some efficiency.

Software Engineering

OpenStack REST API

There are some high quality resources that already cover the OpenStack API, so this is a YEA (yet another example) post. See the resources section below for some helpful links.

OpenStack APIs provide access to all OpenStack components, such as nova (compute), glance (VM images), swift (object storage), cinder (block storage), keystone (authentication) and neutron (networking). Authentication tokens are valid for a fixed duration, after which they expire and must be replaced. Each service requires it’s own token. Services that are hosted on the same logical server are typically accessible over different ports.

OpenStack APIs are RESTful, which means there are many ways to use them. In this post I’ll demonstrate three approaches that should provide clarity into their structure.

  • Command Line Interface (CLI)
  • cURL
  • REST Client

In this post I don’t cover programming against the REST APIs, but instead focus just on how they work. This work builds on my OpenStack development post.

Command Line Interface (CLI)

Command Line Interfaces used to manage OpenStack components make use of the REST APIs behind the scenes: a rather smart design choice on the part of the OpenStack community. This brings consistency to OpenStack management efforts and discourages disparity between standard tooling (CLI) and custom tooling (direct API access).

Credentials are required to access the REST APIs. For the command line client, these credentials are stored as environment variables. If you’re using DevStack, you can use the openrc script to automatically setup your environment.

source openrc admin admin

Some clients support a debug option that will output full details about the request and response cycle. Raw request and response details can be helpful when learning the APIs or creating programmatic access libraries that wrap the APIs. Here’s an example that will list the flavors available.

$ nova --debug flavor-list
REQ: curl -i 'http://openstack.danielwatrous.com:5000/v2.0/tokens' -X POST -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "{SHA1}95397c42a173838417806ce19d78f133ae6baa24"}}}'
INFO (connectionpool:258) Starting new HTTP connection (1): proxy.company.com
DEBUG (connectionpool:375) Setting read timeout to 600.0
DEBUG (connectionpool:415) "POST http://openstack.danielwatrous.com:5000/v2.0/tokens HTTP/1.1" 200 6823
RESP: [200] CaseInsensitiveDict({'content-length': '6823', 'proxy-connection': 'Keep-Alive', 'vary': 'X-Auth-Token', 'server': 'Apache/2.4.7 (Ubuntu)', 'connection': 'Keep-Alive', 'date': 'Thu, 21 Aug 2014 19:09:21 GMT', 'content-type': 'application/json'})
RESP BODY: {"access": {"token": {"issued_at": "2014-08-21T19:09:21.692110", "expires": "2014-08-21T20:09:21Z", "id": "{SHA1}99ff604f28f5706bfd82a00c21e099cba7fafab2", "tenant": {"enabled": true, "description": null, "name": "admin", "id": "32c13e88d51e49179c28520f688fa74d"}}, "serviceCatalog": [{"endpoints_links": [], "endpoints": [{"adminURL": "http://openstack.danielwatrous.com:8774/v2/32c13e88d51e49179c28520f688fa74d", "region": "RegionOne", "publicURL": "http://openstack.danielwatrous.com:8774/v2/32c13e88d51e49179c28520f688fa74d", "internalURL": "http://openstack.danielwatrous.com:8774/v2/32c13e88d51e49179c28520f688fa74d", "id": "03d570ce41c04daeb7ffa274c20435f0"}], "type": "compute", "name": "nova"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://openstack.danielwatrous.com:8776/v2/32c13e88d51e49179c28520f688fa74d", "region": "RegionOne", "publicURL": "http://openstack.danielwatrous.com:8776/v2/32c13e88d51e49179c28520f688fa74d", "internalURL": "http://openstack.danielwatrous.com:8776/v2/32c13e88d51e49179c28520f688fa74d", "id": "20d2caebf4814e1bb2c05f30a4802a2c"}], "type": "volumev2", "name": "cinderv2"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://openstack.danielwatrous.com:8774/v3", "region": "RegionOne", "publicURL": "http://openstack.danielwatrous.com:8774/v3", "internalURL": "http://openstack.danielwatrous.com:8774/v3", "id": "47f43a622264422f8980f3b0fbac5f00"}], "type": "computev3", "name": "novav3"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://openstack.danielwatrous.com:3333", "region": "RegionOne", "publicURL": "http://openstack.danielwatrous.com:3333", "internalURL": "http://openstack.danielwatrous.com:3333", "id": "149e00e61cc543cf94ae6162f79d9f00"}], "type": "s3", "name": "s3"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://openstack.danielwatrous.com:9292", "region": "RegionOne", "publicURL": "http://openstack.danielwatrous.com:9292", "internalURL": "http://openstack.danielwatrous.com:9292", "id": "1b7a45b6d1c840978491250fd1a67204"}], "type": "image", "name": "glance"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://openstack.danielwatrous.com:8000/v1", "region": "RegionOne", "publicURL": "http://openstack.danielwatrous.com:8000/v1", "internalURL": "http://openstack.danielwatrous.com:8000/v1", "id": "0b8abc323d884a0aa657bcb2f0274ee5"}], "type": "cloudformation", "name": "heat-cfn"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://openstack.danielwatrous.com:8776/v1/32c13e88d51e49179c28520f688fa74d", "region": "RegionOne", "publicURL": "http://openstack.danielwatrous.com:8776/v1/32c13e88d51e49179c28520f688fa74d", "internalURL": "http://openstack.danielwatrous.com:8776/v1/32c13e88d51e49179c28520f688fa74d", "id": "63675bf8e9a04d199cffafb7b8354b05"}], "type": "volume", "name": "cinder"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://openstack.danielwatrous.com:8773/services/Admin", "region": "RegionOne", "publicURL": "http://openstack.danielwatrous.com:8773/services/Cloud", "internalURL": "http://openstack.danielwatrous.com:8773/services/Cloud", "id": "520de40a96ea47c4a08c3ae5e0a8243c"}], "type": "ec2", "name": "ec2"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://openstack.danielwatrous.com:8004/v1/32c13e88d51e49179c28520f688fa74d", "region": "RegionOne", "publicURL": "http://openstack.danielwatrous.com:8004/v1/32c13e88d51e49179c28520f688fa74d", "internalURL": "http://openstack.danielwatrous.com:8004/v1/32c13e88d51e49179c28520f688fa74d", "id": "0dcff3f004ff4c9b9b96d012e47d2edb"}], "type": "orchestration", "name": "heat"}, {"endpoints_links": [], "endpoints": [{"adminURL": "http://openstack.danielwatrous.com:35357/v2.0", "region": "RegionOne", "publicURL": "http://openstack.danielwatrous.com:5000/v2.0", "internalURL": "http://openstack.danielwatrous.com:5000/v2.0", "id": "6f43e35702844e149dde900124c352bf"}], "type": "identity", "name": "keystone"}], "user": {"username": "admin", "roles_links": [], "id": "b9936b16c5d343588f5a19d31a55c1ea", "roles": [{"name": "_member_"}, {"name": "heat_stack_owner"}, {"name": "admin"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab", "c28444beb7e64b4ea2ea223a6efcba6a", "3ea10423929b47779f977e11015fe480"]}}}
 
REQ: curl -i 'http://openstack.danielwatrous.com:8774/v2/32c13e88d51e49179c28520f688fa74d/flavors/detail' -X GET -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: admin" -H "X-Auth-Token: {SHA1}99ff604f28f5706bfd82a00c21e099cba7fafab2"
INFO (connectionpool:258) Starting new HTTP connection (1): proxy.company.com
DEBUG (connectionpool:375) Setting read timeout to 600.0
DEBUG (connectionpool:415) "GET http://openstack.danielwatrous.com:8774/v2/32c13e88d51e49179c28520f688fa74d/flavors/detail HTTP/1.1" 200 3337
RESP: [200] CaseInsensitiveDict({'content-length': '3337', 'proxy-connection': 'Keep-Alive', 'x-compute-request-id': 'req-802ef8c9-d4a3-41e5-a93d-7ab2120089db', 'connection': 'Keep-Alive', 'date': 'Thu, 21 Aug 2014 19:09:24 GMT', 'content-type': 'application/json', 'age': '0'})
RESP BODY: {"flavors": [{"name": "m1.tiny", "links": [{"href": "http://openstack.danielwatrous.com:8774/v2/32c13e88d51e49179c28520f688fa74d/flavors/1", "rel": "self"}, {"href": "http://openstack.danielwatrous.com:8774/32c13e88d51e49179c28520f688fa74d/flavors/1", "rel": "bookmark"}], "ram": 512, "OS-FLV-DISABLED:disabled": false, "vcpus": 1, "swap": "", "os-flavor-access:is_public": true, "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 0, "disk": 1, "id": "1"}, {"name": "m1.small", "links": [{"href": "http://openstack.danielwatrous.com:8774/v2/32c13e88d51e49179c28520f688fa74d/flavors/2", "rel": "self"}, {"href": "http://openstack.danielwatrous.com:8774/32c13e88d51e49179c28520f688fa74d/flavors/2", "rel": "bookmark"}], "ram": 2048, "OS-FLV-DISABLED:disabled": false, "vcpus": 1, "swap": "", "os-flavor-access:is_public": true, "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 0, "disk": 20, "id": "2"}, {"name": "m1.medium", "links": [{"href": "http://openstack.danielwatrous.com:8774/v2/32c13e88d51e49179c28520f688fa74d/flavors/3", "rel": "self"}, {"href": "http://openstack.danielwatrous.com:8774/32c13e88d51e49179c28520f688fa74d/flavors/3", "rel": "bookmark"}], "ram": 4096, "OS-FLV-DISABLED:disabled": false, "vcpus": 2, "swap": "", "os-flavor-access:is_public": true, "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 0, "disk": 40, "id": "3"}, {"name": "m1.large", "links": [{"href": "http://openstack.danielwatrous.com:8774/v2/32c13e88d51e49179c28520f688fa74d/flavors/4", "rel": "self"}, {"href": "http://openstack.danielwatrous.com:8774/32c13e88d51e49179c28520f688fa74d/flavors/4", "rel": "bookmark"}], "ram": 8192, "OS-FLV-DISABLED:disabled": false, "vcpus": 4, "swap": "", "os-flavor-access:is_public": true, "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 0, "disk": 80, "id": "4"}, {"name": "m1.nano", "links": [{"href": "http://openstack.danielwatrous.com:8774/v2/32c13e88d51e49179c28520f688fa74d/flavors/42", "rel": "self"}, {"href": "http://openstack.danielwatrous.com:8774/32c13e88d51e49179c28520f688fa74d/flavors/42", "rel": "bookmark"}], "ram": 64, "OS-FLV-DISABLED:disabled": false, "vcpus": 1, "swap": "", "os-flavor-access:is_public": true, "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 0, "disk": 0, "id": "42"}, {"name": "m1.heat", "links": [{"href": "http://openstack.danielwatrous.com:8774/v2/32c13e88d51e49179c28520f688fa74d/flavors/451", "rel": "self"}, {"href": "http://openstack.danielwatrous.com:8774/32c13e88d51e49179c28520f688fa74d/flavors/451", "rel": "bookmark"}], "ram": 512, "OS-FLV-DISABLED:disabled": false, "vcpus": 1, "swap": "", "os-flavor-access:is_public": true, "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 0, "disk": 0, "id": "451"}, {"name": "m1.xlarge", "links": [{"href": "http://openstack.danielwatrous.com:8774/v2/32c13e88d51e49179c28520f688fa74d/flavors/5", "rel": "self"}, {"href": "http://openstack.danielwatrous.com:8774/32c13e88d51e49179c28520f688fa74d/flavors/5", "rel": "bookmark"}], "ram": 16384, "OS-FLV-DISABLED:disabled": false, "vcpus": 8, "swap": "", "os-flavor-access:is_public": true, "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 0, "disk": 160, "id": "5"}, {"name": "m1.micro", "links": [{"href": "http://openstack.danielwatrous.com:8774/v2/32c13e88d51e49179c28520f688fa74d/flavors/84", "rel": "self"}, {"href": "http://openstack.danielwatrous.com:8774/32c13e88d51e49179c28520f688fa74d/flavors/84", "rel": "bookmark"}], "ram": 128, "OS-FLV-DISABLED:disabled": false, "vcpus": 1, "swap": "", "os-flavor-access:is_public": true, "rxtx_factor": 1.0, "OS-FLV-EXT-DATA:ephemeral": 0, "disk": 0, "id": "84"}]}
 
+-----+-----------+-----------+------+-----------+---------+-------+-------------+-----------+
| ID  | Name      | Memory_MB | Disk | Ephemeral | Swap_MB | VCPUs | RXTX_Factor | Is_Public |
+-----+-----------+-----------+------+-----------+---------+-------+-------------+-----------+
| 1   | m1.tiny   | 512       | 1    | 0         |         | 1     | 1.0         | True      |
| 2   | m1.small  | 2048      | 20   | 0         |         | 1     | 1.0         | True      |
| 3   | m1.medium | 4096      | 40   | 0         |         | 2     | 1.0         | True      |
| 4   | m1.large  | 8192      | 80   | 0         |         | 4     | 1.0         | True      |
| 42  | m1.nano   | 64        | 0    | 0         |         | 1     | 1.0         | True      |
| 451 | m1.heat   | 512       | 0    | 0         |         | 1     | 1.0         | True      |
| 5   | m1.xlarge | 16384     | 160  | 0         |         | 8     | 1.0         | True      |
| 84  | m1.micro  | 128       | 0    | 0         |         | 1     | 1.0         | True      |
+-----+-----------+-----------+------+-----------+---------+-------+-------------+-----------+

The first two sections are calls the REST APIs, first for the keystone service to Authenticate and receive a token. Responses come as JSON due to the Accept header of application/json. If you look closely, you’ll see that the response actually included an access token and entry point URLs for each of the services that are integrated with keystone. These make up the ServiceCatalog and in this case there are ten.

The second section is the actual call to the nova API. In this case it returns a list of eight flavors. The final section is a tabular view of the JSON response created by the nova command line client.

cURL

If you look closely at the debug output of the examples above, you’ll see that the command line clients use cURL to make HTTP requests. We can already see what the authentication call looks like. Calls directly to cURL look similar. For example, here I call the keystone service to get a list of tenants.

$ curl -i -X GET http://openstack.danielwatrous.com:35357/v2.0/tenants -H "User-Agent: linux-command-line" -H "X-Auth-Token: TOKEN"
HTTP/1.1 200 OK
Date: Thu, 21 Aug 2014 20:05:39 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: X-Auth-Token
Content-Length: 546
Content-Type: application/json
Proxy-Connection: Keep-Alive
Connection: Keep-Alive
 
{"tenants_links": [], "tenants": [{"description": null, "enabled": true, "id": "1b7f733fa1394b9fb96838d3d7c6feea", "name": "service"}, {"description": null, "enabled": true, "id": "298cfcec9e9e49858e9b8e83d6b7d14e", "name": "demo"}, {"description": null, "enabled": true, "id": "32c13e88d51e49179c28520f688fa74d", "name": "admin"}, {"description": null, "enabled": true, "id": "8536da0aee8149d48e1fe6078dade4bf", "name": "alt_demo"}, {"description": null, "enabled": true, "id": "e66e6a80a6014dd28c7b4c1fcad19448", "name": "invisible_to_admin"}]}

REST Client

On Windows, the tool Fiddler can be used to create REST calls. When fiddler is first started, you may need to turn off capturing of traffic. You can do this from the File menu or by pressing F12. In the right side of the window, choose the composer tab. There you can provide the URL, headers and other HTTP request details. Below you can see a call to Keystone for tokens.

openstack-rest-api-fiddler-1

The response can be viewed by selecting the resulting request in the left pane and choosing the Inspectors tab in the right pane. The results can be viewed raw, as shown here.

openstack-rest-api-fiddler-2

Fiddler also provides various parsers, including JSON, to make the content easier to visualize.

openstack-rest-api-fiddler-3

Resources

The quality of the documentation available for OpenStack APIs is really amazing. Here are a couple of starting points for you.

http://developer.openstack.org/
http://docs.openstack.org/api/quick-start/content/

Software Engineering

Design principles for REST APIs

I have recently had to work with a few REST APIs that exhibited some poor design choices that I had previously assumed would be obvious. Since they may not be obvious to everyone, I wanted to highlight them.

Idempotent operations

When an operation is idempotent that means that an end state will be identical regardless of how many times the operation is executed. If the end state is dependent on the number of times an operation is executed, then it is not idempotent.

Why is this important?

REST interfaces should assume unreliable networks (which isn’t hard). A consumer of a REST API may issue the same call multiple times if a call fails, or times out or for any other reason. An idempotent operation ensures that the end result of a system is identical in cases where an operation is called multiple times.

The most common violation of this in REST APIs is a call to create a record using a POST. I do agree that in some cases it is unavoidable, but there are many instances where a PUT would be a good replacement. PUT semantics simply make the server representation of a resource match what is sent in. When a POST is unavoidable, it may be possible to add some throttle or comparison function in the API that monitors for replays and either drops them or flags them as potential duplicates.

Authentication

Authentication is a critical element of any application. Fortunately there are some well established standards which also make your application more interoperable, such as oAuth. Unfortunately, many developers tend to roll their own authentication code, which can be a resource drain and present security risks.

What is this important?

Security is a big deal and has a huge potential impact on business for better or for worse. More and more businesses are getting hacked, which comes with a huge cost to both businesses and consumers.

Accepted standards are more likely to be secure. It is also more likely that there is an existing implementation that can be dropped in to your application.

Personal Information

Personally Identifiable Information (PII), such as username, email, user ID, address, phone, etc. should NEVER be included in a URI. Not only does this muddy the concept that a URI identifies a resource (since a userid in the URI would make it appear to be many resources, one for each userid), but it also compromises personal information. Any necessary personal information should be transmitted in the HEADERS or BODY of the request, not in the URI.

What is this important?

In REST, it’s important to make sure that a given URI identifies one resource, whether that resource is identified as a single item or a collection of items. It must also be reproducible. When personal information is included in a URI, the resource correlation is lost.

Proxy and cache servers often use the URI as the cache key to facilitate lookups. Most web servers also include full URIs in log files. For these reasons, any personal information that is included in the URI may end up persistently stored on any number of systems, including the local browser.

Response Codes

HTTP response codes are standard. Unfortunately many REST APIs make poor use of the standard. Some lump nearly all response states into just one or two response codes, like 500 and 404. Others deviate from the standard and create their own response codes. I’m not going to argue that the available response codes are sufficient for every case. What I will argue is that there is more value in terms of interoperability and adoption when a REST API conforms to existing response codes.

What is this important?

REST APIs are designed to accommodate integration with other systems and applications. Developers of those systems and applications are more likely to be familiar with the standards. It may even be that they are using a library that only accommodates standard response codes (possibly omitting 418 I’m a teapot). Sticking with standard response codes makes integration more straight forward.

Software Engineering

Echo for REST client debugging

Recently I was debugging a REST client class in Java. The service I was calling simply returned that the request was incomplete. Without access to the service logs and without more meaningful feedback from the service, I decided to create an echo script that would allow me to see exactly what my REST client was sending.

For simplicity I used a LAMP installation on Ubuntu running inside virtual box on my Windows 7 development machine. I could have installed something like WAMPServer, but I prefer to leave my machine as clean as possible. Virtual Box is an ideal choice for this since I can create any type of host with any configuration and leave my own development machine unaffected.

I wrote my echo script in PHP. Here’s the code:

1
2
3
4
5
<?php
print_r($_SERVER);
$raw_post = file_get_contents("php://input");
echo $raw_post;
?>

The script is pretty simple. I just echos back to me what it sees. This includes the headers and the data contained in the BODY of the request. Note that the body will be empty for GET and DELETE (or should be anyway).

Here’s what a call to my service looks like:

As you can see, I can now see exactly what my REST client is sending to any service. This enables me to debug the client until I know that what I’m sending just right.

Note that if it becomes necessary to contact the owner of the REST service, I now have additional details to provide him and help him identify potential problems with his service. This is much better than claiming his service doesn’t work right and later finding that I actually wasn’t sending the request properly.

Resources

Java code to send a POST to a REST client: http://www.exampledepot.com/egs/java.net/post.html

Software Engineering

RESTful Java Servlet: Serializing to/from JSON with Jackson

In a previous article I demonstrated one way to create a RESTful interface using a plain Java Servlet. In this article I wanted to extend that to include JSON serialization using Jackson.

I found a very simple article showing a basic case mapping a POJO to JSON and back again. However, when I copied this straight over I got the following error:

org.codehaus.jackson.map.JsonMappingException: No serializer found for class DataClass and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationConfig.Feature.FAIL_ON_EMPTY_BEANS)

I found there were two ways to get past that error. The first was to use the Jackson annotations to define properties more directly. The other was to add getters and setters to the class. Here is my DataClass with both configurations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import java.util.ArrayList;
import java.util.List;
 
import org.codehaus.jackson.annotate.JsonAutoDetect;
import org.codehaus.jackson.annotate.JsonIgnoreProperties;
import org.codehaus.jackson.annotate.JsonProperty;
 
@JsonAutoDetect   // use this annotation if you don't have getters and setters for each JsonProperty
//@JsonIgnoreProperties(ignoreUnknown = true)    // use this if there isn't an exact correlation between JSON and class properties
public class DataClass {
 
  //@JsonProperty("theid") // use this if the property in JSON has a different identifier than your class
  @JsonProperty
  private int id = 1;
 
  @JsonProperty
  private String name = "Test Class";
 
  @JsonProperty
  private List<String> messages = new ArrayList<String>() {
    {
      add("msg 1");
      add("msg 2");
      add("msg 3");
    }
  };
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
import java.util.ArrayList;
import java.util.List;
 
public class DataClass {
 
  private int id = 1;
 
  private String name = "Test Class";
 
  private List<String> messages = new ArrayList<String>() {
    {
      add("msg 1");
      add("msg 2");
      add("msg 3");
    }
  };
 
  public String toString() {
    return "User [id=" + id + ", name=" + name + ", " + "messages=" + messages + "]";
  }
  public int getId() {
    return id;
  }
  public void setId(int id) {
    this.id = id;
  }
  public String getName() {
    return name;
  }
  public void setName(String name) {
    this.name = name;
  }
  public List<String> getMessages() {
    return messages;
  }
  public void setMessages(List<String> messages) {
    this.messages = messages;
  }
}

We are now free to modify the doGet and doPost methods in our original servlet to serialize and deserialize the DataClass to and from JSON. Here’s the code for that:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
...
  protected void doGet(HttpServletRequest request, HttpServletResponse response)
      throws ServletException, IOException {
    PrintWriter out = response.getWriter();
 
    DataClass mydata = new DataClass();
    ObjectMapper mapper = new ObjectMapper();
 
    try {
      // display to console
      out.println(mapper.writeValueAsString(mydata));
    } catch (JsonGenerationException e) {
      e.printStackTrace();
    } catch (JsonMappingException e) {
      e.printStackTrace();
    } catch (IOException e) {
      e.printStackTrace();
    }
    out.close();
  }
 
  protected void doPost(HttpServletRequest request, HttpServletResponse response)
      throws ServletException, IOException {
    PrintWriter out = response.getWriter();
 
    ObjectMapper mapper = new ObjectMapper();
 
    try {
      // read from file, convert it to user class
      DataClass user = mapper.readValue(request.getReader(), DataClass.class);
      // display to console
      out.println(user);
    } catch (JsonGenerationException e) {
      e.printStackTrace();
    } catch (JsonMappingException e) {
      e.printStackTrace();
    } catch (IOException e) {
      e.printStackTrace();
    }
    out.close();
  }
...

Now you can easily serialize data to and from JSON using Jackson and POJOs without the need for a mapping file. There are even convenient annotations available that allow you to accommodate differences between the JSON and POJO properties.

Software Engineering

RESTful Java Servlet

For a recent project I found that a RESTful interface would be appropriate. My first inclination was to use Jersey (or one of the JAX-RS implementations available). The environment where this new REST API would deploy is still using Java 1.5. This became a major roadblock when I was found that none of the JAX-RS implementations provide support for the Java 1.5 virtual machine. This is not surprising since it’s few YEARS past EOSL (end of support life) for Java 1.5, but disappointing still the same.

After spending a day or so with the available frameworks, trying to get one of them to work in Java 1.5, I grew nervous that even if I did succeed, that puts my implementation squarely in a non-support region, even on the mailing lists. So I decided to see how close I could come to a simple, reliable RESTful interface using a plain old Java servlet (POJS?).

While I’m disappointed that I’m unable to leverage an existing, mature framework, I was happy with the outcome. Here are the details

HTTP Methods built in

To begin with, the servlet specification defines methods for the HTTP actions: GET, POST, PUT, DELETE, OPTIONS and HEAD.

Each of these methods receives a HttpServletRequest and HttpServletResponse object, which makes it easy to access the payload of the request and construct a response. Manipulation of return headers, payload content, response codes (200, 404, etc.) are all available directly through the HttpServletResponse object.

The only imports required to accomplish this were available in the standard Java environment, which relieved me of the need to worry about interoperability.

1
2
3
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

URI mapping

The next challenge was how to map RESTful URIs on to the methods in my servlet. At first glance, the conventional mapping done in the web.xml file wouldn’t be sufficient. The problem was with the extra parameters that were embedded in the URI. This goes just beyond what the mapping delineates in web.xml.

What I found was that a wildcard mapping and use of HttpServletRequest.getPathInfo() gave me all the information that I needed to complete the mapping. This is what my web.xml file looks like

1
2
3
4
5
6
7
8
9
10
11
12
13
14
<?xml version="1.0" encoding="UTF-8"?>
<web-app ...>
	<display-name>testrestservlet</display-name>
	<servlet>
		<description></description>
		<display-name>TestRestServlet</display-name>
		<servlet-name>TestRestServlet</servlet-name>
		<servlet-class>com.danielwatrous.testrestservlet.TestRestServlet</servlet-class>
	</servlet>
	<servlet-mapping>
		<servlet-name>TestRestServlet</servlet-name>
		<url-pattern>/api/v1/*</url-pattern>
	</servlet-mapping>
</web-app>

The call I mentioned above, getPathInfo(), returns a string with everything that is in place of the asterisk (*) in my url-pattern above. This means that a URI of the form http://server/testrestservlet/api/vi/resource/id would return a string of “/resource/id” when getPathInfo() is called.

Avoid String Handling

I wanted to avoid messy string manipulation or analysis to parse out the details of this resource identification information. This ruled out any string splitting and checking for indexOf or startsWith.

Instead I wanted to be deliberate and reduce the possibility that a mis-mapping would slip through or that a future developer would misunderstand the exact nature of what to expect. For this reason I chose to use regular expressions. This provides a clear pattern of what my URI should look like, and it will be obvious to other developers. It also reduces the chances of a bad URI resulting in an inconsistent result.

I always use Kodos to develop regular expressions. After I got the regular expressions worked out for each class, I created an inner class inside the servlet that would help me map the URI to a specific request and give me access to the parameters embedded in the request URI. Here’s what that inner class looks like.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
  private class RestRequest {
    // Accommodate two requests, one for all resources, another for a specific resource
    private Pattern regExAllPattern = Pattern.compile("/resource");
    private Pattern regExIdPattern = Pattern.compile("/resource/([0-9]*)");
 
    private Integer id;
 
    public RestRequest(String pathInfo) throws ServletException {
      // regex parse pathInfo
      Matcher matcher;
 
      // Check for ID case first, since the All pattern would also match
      matcher = regExIdPattern.matcher(pathInfo);
      if (matcher.find()) {
        id = Integer.parseInt(matcher.group(1));
        return;
      }
 
      matcher = regExAllPattern.matcher(pathInfo);
      if (matcher.find()) return;
 
      throw new ServletException("Invalid URI");
    }
 
    public Integer getId() {
      return id;
    }
 
    public void setId(Integer id) {
      this.id = id;
    }
  }

Now I’m set to override the methods for each of my HTTP actions.

The Servlet

At this point, creating the servlet is trivial. The mapping provides all path (resource) information to the servlet and our inner class is responsible to determining whether the URI makes sense for our application and to make that information available in a sensible way for processing of the request in the action methods. Here’s what I came up with:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
package com.danielwatrous.testrestservlet;
 
import java.io.IOException;
import java.io.PrintWriter;
 
import java.util.regex.Pattern;
import java.util.regex.Matcher;
 
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
 
public class TestRestServlet extends HttpServlet {
 
  private class RestRequest {
    // Accommodate two requests, one for all resources, another for a specific resource
    private Pattern regExAllPattern = Pattern.compile("/resource");
    private Pattern regExIdPattern = Pattern.compile("/resource/([0-9]*)");
 
    private Integer id;
 
    public RestRequest(String pathInfo) throws ServletException {
      // regex parse pathInfo
      Matcher matcher;
 
      // Check for ID case first, since the All pattern would also match
      matcher = regExIdPattern.matcher(pathInfo);
      if (matcher.find()) {
        id = Integer.parseInt(matcher.group(1));
        return;
      }
 
      matcher = regExAllPattern.matcher(pathInfo);
      if (matcher.find()) return;
 
      throw new ServletException("Invalid URI");
    }
 
    public Integer getId() {
      return id;
    }
 
    public void setId(Integer id) {
      this.id = id;
    }
  }
 
  protected void doGet(HttpServletRequest request, HttpServletResponse response)
      throws ServletException, IOException {
    PrintWriter out = response.getWriter();
 
    out.println("GET request handling");
    out.println(request.getPathInfo());
    out.println(request.getParameterMap());
    try {
      RestRequest resourceValues = new RestRequest(request.getPathInfo());
      out.println(resourceValues.getId());
    } catch (ServletException e) {
      response.setStatus(400);
      response.resetBuffer();
      e.printStackTrace();
      out.println(e.toString());
    }
    out.close();
  }
 
  // implement remaining HTTP actions here
  ... 
 
}

Conclusion

This turned out to be a clear enough way to capture REST like URIs and map those onto a servlet for processing. Regular expressions provide for reliable and clear access to resource details and should be easy for future developers to quickly understand and extend.

In my next article, I’ll show you how I extend this servlet using the Jackson JSON library and a helper object to serialize and deserialize JSON payloads in RESTful requests.

Software Engineering

Software licensing: Jersey framework for REST

I did some investigation into building the RESTful API for the software licensing system using wicket. At first I was encouraged that URL management was so easy (nevermind that really old article I just linked to).

Wicket Not Ideal for RESTful APIs

In the end I decided that wicket was not a good choice for the RESTful API. The crux came in treating HTTP methods differently. In other words, for a given resource URI, I want to do something different for GET, POST, PUT and DELETE. That’s not one of wicket’s strengths, so I moved on.

The first framework I looked at had a really slick look. I know that’s not necessarily an indication of technical superiority, but the simplicity of the homepage made me believe that Restlet would be a good choice. But before making my choice, I looked for references and found this stackoverflow discussion.

Jersey Wins

I had seen Jersey before reading that, but was turned off thinking I had to dive deep into glassfish and other technologies. The stackoverflow discussion convinced me to give Jersey a try and I’m so glad that I did. In about a half an hour I had my initial API calls framed and responding to web calls in Google App Engine.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
package com.danielwatrous.softwarelicensing;
 
import javax.ws.rs.GET;
import javax.ws.rs.PUT;
import javax.ws.rs.DELETE;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
 
//The Java class will be hosted at the URI path "/sitelicenses"
@Path("/sitelicenses")
public class SiteLicenses {
 
	@GET
	@Produces("text/plain")
	public String getAllSiteLicenses() {
		// Return some cliched textual content
		return "Hello World GET";
	}
 
	@POST
	@Produces("text/plain")
	public String createNewSiteLicense() {
		return "Hello World POST";
	}
 
	// @PUT
	// @DELETE
}

I followed this extremely concise Jersey + GAE tutorial to get things going. There were more jar files required that are shown there. Otherwise I can’t believe how easy it was to get my RESTful API framed in and working.

TIP:

About a year ago I found a great tool to explore REST and test RESTful services. It’s a Chrome plugin named Simple REST Client. This makes it possible to send requests to a URI using any of the HTTP methods and see the full response, including headers. Here’s a screenshot of my first API response:

Software Engineering

Software licensing: RESTful API, resources vs. parameters

I’m starting to build out the RESTful API for the software licensing system. The first api call to implement is this

http://mysite/api/sitelicenses/?apikey=KEYVAL

That came from the initial design documents and implicitly suggests that there is a difference between the resource and parameters passed in when asking for the resource.

This distinction is important and came up today when I was thinking about pagination. As the API is defined currently, the above URI would return all site licenses associated with an apikey. The first thought that came to mind was that the number of licenses could be very large and so there needs to be some way to paginate the results.

Difference between resource and parameters

The pagination itself shouldn’t be too difficult. However, I wanted to be sure that the API calls made sense and would be intuitive. That brought me to this discussion about pagination in a REST web application. What I take away from that post is that it’s fine to mix query string parameters with RESTful identification of a web resource. That wasn’t clear to me before.

One helpful tip given in one of the comments was to look at the Twitter API. I did have a look and I do agree that it’s very clean. They have a clean and clear style of documenting the API and the structure of their RESTful calls, which includes both identification of resources and parameters to paginate, limit or otherwise clarify aspects of that resource.

Structuring my API so that it clearly identifies resources and accepts parameters in a query string in the URI is how I’ll approach development for the current software licensing plugin going forward.

Software Engineering

WordPress plugin licensing: API and class diagram

In the first article in this series, I sketched out some of the basic design ideas for the wordpress plugin licensing system.

The next step is to come up with some more concrete details about how the application will work, what information it will manage and what the interfaces will look like when accessing it. I really like using the UML as I flesh out a design. In particular I find the class diagram and activity diagram useful.

I frequently make use of stereotypes to extend the meaning of my classes. I’m not sure if this is exactly how the UML was intended to be used, but it’s really helpful. At times I also use simple tables, hand drawings, etc. if it helps capture an important point.

Here are the first design documents showing the classes I have in mind for the application and the RESTful API that will provide access to them.

Classes and API

Design choices

My initial design included normalized classes for Licensee, Product and LicenseTemplate. This would prevent duplication of licensee data across multiple licenses and would facilitate reporting by liensee and product. I also had APICall associated with AdministratorAccount since each API call could correspond to various objects in the system.

As I fleshed out the API, a few drawbacks came to mind regarding the normalization I mentioned above. First was the additional complexity in managing all of the normalized classes and associating parent and child objects. Since I am not targeting an RDBMS, the traditional SQL joins and other query approaches don’t benefit me with properly normalized design. I can make the application much more performant and simple by keeping everything in a single SiteLicenses container as shown.

Next design steps

You can see from the diagram above that I haven’t begun assigning responsibilities yet. However, it’s easy to see already where some of the key responsibilities, like isPremiumAccount(), getSiteLicenses(), revokeLicense(), getAPICallsMinutes(Integer minutesBack), etc. will end up.