Daniel Watrous on Software Engineering

A Collection of Software Problems and Solutions

Posts tagged code reuse

Software Engineering

Microservices are not as new as you think

All around me architects and business managers are beginning to mandate that internal software applications be built (and sometimes rebuilt) as microservices so that we can reuse them and compose applications more quickly. I do admit that the idea of a curated catalog of well designed microservices is attractive. Contrary to all the buzz that this is a new approach and will produce huge efficiencies, there is a lot of history that points to serious barriers to a microservice economy.

Microservices are Not New

What is being referred to as microservices today is just another name for abstraction and isolation, or code reuse. The idea is that a service should be designed to solve a single problem well in a way that is general (or abstract) enough to be broadly useful. This service is then isolated from its consumers which simplifies development and releases. Concepts around code reuse have date back at least 45 years.

The same goals of abstraction, isolation and reuse were the focus of Enterprise JavaBeans (EJB), which released its original specification in 1997. Much like microservices, EJBs were adopted by large enterprises who wanted all the benefits of code reuse and rapid application development through a type of composition. Unfortunately the complexity of the EJB APIs, and the apparent difficulty in designing easy to use interfaces reduced the value to the developer. Developers considered EJBs as adding complexity without delivering tangible benefit.

Similar efforts toward reuse have emerged as language libraries, drivers, patterns and so on. Many SaaS solutions today deliver on this same promise at a higher level making it possible to compose Enterprises from a collection of SaaS offerings.

But microservices are language independent…

Most discussion of microservices focuses on a RESTful interface (HTTP based) rather than a language specific interface (EJB). While this can make a microservice programming language agnostic, it requires an application developer to translate between the HTTP interaction with the service and their application code. Microservices must also deliver on the promise of simplifying, securing and scaling a service that is sufficiently abstract to be broadly useful, yet specific enough to solve a non-trivial problem.

Start with real problems, then abstract

It’s uncommon to see the most appropriate and useful abstraction for broadly useful services at the beginning of a project. The most useful abstractions often emerge over time as similar problems/solutions find their way into many different applications. Identifying these opportunities for reuse should start with existing solutions to the problems in your enterprise, but that alone isn’t enough. When recurring problems/solutions are identified, they still need to be useful when generalized. Excessive parameterization and configuration can limit the usefulness of the service. In other words, a microservice needs to be at least as easy to use (probably easier), and deliver at least as much value (probably more) than directly implementing the same functionality.

Some examples of microservices that are probably a bad idea:

  • spell checking
  • string formatting
  • email validation

Efficient language specific solutions already exist and have low latency for the cases listed above. They are also likely to have application specific implementations that interfere with making them generally useful. These functions are also unlikely to have associated enterprise wide standards.

Some examples that might make good microservices:

  • Fraud detection
  • Software licensing
  • Authentication and Authorization

In these cases, there are likely to be business specific rules that are common to the whole enterprise, which is benefited by centralizing a service. These types of services do not complete with well established norms or existing language specific libraries. Similar services are also offered through vendors and as SaaS, which further supports the possibility that it could be broadly useful and non-trivial to implement directly.

Start with a monolithic application

One of my favorite authors, Martin Fowler, recently published a follow up article to his previously published discussion of microservices architecture, which suggests the best way to develop microservices is to start by developing monolithic applications.

As he points out, if you already have a clear view of the boundaries for your service and are convinced that it can stand on it’s own and be broadly useful, there’s no reason not to start with microservices. What should be avoided is pre-mature optimization which drives the solution independent of the problem being solved.

Treat everything as a real product

One technique to make consistent decisions is to have a product owner who can align possible solutions with the actual product goals. Creating a microservice is a type of segmentation. The product is being split into two products and the would be product owner for the new microservice should be able to identify his consumer’s needs and where his product fits.

Follow the old rules

If I’ve convinced you that microservices are just an exercise in abstraction, isolation and code reuse, then you might find a lot of value in going back to basics for tips on how to design applications. Some of my favorite books on the subject are

http://www.amazon.com/Expert-One—One-Design-Development/dp/0764543857
http://www.amazon.com/Patterns-Enterprise-Application-Architecture-Martin/dp/0321127420

Software Engineering

Wicket + Guice including unittests

Last week I spent way too much time integrating Apache Wicket and Google Guice. Yikes! The most difficult part for me was getting the initialization to happen in the right order. A big Thank You to Dan Retzlaff on the Wicket list for helping work through these details.

The details below were applied to a Wicket quickstart project for Wicket 6.0.0.

Design Decisions

It was important to me to keep the application tier separate from web tier. I actually maintain each in a separate repository. I have several motivations for this, such as:

  • Clean separation of concerns. In other words, prevent logic from ending up in my Wicket pages
  • Independent revisions and release cycles between web tier and application tier
  • Easier to divide work between scrum teams

I include the application tier into the Wicket front end as a jar. By the way, this also makes it easy to include my application tier into a Jersey REST interface and other legacy servlets.

It was also important to maintain a mock package for fast unittests. Since the data providers are managed in the application tier, the mock package lives there, further enforcing the separation of concerns.

Implementation Approach

After some experimentation I decided to use the GuiceServlet approach. I started by adding the following maven dependencies to my pom.xml for the Wicket quickstart.

        <dependency>
            <groupId>com.google.inject</groupId>
            <artifactId>guice</artifactId>
            <version>3.0</version>
        </dependency>
        <dependency>
            <groupId>com.google.inject.extensions</groupId>
            <artifactId>guice-servlet</artifactId>
            <version>3.0</version>
            <type>jar</type>
        </dependency>
        <dependency>
            <groupId>org.apache.wicket</groupId>
            <artifactId>wicket-guice</artifactId>
            <version>6.7.0</version>
            <type>jar</type>
        </dependency>

My web.xml defines only the GuiceFilter and a listener to initialize the injector when the servlet context is created.

<?xml version="1.0" encoding="ISO-8859-1"?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
	version="2.5">
 
	<display-name>guicewickettest</display-name>
 
        <listener>
            <listener-class>com.danielwatrous.myapp.web.MyGuiceServletConfig</listener-class>
        </listener>
 
        <filter>
            <filter-name>guiceFilter</filter-name>
            <filter-class>com.google.inject.servlet.GuiceFilter</filter-class>
        </filter>
 
        <filter-mapping>
            <filter-name>guiceFilter</filter-name>
            <url-pattern>/*</url-pattern>
        </filter-mapping>
</web-app>

As you’ll see, this keeps configuration details in code, rather than XML. The web.xml remains simple. That brings us to the GuiceServletConfig, which is where we initialize the injector.

You may recall that listeners receive event notifications at well defined points in the life cycle of a web application or session. Here’s the MyGuiceServletConfig that’s referenced in web.xml:

package com.danielwatrous.myapp.web;
 
import com.google.inject.Guice;
import com.google.inject.Injector;
import com.google.inject.servlet.GuiceServletContextListener;
import com.danielwatrous.myapp.modules.MongoMyappModule;
 
public class MyGuiceServletConfig extends GuiceServletContextListener {
 
    @Override
    protected Injector getInjector() {
        return Guice.createInjector(new MyappServletModule(), new MongoMyappModule());
    }
 
}

The GuiceServletContextListener implements ServletContextListener, which ends up executing when the application context is created (or destroyed). This way we have the injector available before executing any application code.

Another thing you may notice is that I create the injector with two modules, not just one. The first module is the ServletModule, and I’ll show that to you in a minute. The next module is the application tier module that has been included as a jar in the Wicket application. The single injector available throughout the Wicket application will be able to inject servlet/Wicket related components in addition to application tier components.

Let’s have a closer look at MyappServletModule:

package com.danielwatrous.myapp.web;
 
import com.google.inject.Inject;
import com.google.inject.Provider;
import com.google.inject.Singleton;
import com.google.inject.servlet.ServletModule;
import java.util.HashMap;
import java.util.Map;
import org.apache.wicket.protocol.http.IWebApplicationFactory;
import org.apache.wicket.protocol.http.WebApplication;
import org.apache.wicket.protocol.http.WicketFilter;
 
public class MyappServletModule extends ServletModule {
    @Override
    protected void configureServlets() {
        filter("/*").through(WicketFilter.class, createWicketFilterInitParams());
        bind(WebApplication.class).to(WicketApplication.class);
        bind(WicketFilter.class).to(CustomWicketFilter.class).in(Scopes.SINGLETON);
    }
 
    @Singleton
    private static class CustomWicketFilter extends WicketFilter {
 
        @Inject
        private Provider<WebApplication> webApplicationProvider;
 
        @Override
        protected IWebApplicationFactory getApplicationFactory() {
            return new IWebApplicationFactory() {
                @Override
                public WebApplication createApplication(WicketFilter filter) {
                    return webApplicationProvider.get();
                }
 
                @Override
                public void destroy(WicketFilter filter) {
                }
            };
        }
    }
 
    private Map<String, String> createWicketFilterInitParams() {
        Map<String, String> wicketFilterParams = new HashMap<String, String>();
        wicketFilterParams.put(WicketFilter.FILTER_MAPPING_PARAM, "/*");
        wicketFilterParams.put("applicationClassName", "com.danielwatrous.myapp.web.WicketApplication");
        return wicketFilterParams;
    }
}

You may notice that I added a mechanism to provide WicketFilter with additional parameters. Next I bind my WebApplication and WicketFilter classes to specific implementations. The CustomWicketFilter overrides the typical behavior of the WicketFilter which usually takes a string reference to the WebApplication class. Instead it now uses the injected WebApplication object.

As you’ll see below, this step of injecting the desired WebApplication is critical to enabling unittests, primarily because it allows us to construct the WebApplication with an injector.

package com.danielwatrous.myapp.web;
 
import com.google.inject.Inject;
import com.google.inject.Injector;
import org.apache.wicket.guice.GuiceComponentInjector;
import org.apache.wicket.protocol.http.WebApplication;
 
public class WicketApplication extends WebApplication {    	
    private final Injector injector;
 
    @Inject
    public WicketApplication(Injector injector) {
        this.injector = injector;
    }
 
    @Override
    public Class<HomePage> getHomePage() {
        return HomePage.class;
    }
 
    @Override
    public void init() {
        super.init();
        getComponentInstantiationListeners().add(new GuiceComponentInjector(this, injector));
    }
}

At this point Wicket and Guice are successfully integrated. Let’s have a look at what needs to happen to make the unittests work.

Unittests

The only real change that’s required to make the unittests work is in the setUp function of the unittest. Since the WebApplication above was modified to receive an injector, all we need to do is create an injector and provide it at instantiation.

package com.danielwatrous.myapp;
 
import com.google.inject.Guice;
import com.google.inject.Injector;
import com.danielwatrous.myapp.modules.TestMyappModule;
import com.danielwatrous.myapp.web.HomePage;
import com.danielwatrous.myapp.web.WicketApplication;
import org.apache.wicket.util.tester.WicketTester;
import org.junit.Before;
import org.junit.Test;
 
public class TestHomePage {
 
    private WicketTester tester;
 
    @Before
    public void setUp() {
        Injector injector = Guice.createInjector(new TestMyappModule());
        tester = new WicketTester(new WicketApplication(injector));
    }
 
    @Test
    public void homepageRendersSuccessfully() {
        //start and render the test page
        tester.startPage(HomePage.class);
 
        //assert rendered page class
        tester.assertRenderedPage(HomePage.class);
    }
}

You’ll notice in this case I create an injector that doesn’t include the ServletModule and uses the TestMyappModule. Since the Wicket unittests don’t operate within a full web context I don’t need the ServletModule. Additionally my TestMayappModule makes use of my mock package. This allows the tests to run without access to any external resources. It also keeps the tests very fast!

Wicket Pages

Accessing the injector in your Wicket pages is easy. All you need to do is inject it. Here’s how that looks:

package com.danielwatrous.myapp.web;
 
import com.google.inject.Inject;
import com.google.inject.Injector;
import com.danielwatrous.myapp.domain.QuickLink;
import org.apache.wicket.markup.html.WebPage;
import org.apache.wicket.markup.html.basic.Label;
import org.apache.wicket.request.mapper.parameter.PageParameters;
 
public class HomePage extends WebPage {
    private static final long serialVersionUID = 1L;
    @Inject private Injector injector;
 
    public HomePage(final PageParameters parameters) {
	super(parameters);
 
	add(new Label("version", getApplication().getFrameworkSettings().getVersion()));
 
        // TODO Add your page's components here
        QuickLink quickLink = injector.getInstance(QuickLink.class);
	add(new Label("quickLink ", quickLink.buildQuickLink()));
    }
}

Great Combination

After working through the particulars, this implementation feels clean and flexible. Configuration is in the code and benefits from compile time type checking. Unittests are working and fast with very little modification to the traditional Wicket approach, which keeps the application testable.

Resources:

http://apache-wicket.1842946.n4.nabble.com/Wicket-Guice-unittests-td4652853.html
https://gist.github.com/3880246

Software Engineering

Refactoring to interfaces

The information below was delivered to one of my programmers as direction for how to implement a rather big change in an existing software product that I sell. I thought it was potentially useful to a broader audience, so I’m posting it here:


…The rest of this is rather complicated to explain online. I’ll do my best. I’m going to look at this in a simplistic way and let you work through the details.

First imagine that we have an Authorize.net processing class based largely on their API.

class AuthnetProcessAIMPayment {
	protected $apiKey;
	protected $transactionKey;
	protected $x_first_name;
	protected $cardNumber;
	protected $expDate;
	protected $ccv;
 
	function __construct($apiKey, $transactionKey) {
		// initialize object instance
	}
 
	// getter and setter functions
	function setX_first_name ($x_first_name) {
		$this->x_first_name = $x_first_name;
	}
	function getX_first_name () {
		return $this->x_first_name;
	}
	...
 
	function execute () {
		// process payment here
	}
}

I could use this in a script like authnet_process.php

// get $myApiKey and $myTransactionKey values
$payment = new AuthnetProcessAIMPayment($myApiKey, $myTransactionKey);
$payment->setX_first_name("Daniel");
...
$payment->execute();

Now there’s a problem with this approach when it comes to extending our software. You’re about to create an integration with payflowpro. Let’s imaging that the payflowpro class looks like this.

classPayPalPayflowProPayment {
	protected $securityToken;
	protected $paypalEmailAddress;
	protected $name_first;
	protected $cardNumber;
	protected $expDate;
	protected $ccv;
 
	function __construct($securityToken, $paypalEmailAddress) {
		// initialize object instance
	}
 
	// getter and setter functions
	function setName_first ($name_first) {
		$this->name_first = $name_first;
	}
	function getName_first () {
		return $this->name_first;
	}
	...
 
	function execute () {
		// process payment here
	}
}

Here is what you’ll be tempted to do in authnet_process.php (but it’s a mistake)

$paymentProcessor = get_option("authnet_payment_processor");
if ($paymentProcessor == "authnet") {
	// get $myApiKey and $myTransactionKey values
	$payment = new AuthnetProcessAIMPayment($myApiKey, $myTransactionKey);
	$payment->setX_first_name("Daniel");
	...
	$payment->execute();
} else if ($paymentProcessor == "paypal") {
	// get $mySecurityToken and $myPaypalEmailAddress values
	$payment = new classPayPalPayflowProPayment($mySecurityToken, $myPaypalEmailAddress);
	$payment->setName_first("Daniel");
	...
	$payment->execute();
} else {
	// error out, invalid payment processor
}

There are a few problems with this. First is that you now have several lines of code that are almost duplicate to set name, cardNumber, etc. The other is that for each new payment processor (like bluepay coming up) you now add more conditionals. Conditionals increase the chances of bugs and they make the code less clear. Another problem is that the names for common values are likely to be different between implementations. For example, authnet may call it “x_first_name” and paypal might call it “name_first”. Even though you’re providing the same value to it, each payment processor class receives it a different way. This divergence can make it difficult to identify bugs and know which token to search for when you’re making changes.

The ideal that we’re looking for in this case is a solution that will allow authnet_process.php to remain the same regardless of how many payment processors we implement. In answer to one of your questions, we also want our settings interface to be as insulated as possible. In other words, we don’t want to have a different settings mechanism for each payment processor. In even other words, we want each payment processor to accommodate as many of the settings that we’ve already implemented, so that the user has the most seamless experience possible.

You can think of this as being similar to your autoresponder integration in subscription mate. That was done very well. and it didn’t bleed into other settings. You could change the autoresponder provider without requiring changes anywhere else.

So, how do we do this?

Wouldn’t it be great if we could just give a different processing object to authnet_process.php and it knew what to do with it? That’s exactly what we can do with interfaces. Dependency Injection (DI) can be a huge benefit too, but PHP isn’t wired for DI like other languaes, so let’s start with the interface.

An interface cannot be instantiated. Instead, all it does is say that anyone implementing the interface must provide certain functions. Mind you it doesn’t dictate how those functions should be implemented, but you must provide an implementation, even if it does nothing (e.g. a function that just adds a log entry or simply returns an empty value).

The mapping that you were tempted to do in the conditionals above, meaning mapping the clients first name on to one member function for authnet and another for payflowpro, should instead happen one time in the interface implementation.

Here’s what an interface for the above case might look like.

interface OneTimePayment {
	public function setName($name);
	public function getName();
	...
	public function execute();
}

The payment processing classes above would now look like this (I’m showing only the lines that would change).

class AuthnetProcessAIMPayment implements OneTimePayment {
	...
	// getter and setter functions
	function setName ($x_first_name) {
		$this->x_first_name = $x_first_name;
	}
	function getName () {
		return $this->x_first_name;
	}
	...
}

and for payflow pro

class PayPalPayflowProPayment implements OneTimePayment {
	...
	// getter and setter functions
	function setName ($name_first) {
		$this->name_first = $name_first;
	}
	function getName () {
		return $this->name_first;
	}
	...
}

Now that the method names to set and get a name are identical, the code within the conditionals really is duplicate. authnet_process.php can now be simplified to this:

$paymentProcessor = get_option("authnet_payment_processor");
if ($paymentProcessor == "authnet") {
	// get $myApiKey and $myTransactionKey values
	$payment = new AuthnetProcessAIMPayment($myApiKey, $myTransactionKey);
} else if ($paymentProcessor == "paypal") {
	// get $mySecurityToken and $myPaypalEmailAddress values
	$payment = new classPayPalPayflowProPayment($mySecurityToken, $myPaypalEmailAddress);
} else {
	// error out, invalid payment processor
}
// once you create the object, everything after this 
// point can assume it's dealing with a OneTimePayment
// insead of a specific payment processor
$payment->setName("Daniel");
...
$payment->execute();

At this point, even if that conditional grows to accommodate additional payment processors, each one adds only a handful of lines of code to get authentication values and create the object. None of the work to set values and execute transactions will ever need to change, since it treats everything like a OneTimePayment.

How could DI make this even better?

I haven’t seen any good DI frameworks for PHP. In some ways it goes a bit against the grain for PHP development. If there were, this might be how authnet_process.php would look.

$payment = diFramework.get(OneTimePayment);
$payment->setName("Daniel");
...
$payment->execute();

Elsewhere in the DI framework you would define what a concrete instance of OneTimePayment should be. In other words, something like this might be in an XML file (Spring like) or a module (Guice like).

bind(OneTimePayment).to(AuthnetProcessAIMPayment);

Now, anytime you ask for a OneTimePayment, you’ll get an AuthnetProcessAIMPayment object. Similar injection can be used to provide the authentication values, so that in authnet_process.php you really only ask for a OneTimePayment and you get back a functional, ready to use object.

Refactoring

Refactoring is a key aspect of developing software. In this case, where we started with code that creates instances of Authnet classes and uses them directly, changes to authnet_process.php will be necessary. Changes will also be necessary to the Authnet classes to conform with the new interface that we come up with. That may sound like a lot of work.

It is a lot of work, and may actually be more work than just creating a payflowpro class and adding the conditional that I showed at the top. However, there are some concrete benefits and gains we get from doing this additional work.

First is that once the work is done and authnet_process.php uses the interface based calls, all future changes for new payment processors will be very small and won’t functionally endanger any working code for other payment processors.

Second is that there’s a clear scaffolding for adding new payment processors. You can still add as many private internal functions as are necessary and desired to create an “execute” function, but you know that you need one and when it’s done it will work anywhere in your app that you call execute.

Third unittests can more easily test new payment processors without duplicating a lot of code. You simply provide a different OneTimePayment implementation and run the same test. Test coverage stays high and maintenance remains low.

Where do you start?

The best place to start is by coming up with the interface. You want to look at all the values that the current implementation needs and ask if it can be generalized. For example, “x_first_name” might prompt you to design for “first_name” in your interface. A quick check with the documentation for the other payment processor(s) will help you arrive at a comprehensive interface.

Next, make the changes to the Authnet classes so that they implement the new interface(s). Finally work with your code until the unittests pass.

Now you’re ready to implement the interface for payflowpro. As soon as you have done that, you run your unittests and provide an instance of your payflowpro OneTimePayment object.

Once you’re done there and have passing unittests, move on to authnet_process.php.

At some point you’ll need to modify the settings page so that you can provide payflowpro credentials rather than authnet. I really like how you did multiple autoresponder vendors in subscription mate, so I would suggest that as a possible approach for this.

Feel free to create a branch and be daring. You have the security of unittests and the isolation of a branch in svn. If you end up with a mess the first time through, you can start over. I personally think you’ll do great.

Software Engineering

Software licensing: Wicket pages and panels

My initial perception of wicket panels was that they were like includes. This naturally lends itself to header, footer and style content that will be the same throughout a web application and avoids duplicating code.

It seems wicket panels are more widely used than I first expected. For example, I keep seeing examples of a very lightweight page where panels represent the majority of the content too. This is a more segmented approach than I have seen in the past, and I can see some benefits and drawbacks to it.

Based on other frameworks I would typically structure each page with the following files:

  • header [common]
  • footer [common]
  • page

This would be repeated for additional pages with the header and footer representing the only elements common among all pages. In wicket the design changes by adding a new file

  • header [common]
  • footer [common]
  • page [common]
  • panel

By segmenting the content of the page out into panels, the page frame itself becomes reusable. It also makes it possible to use some clever wicket extensions, like breadcrumbs and tabbed ajax panels. These can save a lot of time in development. Another benefit is that it keeps you in the habit of creating panels for page sub-components, which can increase code reuse and maintainability.

The biggest pitfall I have found so far is that panels rely on state and the URLs are not bookmarkable. I know that’s what wicket is all about, but in terms of search engines and bookmarking pages, this can be a big drawback.

For example, when implementing the documentation pages for the API of the software licensing appication, I made use of the breadcrumb extension and put all the content in panels. I found afterward that based on the way it is implemented, it’s not possible to bookmark a link to a specific API document. The same lack of bookmarkable links exists in the tabbedpanel that I planned to use. Unfortunately this won’t do.

While I may leave the content as a panel, I now see that my implementation needs to be pages so that I can map them to specific URLs.

The alternative would be to make use of the extensions and give up the bookmarkable URLs. However, when I’m working with an API, emailing a support or user list or creating tutorials I frequently link to specific API documentation and I rely on Google to know how to find that documentation when I search for it.

I have sent an email to the wicket user list about the whether the breadcrumb extension can be made to work with bookmarkable URLs, but I didn’t get a response. If anyone responds then I’ll update this post with the solution.