Daniel Watrous on Software Engineering

A Collection of Software Problems and Solutions

Posts tagged productivity

Software Engineering

Developer Productivity and Vertical vs Horizontal Deployments

I’ve recently had many conversations related to developer productivity. In order for a developer to be productive, he must have control over enough of the application lifecycle to complete his work. When a developer gets stuck at any point in the application lifecycle, his productivity drops, which can often reduce morale too.

One question I’ve been asking is: how much of the application lifecycle needs to fall under the scope of the developer? In other words, how broad is the scope of the application lifecycle that needs to be available to a developer in order to keep him productive. Does the developer need to be able to create and configure his own server? If there is an application stack, should he also be empowered to deploy other applications and services in the stack on which his component depends? As development efforts increase, should capacity be increased to accommodate the individual development environments for each developer?

Vertical vs Horizontal deployments

As I was working through these questions with some colleagues, I began to make a distinction between a vertical and a horizontal deployment.

A vertical deployment is one that requires deploying all tiers and components in order to test any one of them. While this can create a less volatile development environment, it also increases the complexity and resource footprint required to develop an application. It also complicates integration, since any work done on other components or tiers in the stack are not available until the vertical development deployment is refreshed.

A horizontal deployment is one that focuses only on one component or tier. It is assumed that other application dependencies are provided elsewhere. This decreases development overhead and resource needs. It also speeds up integration, since changes made to other horizontal components become available more quickly. This can also increase developer productivity, since a developer is only required to understand his application, not the full stack.

In the above diagram I illustrate that many applications now have dependencies on other applications. This is especially true for microservices. However, it should not be necessary to deploy all related applications in order to develop one of them. I propose instead a horizontal deployment where all related applications are moving toward an integration deployment and that all development, QA and other validation work operate against the integration layer. For a team following the github flow, the initial branch, the pull request and finally the merge should represent stages in the horizontal progress toward production ready code. This also has the advantage of catching most integration problems in the development and QA stages, because production ready code can make it more quickly into the integration tier and is immediately available to any integrating applications.

Capacity benefits

One of the most obvious benefits to the horizontal approach is a reduced strain on compute and storage capacity. Sharing more of the vertical stack leaves available infrastructure resources free for other application teams. Naturally containers would accentuate this benefit even more.

When to go vertical

There are times when a developer will need to deploy other elements in the vertical stack. These may include database changes that would interfere with other development teams or coordinated modifications to interdependent applications. Even in these scenarios, it may be beneficial to develop against another team’s development deployment rather than their integration deployment.

Software Engineering

PHP CodeSniffer and Mess Detecter in Netbeans

Code quality and structure becomes extremely significant as the number of developers on a project increases. It can also be helpful when the code is maintained or refactored infrequently. In both cases, it can reduce the time required to get your head back in the code.

Today I found a plugin for Netbeans that uses phpMD and PHP CodeSniffer to examine the code for a project and make recommendations to improve it. The feedback is visible in the tasks panel and will take you right to the place it suggests you change.

Here’s how I installed it on Windows using WAMPServer. I first installed PEAR on WAMPServer. Then I installed a few libraries using the PEAR installer as follows:

C:\wamp2.1e\bin\php\php5.2.11>pear install PHP_CodeSniffer
Unknown remote channel: pear.phpunit.de
Did not download optional dependencies: channel://pear.phpunit.de/PHP_Timer, use --alldeps to download automatically
pear/PHP_CodeSniffer can optionally use package "channel://pear.phpunit.de/PHP_Timer"
downloading PHP_CodeSniffer-1.3.2.tgz ...
Starting to download PHP_CodeSniffer-1.3.2.tgz (328,845 bytes)
....................................................................done: 328,845 bytes
install ok: channel://pear.php.net/PHP_CodeSniffer-1.3.2
 
C:\wamp2.1e\bin\php\php5.2.11>phpcs --version
PHP_CodeSniffer version 1.3.2 (stable) by Squiz Pty Ltd. (http://www.squiz.net)
 
C:\wamp2.1e\bin\php\php5.2.11>pear channel-discover pear.pdepend.org
Adding Channel "pear.pdepend.org" succeeded
Discovery of channel "pear.pdepend.org" succeeded
 
C:\wamp2.1e\bin\php\php5.2.11>pear remote-list -c pdepend
CHANNEL PDEPEND AVAILABLE PACKAGES:
===================================
PACKAGE                            VERSION
PHP_CodeSniffer_Standards_PDepend2 1.0.0
PHP_Depend                         1.0.1
PHP_Depend_Log_Arbit               1.0.0
staticReflection                   1.0.0
 
C:\wamp2.1e\bin\php\php5.2.11>pear install pdepend/PHP_Depend
Did not download optional dependencies: pecl/imagick, use --alldeps to download automatically
pdepend/PHP_Depend can optionally use package "pecl/imagick" (version >= 2.2.0b2)
downloading PHP_Depend-1.0.1.tgz ...
Starting to download PHP_Depend-1.0.1.tgz (181,720 bytes)
......................................done: 181,720 bytes
install ok: channel://pear.pdepend.org/PHP_Depend-1.0.1
 
C:\wamp2.1e\bin\php\php5.2.11>pear channel-discover pear.phpmd.org
Adding Channel "pear.phpmd.org" succeeded
Discovery of channel "pear.phpmd.org" succeeded
 
C:\wamp2.1e\bin\php\php5.2.11>pear remote-list -c phpmd
CHANNEL PHPMD AVAILABLE PACKAGES:
=================================
PACKAGE VERSION
PHP_PMD 1.3.0
 
C:\wamp2.1e\bin\php\php5.2.11>pear install phpmd/PHP_PMD
downloading PHP_PMD-1.3.0.tgz ...
Starting to download PHP_PMD-1.3.0.tgz (45,722 bytes)
.............done: 45,722 bytes
install ok: channel://pear.phpmd.org/PHP_PMD-1.3.0
 
C:\wamp2.1e\bin\php\php5.2.11>

You could also install phpMD and PHP CodeSniffer separately if you wanted to.

I then had to tell the Netbeans plugin where the batch files were. I did that by opening the Tools -> Options window and setting the phpMD and PHP CodeSniffer tabs as shown below.

At this point all of the output in my tasks view showed errors. After restarting Netbeans it all worked like a charm. Here’s what my output looked like.

Conclusion

That little process took me slightly longer than I would like to admit. At the end of it, most of the recommendations were about comment tags, white space and underscores for variables. It turns out I don’t really care much about those items, so I didn’t spend any time going through them.

It did identify a few variable names that it thought were too long, but then I prefer to be descriptive for the same reasons I mentioned at the beginning of this post, so I ignored those suggestions too.

Perhaps I’ll find other projects where this type of automated review gives me more actionable work, but not this time.

Software Engineering

Wicket + GAE automatic reload

One disappointment of developing for Wicket and Google App Engine (GAE) is that the automatic monitoring and reloading of modified HTML files didn’t work. It had something to do with the single threaded nature of the GAE platform.

I had found a few previous efforts to make this work, but none of them worked with the current version of Wicket and GAE. I went without it for a while, but restarting the web server after every markup change finally drove me to figure it out.

Working with the project that I setup using my Wicket + GAE tutorial, I added two new files and modified the WicketApplication. Here are the details.

GaeModificationWatcher.java

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
package com.danielwatrous.softwarelicensing.web;
 
import java.util.HashSet;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.Map.Entry;
 
import org.apache.wicket.util.listener.IChangeListener;
import org.apache.wicket.util.time.Duration;
import org.apache.wicket.util.time.Time;
import org.apache.wicket.util.watch.IModifiable;
import org.apache.wicket.util.watch.IModificationWatcher;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
 
public class GaeModificationWatcher implements IModificationWatcher {
 
	private static final Logger LOG = (Logger) LoggerFactory
			.getLogger(GaeModificationWatcher.class);
 
	ConcurrentHashMap<IModifiable, Set<IChangeListener>> listenersMap = new ConcurrentHashMap<IModifiable, Set<IChangeListener>>();
	Duration pollFrequency;
	Time lastCheckTime;
	Object timeCheckLock = new Object();
 
	public boolean add(IModifiable modifiable, IChangeListener listener) {
		checkResources();
		HashSet<IChangeListener> listenerSet = new HashSet<IChangeListener>();
		Set<IChangeListener> listeners = listenersMap.putIfAbsent(modifiable,
				listenerSet);
		if (listeners != null) {
			return listeners.add(listener);
		} else
			return listenerSet.add(listener);
	}
 
	public IModifiable remove(IModifiable modifiable) {
		if (listenersMap.remove(modifiable) != null) {
			return modifiable;
		} else {
			return null;
		}
	}
 
	public void start(Duration pollFrequency) {
		LOG.debug("Starting watcher");
		synchronized (timeCheckLock) {
			lastCheckTime = Time.now();
			this.pollFrequency = pollFrequency;
		}
	}
 
	public void destroy() {
		// do nothing
	}
 
	public Set<IModifiable> getEntries() {
		return listenersMap.keySet();
	}
 
	public void checkResources() {
		Time now = Time.now();
 
		Time timeCheck;
		synchronized (timeCheckLock) {
			if (lastCheckTime == null) {
				return; // not started
			}
 
			Time nextTimeCheck = lastCheckTime.add(pollFrequency);
			if (nextTimeCheck.after(now)) {
				return; // nothing to do, not ready
			}
 
			// lets go
			timeCheck = this.lastCheckTime;
			this.lastCheckTime = now;
		}
 
		Set<Entry<IModifiable, Set<IChangeListener>>> entrySet = new HashSet<Entry<IModifiable, Set<IChangeListener>>>(
				listenersMap.entrySet());
 
		for (Entry<IModifiable, Set<IChangeListener>> entry : entrySet) {
			if (timeCheck.before(entry.getKey().lastModifiedTime())) {
				LOG.debug("Found modification, notifying listeners of change");
				for (IChangeListener listener : entry.getValue()) {
					listener.onChange();
				}
			}
		}
	}
}

GaeReloadRequestCycleListener.java

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
package com.danielwatrous.softwarelicensing.web;
 
import org.apache.wicket.Application;
import org.apache.wicket.RuntimeConfigurationType;
import org.apache.wicket.request.cycle.AbstractRequestCycleListener;
import org.apache.wicket.request.cycle.RequestCycle;
 
public class GaeReloadRequestCycleListener extends AbstractRequestCycleListener {
 
	public void onBeginRequest(RequestCycle cycle) {
		if (Application.get().getConfigurationType().equals(RuntimeConfigurationType.DEVELOPMENT)) {
			final GaeModificationWatcher resourceWatcher = (GaeModificationWatcher) Application.get()
					.getResourceSettings().getResourceWatcher(true);
			resourceWatcher.checkResources();
		}	
	}
}

WicketApplication.java

I haven’t provided all the details for this class, but this should show you how to implement it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
public class WicketApplication extends WebApplication
{    	
	...
 
	/**
	 * @see org.apache.wicket.Application#init()
	 */
	@Override
	public void init()
	{
		super.init();
 
		...
 
		// add your configuration here
		getRequestCycleListeners().add(new GaeReloadRequestCycleListener());
		IModificationWatcher watcher = new GaeModificationWatcher();
		watcher.start(Duration.ONE_SECOND);
		getResourceSettings().setResourceWatcher(watcher);
	}
}

Resources

http://agilewombat.blogspot.com/2010/01/wicket-on-google-app-engine.html
http://apache-wicket.1842946.n4.nabble.com/How-can-I-reload-HTML-in-app-engine-td3005241.html
http://apache-wicket.1842946.n4.nabble.com/Reload-html-in-Wicket-GAE-td4363236.html

Software Engineering

Software licensing: The value of good books

I have a large budget for books (but thanks to Amazon it doesn’t have to be as big as it could be). Sure it’s true that most of the information in programming books is online and available for free. There may even be substance to the argument that most books are out of date as soon as they hit the shelf because technology moves so fast. Oh well.

I get huge value from books. They save me many hours of time that I might spend scouting around for a snippet here or an explanation there. One of my favorite publishers of technology books is O’Reilly.

For this project I purchased Programming Google App Engine by O’Reilly. It’s a fantastic book so far and covers a lot of ground. Bookmarking, highlighting and so on gives me a quick path back to bits that I’ve learned.

Another book I purchased for this project and for my shelf is Thinking in Java (4th Edition) by Bruce Eckel. I previously read the free downloadable version of his 3rd edition. He provides uncommon depth in his approach, tying Java back into the other languages that inspired it. That context is extremely valuable!

If you have a hard time spending $100 or more on books for a project, just ask yourself how many hours you would have to save in order to justify the cost. At today’s contractor rates that might only be two or three hours to hit the break even point. Across the life of a project, a well written and edited book from a trusted publisher can save you many more hours than that.