Selenium, HtmlUnit, JavaScript and JQuery - Writing Performant Tests

I'm working on a project that uses a certain amount of JavaScript to build a dynamic UI, the project team were of the opinion that we used barely any JavaScript and were flummoxed as to why the integration tests using Selenium with the HtmlUnit Driver were so slow.

I finally got time a couple of weeks ago to investigate the problem and because I tend to avoid JavaScript like the plague, it took me a little while to garner the necessary skills and utilities to diagnose the problems.

I thought I would share the lessons I learnt in case anyone else comes across similar problems.

The first thing I did was to upgrade all the libraries to their latest versions this often fixes performance bugs and can highlight problems with the code base. In my case it only garnered a minor speed up but highlighted the fact that the old version of Selenium we were using allowed interaction with hidden and disabled HTML elements and a number of our tests were badly written.

The second thing I discovered is that the Selenium / HtmlUnit event model means that SELECT element change events were only fired when focus changed to another element in the page.

Seeing no major speed increase I instead reverted to profiling the Java processes. We don't have access to a commercial profiler but a remarkable amount can be achieved using VisualVM which is included as part of the Sun JDK since about version 6.012.

The first thing you should always do when installing this utility is to increase the memory available using the VisualVM options file in the JDK configuration folders.

I was very careful to limit what I profiled to only a certain subset of the classes involved as even with the tweaked memory settings VisualVM is a little flaky when profiling large, long-running processes.

I used the CPU profiling capability and focused in on the projects test classes, the Selenium classes and the HtmlUnit classes.

After a certain amount of faffing around I discovered that a large amount of time was spent in the JavaScript and particularly around event bubbling and DOM change listening.

Unfortunately there is a certain level of indirection involved in the JavaScript implementation and could not discover precisely which JavaScript elements were involved even when using Java debugging.

I instead reverted to using FireBug and FireQuery to profile and navigate through the JavaScript within Firefox. I quickly discovered some obvious JavaScript performance problems and fixed them. Equally rapidly I discovered that the performance characteristics of the Firefox JavaScript engine is very different to those of the HtmlUnit JavaScript engine - Mozilla Rhino.

FireBug and FireQuery were excellent for helping me to understand what was executing and would be perfect for profiling browser performance issues, I would have to look elsewhere for the root cause of the HtmlUnit performance problems.

I resorted to old-school - comment stuff out until it broke or performed faster.

The problem was rapidly resolved to being the use of JQuery live() methods. They are designed to handle events highly dynamic DOMs by listening to events that have bubbled all the way up to the root and then matching the source of the event to a selector before calling the method intended to handle the event.

I replaced them with JQuery bind() and delegate() methods and saw a massive performance increase: A UI test suite that took 26 minutes to run completed in 10.

I learnt that highly dynamic JQuery based scripting could cause severe test performance problems and learnt a lot about how it worked. Longer term I'm going to move to statically bound JavaScript as much as possible particularly when I need to be able to test it.

A framework like JQuery is a little like an iceberg, it may look like you're barely using any JavaScript but under the surface...

- Posted using BlogPress from my iPad

On the Insanity of Expecting Nothing to Happen...

On my current project we're building an asynchronous message based system with various outputs, in particular a file for consumption by a mainframe.

On a number of occasions tests have been written that assert that something has not happened.

When you're testing synchronous processes this is a perfectly reasonable assertion to make, when the process returns all the work will be done.

For asynchronous processes?When the process returns then you have no direct way of knowing if it has completed.

For many people the most obvious way of checking that something has not happened in an asynchronous process is the counterpart for checking that something has happened asynchronously. You put in a timeout.

The problem with this is the likelihood of a given outcome. When expecting something to happen asynchronously you only have to wait until it has happened which means that you only ever wait until the timeout when something has gone wrong. When expecting something not to happen, you end up waiting for the time out most of the time. This means that for every assertion that nothing will happen you end up adding a fixed timeout that causes your test phase to increase in duration. 

When you use a time out to wait for something not to happen, you risk setting the timeout too short and the test becoming invalid because the thing you asserted wouldn't happen occurs after the test completes. This can lead to further increases in the timeouts to ensure that the tests remain valid.

There is a related problem with testing asynchronous processes where the test only checks an intervening step in the process before moving on the next test. This leaves you open to side effects from the processes started by the preceding tests causing hard to diagnose problems with your later tests. This is particularly likely with tests that assert that something doesn't happen as they have a greater chance of leaving the test before the process under test has completed. 

How do we avoid these problems and still be able to make assertions that something hasn't happened?

The 'Sentinel' pattern is a good place to start. 

Send a second request after the first that will produce a result after the first has completed, in this way you can assert that if you only see the result of the second request then you can state that the first request hasn't produced anything. Maybe you can ask the asynchronous process to do two things in one request and can then assert that only the effects of the additional work are visible. 

Alternatively you can look at other ways of knowing that the process under test has completed. Check the logging for a statement that is only produced when the process has completed or maybe introduce an event or message that is produced when the process completes regardless of whether the other output is produced. 

It is always a bad idea to expect 'nothing' to happen asynchronously, always expect something. 

Debugging Neo4J Spatial - Update 3 on Neo4J Spatial and British Isles OSM Data import.

I have been having a long discussion with Craig Taverner over an issue that I think that I have identified in the Neo4J spatial codebase with particular reference to the OSM code. Unfortunately I have completely failed to communicate the issue that I have perceived. Craig has variously believed that the issue is in my code or my misperception of the Neo4J spatial unit test code.

The actual issue is buried deep in private and protected methods within the Neo4J Spatial codebase in particular in the interaction between the spatial search classes and the OSM layer and index. I have yet think of a way to write a simple and direct unit test that will illustrate the problem so instead I am going to write this blog entry to explain what I found and how I found it.

I actually found the issue when I was attempting to debug why some of my unit test code was failing to return the results that I expected. It turned out that my problem was because I had swapped latitude and longitude on the point that I passed in. While debugging I spotted another issue in the way that the spatial search classes (in particular the org.neo4j.gis.spatial.AbstractSearch and org.neo4j.gis.spatial.query.AbstractSearchIntersection classes) interact with the OSM Layer.

The actual issue is that the AbstractSearch class implements its own getEnvelope(Node geomNode) method that takes a node as a parameter and uses the related layer's GeometryEncoder to parse the node and create the envelope. This method is used in AbstractSearchIntersection's needsToVisit(Node geomNode) method to determine whether to visit the node or any of its children.

When I execute the following code which I believe is correct:

SearchContain searchContain = new SearchContain(point);

osmLayer.getIndex().executeSearch(searchContain);

The OSMLayer's GeometryEncoder is used by AbstractSearch to decode the node's envelope and this produces an envelope with the bounding attributes mixed up.

For many searches this is not a huge issue and the search will appear to work. In the case of the Buckinghamshire (Bucks) search the Bucks envelope no longer subtended fractions of a degree but instead subtended 50 odd degrees of latitude and longitude which included Bucks' true envelope.

As far as I can make out the correct results are returned because within the OSM layer the nodes coordinates and bounding boxes are handled differently so even if the spatial search classes visit unnecessary nodes, the correct answers are returned. This means however that there may be performance issues due to unnecessary visits. Also I think that there may be situations where index nodes that should be visited aren't.

At a fundamental level I think that there is some pretty bad schizophrenia between the way that OSMGeometryEncoder handles envelopes in the Envelope decodeEnvelope(PropertyContainer container) method and the way that the RTreeIndex handles envelopes in the private Envelope bboxToEnvelope(double[] bbox) method.

My gut feeling is that all envelopes should be handled by the relevant GeometryEncoder rather than by a private method on an index, However the OSMGeometryEncoder just plain seems to get it wrong.

Update 2 on Neo4J Spatial and British Isles OSM Data import.

Well even 6GB of heap space isn't enough to fully import and re-index the British Isles Open Street Map Data set. I'm thinking that there will have to be some work on memory management somewhere. First thing I'm going to need to do is profile the import and work out where the memory is being used (I'm assuming it isn't a memory leak per se). Basically I want to determine whether it is because of a misconfiguration by me or there is a need to look at the memory usage in the OSMImporter class.

I'm also thinking that for reverse geo and local searches I really don't need the full OSM data set so a customised version of the OSM importer would be a good idea. I really don't need ways or the node data associated with them. I suspect a multi-pass approach would be a good idea. The first pass through the OSM data set determines which nodes are needed for the features that I want to import, the second pass imports the required nodes and their associated features, followed by a final re-indexing.

I've been in discussions with one of the people behind Neo4J Spatial, Craig Taverner, on the Neo4J User List. I found an issue with the bounding box used in the spatial search index visitor pattern. It's been quite illuminating to dig into this. I'd been thinking for a while that a number of the techniques that I encountered working with 3D graphics and scene graphs would be relevant and it definitely seems to be the case. I'm beginning to think that OpenGL or OpenCL backed indices could prove very useful in high throughput micro-batched situations where the bus latency can be hidden.

Update 1 on Neo4J Spatial and British Isles OSM Data import.

Successes and failures...

I managed to shepherd the import all the way through to import completion with 4096 MB of memory but unfortunately fell short when it came to the re-indexing. As I commented on my post I upped the memory to 6144 MB and restarted and it has again successfully imported and has been re-indexing overnight without falling over. I'm waiting to see how this goes.

During the past import I did a little more characterisation of the system utilisation and came up with the following: As reported before the node import process is single threaded and does not saturate my disk IO, what is more interesting is the way import which is heavily reliant on the already imported nodes. The way import saturated neither IO nor a single core on my CPU (again it appeared largely single threaded), I'm not sure what the problem is, it could be that the limit is the disk seek time / latency as the ways are composed of nodes already imported. I'm guessing that there would be a need to look at how the nodes are stored and whether any useful caching can be done to accelerate the way import.

The 'relation' import phase was too short for me to characterise.

The indexing phase does appear to be multi-threaded during a brief glance this morning before I left. using 2 cores quite consistently (4 cores are available).

Aside from the importing I've started playing with the API, using the Buckinhamshire data set that I imported on my laptop. So far not too impressed by performance, I used the code from some of the Neo4J spatial unit tests to do a way (highway) search from a given point. I chose coordinate in the middle of a lake (51.808721,-0.689735) and gradually expanded the search from 10m, 100m, 1000m, 10000m. quite slow but at least not an exponential increase in time - 14.4s, 85.8s, 88.7s, 86.1s.

Interesting to note that the OSM parts of the Neo4J Spatial codebase so far does not seem to have any convenience methods for area and boundary searches, only for way searches.

Here is my current (very rough) code below:

package com.presynt.neo4j;

import static org.junit.Assert.*;

import com.vividsolutions.jts.geom.Coordinate;
import com.vividsolutions.jts.geom.GeometryFactory;
import com.vividsolutions.jts.geom.Point;
import org.geotools.referencing.CRS;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Ignore;
import org.junit.Test;
import org.neo4j.gis.spatial.Layer;
import org.neo4j.gis.spatial.SpatialDatabaseService;
import org.neo4j.gis.spatial.SpatialTopologyUtils;
import org.neo4j.gis.spatial.osm.OSMImporter;
import org.neo4j.gis.spatial.osm.OSMLayer;
import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.kernel.EmbeddedGraphDatabase;
import org.neo4j.kernel.impl.batchinsert.BatchInserterImpl;

import javax.xml.stream.XMLStreamException;
import java.io.IOException;


public class TestNeo4JSpatial {

   private static GraphDatabaseService graphDB;
   private static SpatialDatabaseService spatialDB;
  
   @Test
   @Ignore
   public void createSpatialDB() throws XMLStreamException, IOException {
       OSMImporter importer = new OSMImporter("OSM-BUCKS");
       BatchInserterImpl inserter = new BatchInserterImpl("/testDB");
       importer.importFile(inserter ,"/OSMData/buckinghamshire.osm");//"british_isles.osm"); // "C:\\british_isles.osm");
       inserter.getGraphDbService().shutdown();
       graphDB = new EmbeddedGraphDatabase("/testDB");
       importer.reIndex(graphDB, 100000);
       graphDB.shutdown();
   }

   @Test
   public void retrieveLayer(){
       final Layer layer =
               spatialDB.getLayer("OSM-BUCKS");
       assertNotNull(layer);
       assertEquals(OSMLayer.class, layer.getClass());
   }

   @Test
   public void useLayer() {
       final OSMLayer osmLayer = (OSMLayer)spatialDB.getLayer("OSM-BUCKS");
       final GeometryFactory factory = osmLayer.getGeometryFactory();
       System.out.println("Unit of measure: " + CRS.getEllipsoid(osmLayer.getCoordinateReferenceSystem()).getAxisUnit().toString());

       final Point point = factory.createPoint(new Coordinate(51.808721,-0.689735));

//        final Layer boundaryLayer = osmLayer.addSimpleDynamicLayer("boundary", "administrative");
//        SearchContain searchContain = new SearchContain(point);
//        boundaryLayer.getIndex().executeSearch(searchContain);
//        for(SpatialDatabaseRecord record: searchContain.getResults()){
//            System.out.println("Container:" + record);
//        }

       final Layer highwayLayer = osmLayer.addSimpleDynamicLayer("highway", null);
       long startTime = System.currentTimeMillis();
       SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 10.0);
       System.out.println("10m took:" + ((System.currentTimeMillis() - startTime) / 1000d) +"s");
       startTime = System.currentTimeMillis();
       SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 10.0);
       System.out.println("10m took:" + ((System.currentTimeMillis() - startTime) / 1000d) +"s");

       startTime = System.currentTimeMillis();
       SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 100.0);
       System.out.println("100m took:" + ((System.currentTimeMillis() - startTime) / 1000d) +"s");
       startTime = System.currentTimeMillis();
       SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 100.0);
       System.out.println("100m took:" + ((System.currentTimeMillis() - startTime) / 1000d) +"s");

       startTime = System.currentTimeMillis();
       SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 1000.0);
       System.out.println("1000m took:" + ((System.currentTimeMillis() - startTime) / 1000d) +"s");
       startTime = System.currentTimeMillis();
       startTime = System.currentTimeMillis();
       SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 1000.0);
       System.out.println("1000m took:" + ((System.currentTimeMillis() - startTime) / 1000d) +"s");

       SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 10000.0);
       System.out.println("10000m took:" + ((System.currentTimeMillis() - startTime) / 1000d) +"s");
       startTime = System.currentTimeMillis();
       SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 10000.0);
       System.out.println("10000m took:" + ((System.currentTimeMillis() - startTime) / 1000d) +"s");
       startTime = System.currentTimeMillis();
//        for(SpatialTopologyUtils.PointResult result : SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 10000.0)){
//            System.out.println(result);
//        }
   }

   @BeforeClass
   public static void initialiseDatabase() {
       graphDB = new EmbeddedGraphDatabase("target/test-classes/spatialTestDB");
       spatialDB = new SpatialDatabaseService(graphDB);
   }

   @AfterClass
   public static void shutdownDatabase() {
       graphDB.shutdown();
       spatialDB = null;
       graphDB = null;
   }
}

Neo4J Spatial and Importing the British Isles Open Street Map Data

I know, I know. Overly ambitious and all that.

While we've been working on Presynt we've been having fun and games with Geo Data; both for Local Search and Reverse Geo lookups. We've also got a number of ambitious new Geo features planned for future versions that needs needs some form of Geo store.

So with that in mind I started playing with Neo4J Spatial and the Open Street Map data set.

It's been an interesting experience with some gotchas as Neo4J Spatial is still not fully ready for prime time, so I thought I would share my experience with those who are interested.

First off I thought I would try to import the full global data set from Open Street Map. Not necessarily a mistake, but a huge undertaking as we are talking about over 200GB of map data in raw OSM format, billions of data points and relationships.

The first gotcha is that if you wish to import the full OpenStreetMap data set I'd recommend starting with at least Neo4j version 1.3 Milestone 4 as that version was the first one to hugely expand the number of nodes supported from 4 billion to 128 billion.

Of course attempting this type of import I also started running into memory problems. I ended up running with -Xmx4096m and -XX:+UseConcMarkSweepGC. The concurrent mark and sweep garbage collector proved most resilient to the loads placed on it. The default parallel GC would fall over and cry quite regularly due to an issue with consuming too much CPU to too little effect.

In the end I discovered that the full global import would take too long to run and set my sights a little lower on just the British Isles data set which is all that Presynt needs right now.

In order to test my set up I decided to use the Buckinghamshire data set which is the smallest county data set in the UK and this highlighted another gotcha: Not all the dependencies required are actually included in the Neo4J POMs.

In the end I used the following dependencies:

• org.neo4j:neo4j:1.3.M04
• org.neo4j:neo4j-spatial:0.5-SNAPSHOT
• org.geotools:gt-referencing:2.6.5
• org.geotools:gt-main:2.6.5
• org.geotools:gt-cql:2.6.5
• org.geotools:gt-epsg-hsql:2.6.5
• com.vividsolutions:jts:1.11

They may not all be necessary but they certainly work...

Once I had resolved these issues then I was able to start the full British Isles import. I was running it on a quad-core machine and found that the OSMImporter/BatchInserter combination (all default configuration mind) was using just one CPU. Looking at the OSMImporter codebase it became obvious that it was written to stream read the OSM xml data and synchronously write it out to Neo4J. I think that there may be some room to parallelise it as large amounts of the GPS node data is utterly independent and the XML data set is structured in such a way that it could be broken down into independent units of work. The only points where serialised behaviour are required are during the parsing of the XML, the construction of the Way and Relation data and possibly the writing to Neo4J.

The process seems to be largely CPU bound as I ran it on SATA II RAID 0 SSDs and their read and write capabilities were barely touched while the single CPU in use was pegged at 100% almost continuously.

The import of the British Isles dataset seems to take about 48 hours.

I used the vanilla OSMImporter and BatchInserter configurations as I am still a Neo4J Neophyte (bad pun intended ;-)) I will be reading up on alternative configurations to see if I can improve the performance. I will also look at the import of changesets so that I shouldn't need to do a full inport again.

My Web Comics...

I've been lifelong comic reader, starting with Peanuts, Giles and Thelwell but then graduating to the delights of The Far Side, Calvin and Hobbes and Bloom County. In many ways they were as influential to me as all the conventional books I read as among their humour was much wisdom.

With such strong early influences it's little wonder that I've kept on reading cartoons though I've transferred my allegiance from the funny pages to the web comic. I think that this is because syndicated cartoons have lost their way - both as a result of lack of courage and the pressures of the modern newspaper business.

I now regularly read a number of web comics and I would recommend these whole heartedly to anyone:

• Butternutquash - Telling the tales of a group of friends that hang around a comic strip. It is far more cynical than you would expect but always funny. I love the Narcoleptic Dog.
• Ctrl+Alt+Del - A delightfully odd ball comic strip following the personal and imaginary life of a dedicated gamer.
• Diesel Sweeties - The life and loves of an assorted group of people, including a fading porn actress her sometime boyfriend - a robot, a chronic hipster, a Canadian, a very disturbing furry and the list goes on. This comic strip has resulted in some of my favourite T-Shirts.
• Girl Genius - A gloriously drawn story of an orphaned. young woman who discovers that she has inherited the 'Spark' from her parents; the ability to build amazing steam-punk devices. This is a complex and involving tale set in a richly realised world.
• Gunnerkrigg Court - I only discovered this recently - A comic that becomes more beautiful as the artist develops his skill. telling the story of a conflict between science and magic through the eyes of a young girl thrown into the middle.
• Kukuburi - By the artist of Butternutsquash. Again a fabulously beautiful comic strip set in a seemingly dream world. Now sadly on hiatus but well worth reading.
• Orneryboy - A richly humorous gothic story about a couple who live with Brian the Zombie and three cats, one of whom is a ghost.
• Penny Arcade - A pure geek gamers comic strip but highly entertaining if you get the jokes. I'm a particular fan of Catsby and Twisp who crop up from time to time.
• PvP - The Godfather of web comics. Many web cartoonists cite Scott Kurtz as their influence both artistically and professionally. He is credited with pioneering making web cartooning a viable career. The comic itself follows the adventures of a bunch of journalists working at a games magazine.
• Scary-Go-Round - A very British comic strip and all the better for it. Over the years it has mutated but always centres around ordinary people living very un-ordinary lives.
• Sluggy Freelance - For a comic strip that started as a little story about a group of house mates this has developed into one of the richest and most perpetually rewarding comic strips that I read. Dimension hopping, big guns and humour make this a firm favourite. One character in particular should always be mentioned: Bun-Bun a psychotic dwarf lop rabbit with a flick-knife.
• The Abominable Charles Christopher - Tough to describe as it follows the perambulations of a perpetually innocent hairy beast that is largely man-shaped. He says nothing but still converses freely with the animals of the forest. Just before the summer break he has found himself face to face with a human town. I love it for the art and the crazy animal characters.
• User Friendly - One of the oldest web comics. I was introduced to this many years ago by my friend Neil. Gloriously anarchic tales told of an Internet Service Provider's support team with occasional visits by characters of the Cthulhu pantheon.

I have read many more over the years but these are particular favourites. What web comics do you read and if not why not?

Further Forays Into the HTML5 Stack - Animation

Short update this one.

I've been doing some simple animations replacing one image with another every 5 seconds using the SVG SMIL animation framework. I'm using the 'defs' element to define a number of image elements that will not be immediately rendered. I then use an 'use' element to refer to the first element in the sequence before using the 'animate' element to alter the 'xlink:href' attribute of the 'use' element every 5 seconds from a set of values in the 'animate' element.

Unfortunately this approach seems to be very CPU intensive even between changes to the 'use' element's 'xlink:href' attribute.

I'm wondering whether the animation timer does not take into account the down time between the changes and just spins its wheels, wasting CPU, all the time.

I think that I need to do some rough benchmarking to work out whether to use CSS3 or SVG SMIL animation and to work out which techniques are more CPU friendly.

Forays into the HTML5 stack.

While working on Cazcade with Neil, Dan C., Jon et al I'm also working on my own little project - DiarWise. It's still pretty stealth so I'm not going to discuss it in detail on an open forum but I think that now is a good time to discuss some of the technology that I've been using.

For DiarWise I've been looking at a much more rich UI experience highly interactive and completely resolution independent. I wanted to be able to create a UI that would be equally renderable on a smartphone, a tablet and a PC without extensive customisation for each. I also wanted a UI that would would be able to interact well with the browser event models for mouse and multi-touch.

I've been focussing on SVG embedded in HTML over the HTML 5 Canvas element for a number of reasons:

• It is significantly more mature as a specification than the HTML 5 Canvas element.
• The HTML 5 canvas element is procedural in its rendering so with a command it renders pixels directly to screen and so does not have a screen graph; the SVG xml elements are the scene graph and are embedded directly in the DOM and so are manipulable using all the tricks that we are familiar with. This also means that a lot of the heavy lifting for re-drawing is done for us.
• Text is more of a first class citizen in SVG and so is easier to make available to Accessible browsers.
• SVG graphics elements are style-able using CSS.
• HTML can also be embedded back into SVG (and manipulated via the DOM) using the foreignObject element

There are a few weaknesses as I've discovered as I've worked with it.

• The default SVG animation framework is built around the SMIL standard which is pretty low level and is not always the easiest to work with though it is very powerful. I'm intending to see how well SVG interacts with the CSS3 animation effects which are much nicer to work with.
• SVG support is present in the latest iteration of all the browsers but is patchy to say the least. The best of the moment is the WebKit family, Firefox is the next best though it has big issues with the text rendering. IE9 claims to be a huge step forwards over IE8 but I have yet to try out the beta and there are question marks over how well it will support SVG animation.
• While its text support is good there is one glaring omission - word wrap. The neatest work-around is to embed an appropriate HTML element in the SVG using foreignObject.

The best trick I've found so far with SVG is the simplicity of decoupling the screen resolution from the render resolution and this cascades down to any embedded HTML elements. The 'svg' element defines a local coordinate space for all its contents and you can override the coordinate system using the viewBox attribute. This means that all embedded elements can be positioned and sized relative to each other in terms of local 'pixels' and the SVG renderer will scale according to the screen resolution. The handles aspect ratios elegantly using the preserveAspectRatio attribute. This gives you the ability to present a consistent interface in a wide range of resolutions and can handle pan and zoom exceedingly smoothly for people with limited eyesight.

So far my grand experiment with SVG is going well and it seems to be well worth using as one of the technologies that make up the HTML 5 stack.

My life with BDD...

This is a blog post that's been very long overdue, but I finally acknowledged that I really need to capture my experiences after a chat with an old friend at Twitter (shameless name drop...).

I first came across Behaviour Driven Development thanks to Mauro Talevi, my first hire on the team that I set up at HSBC back in 2008. He's a major contributor on the JBehave project that is one of the major BDD frameworks. I hope that I'm not going to embarrass him too much by saying that he was an invaluable influence on the project.

I'm not going to talk about the technical implementation as that is well covered elsewhere but instead in this blog I will talk about the general principles and practical experience of working with BDDs.

The basic idea is simple. Any story or feature being implemented in a project should have automated integration tests that proves that the work has been completed successfully. It is important the the tests be comprehensible to the stakeholders requesting the story / feature so that they can confirm that it is complete and done. This provides a very clear goal for the developers and works to build up a suite of regression tests that give a healthy confidence in the application. The tests are defined in terms of business scenarios that represent various paths through the business process.

BDD is complementary to Unit Testing and is in no way a replacement.

The key to building an effective suite of BDD test scenarios is the derivation of a clearly understood Domain Specific Language for the tests. This language bridges the gap between the business concerns and the services and components created by the development team. In one sense this should not be a difficult task as the solution domain being developed should reflect the problem domain defined by the customer. Thus the effort needed to bridge the gap between the DSL and the technical solution is relatively small.

We had a lot of difficulty at first working with the stakeholders to agree and define the DSL as they were used to more traditional development practices and we had to educate them to understand that the scenarios that they were producing were testable. What they had to understand was that the scenarios were the contract between the development team and the project stakeholders. If the scenario was incorrect as signed of by the stakeholders and the analysts then the work produced by the development team would be incorrect too. The scenarios provided a very important structure to the SCRUM methodology we used. The analysis producing the scenarios was typically 1 to 2 sprints ahead of the implementation work.

JBehave provides a set of tools for building the elements of the DSL and for running the scenarios defined in it. Any scenarios in JBehave are written in simplified plain English (or the language of choice for the organisation). A scenario is structured as a list of sentences defining steps in the scenario. These sentences all begin with one of three key words:

• Given: The remainder of the sentence defines pre-conditions for the next steps of the scenario.
• When: Defines activity that the system under test is carrying out as part of the test.
• Then: The assertions that the test should be checking.

These sentences are the reusable elements of your BDD scenarios; the Givens, Whens and Thens can be put together to build different scenarios. The sentences are of course parameterised so that different values can be used in different tests.

What proved very important is the granularity of the concerns addressed by the DSL. Too coarse and you have very short (but deep) scenarios made of elements which are almost impossible to reuse; too fine and the size of the scenarios balloon and they become unmaintainable because of the sheer volume of change required as the system evolves.

In the end on the project at HSBC the appropriate level emerged organically, we paid close attention to the tests and worked hard to refactor and leverage common elements that emerged.

Another critical element is the data representation. When we started we bootstrapped the BDD test data management by using XStream to dump the objects help in memory to be compared with saved text files. This proved very helpful in the early stages but proved an increasing burden as the suite of tests broadened and deepened. The problem was that the test data was acutely sensitive to the domain data model and as that evolved we found ourselves expending major effort to correct tests that were not testing the changes but were affected because of our dumping the whole domain model.

The solution was to move to a factory and builder based mechanism. All tests used the same basic data sets generated by factories which then used builders to alter them to match the test requirements. When we came to make assertions we found that it was best to focus on the data elements required by the specific test and allow other tests to check their own data elements. This dramatically reduced the amount of data that needed incidental maintenance.

As we worked we found that after every major release it was worth spending some time on consolidating the BDD scenarios. The earlier scenarios for new functionality tended to be supplanted by scenarios testing more elaborate versions of the functionality. We worked to actively identify situations where this occurred and reduced the volume of tests while keeping their coverage.

JBehave integrated very effectively with Selenium and proved very effective at automating Web UI testing but rapidly exposed shortcomings in Selenium's implementation. By the time I left work was underway to use the WebDriver integration to address this.

By working to continually polish the DSL and the data representation we started seeing great reuse and efficiencies in developing provably tested new functionality. Where we originally spent anything up to 50% of the development time on building the BDD scenarios we saw that fall to as little as 5% of the development time.

Where this really began to pay dividends was that we had the tools that the test team needed to build out a much wider suite of tests and as they developed them we had probably the most thoroughly tested system I have ever worked on. The regression tests were run on our continuous build server producing near real time feedback as we developed which meant that problems were fixed as they occurred rather than after weeks of formal testing. This meant a much smaller testing team was required to thoroughly test the system at release. Fewer regressions were found and we were much more confident that the acceptance testing cycle would run on budget.

BDD is a key technique that I will use on future projects. It provides real benefits for the work invested and contributes materially to successful delivery.

Some New Social Networking Infrastructure

I'm a long time Facebook and LinkedIn user but as part of my involvement in developing Cazcade I've had to get my feet wet with Twitter.

I've been running FlipBoard for a while and I've found an outlet for sharing what I find on it by adding my Twitter account. I've now joined my blog to my Twitter account by using TwitterFeed and this post will test the whole process.

Twittering is very addictive and I suspect that I will begin to make more and more use of Twitter both to publish and consume content. Certainly Twitter together with FlipBoard makes consumption entirely easy and pleasurable. I'm very keen to bring together a number of my favourite RSS feeds in one place and read them in FlipBoard which has yet to have direct RSS support so TwitterFeed will be further utilised to aggregate them into one Twitter account.

What's interesting to me is how Cazcade fits into all of this. FlipBoard provides a brilliant mechanism for consuming and re-tweeting individual pieces of information. The limitation is that this is a piecemeal mechanism, you can only present one piece of information at a time. Cazcade's analogy by contrast allows you to accumulate various sources (web pages, videos, images and Twitter) into pools and present the information as a wider vision or argument and then pass them on via Twitter.

Cazcade is attempting to provide a mechanism above and beyond the current transient social network conversations.

The question I am asking myself is whether this will meet an unsatisfied need. Only time will tell.

Lyrical Ambiguity in the Kinks' Lola

I greatly enjoy a number of the Kinks songs, they were a brilliant group that created a number of classics that still resonate today.

One thing that always makes me stop and wonder is how many people believe firmly that they know the gender of Lola.

To me it seems that they have missed the brilliance of the lyrical ambiguity. The lines that really seem to be misunderstood are: "But I know what I am and I'm glad I'm a man and so is Lola.". Most people I talk to who have yet to get it believe firmly that the lines indicate that Lola is a man.

This interpretation arises from the belief that "so is Lola" refers to the fragment "I'm a man". This is a perfectly valid interpretation of the lyric but ignores another equally valid interpretation.

There is another fragment that could be referred to by "so is Lola"; "I'm glad I'm a man". If this is the reference then the interpretation would be that Lola is also glad that the protagonist is a man thus leaving Lola's gender entirely unstated.

In many ways this is the aural equivalent of the face/vase illusion where you can see either the faces or the vase but not both.

Whatever else it is, it is a brilliant song with brilliant lyrics.

- Posted using BlogPress from my iPad

iPhone 4 has arrived...

Faced a rather unique problem... I got two iPhones in my package rather than the one I expected. The nice young customer service rep nearly fell off her seat when I called and told her about it. The first time they had heard of that happening.

I think that I slightly restored her faith in humanity as she thought that no one would have called up and would have just stayed quiet about it. She asked me to drop off the extra to a store and gave me a nice in-store credit.

I'm now on my way to London waiting for the new SIM to activate and just using the iPhone as an iPod. I'm hoping that the rest of the experience is as painless as the iPad was.

- Posted using BlogPress from my iPad

Location:Stoats Nest Rd,Croydon,United Kingdom

My First Track day...

On Monday I took my life into my hands and went on a track day at Bedford Autodrome with EasyTrack.co.uk.

I had a blast! The track was ideal for a novice and my car (an Audi TT RS) proved more than capable. I had asked for a session with an instructor and I got a great guy named Steve. In the end I took a second session in the afternoon. As a result I went from being slowest on the track to not being the slowest by a decent margin. There were some fabulous drivers on the track that day, the two sticking in my mind included one in a GT3 RS and an utter genius in a bog standard MX5.

I've included a photo that was taken by TrackPhoto.co.uk.

- Posted using BlogPress from my iPad

The Height of Geekery

Well, here I am at my favourite cafe for breakfast with my iPad. I'm trying the whole blogging from my iPad thing with an app called BlogPress.

It's really interesting to see how modern integrated technology is making it feasible to do things that would have been awkward if not impossible a few years ago. The opportunities in this space are endless.

The excitement about the potential for location based services is well known. Unfortunately the excitement can obscure the real win. Location-less Services. The fact that I can start doing this kind of thing truly independently of my location is what's exciting. There have been some early mis-fires with the excitement of being able to take a laptop with you on holiday. Lugging a laptop though is really pretty hard work. We're on the cusp of being able to access and interact with the "datasphere" anywhere in the world.

This is where reading science fiction is a great advantage. Writers have been exploring how to use this for years. I feel that I am ready to make use of this and build on their thinking to go places that people who have not been reading will never think of.

How about you?

- Posted using BlogPress from my iPad

Location:Cliffe High St,Lewes,United Kingdom

Constructor Versus Setter Based Dependency Injection

There seems to be two extreme camps here and no middle ground.

I, as usual, agree with neither extreme.

The simple fact is that my preferred approach is to design classes that meet the original intent of object orientation. That the class is fully usable after construction.

That is not to say that I believe that Constructor injection is the one, true way. Rather, I believe that all mandatory dependencies should be configured in the constructor and optional ones should be satisfied by setter methods. This makes the class much easier to understand. It also means that the constructor does not end up completely overloaded with parameters but neither are you left looking at a a sea of setter methods wondering which ones have to be used.

By using this convention the code becomes more self-documenting.

The problem with modern 3D films

What are the problems and limitations of modern 3D? I'm not going to claim any particular expertise in this subject past a decent familiarity with 3D computer graphics. However I do think that there are a couple of key points that are passing people by in the recent excitement about 3D cinema. The pedant in me wants to go on record ;-).

The main thing that is not being said is that the 3D we are experiencing is solely 'Stereoscopic 3D' this means that everyone in the audience is seeing the same 2 images for every frame of the movie.

The best way of highlighting the limitations is to point out the differences in experience between real 3D (i.e. the world around us) and the 3D we experience in the cinemas.

While we do experience the world through two eyes those eyes do have to a degree the ability to sense distance individually through their focus - we do not completely rely on the stereoscopic effect to determine distance and so never lose the ability to determine how far away something is even if we were to lose an eye. This means that the illusion can never be complete as the focus in a 3D movie is determined by the director and camera departments. If something catches your eye that is not in the plane of focus then all you can ever see is a blur though your eyes may strain to focus it. This is both a blessing and a curse as it provides another tool to the director to direct your attention within the frame but can also wreck the illusion.

Another issue is that the stereoscopic effect is achieved by filming the scene from a fixed viewpoint. This means that the frames captured look best from a specific place in the cinema. The further away from that position the less effective the 3D effect is.

A related issue is that the illusion requires one orientation of the spectator. If you lean your head to one side the illusion starts to dissolve as instead of seeing the frames for the top eye and the bottom eye you will keep on seeing the frames for the left and right eyes. This gets worse if you were to continue the rotation as the eyes would then be reversed... Not many people will see the movie standing on their heads but couples leaning their heads on each other's shoulders will have a problem.

The final issue is that of movement. In dealing with a 3D world we are used to moving around it and seeing different perspectives. Despite the fact that we are largely fixed in our seats in a cinema we still have the impulse to crane up or slump down, lean left or lean right just to see that little bit more. Unfortunately the Stereoscopic 3D effect cannot handle that behaviour.

It worries me a little that there is talk of bringing Stereoscopic 3D into the home as there may be an effect where people are trained to sit absolutely still in one orientation to maximise the effect. There are already concerns about how children sitting still in front of the traditional 2 D television and not fidgeting are contributing to the obesity epidemic, how much worse may it become with Stereoscopic 3D TV?

What do I think about Business Analysis?

At work we're currently looking for a new Business Analyst and it has given me pause to review my thoughts on Business Analysis. I originally trained as an Analyst/Programmer and have during the course of my career had cause to do rather a lot of Business and Systems Analysis even though you will not find a huge amount about it on my CV.

I started off using Entity Relationship Diagrams as an Oracle Specialist and, as I encountered more types of formal analysis, made a point of learning about all of them.

As a result of my Java work I became exposed to UML as a diagramming notation for OO programming concepts but over time came to appreciate it as a fairly complete set of diagrams for supporting Business and Systems Analysis.

The recent search for a Business Analyst has highlighted to me that an awful lot of people are described as Business Analysts who, to my mind, do not have the skills to perform the job.

A good Business Analyst can add a great deal of value to a project. They are the front runner in terms of understanding the business domain. They have the following key roles to perform:

• Working with the business to understand the business domain.
• Capturing and documenting business requirements.
• Analyse the business domain to highlight areas of inconsistency and identify areas of uncertainty.
• Distill and communicate the business requirements to the implementation team.

There is a continuum of competency between the parrot and the professional.

The parrot is the so-called Business Analyst that just sits between the implementation team and the business and just acts as a conduit for questions and answers - actively subtracting value from the exchange by slowing it down.

The professional is, for me, the Business Analyst that can work with a business that has yet to fully formulate a problem and work with them to produce a full and clear definition of the problem and the business solution that can be used by the development team. The professional actively helps the Business think about the problem space and gives them the tools to properly understand it themselves. The professional will be able to answer a very high percentage of the implementation team's questions without recourse to the business as the questions have already been addressed by the analysis.

As far as I can tell the vast majority of Business Analysts in the Investment Banking sector fall perilously close to being parrots. While I am most used to UML, any business analyst that can use a clear approach to formally analyse the business domain and its requirements adding value all the while is fine by me.

As many of my colleagues will know I firmly believe that 'code is cheap' and I have come to realise that the same holds true of analysis documentation. What is truly valuable is the knowledge it embodies. While the concept of Bit-Rot is well understood in code; the same concept applies to analysis documentation (Word-Rot?). The real value is held in the analyst's head and in the heads of the business and implementation team at the end of the whole process.

I no longer believe that huge repositories of analysis have any real value - what is valuable is the process of producing the analysis and the knowledge that process imparts. A certain level of high level analysis documentation for invariant concepts can have longer term value but that will only be a small fraction of all that is produced.

I believe that there is no value to analysis artefacts that cannot be formally tested. Once an unit of analysis is complete it should be handed over to a tester for formal testing and then be used as the basis for User Acceptance Tests for the delivered system.

When a business analyst is required then you had better find yourself a professional and once the analysis is done you had better hold onto that professional. Inevitably team members move on, but perhaps the best hand over will be for the outgoing analyst to guide the replacement through a complete analysis of the domain.

I have worked with excellent analysts who have added immense value to a project and I have worked with some complete incompetents. Guess which ones I actively seek to work with again.

A First Attempt at Haiku

Normally I'm shy about my poetry but I think I can be brave enough to share 3 lines and 17 syllables.

Talking to my Project Manager last night, I said that a particularly gnarly problem would probably be solved by about 3 lines of code, but which 3 lines of code? He laughed and said that was going to be a piece of software haiku. He went to a neighbouring desk tossing out a line that he would like to see some haiku from me and this is what I produced in the 3 minutes he was away.

Code glows on white screen;

Incandescent thought laid bare,

Human to Machine.

I hope that you like it.

Octopussies...

Nope - I'm not referring to multiples of a James Bond Film - I'm referring instead to a family tradition.

My mother has for as long as I can remember given an 'Octopussy' to the young children of family and friends. An Octopussy is a friendly octopus crocheted out of wool:

To start the body:

• Make a chain of 6 or 7 stitches and then form a circle using whatever colour wool you have to hand..
• Using double crochet make a flat circle about 2 or 3 inches in diameter.
• Continue adding rows to the circle without increasing the number of stitches and this will form a cup shape.
• Change to white to continue in order to make a space for the eyes to be embroidered.
• Continue until you have a nice wide band for the eyes.

Now make 8 thick crochet legs as long as you like:

• Start trimming the edge with double crochet in a contrasting colour going through the middle of the chain.
• Continue down to one end and crochet in the long strands.
• Turn and do a second row of double crochet and turn again to give a 3 row 'foot'.
• Continue until you reach the end that you started with.
• Using the loose strands from each of the feet pull them through the flat circle at the base of the body and tie them off to attach the legs.

Finish the body;

• Embroider the eyes.
• Change colour from the white to whatever colour you like.
• Continue with crochet decreasing with each row.
• When you are close to the end stuff the body with a non-toxic filling (my mother uses a form of foam chips).
• finish the top of the body and pull final thread through to lose it in the body.
• Make a long chain with two or three strands of wool.
• Pull the chain halfway through the top of the Octopussy placing a knot so that it cannot be shifted.
• The chain can then be tied onto a convenient point so that the Octopussy cannot be thrown out of a pram or cot.

The Octopussy contains only wool, thread and stuffing so there are no small hard parts that can be pulled off and cause a hazard. It is very robust to chewing and is fully washable.

It also has the advantage of leaving the child with a friendly view of Octopi!