Showing posts with label Technical. Show all posts
Showing posts with label Technical. Show all posts

March 31, 2011

Update 2 on Neo4J Spatial and British Isles OSM Data import.

Well even 6GB of heap space isn't enough to fully import and re-index the British Isles Open Street Map Data set. I'm thinking that there will have to be some work on memory management somewhere. First thing I'm going to need to do is profile the import and work out where the memory is being used (I'm assuming it isn't a memory leak per se). Basically I want to determine whether it is because of a misconfiguration by me or there is a need to look at the memory usage in the OSMImporter class.

I'm also thinking that for reverse geo and local searches I really don't need the full OSM data set so a customised version of the OSM importer would be a good idea. I really don't need ways or the node data associated with them. I suspect a multi-pass approach would be a good idea. The first pass through the OSM data set determines which nodes are needed for the features that I want to import, the second pass imports the required nodes and their associated features, followed by a final re-indexing.

I've been in discussions with one of the people behind Neo4J Spatial, Craig Taverner, on the Neo4J User List. I found an issue with the bounding box used in the spatial search index visitor pattern. It's been quite illuminating to dig into this. I'd been thinking for a while that a number of the techniques that I encountered working with 3D graphics and scene graphs would be relevant and it definitely seems to be the case. I'm beginning to think that OpenGL or OpenCL backed indices could prove very useful in high throughput micro-batched situations where the bus latency can be hidden.

March 21, 2011

Neo4J Spatial and Importing the British Isles Open Street Map Data

I know, I know. Overly ambitious and all that.

While we've been working on Presynt we've been having fun and games with Geo Data; both for Local Search and Reverse Geo lookups. We've also got a number of ambitious new Geo features planned for future versions that needs needs some form of Geo store.

So with that in mind I started playing with Neo4J Spatial and the Open Street Map data set.

It's been an interesting experience with some gotchas as Neo4J Spatial is still not fully ready for prime time, so I thought I would share my experience with those who are interested.

First off I thought I would try to import the full global data set from Open Street Map. Not necessarily a mistake, but a huge undertaking as we are talking about over 200GB of map data in raw OSM format, billions of data points and relationships.

The first gotcha is that if you wish to import the full OpenStreetMap data set I'd recommend starting with at least Neo4j version 1.3 Milestone 4 as that version was the first one to hugely expand the number of nodes supported from 4 billion to 128 billion.

Of course attempting this type of import I also started running into memory problems. I ended up running with -Xmx4096m and -XX:+UseConcMarkSweepGC. The concurrent mark and sweep garbage collector proved most resilient to the loads placed on it. The default parallel GC would fall over and cry quite regularly due to an issue with consuming too much CPU to too little effect.

In the end I discovered that the full global import would take too long to run and set my sights a little lower on just the British Isles data set which is all that Presynt needs right now.

In order to test my set up I decided to use the Buckinghamshire data set which is the smallest county data set in the UK and this highlighted another gotcha: Not all the dependencies required are actually included in the Neo4J POMs.

In the end I used the following dependencies:
  • org.neo4j:neo4j:1.3.M04
  • org.neo4j:neo4j-spatial:0.5-SNAPSHOT
  • org.geotools:gt-referencing:2.6.5
  • org.geotools:gt-main:2.6.5
  • org.geotools:gt-cql:2.6.5
  • org.geotools:gt-epsg-hsql:2.6.5
  • com.vividsolutions:jts:1.11
They may not all be necessary but they certainly work...

Once I had resolved these issues then I was able to start the full British Isles import. I was running it on a quad-core machine and found that the OSMImporter/BatchInserter combination (all default configuration mind) was using just one CPU. Looking at the OSMImporter codebase it became obvious that it was written to stream read the OSM xml data and synchronously write it out to Neo4J. I think that there may be some room to parallelise it as large amounts of the GPS node data is utterly independent and the XML data set is structured in such a way that it could be broken down into independent units of work. The only points where serialised behaviour are required are during the parsing of the XML, the construction of the Way and Relation data and possibly the writing to Neo4J.

The process seems to be largely CPU bound as I ran it on SATA II RAID 0 SSDs and their read and write capabilities were barely touched while the single CPU in use was pegged at 100% almost continuously.

The import of the British Isles dataset seems to take about 48 hours.

I used the vanilla OSMImporter and BatchInserter configurations as I am still a Neo4J Neophyte (bad pun intended ;-)) I will be reading up on alternative configurations to see if I can improve the performance. I will also look at the import of changesets so that I shouldn't need to do a full inport again.

September 28, 2010

Further Forays Into the HTML5 Stack - Animation

Short update this one.

I've been doing some simple animations replacing one image with another every 5 seconds using the SVG SMIL animation framework. I'm using the 'defs' element to define a number of image elements that will not be immediately rendered. I then use an 'use' element to refer to the first element in the sequence before using the 'animate' element to alter the 'xlink:href' attribute of the 'use' element every 5 seconds from a set of values in the 'animate' element.

Unfortunately this approach seems to be very CPU intensive even between changes to the 'use' element's 'xlink:href' attribute.

I'm wondering whether the animation timer does not take into account the down time between the changes and just spins its wheels, wasting CPU, all the time.

I think that I need to do some rough benchmarking to work out whether to use CSS3 or SVG SMIL animation and to work out which techniques are more CPU friendly.

September 20, 2010

Forays into the HTML5 stack.

While working on Cazcade with Neil, Dan C., Jon et al I'm also working on my own little project - DiarWise. It's still pretty stealth so I'm not going to discuss it in detail on an open forum but I think that now is a good time to discuss some of the technology that I've been using.

For DiarWise I've been looking at a much more rich UI experience highly interactive and completely resolution independent. I wanted to be able to create a UI that would be equally renderable on a smartphone, a tablet and a PC without extensive customisation for each. I also wanted a UI that would would be able to interact well with the browser event models for mouse and multi-touch.

I've been focussing on SVG embedded in HTML over the HTML 5 Canvas element for a number of reasons:
  • It is significantly more mature as a specification than the HTML 5 Canvas element.
  • The HTML 5 canvas element is procedural in its rendering so with a command it renders pixels directly to screen and so does not have a screen graph; the SVG xml elements are the scene graph and are embedded directly in the DOM and so are manipulable using all the tricks that we are familiar with. This also means that a lot of the heavy lifting for re-drawing is done for us.
  • Text is more of a first class citizen in SVG and so is easier to make available to Accessible browsers.
  • SVG graphics elements are style-able using CSS.
  • HTML can also be embedded back into SVG (and manipulated via the DOM) using the foreignObject element
There are a few weaknesses as I've discovered as I've worked with it.
  • The default SVG animation framework is built around the SMIL standard which is pretty low level and is not always the easiest to work with though it is very powerful. I'm intending to see how well SVG interacts with the CSS3 animation effects which are much nicer to work with.
  • SVG support is present in the latest iteration of all the browsers but is patchy to say the least. The best of the moment is the WebKit family, Firefox is the next best though it has big issues with the text rendering. IE9 claims to be a huge step forwards over IE8 but I have yet to try out the beta and there are question marks over how well it will support SVG animation.
  • While its text support is good there is one glaring omission - word wrap. The neatest work-around is to embed an appropriate HTML element in the SVG using foreignObject.
The best trick I've found so far with SVG is the simplicity of decoupling the screen resolution from the render resolution and this cascades down to any embedded HTML elements. The 'svg' element defines a local coordinate space for all its contents and you can override the coordinate system using the viewBox attribute. This means that all embedded elements can be positioned and sized relative to each other in terms of local 'pixels' and the SVG renderer will scale according to the screen resolution. The handles aspect ratios elegantly using the preserveAspectRatio attribute. This gives you the ability to present a consistent interface in a wide range of resolutions and can handle pan and zoom exceedingly smoothly for people with limited eyesight.

So far my grand experiment with SVG is going well and it seems to be well worth using as one of the technologies that make up the HTML 5 stack.

September 15, 2010

My life with BDD...

This is a blog post that's been very long overdue, but I finally acknowledged that I really need to capture my experiences after a chat with an old friend at Twitter (shameless name drop...).

I first came across Behaviour Driven Development thanks to Mauro Talevi, my first hire on the team that I set up at HSBC back in 2008. He's a major contributor on the JBehave project that is one of the major BDD frameworks. I hope that I'm not going to embarrass him too much by saying that he was an invaluable influence on the project.

I'm not going to talk about the technical implementation as that is well covered elsewhere but instead in this blog I will talk about the general principles and practical experience of working with BDDs.

The basic idea is simple. Any story or feature being implemented in a project should have automated integration tests that proves that the work has been completed successfully. It is important the the tests be comprehensible to the stakeholders requesting the story / feature so that they can confirm that it is complete and done. This provides a very clear goal for the developers and works to build up a suite of regression tests that give a healthy confidence in the application. The tests are defined in terms of business scenarios that represent various paths through the business process.

BDD is complementary to Unit Testing and is in no way a replacement.

The key to building an effective suite of BDD test scenarios is the derivation of a clearly understood Domain Specific Language for the tests. This language bridges the gap between the business concerns and the services and components created by the development team. In one sense this should not be a difficult task as the solution domain being developed should reflect the problem domain defined by the customer. Thus the effort needed to bridge the gap between the DSL and the technical solution is relatively small.

We had a lot of difficulty at first working with the stakeholders to agree and define the DSL as they were used to more traditional development practices and we had to educate them to understand that the scenarios that they were producing were testable. What they had to understand was that the scenarios were the contract between the development team and the project stakeholders. If the scenario was incorrect as signed of by the stakeholders and the analysts then the work produced by the development team would be incorrect too. The scenarios provided a very important structure to the SCRUM methodology we used. The analysis producing the scenarios was typically 1 to 2 sprints ahead of the implementation work.

JBehave provides a set of tools for building the elements of the DSL and for running the scenarios defined in it. Any scenarios in JBehave are written in simplified plain English (or the language of choice for the organisation). A scenario is structured as a list of sentences defining steps in the scenario. These sentences all begin with one of three key words:
  • Given: The remainder of the sentence defines pre-conditions for the next steps of the scenario.
  • When: Defines activity that the system under test is carrying out as part of the test.
  • Then: The assertions that the test should be checking.
These sentences are the reusable elements of your BDD scenarios; the Givens, Whens and Thens can be put together to build different scenarios. The sentences are of course parameterised so that different values can be used in different tests.

What proved very important is the granularity of the concerns addressed by the DSL. Too coarse and you have very short (but deep) scenarios made of elements which are almost impossible to reuse; too fine and the size of the scenarios balloon and they become unmaintainable because of the sheer volume of change required as the system evolves.

In the end on the project at HSBC the appropriate level emerged organically, we paid close attention to the tests and worked hard to refactor and leverage common elements that emerged.

Another critical element is the data representation. When we started we bootstrapped the BDD test data management by using XStream to dump the objects help in memory to be compared with saved text files. This proved very helpful in the early stages but proved an increasing burden as the suite of tests broadened and deepened. The problem was that the test data was acutely sensitive to the domain data model and as that evolved we found ourselves expending major effort to correct tests that were not testing the changes but were affected because of our dumping the whole domain model.

The solution was to move to a factory and builder based mechanism. All tests used the same basic data sets generated by factories which then used builders to alter them to match the test requirements. When we came to make assertions we found that it was best to focus on the data elements required by the specific test and allow other tests to check their own data elements. This dramatically reduced the amount of data that needed incidental maintenance.

As we worked we found that after every major release it was worth spending some time on consolidating the BDD scenarios. The earlier scenarios for new functionality tended to be supplanted by scenarios testing more elaborate versions of the functionality. We worked to actively identify situations where this occurred and reduced the volume of tests while keeping their coverage.

JBehave integrated very effectively with Selenium and proved very effective at automating Web UI testing but rapidly exposed shortcomings in Selenium's implementation. By the time I left work was underway to use the WebDriver integration to address this.

By working to continually polish the DSL and the data representation we started seeing great reuse and efficiencies in developing provably tested new functionality. Where we originally spent anything up to 50% of the development time on building the BDD scenarios we saw that fall to as little as 5% of the development time.

Where this really began to pay dividends was that we had the tools that the test team needed to build out a much wider suite of tests and as they developed them we had probably the most thoroughly tested system I have ever worked on. The regression tests were run on our continuous build server producing near real time feedback as we developed which meant that problems were fixed as they occurred rather than after weeks of formal testing. This meant a much smaller testing team was required to thoroughly test the system at release. Fewer regressions were found and we were much more confident that the acceptance testing cycle would run on budget.

BDD is a key technique that I will use on future projects. It provides real benefits for the work invested and contributes materially to successful delivery.

September 12, 2010

Some New Social Networking Infrastructure

I'm a long time Facebook and LinkedIn user but as part of my involvement in developing Cazcade I've had to get my feet wet with Twitter.

I've been running FlipBoard for a while and I've found an outlet for sharing what I find on it by adding my Twitter account. I've now joined my blog to my Twitter account by using TwitterFeed and this post will test the whole process.

Twittering is very addictive and I suspect that I will begin to make more and more use of Twitter both to publish and consume content. Certainly Twitter together with FlipBoard makes consumption entirely easy and pleasurable. I'm very keen to bring together a number of my favourite RSS feeds in one place and read them in FlipBoard which has yet to have direct RSS support so TwitterFeed will be further utilised to aggregate them into one Twitter account.

What's interesting to me is how Cazcade fits into all of this. FlipBoard provides a brilliant mechanism for consuming and re-tweeting individual pieces of information. The limitation is that this is a piecemeal mechanism, you can only present one piece of information at a time. Cazcade's analogy by contrast allows you to accumulate various sources (web pages, videos, images and Twitter) into pools and present the information as a wider vision or argument and then pass them on via Twitter.

Cazcade is attempting to provide a mechanism above and beyond the current transient social network conversations.

The question I am asking myself is whether this will meet an unsatisfied need. Only time will tell.

June 27, 2010

The Height of Geekery

Well, here I am at my favourite cafe for breakfast with my iPad. I'm trying the whole blogging from my iPad thing with an app called BlogPress.

It's really interesting to see how modern integrated technology is making it feasible to do things that would have been awkward if not impossible a few years ago. The opportunities in this space are endless.

The excitement about the potential for location based services is well known. Unfortunately the excitement can obscure the real win. Location-less Services. The fact that I can start doing this kind of thing truly independently of my location is what's exciting. There have been some early mis-fires with the excitement of being able to take a laptop with you on holiday. Lugging a laptop though is really pretty hard work. We're on the cusp of being able to access and interact with the "datasphere" anywhere in the world.

This is where reading science fiction is a great advantage. Writers have been exploring how to use this for years. I feel that I am ready to make use of this and build on their thinking to go places that people who have not been reading will never think of.

How about you?


- Posted using BlogPress from my iPad

Location:Cliffe High St,Lewes,United Kingdom

June 17, 2010

Constructor Versus Setter Based Dependency Injection

There seems to be two extreme camps here and no middle ground.

I, as usual, agree with neither extreme.

The simple fact is that my preferred approach is to design classes that meet the original intent of object orientation. That the class is fully usable after construction.

That is not to say that I believe that Constructor injection is the one, true way. Rather, I believe that all mandatory dependencies should be configured in the constructor and optional ones should be satisfied by setter methods. This makes the class much easier to understand. It also means that the constructor does not end up completely overloaded with parameters but neither are you left looking at a a sea of setter methods wondering which ones have to be used.

By using this convention the code becomes more self-documenting.

February 10, 2010

What do I think about Business Analysis?

At work we're currently looking for a new Business Analyst and it has given me pause to review my thoughts on Business Analysis. I originally trained as an Analyst/Programmer and have during the course of my career had cause to do rather a lot of Business and Systems Analysis even though you will not find a huge amount about it on my CV.

I started off using Entity Relationship Diagrams as an Oracle Specialist and, as I encountered more types of formal analysis, made a point of learning about all of them.

As a result of my Java work I became exposed to UML as a diagramming notation for OO programming concepts but over time came to appreciate it as a fairly complete set of diagrams for supporting Business and Systems Analysis.

The recent search for a Business Analyst has highlighted to me that an awful lot of people are described as Business Analysts who, to my mind, do not have the skills to perform the job.

A good Business Analyst can add a great deal of value to a project. They are the front runner in terms of understanding the business domain. They have the following key roles to perform:
  • Working with the business to understand the business domain.
  • Capturing and documenting business requirements.
  • Analyse the business domain to highlight areas of inconsistency and identify areas of uncertainty.
  • Distill and communicate the business requirements to the implementation team.
There is a continuum of competency between the parrot and the professional.

The parrot is the so-called Business Analyst that just sits between the implementation team and the business and just acts as a conduit for questions and answers - actively subtracting value from the exchange by slowing it down.

The professional is, for me, the Business Analyst that can work with a business that has yet to fully formulate a problem and work with them to produce a full and clear definition of the problem and the business solution that can be used by the development team. The professional actively helps the Business think about the problem space and gives them the tools to properly understand it themselves. The professional will be able to answer a very high percentage of the implementation team's questions without recourse to the business as the questions have already been addressed by the analysis.

As far as I can tell the vast majority of Business Analysts in the Investment Banking sector fall perilously close to being parrots. While I am most used to UML, any business analyst that can use a clear approach to formally analyse the business domain and its requirements adding value all the while is fine by me.

As many of my colleagues will know I firmly believe that 'code is cheap' and I have come to realise that the same holds true of analysis documentation. What is truly valuable is the knowledge it embodies. While the concept of Bit-Rot is well understood in code; the same concept applies to analysis documentation (Word-Rot?). The real value is held in the analyst's head and in the heads of the business and implementation team at the end of the whole process.

I no longer believe that huge repositories of analysis have any real value - what is valuable is the process of producing the analysis and the knowledge that process imparts. A certain level of high level analysis documentation for invariant concepts can have longer term value but that will only be a small fraction of all that is produced.

I believe that there is no value to analysis artefacts that cannot be formally tested. Once an unit of analysis is complete it should be handed over to a tester for formal testing and then be used as the basis for User Acceptance Tests for the delivered system.

When a business analyst is required then you had better find yourself a professional and once the analysis is done you had better hold onto that professional. Inevitably team members move on, but perhaps the best hand over will be for the outgoing analyst to guide the replacement through a complete analysis of the domain.

I have worked with excellent analysts who have added immense value to a project and I have worked with some complete incompetents. Guess which ones I actively seek to work with again.

October 13, 2009

GPGPU Mandelbrot with OpenCL and Java

I've done a fair amount of learning since I last posted. OpenCL stopped being just a specification and now has a concrete implementation.

I upgraded my MacBook to Snow Leopard and got access to a fairly solid implementation. I say 'fairly' because it exhibits a few interesting quirks... I can pretty reliably get it to throw memory access errors even when I am doing nothing exotic. More about this if I can nail down why it is happening.

I've recently installed ATI Stream SDK 2.0 beta 2 to my 'Windows Beast' only to discover that beta 3 is the only one so far that works with the toolkit that I am using.

At present I am using OpenCL4Java to provide the Java bindings to OpenCL they are developing fast and have the advantage of using JNA so there needs to be no native installs past the OpenCL drivers.

It has proved both easier and harder than I expected - I hadn't realised how much C I had forgotten, but then again the syntactical similarities with Java made it easier for me to get to grips with.

I walked through the various examples and tinkered with them until I had at least a basic grip. For my first program I decided to use a nearly perfect algorithm for scaling on multiple threads, the Mandelbrot set. Pseudo code for the algorithm is widely disseminated but to be honest the OpenCL implementation is clear enough. I'll present my take here and with it I will highlight some of the interesting features of OpenCL.
__kernel mandelbrot(
const float deltaReal,
const float deltaImaginary,
const float realMin,
const float imaginaryMin,
const unsigned int maxIter,
const unsigned int magicNumber,
const unsigned int hRes,
__global int* outputi
)
{
int xId = get_global_id(0);
int yId = get_global_id(1);

float realPos = realMin + (xId * deltaReal);
float imaginaryPos = imaginaryMin + (yId * deltaImaginary);
float real = realPos;
float imaginary = imaginaryPos;
float realSquared = real * real;
float imaginarySquared = imaginary * imaginary;

int iter = 0;
while ( (iter < maxIter) && ((realSquared + imaginarySquared) < magicNumber) )
{
imaginary = (2 * (real * imaginary)) + imaginaryPos;
real = realSquared - imaginarySquared + realPos;
realSquared = real * real;
imaginarySquared = imaginary * imaginary;
iter++;
}
if(iter >= maxIter){
iter = 0;
}
outputi[(yId * hRes) + xId] = iter;
}
First thing I'll highlight is the fact that I've only non-pixel specific parameters - all pixel specific variables are calculated in the OpenCL program. At present I'm only using floats - the implementations have yet to support doubles.

The coordinates of the pixel / work group are retrieved using the OpenCL specific function : get_global_id(int dimension). Currently Open CL supports 1,2 and 3 dimensional coordinate spaces.

The OpenCL language is mainly derived from C99 and should be pretty easy for C/C++/Java developers to read. The main difference is exclusions - a lot of the default libraries and some control of flow instructions. There are a few additions such as the work group / unit functions and the '__kernel' keyword.

The OpenCL4Java source code is pretty simple:
package bbbob.gparallel.mandelbrot;
import com.nativelibs4java.opencl.*;
import static com.nativelibs4java.opencl.OpenCL4Java.*;
import com.nativelibs4java.util.NIOUtils;

import javax.imageio.ImageWriter;
import javax.imageio.ImageIO;
import javax.imageio.stream.ImageOutputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.io.IOException;
import java.io.File;
import java.nio.IntBuffer;
import java.awt.image.BufferedImage;

public class Mandelbrot {
//boundary of view on mandelbrot set

public static void main(String[] args) throws IOException, CLBuildException {

//Setup variables for parameters

//Boundaries
float realMin = -2.25f; //-0.19920f; // -2.25
float realMax = 0.75f; //-0.12954f; // 0.75
float imaginaryMin = -1.5f; //1.01480f; // -1.5
float imaginaryMax = 1.5f; //1.06707f; // 1.5

//Resolution
int realResolution = 640; // TODO validate against device capabilities
int imaginaryResolution = 640;

//The maximum iterations to perform before returning and assigning 0 to a pixel (infinity)
int maxIter = 64;

//TODO describe what this number means...
int magicNumber = 4;

//Derive the distance in imaginary / real coordinates between adjacent pixels of the image.
float deltaReal = (realMax - realMin) / (realResolution-1);
float deltaImaginary = (imaginaryMax - imaginaryMin) / (imaginaryResolution-1);

//Setup output buffer
int size = realResolution * imaginaryResolution;
IntBuffer results = NIOUtils.directInts(size);

//TODO use an image object directly.
//CL.clCreateImage2D(context.get(), 0, OpenCLLibrary);
//TODO set up a Float4 array in order to be able to provide a colour map.
//This depends on whether we will be able to pass in a Float4 array as an argument in the future.

//Read the source file.
String src = readFully(
new InputStreamReader(Mandelbrot.class.getResourceAsStream("opencl/mandelbrot.cl")),
new StringBuilder()
).toString();

buildAndExecuteKernel(realMin, imaginaryMin, realResolution, imaginaryResolution, maxIter, magicNumber,
deltaReal, deltaImaginary, results, src);


outputResults(realResolution, imaginaryResolution, results);
}

private static void outputResults(int realResolution, int imaginaryResolution,
IntBuffer results) {
int[] outputResults = new int[realResolution * imaginaryResolution];

results.get(outputResults);
BufferedImage image = new BufferedImage(realResolution, imaginaryResolution, BufferedImage.TYPE_INT_RGB);
for(int y = 0; y < imaginaryResolution; y++){
int rowPos = y * imaginaryResolution;
for(int x = 0; x < realResolution; x++){
image.setRGB(x,y,outputResults[rowPos + x] * 32 );
}
}

try {
ImageWriter writer = ImageIO.getImageWritersByFormatName("gif").next();
ImageOutputStream stream = ImageIO.createImageOutputStream(new File("test.gif"));
writer.setOutput(stream);
writer.write(image);
} catch (IOException e) {
e.printStackTrace(); //To change body of catch statement use File | Settings | File Templates.
}


// for (int i = 0; i < outputResults.length; i++) {
// if((i % realResolution) == 0 ){
// System.out.print("\n");
// }
// int outputResult = outputResults[i];
// if(outputResult == 0){
// System.out.print("0");
// }else{
// System.out.print("" + outputResult % 10);
// }
// }
}

private static void buildAndExecuteKernel(float realMin, float imaginaryMin, int realResolution,
int imaginaryResolution, int maxIter, int magicNumber, float deltaReal,
float deltaImaginary, IntBuffer results, String src) throws CLBuildException {
//TODO build some intelligence into the mechanism for pulling out platforms and devices.
CLPlatform[] platforms = listPlatforms();
CLDevice[] devices = platforms[0].listGPUDevices(false);

//Create a context and program using the devices discovered.
CLContext context = platforms[0].createContext(devices);
CLProgram program = context.createProgram(src).build();

//Create a kernel instance from the mandelbrot kernel, passing in parameters.
CLKernel kernel = program.createKernel(
"mandelbrot",
deltaReal,
deltaImaginary,
realMin,
imaginaryMin,
maxIter,
magicNumber,
realResolution,
context.createIntBuffer(CLMem.Usage.Output, results, false)
);

//Enqueue and complete work using a 2D range of work groups corrsponding to individual pizels in the set.
//The work groups are 1x1 in size and their range is defined by the desired resolution. This corresponds
//to one device thread per pixel.
CLQueue queue = context.createDefaultQueue();
kernel.enqueueNDRange(queue, new int[]{realResolution, imaginaryResolution}, new int[]{1,1});
queue.finish();
}

public static CharSequence readFully(Reader reader, StringBuilder builder) throws IOException {

char[] buffer = new char[8192];
for (int readLen = reader.read(buffer); readLen >= 0; readLen = reader.read(buffer)){
builder.append(buffer, 0, readLen);
}
return builder;
}

}

I'm finding it pretty easy to work with OpenCL - now I need to identify some more complex problems to solve.

May 07, 2009

Balance in Business and Technical Architectures

I've worked in a number of organisations over the years. These organisations were many and varied but one pattern seemed to be repeated without fail.

The cycle of centralisation and de-centralisation.

This seems to be a very disruptive and expensive cycle.. The time and effort wasted on reorganising must detract from the company bottom line. I've spent some time trying to work out why this cycle happens and how to resolve it.

Both centralisation and de-centralisation has their positives and their negatives. Centralisation allows the rationalisation of processes and resources but increases the rigidity of the organisation, reducing its ability to respond to changing conditions. De-centralisation can make the company more flexible and agile but risks different segments of the organisation wasting resources through unnecessary competition.

The trouble seems to be that the people at the helm don't seem to understand the interplay of benefits and consequences. This may be due to the way that information about the organisation is presented to them. There is a belief that only simple messages can be delivered at the executive level and so they hear stark messages like 'We're too centralised, we must de-centralise' or 'We're too de-centralised we must de-centralise'. They aren't told the detail and texture.

When they hear these messages, they feel they must act decisively.

It may also be that the hardest thing in business is the same as the hardest thing in politics: Be seen to do nothing. Perhaps the only way out of this cycle is for the bosses to make smaller, more considered changes but to do that takes better communications and finer filters.

I wonder whether Service Oriented Architecture is an example of a potential business architecture that can be used to break this cycle. It balances by centralising the control of the interfaces between business units and systems, but de-centralises the system implementations. The technologies of the interfaces are centralised and controlled but the technologies of the implementations can be the ones most suited to their requirements.

Of course the SOA model can be centralised when someone imposes the implementation technologies on the business units.

I think that there is a balance that must be struck between centralisation and de-centralisation. When I find a large company that manages it I'll be fascinated to see how they achieve it.

February 10, 2009

Pair Programming Interviews

My current project has undergone an highly protracted round of interviews due in part to the hiring freeze that for some strange reason started in September.

This was my first set of interviews where I used pair programming as part of the interview process.

It was extremely effective. We used a highly simplified 'story' to test the candidate's design and coding skills. The story defined an UserManager middleware component that was used to register new users given a login and an e-mail address. The inputs were to be validated and an e-mail sent to the e-mail address containing an auto-generated password.

An empty Eclipse or IntelliJ project was provided with Spring, JUnit and all the mocking frameworks available as dependencies. A working Maven 2 build was implemented on the command line. The candidate was asked to provide an implementation of the middleware component and provide interfaces for all the dependencies that this component required.

There were two interviewers with the candidate, one acting as pair and the other as the product owner.

We were looking for a clear understanding of interface-implementation separation, understanding of how unit testing and dependency injection fitted together and a clear understanding of the IDE tooling and the various refactorings.

It was certainly effective. It was surprising how many candidates that seemed technically competent verbally were unable to make any headway with this exercise. A lot of candidates just couldn't handle the unfamiliar interview situation; others really had not learnt the tools of their trade and were unable to use the IDE to speed up their coding; other candidates had claimed TDD experience and were unable to write a decent unit test or use mocks; many candidates were unable to handle the design element, defining interfaces and making clear, logical decisions about contracts and responsibilities.

The successful candidates usually sketched out the domain either as interfaces in code or on paper, they asked good questions of their pair and the product owner and they always were very clear about what testing was needed. They also picked up the fact that the story was incomplete and asked about details of the validation.

A pair programming interview highlights the candidates technical skills but provides great insight into their characters and how they will fit into a team.

December 12, 2008

Thoughts on Scrum

The project that I'm on is being run as a pure Scrum project. I'm still forming my thoughts, but I think it's still worth sharing my first impressions.

I'm something of a reactionary old architect; I still instinctively like to design a lot up front. I'm fully aware that design still has a part to play in Scrum, but I do feel better with more up-front thought about the problems.

I've been surprised at how heavyweight the Sprint meetings feel (Planning, Review and Retrospective). We were running a 2 week sprint and it felt like we spent far too much time in these meetings, it felt a lot better when we moved to a 3 week sprint.

The Scrum Owner role is critical. He or she needs to keep all the meetings very well focussed. The meetings can become very wasteful if they are allowed to drift.

On this project I've been introduced to Behaviour Driven Development using JBehave and will use BDD on any project that I can as it has made developing and refactoring the system infinitely easier. Being able to form a contract in English with the customer for the functionality and automatically regression test it brings to the integrated system what Unit Testing did for individual classes.

Speaking of Unit Testing, I've been doing purer and purer Test Driven Development. I'm don't think I'll ever be a purist as I find that sometimes you need to do exploratory programming.

I do find it sensible to limit the scope of my firm estimating to just the upcoming Sprint; I'm fed up with being forced to give estimates for a poorly understood system 6 months or a year in advance. I also like the fact that Scrum works so well in the face of changing requirements. There has been no project I've ever worked on without chanign requirements and scope creep and Scrum keeps the effects of those changes very visible to the stakeholders.

As the project moves towards production I'll blog again on how well Scrum has worked.

December 09, 2008

Rules with Jess

As part of my recent contract with I've been working on a project that is likely to have a large rulebase requiring frequent change. As a result we've decided to use a rules engine to manage them. We won't necessarily be allowing real time changes to rules, but we may well allow some changes between releases.

Jess was selected as it is already used within the organisation. It's not open source but meets the JSR-94 specification. It's based on the Rete algorithm and uses a CLIPS based language to define the rules. (CLIPS is a derivative of LISP so be ready for a lot of brackets...).

I have to say that it has taken me a while to get to grips with Jess as, for whatever reason, I ended up with the work of integrating it with our codebase. Trying to write the rules was incredibly difficult until I had a key realisation: Jess is not object oriented.

In fact the data (facts in Jess-speak) needs to be presented in a relational manner. In effect instead of references you need to create Primary and Foreign keys. Then you can create rules that navigate graphs of facts relatively easily. In effect you can start to leverage your experience with relational databases, the syntax is different but the thinking abou relationships is the same.

Once I had that realisation I produced a set of 'Fact' classes that present the Object Oriented data model in a relational manner. In order to bind the facts I define Jess templates with meaningful names from the 'Fact' classes and then use the Rete.addAsTemplate method to add the objects representing the data as Facts to Jess.

Jess is now much easier to work with.

December 02, 2008

Are We Building a Biplane or a Jumbo Jet?

This is a question that I nowadays find myself asking very frequently; of myself and of my end clients. It's an important question as it frames the scope of the piece of work. The very technical versions of this question ('What percent Disaster Recovery do we need?' or 'What percent up time do we want to guaranteer?') tend to cause non-technical people to boggle and not come up with any useful answers.

My technique nowadays tends to lead people gently into the whole topic.

The analogy of building a Biplane or a Jumbo Jet is a good one and can be extended quite a long way. You can get across the English Channel safely in both, but you could only take a few people in a Biplane and the safety levels are not as good as in a Jumbo. A Jumbo tends to be faster but limited in where it can land, it tends to be more robust but costs much more to build and maintain. I'd happily cross the Atlantic in a Jumbo, but would be much less happy in a Biplane.

Once you've used this analogy to lead the client gently into thinking about the robustness and availability of the system, I then tend to get them thinking about the cost implications of the various levels of Service that the system could be offering. Point out that 99.999% availability would cost several tens of millions and do they really want to spend that much to build a system that would guarantee only a few minutes of downtime in a year? When you frame it that way they tend to become more amenable to 99.9% or less availability. It is also worth coming prepared with the staffing costs for a 24/7 system, they often realise that the system is only needed 5 days a week and for only 12 hours a day...

Until you know the Disaster Recovery and Availability requirements for a system you can't architect or design it. However you must be careful in asking the right questions so that your clients think thoroughly about what they need to be. 

September 22, 2008

Recycling Factory Pattern.

I've been thinking a little about patterns recently as a result of having to interview far too many people in the past week. I've been asking them to tell me about patterns and their relationships.

It's been interesting hearing people talk about the 'evil Singleton' pattern, get muddled over the Factory pattern and refer to the Pool pattern.

The thing that these patterns all have in common is that they provide a mechanism for accessing an object for immediate use,

These patterns all feel like special cases of a more general pattern. More specifically they are implementation patterns of that general pattern. What makes the Singleton evil is not that one instance exists in the JVM but that dependent code knows that a single instance exists in the JVM.

I'm thinking that there is a more general pattern for managing objects that includes a get instance method and a return instance method. The dependent code on this pattern becomes responsible for getting an object, using it and returning it (though a callback could be used too). The dependent code then becomes less aware of whether it is working with an object from a pool, a factory or a singleton and just gets on with using the object. The implementation of the pattern can be switched to any implementation pattern at any time according to need.

This pattern could be called the Recycling Factory Pattern.

On a more general basis this leads me to think that patterns need to be classified into whether they are implementation patterns or something else (can't think of a name yet).

August 26, 2008

Distributed Transactions

One of the hardest things to design in distributed systems is the transactional behaviour.

I am perpetually surprised at the naivety with which people approach this set of problems. There does not seem to be any formalism to approaching this kind of design. What I do is deliberately take as step back and design the 'transactional architecture':
  1. Identify parts of the architecture that may participate in distributed transactions; components with storage that will be changed irrevocably as a result of a transaction. The obvious ones are databases and message queues which are themselves transactionally aware but one must also take into account non-transactional components such as the file systems and LDAP servers that may be changed in the scope of a transaction.
  2. Document the expected transactional behaviours involving these components.
  3. Design transactional schemes that meet the expected behaviours.
At each step I revisit the designs of the system based.
  1. Should I really be changing this component? I may stop using the file system for storage and instead move it into a database table.
  2. Does the expected transactional behaviour introduce circular dependencies between components? Is ACID-ity expected between two widely divergent transactional domains?  I may move whole entities from one component to another.
  3. If there is no way to implement the behaviour can I make some change to produce the same business behaviour but avoid the implementation problems.
In general I believe that using two-phase commits (XA and the like) is the sign of a poorly designed solution architecture as they are only necessary if the system requires 'immediate' consistency between two or more transactional domains. I will use other transactional management techniques such as idempotency and compensating transactions which provide 'eventual' consistency not 'immediate'.

July 30, 2008

An Abstract View of Documentation

I've been thinking a little about the documentation of a system.

I think that there are two sorts:
  • Descriptive
  • Transformative
Descriptive documentation describes a system at a given state of time.

Transformative documentation describes the transformation of a system as a result of change.

Transformative documentation is only maintained for the duration of the change and is thereafter left as a record of the change. The work defined by transformative documentation results in change to the descriptive documentation.

Descriptive documentation is maintained for the lifetime of the system and is changed as the system is changed.

I think that by classifying documentation in this way one can start to define what documentation is required.

Ideally descriptive documentation should provide a clear, succinct and easily maintainable view of the system in question. Transformative documentation needs to clearly define the necessary changes and only be maintained to correct mistakes.

July 21, 2008

Prototyping

I've been working on defining development processes for NHSBT and have been discussing useful forms of prototypes.

The way I see it there are two main classifications:
  • User Interface Prototype
  • Technology Prototype
One typically prototypes to communicate concepts to a user or to prove a technology.

I tend to use two types of User Interface Prototype:
  • Wireframe
  • Ghost Town
A wireframe prototype is used in most projects that have a UI. It is typically a set of drawings (electronic or paper) that show the layout and flow of the UI screens. It is great when you have experienced users who are familiar with a broadly similar application and can visualise the final functionality. It falls down badly when the users are unfamiliar with the type of functionality.

A ghost town prototype fills in the gap. The UI is implemented in actual code as a shell with no actual business logic or persistence. It is called a 'ghost town' because it is like the towns in western shows and films where the entire street is just a set of thin facades held up by beams and boxes. The UI is fully interactive and it makes it very easy for the user to grasp the functionality. Traditionally ghost towns were considered 'throw away'; nowadays with the Model View Controller pattern it is very easy to implement ghost towns where the majority of code can be reused. The only real drawback is that they can give a false impression to the user that the application is finished and ready.

There are also only two technology prototype forms that I use:
  • Proof of Concept
  • Tracer Bullet
The proof of concept is a purely exploratory prototype which is done to investigate a new technology. It follows a simple process of writing code and seeing if it works in the expected way. This prototype can be very dangerous if clear goals and criteria are not established early on as it can become very undirected and any decisions made as a result can become matters of opinion that may not be in the best interests of the wider project or programme. The proof of concept should always be written expecting to throw the code away as the technology being investigated may not be suitable; that does not mean that the code should be written as if it were expected to be thrown away. A poorly written proof of concept can be less than useless in evaluating a technology, also with modern refactoring a successful proof of concept can be turned into production code.

The tracer bullet is used when the choice of technology is well established but the particular technology stack has never been used before. A very thin vertical slice of functionality is selected that touches all the novel integrations and the prototype is written to implement it. This prototype should always be of production quality as the code it produces will define the patterns and standards for the finished application and will normally be the first check in to the development repository.

These are the types of prototype that I use on a regular basis, are there any that I should be using that I have missed?

July 17, 2008

I've got to be careful not to let it grow out of control...

I've been working and thinking about my encrypted zip framework and in the process of refactoring and abstracting I find myself teetering on the brink of creating a grand unified storage framework for Java, where one defines 'entries' and those can be stored to filesystem, database, zip file, etc.

I think that a Grand Unified Storage Framework would not be a bad thing as I have long been frustrated at the Java API differences in writing data to the different storage media, but I really should focus on delivering the encrypted zip framework. I think that I need to keep the GUSF in mind so that whatever I produce for encrypted zip can be rolled in.