A blog about my life, thoughts and work. This blog will consist of programming, philosophy, politics, poetry and anything else that I want to talk about.
March 31, 2011
Update 2 on Neo4J Spatial and British Isles OSM Data import.
March 21, 2011
Neo4J Spatial and Importing the British Isles Open Street Map Data
- org.neo4j:neo4j:1.3.M04
- org.neo4j:neo4j-spatial:0.5-SNAPSHOT
- org.geotools:gt-referencing:2.6.5
- org.geotools:gt-main:2.6.5
- org.geotools:gt-cql:2.6.5
- org.geotools:gt-epsg-hsql:2.6.5
- com.vividsolutions:jts:1.11
September 28, 2010
Further Forays Into the HTML5 Stack - Animation
September 20, 2010
Forays into the HTML5 stack.
- It is significantly more mature as a specification than the HTML 5 Canvas element.
- The HTML 5 canvas element is procedural in its rendering so with a command it renders pixels directly to screen and so does not have a screen graph; the SVG xml elements are the scene graph and are embedded directly in the DOM and so are manipulable using all the tricks that we are familiar with. This also means that a lot of the heavy lifting for re-drawing is done for us.
- Text is more of a first class citizen in SVG and so is easier to make available to Accessible browsers.
- SVG graphics elements are style-able using CSS.
- HTML can also be embedded back into SVG (and manipulated via the DOM) using the foreignObject element
- The default SVG animation framework is built around the SMIL standard which is pretty low level and is not always the easiest to work with though it is very powerful. I'm intending to see how well SVG interacts with the CSS3 animation effects which are much nicer to work with.
- SVG support is present in the latest iteration of all the browsers but is patchy to say the least. The best of the moment is the WebKit family, Firefox is the next best though it has big issues with the text rendering. IE9 claims to be a huge step forwards over IE8 but I have yet to try out the beta and there are question marks over how well it will support SVG animation.
- While its text support is good there is one glaring omission - word wrap. The neatest work-around is to embed an appropriate HTML element in the SVG using foreignObject.
September 15, 2010
My life with BDD...
- Given: The remainder of the sentence defines pre-conditions for the next steps of the scenario.
- When: Defines activity that the system under test is carrying out as part of the test.
- Then: The assertions that the test should be checking.
September 12, 2010
Some New Social Networking Infrastructure
June 27, 2010
The Height of Geekery
Well, here I am at my favourite cafe for breakfast with my iPad. I'm trying the whole blogging from my iPad thing with an app called BlogPress.
It's really interesting to see how modern integrated technology is making it feasible to do things that would have been awkward if not impossible a few years ago. The opportunities in this space are endless.
The excitement about the potential for location based services is well known. Unfortunately the excitement can obscure the real win. Location-less Services. The fact that I can start doing this kind of thing truly independently of my location is what's exciting. There have been some early mis-fires with the excitement of being able to take a laptop with you on holiday. Lugging a laptop though is really pretty hard work. We're on the cusp of being able to access and interact with the "datasphere" anywhere in the world.
This is where reading science fiction is a great advantage. Writers have been exploring how to use this for years. I feel that I am ready to make use of this and build on their thinking to go places that people who have not been reading will never think of.
How about you?
- Posted using BlogPress from my iPad
Location:Cliffe High St,Lewes,United Kingdom
June 17, 2010
Constructor Versus Setter Based Dependency Injection
There seems to be two extreme camps here and no middle ground.
I, as usual, agree with neither extreme.
The simple fact is that my preferred approach is to design classes that meet the original intent of object orientation. That the class is fully usable after construction.
That is not to say that I believe that Constructor injection is the one, true way. Rather, I believe that all mandatory dependencies should be configured in the constructor and optional ones should be satisfied by setter methods. This makes the class much easier to understand. It also means that the constructor does not end up completely overloaded with parameters but neither are you left looking at a a sea of setter methods wondering which ones have to be used.
By using this convention the code becomes more self-documenting.
February 10, 2010
What do I think about Business Analysis?
- Working with the business to understand the business domain.
- Capturing and documenting business requirements.
- Analyse the business domain to highlight areas of inconsistency and identify areas of uncertainty.
- Distill and communicate the business requirements to the implementation team.
October 13, 2009
GPGPU Mandelbrot with OpenCL and Java
I upgraded my MacBook to Snow Leopard and got access to a fairly solid implementation. I say 'fairly' because it exhibits a few interesting quirks... I can pretty reliably get it to throw memory access errors even when I am doing nothing exotic. More about this if I can nail down why it is happening.
I've recently installed ATI Stream SDK 2.0 beta 2 to my 'Windows Beast' only to discover that beta 3 is the only one so far that works with the toolkit that I am using.
At present I am using OpenCL4Java to provide the Java bindings to OpenCL they are developing fast and have the advantage of using JNA so there needs to be no native installs past the OpenCL drivers.
It has proved both easier and harder than I expected - I hadn't realised how much C I had forgotten, but then again the syntactical similarities with Java made it easier for me to get to grips with.
I walked through the various examples and tinkered with them until I had at least a basic grip. For my first program I decided to use a nearly perfect algorithm for scaling on multiple threads, the Mandelbrot set. Pseudo code for the algorithm is widely disseminated but to be honest the OpenCL implementation is clear enough. I'll present my take here and with it I will highlight some of the interesting features of OpenCL.
__kernel mandelbrot(First thing I'll highlight is the fact that I've only non-pixel specific parameters - all pixel specific variables are calculated in the OpenCL program. At present I'm only using floats - the implementations have yet to support doubles.
const float deltaReal,
const float deltaImaginary,
const float realMin,
const float imaginaryMin,
const unsigned int maxIter,
const unsigned int magicNumber,
const unsigned int hRes,
__global int* outputi
)
{
int xId = get_global_id(0);
int yId = get_global_id(1);
float realPos = realMin + (xId * deltaReal);
float imaginaryPos = imaginaryMin + (yId * deltaImaginary);
float real = realPos;
float imaginary = imaginaryPos;
float realSquared = real * real;
float imaginarySquared = imaginary * imaginary;
int iter = 0;
while ( (iter < maxIter) && ((realSquared + imaginarySquared) < magicNumber) )
{
imaginary = (2 * (real * imaginary)) + imaginaryPos;
real = realSquared - imaginarySquared + realPos;
realSquared = real * real;
imaginarySquared = imaginary * imaginary;
iter++;
}
if(iter >= maxIter){
iter = 0;
}
outputi[(yId * hRes) + xId] = iter;
}
The coordinates of the pixel / work group are retrieved using the OpenCL specific function : get_global_id(int dimension). Currently Open CL supports 1,2 and 3 dimensional coordinate spaces.
The OpenCL language is mainly derived from C99 and should be pretty easy for C/C++/Java developers to read. The main difference is exclusions - a lot of the default libraries and some control of flow instructions. There are a few additions such as the work group / unit functions and the '__kernel' keyword.
The OpenCL4Java source code is pretty simple:
package bbbob.gparallel.mandelbrot;
import com.nativelibs4java.opencl.*;
import static com.nativelibs4java.opencl.OpenCL4Java.*;
import com.nativelibs4java.util.NIOUtils;
import javax.imageio.ImageWriter;
import javax.imageio.ImageIO;
import javax.imageio.stream.ImageOutputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.io.IOException;
import java.io.File;
import java.nio.IntBuffer;
import java.awt.image.BufferedImage;
public class Mandelbrot {
//boundary of view on mandelbrot set
public static void main(String[] args) throws IOException, CLBuildException {
//Setup variables for parameters
//Boundaries
float realMin = -2.25f; //-0.19920f; // -2.25
float realMax = 0.75f; //-0.12954f; // 0.75
float imaginaryMin = -1.5f; //1.01480f; // -1.5
float imaginaryMax = 1.5f; //1.06707f; // 1.5
//Resolution
int realResolution = 640; // TODO validate against device capabilities
int imaginaryResolution = 640;
//The maximum iterations to perform before returning and assigning 0 to a pixel (infinity)
int maxIter = 64;
//TODO describe what this number means...
int magicNumber = 4;
//Derive the distance in imaginary / real coordinates between adjacent pixels of the image.
float deltaReal = (realMax - realMin) / (realResolution-1);
float deltaImaginary = (imaginaryMax - imaginaryMin) / (imaginaryResolution-1);
//Setup output buffer
int size = realResolution * imaginaryResolution;
IntBuffer results = NIOUtils.directInts(size);
//TODO use an image object directly.
//CL.clCreateImage2D(context.get(), 0, OpenCLLibrary);
//TODO set up a Float4 array in order to be able to provide a colour map.
//This depends on whether we will be able to pass in a Float4 array as an argument in the future.
//Read the source file.
String src = readFully(
new InputStreamReader(Mandelbrot.class.getResourceAsStream("opencl/mandelbrot.cl")),
new StringBuilder()
).toString();
buildAndExecuteKernel(realMin, imaginaryMin, realResolution, imaginaryResolution, maxIter, magicNumber,
deltaReal, deltaImaginary, results, src);
outputResults(realResolution, imaginaryResolution, results);
}
private static void outputResults(int realResolution, int imaginaryResolution,
IntBuffer results) {
int[] outputResults = new int[realResolution * imaginaryResolution];
results.get(outputResults);
BufferedImage image = new BufferedImage(realResolution, imaginaryResolution, BufferedImage.TYPE_INT_RGB);
for(int y = 0; y < imaginaryResolution; y++){
int rowPos = y * imaginaryResolution;
for(int x = 0; x < realResolution; x++){
image.setRGB(x,y,outputResults[rowPos + x] * 32 );
}
}
try {
ImageWriter writer = ImageIO.getImageWritersByFormatName("gif").next();
ImageOutputStream stream = ImageIO.createImageOutputStream(new File("test.gif"));
writer.setOutput(stream);
writer.write(image);
} catch (IOException e) {
e.printStackTrace(); //To change body of catch statement use File | Settings | File Templates.
}
// for (int i = 0; i < outputResults.length; i++) {
// if((i % realResolution) == 0 ){
// System.out.print("\n");
// }
// int outputResult = outputResults[i];
// if(outputResult == 0){
// System.out.print("0");
// }else{
// System.out.print("" + outputResult % 10);
// }
// }
}
private static void buildAndExecuteKernel(float realMin, float imaginaryMin, int realResolution,
int imaginaryResolution, int maxIter, int magicNumber, float deltaReal,
float deltaImaginary, IntBuffer results, String src) throws CLBuildException {
//TODO build some intelligence into the mechanism for pulling out platforms and devices.
CLPlatform[] platforms = listPlatforms();
CLDevice[] devices = platforms[0].listGPUDevices(false);
//Create a context and program using the devices discovered.
CLContext context = platforms[0].createContext(devices);
CLProgram program = context.createProgram(src).build();
//Create a kernel instance from the mandelbrot kernel, passing in parameters.
CLKernel kernel = program.createKernel(
"mandelbrot",
deltaReal,
deltaImaginary,
realMin,
imaginaryMin,
maxIter,
magicNumber,
realResolution,
context.createIntBuffer(CLMem.Usage.Output, results, false)
);
//Enqueue and complete work using a 2D range of work groups corrsponding to individual pizels in the set.
//The work groups are 1x1 in size and their range is defined by the desired resolution. This corresponds
//to one device thread per pixel.
CLQueue queue = context.createDefaultQueue();
kernel.enqueueNDRange(queue, new int[]{realResolution, imaginaryResolution}, new int[]{1,1});
queue.finish();
}
public static CharSequence readFully(Reader reader, StringBuilder builder) throws IOException {
char[] buffer = new char[8192];
for (int readLen = reader.read(buffer); readLen >= 0; readLen = reader.read(buffer)){
builder.append(buffer, 0, readLen);
}
return builder;
}
}
I'm finding it pretty easy to work with OpenCL - now I need to identify some more complex problems to solve.
May 07, 2009
Balance in Business and Technical Architectures
The cycle of centralisation and de-centralisation.
This seems to be a very disruptive and expensive cycle.. The time and effort wasted on reorganising must detract from the company bottom line. I've spent some time trying to work out why this cycle happens and how to resolve it.
Both centralisation and de-centralisation has their positives and their negatives. Centralisation allows the rationalisation of processes and resources but increases the rigidity of the organisation, reducing its ability to respond to changing conditions. De-centralisation can make the company more flexible and agile but risks different segments of the organisation wasting resources through unnecessary competition.
The trouble seems to be that the people at the helm don't seem to understand the interplay of benefits and consequences. This may be due to the way that information about the organisation is presented to them. There is a belief that only simple messages can be delivered at the executive level and so they hear stark messages like 'We're too centralised, we must de-centralise' or 'We're too de-centralised we must de-centralise'. They aren't told the detail and texture.
When they hear these messages, they feel they must act decisively.
It may also be that the hardest thing in business is the same as the hardest thing in politics: Be seen to do nothing. Perhaps the only way out of this cycle is for the bosses to make smaller, more considered changes but to do that takes better communications and finer filters.
I wonder whether Service Oriented Architecture is an example of a potential business architecture that can be used to break this cycle. It balances by centralising the control of the interfaces between business units and systems, but de-centralises the system implementations. The technologies of the interfaces are centralised and controlled but the technologies of the implementations can be the ones most suited to their requirements.
Of course the SOA model can be centralised when someone imposes the implementation technologies on the business units.
I think that there is a balance that must be struck between centralisation and de-centralisation. When I find a large company that manages it I'll be fascinated to see how they achieve it.
February 10, 2009
Pair Programming Interviews
This was my first set of interviews where I used pair programming as part of the interview process.
It was extremely effective. We used a highly simplified 'story' to test the candidate's design and coding skills. The story defined an UserManager middleware component that was used to register new users given a login and an e-mail address. The inputs were to be validated and an e-mail sent to the e-mail address containing an auto-generated password.
An empty Eclipse or IntelliJ project was provided with Spring, JUnit and all the mocking frameworks available as dependencies. A working Maven 2 build was implemented on the command line. The candidate was asked to provide an implementation of the middleware component and provide interfaces for all the dependencies that this component required.
There were two interviewers with the candidate, one acting as pair and the other as the product owner.
We were looking for a clear understanding of interface-implementation separation, understanding of how unit testing and dependency injection fitted together and a clear understanding of the IDE tooling and the various refactorings.
It was certainly effective. It was surprising how many candidates that seemed technically competent verbally were unable to make any headway with this exercise. A lot of candidates just couldn't handle the unfamiliar interview situation; others really had not learnt the tools of their trade and were unable to use the IDE to speed up their coding; other candidates had claimed TDD experience and were unable to write a decent unit test or use mocks; many candidates were unable to handle the design element, defining interfaces and making clear, logical decisions about contracts and responsibilities.
The successful candidates usually sketched out the domain either as interfaces in code or on paper, they asked good questions of their pair and the product owner and they always were very clear about what testing was needed. They also picked up the fact that the story was incomplete and asked about details of the validation.
A pair programming interview highlights the candidates technical skills but provides great insight into their characters and how they will fit into a team.
December 12, 2008
Thoughts on Scrum
I'm something of a reactionary old architect; I still instinctively like to design a lot up front. I'm fully aware that design still has a part to play in Scrum, but I do feel better with more up-front thought about the problems.
I've been surprised at how heavyweight the Sprint meetings feel (Planning, Review and Retrospective). We were running a 2 week sprint and it felt like we spent far too much time in these meetings, it felt a lot better when we moved to a 3 week sprint.
The Scrum Owner role is critical. He or she needs to keep all the meetings very well focussed. The meetings can become very wasteful if they are allowed to drift.
On this project I've been introduced to Behaviour Driven Development using JBehave and will use BDD on any project that I can as it has made developing and refactoring the system infinitely easier. Being able to form a contract in English with the customer for the functionality and automatically regression test it brings to the integrated system what Unit Testing did for individual classes.
Speaking of Unit Testing, I've been doing purer and purer Test Driven Development. I'm don't think I'll ever be a purist as I find that sometimes you need to do exploratory programming.
I do find it sensible to limit the scope of my firm estimating to just the upcoming Sprint; I'm fed up with being forced to give estimates for a poorly understood system 6 months or a year in advance. I also like the fact that Scrum works so well in the face of changing requirements. There has been no project I've ever worked on without chanign requirements and scope creep and Scrum keeps the effects of those changes very visible to the stakeholders.
As the project moves towards production I'll blog again on how well Scrum has worked.
December 09, 2008
Rules with Jess
Jess was selected as it is already used within the organisation. It's not open source but meets the JSR-94 specification. It's based on the Rete algorithm and uses a CLIPS based language to define the rules. (CLIPS is a derivative of LISP so be ready for a lot of brackets...).
I have to say that it has taken me a while to get to grips with Jess as, for whatever reason, I ended up with the work of integrating it with our codebase. Trying to write the rules was incredibly difficult until I had a key realisation: Jess is not object oriented.
In fact the data (facts in Jess-speak) needs to be presented in a relational manner. In effect instead of references you need to create Primary and Foreign keys. Then you can create rules that navigate graphs of facts relatively easily. In effect you can start to leverage your experience with relational databases, the syntax is different but the thinking abou relationships is the same.
Once I had that realisation I produced a set of 'Fact' classes that present the Object Oriented data model in a relational manner. In order to bind the facts I define Jess templates with meaningful names from the 'Fact' classes and then use the Rete.addAsTemplate method to add the objects representing the data as Facts to Jess.
Jess is now much easier to work with.
December 02, 2008
Are We Building a Biplane or a Jumbo Jet?
September 22, 2008
Recycling Factory Pattern.
It's been interesting hearing people talk about the 'evil Singleton' pattern, get muddled over the Factory pattern and refer to the Pool pattern.
The thing that these patterns all have in common is that they provide a mechanism for accessing an object for immediate use,
These patterns all feel like special cases of a more general pattern. More specifically they are implementation patterns of that general pattern. What makes the Singleton evil is not that one instance exists in the JVM but that dependent code knows that a single instance exists in the JVM.
I'm thinking that there is a more general pattern for managing objects that includes a get instance method and a return instance method. The dependent code on this pattern becomes responsible for getting an object, using it and returning it (though a callback could be used too). The dependent code then becomes less aware of whether it is working with an object from a pool, a factory or a singleton and just gets on with using the object. The implementation of the pattern can be switched to any implementation pattern at any time according to need.
This pattern could be called the Recycling Factory Pattern.
On a more general basis this leads me to think that patterns need to be classified into whether they are implementation patterns or something else (can't think of a name yet).
August 26, 2008
Distributed Transactions
- Identify parts of the architecture that may participate in distributed transactions; components with storage that will be changed irrevocably as a result of a transaction. The obvious ones are databases and message queues which are themselves transactionally aware but one must also take into account non-transactional components such as the file systems and LDAP servers that may be changed in the scope of a transaction.
- Document the expected transactional behaviours involving these components.
- Design transactional schemes that meet the expected behaviours.
- Should I really be changing this component? I may stop using the file system for storage and instead move it into a database table.
- Does the expected transactional behaviour introduce circular dependencies between components? Is ACID-ity expected between two widely divergent transactional domains? I may move whole entities from one component to another.
- If there is no way to implement the behaviour can I make some change to produce the same business behaviour but avoid the implementation problems.
July 30, 2008
An Abstract View of Documentation
I think that there are two sorts:
- Descriptive
- Transformative
Transformative documentation describes the transformation of a system as a result of change.
Transformative documentation is only maintained for the duration of the change and is thereafter left as a record of the change. The work defined by transformative documentation results in change to the descriptive documentation.
Descriptive documentation is maintained for the lifetime of the system and is changed as the system is changed.
I think that by classifying documentation in this way one can start to define what documentation is required.
Ideally descriptive documentation should provide a clear, succinct and easily maintainable view of the system in question. Transformative documentation needs to clearly define the necessary changes and only be maintained to correct mistakes.
July 21, 2008
Prototyping
The way I see it there are two main classifications:
- User Interface Prototype
- Technology Prototype
I tend to use two types of User Interface Prototype:
- Wireframe
- Ghost Town
A ghost town prototype fills in the gap. The UI is implemented in actual code as a shell with no actual business logic or persistence. It is called a 'ghost town' because it is like the towns in western shows and films where the entire street is just a set of thin facades held up by beams and boxes. The UI is fully interactive and it makes it very easy for the user to grasp the functionality. Traditionally ghost towns were considered 'throw away'; nowadays with the Model View Controller pattern it is very easy to implement ghost towns where the majority of code can be reused. The only real drawback is that they can give a false impression to the user that the application is finished and ready.
There are also only two technology prototype forms that I use:
- Proof of Concept
- Tracer Bullet
The tracer bullet is used when the choice of technology is well established but the particular technology stack has never been used before. A very thin vertical slice of functionality is selected that touches all the novel integrations and the prototype is written to implement it. This prototype should always be of production quality as the code it produces will define the patterns and standards for the finished application and will normally be the first check in to the development repository.
These are the types of prototype that I use on a regular basis, are there any that I should be using that I have missed?
July 17, 2008
I've got to be careful not to let it grow out of control...
I think that a Grand Unified Storage Framework would not be a bad thing as I have long been frustrated at the Java API differences in writing data to the different storage media, but I really should focus on delivering the encrypted zip framework. I think that I need to keep the GUSF in mind so that whatever I produce for encrypted zip can be rolled in.