October 26, 2009

Octopussies...


Nope - I'm not referring to multiples of a James Bond Film - I'm referring instead to a family tradition.

My mother has for as long as I can remember given an 'Octopussy' to the young children of family and friends. An Octopussy is a friendly octopus crocheted out of wool:

To start the body:
  • Make a chain of 6 or 7 stitches and then form a circle using whatever colour wool you have to hand..
  • Using double crochet make a flat circle about 2 or 3 inches in diameter.
  • Continue adding rows to the circle without increasing the number of stitches and this will form a cup shape.
  • Change to white to continue in order to make a space for the eyes to be embroidered.
  • Continue until you have a nice wide band for the eyes.
Now make 8 thick crochet legs as long as you like:
  • Start trimming the edge with double crochet in a contrasting colour going through the middle of the chain.
  • Continue down to one end and crochet in the long strands.
  • Turn and do a second row of double crochet and turn again to give a 3 row 'foot'.
  • Continue until you reach the end that you started with.
  • Using the loose strands from each of the feet pull them through the flat circle at the base of the body and tie them off to attach the legs.
Finish the body;
  • Embroider the eyes.
  • Change colour from the white to whatever colour you like.
  • Continue with crochet decreasing with each row.
  • When you are close to the end stuff the body with a non-toxic filling (my mother uses a form of foam chips).
  • finish the top of the body and pull final thread through to lose it in the body.
  • Make a long chain with two or three strands of wool.
  • Pull the chain halfway through the top of the Octopussy placing a knot so that it cannot be shifted.
  • The chain can then be tied onto a convenient point so that the Octopussy cannot be thrown out of a pram or cot.
The Octopussy contains only wool, thread and stuffing so there are no small hard parts that can be pulled off and cause a hazard. It is very robust to chewing and is fully washable.

It also has the advantage of leaving the child with a friendly view of Octopi!

October 13, 2009

GPGPU Mandelbrot with OpenCL and Java

I've done a fair amount of learning since I last posted. OpenCL stopped being just a specification and now has a concrete implementation.

I upgraded my MacBook to Snow Leopard and got access to a fairly solid implementation. I say 'fairly' because it exhibits a few interesting quirks... I can pretty reliably get it to throw memory access errors even when I am doing nothing exotic. More about this if I can nail down why it is happening.

I've recently installed ATI Stream SDK 2.0 beta 2 to my 'Windows Beast' only to discover that beta 3 is the only one so far that works with the toolkit that I am using.

At present I am using OpenCL4Java to provide the Java bindings to OpenCL they are developing fast and have the advantage of using JNA so there needs to be no native installs past the OpenCL drivers.

It has proved both easier and harder than I expected - I hadn't realised how much C I had forgotten, but then again the syntactical similarities with Java made it easier for me to get to grips with.

I walked through the various examples and tinkered with them until I had at least a basic grip. For my first program I decided to use a nearly perfect algorithm for scaling on multiple threads, the Mandelbrot set. Pseudo code for the algorithm is widely disseminated but to be honest the OpenCL implementation is clear enough. I'll present my take here and with it I will highlight some of the interesting features of OpenCL.
__kernel mandelbrot(
const float deltaReal,
const float deltaImaginary,
const float realMin,
const float imaginaryMin,
const unsigned int maxIter,
const unsigned int magicNumber,
const unsigned int hRes,
__global int* outputi
)
{
int xId = get_global_id(0);
int yId = get_global_id(1);

float realPos = realMin + (xId * deltaReal);
float imaginaryPos = imaginaryMin + (yId * deltaImaginary);
float real = realPos;
float imaginary = imaginaryPos;
float realSquared = real * real;
float imaginarySquared = imaginary * imaginary;

int iter = 0;
while ( (iter < maxIter) && ((realSquared + imaginarySquared) < magicNumber) )
{
imaginary = (2 * (real * imaginary)) + imaginaryPos;
real = realSquared - imaginarySquared + realPos;
realSquared = real * real;
imaginarySquared = imaginary * imaginary;
iter++;
}
if(iter >= maxIter){
iter = 0;
}
outputi[(yId * hRes) + xId] = iter;
}
First thing I'll highlight is the fact that I've only non-pixel specific parameters - all pixel specific variables are calculated in the OpenCL program. At present I'm only using floats - the implementations have yet to support doubles.

The coordinates of the pixel / work group are retrieved using the OpenCL specific function : get_global_id(int dimension). Currently Open CL supports 1,2 and 3 dimensional coordinate spaces.

The OpenCL language is mainly derived from C99 and should be pretty easy for C/C++/Java developers to read. The main difference is exclusions - a lot of the default libraries and some control of flow instructions. There are a few additions such as the work group / unit functions and the '__kernel' keyword.

The OpenCL4Java source code is pretty simple:
package bbbob.gparallel.mandelbrot;
import com.nativelibs4java.opencl.*;
import static com.nativelibs4java.opencl.OpenCL4Java.*;
import com.nativelibs4java.util.NIOUtils;

import javax.imageio.ImageWriter;
import javax.imageio.ImageIO;
import javax.imageio.stream.ImageOutputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.io.IOException;
import java.io.File;
import java.nio.IntBuffer;
import java.awt.image.BufferedImage;

public class Mandelbrot {
//boundary of view on mandelbrot set

public static void main(String[] args) throws IOException, CLBuildException {

//Setup variables for parameters

//Boundaries
float realMin = -2.25f; //-0.19920f; // -2.25
float realMax = 0.75f; //-0.12954f; // 0.75
float imaginaryMin = -1.5f; //1.01480f; // -1.5
float imaginaryMax = 1.5f; //1.06707f; // 1.5

//Resolution
int realResolution = 640; // TODO validate against device capabilities
int imaginaryResolution = 640;

//The maximum iterations to perform before returning and assigning 0 to a pixel (infinity)
int maxIter = 64;

//TODO describe what this number means...
int magicNumber = 4;

//Derive the distance in imaginary / real coordinates between adjacent pixels of the image.
float deltaReal = (realMax - realMin) / (realResolution-1);
float deltaImaginary = (imaginaryMax - imaginaryMin) / (imaginaryResolution-1);

//Setup output buffer
int size = realResolution * imaginaryResolution;
IntBuffer results = NIOUtils.directInts(size);

//TODO use an image object directly.
//CL.clCreateImage2D(context.get(), 0, OpenCLLibrary);
//TODO set up a Float4 array in order to be able to provide a colour map.
//This depends on whether we will be able to pass in a Float4 array as an argument in the future.

//Read the source file.
String src = readFully(
new InputStreamReader(Mandelbrot.class.getResourceAsStream("opencl/mandelbrot.cl")),
new StringBuilder()
).toString();

buildAndExecuteKernel(realMin, imaginaryMin, realResolution, imaginaryResolution, maxIter, magicNumber,
deltaReal, deltaImaginary, results, src);


outputResults(realResolution, imaginaryResolution, results);
}

private static void outputResults(int realResolution, int imaginaryResolution,
IntBuffer results) {
int[] outputResults = new int[realResolution * imaginaryResolution];

results.get(outputResults);
BufferedImage image = new BufferedImage(realResolution, imaginaryResolution, BufferedImage.TYPE_INT_RGB);
for(int y = 0; y < imaginaryResolution; y++){
int rowPos = y * imaginaryResolution;
for(int x = 0; x < realResolution; x++){
image.setRGB(x,y,outputResults[rowPos + x] * 32 );
}
}

try {
ImageWriter writer = ImageIO.getImageWritersByFormatName("gif").next();
ImageOutputStream stream = ImageIO.createImageOutputStream(new File("test.gif"));
writer.setOutput(stream);
writer.write(image);
} catch (IOException e) {
e.printStackTrace(); //To change body of catch statement use File | Settings | File Templates.
}


// for (int i = 0; i < outputResults.length; i++) {
// if((i % realResolution) == 0 ){
// System.out.print("\n");
// }
// int outputResult = outputResults[i];
// if(outputResult == 0){
// System.out.print("0");
// }else{
// System.out.print("" + outputResult % 10);
// }
// }
}

private static void buildAndExecuteKernel(float realMin, float imaginaryMin, int realResolution,
int imaginaryResolution, int maxIter, int magicNumber, float deltaReal,
float deltaImaginary, IntBuffer results, String src) throws CLBuildException {
//TODO build some intelligence into the mechanism for pulling out platforms and devices.
CLPlatform[] platforms = listPlatforms();
CLDevice[] devices = platforms[0].listGPUDevices(false);

//Create a context and program using the devices discovered.
CLContext context = platforms[0].createContext(devices);
CLProgram program = context.createProgram(src).build();

//Create a kernel instance from the mandelbrot kernel, passing in parameters.
CLKernel kernel = program.createKernel(
"mandelbrot",
deltaReal,
deltaImaginary,
realMin,
imaginaryMin,
maxIter,
magicNumber,
realResolution,
context.createIntBuffer(CLMem.Usage.Output, results, false)
);

//Enqueue and complete work using a 2D range of work groups corrsponding to individual pizels in the set.
//The work groups are 1x1 in size and their range is defined by the desired resolution. This corresponds
//to one device thread per pixel.
CLQueue queue = context.createDefaultQueue();
kernel.enqueueNDRange(queue, new int[]{realResolution, imaginaryResolution}, new int[]{1,1});
queue.finish();
}

public static CharSequence readFully(Reader reader, StringBuilder builder) throws IOException {

char[] buffer = new char[8192];
for (int readLen = reader.read(buffer); readLen >= 0; readLen = reader.read(buffer)){
builder.append(buffer, 0, readLen);
}
return builder;
}

}

I'm finding it pretty easy to work with OpenCL - now I need to identify some more complex problems to solve.

May 07, 2009

Balance in Business and Technical Architectures

I've worked in a number of organisations over the years. These organisations were many and varied but one pattern seemed to be repeated without fail.

The cycle of centralisation and de-centralisation.

This seems to be a very disruptive and expensive cycle.. The time and effort wasted on reorganising must detract from the company bottom line. I've spent some time trying to work out why this cycle happens and how to resolve it.

Both centralisation and de-centralisation has their positives and their negatives. Centralisation allows the rationalisation of processes and resources but increases the rigidity of the organisation, reducing its ability to respond to changing conditions. De-centralisation can make the company more flexible and agile but risks different segments of the organisation wasting resources through unnecessary competition.

The trouble seems to be that the people at the helm don't seem to understand the interplay of benefits and consequences. This may be due to the way that information about the organisation is presented to them. There is a belief that only simple messages can be delivered at the executive level and so they hear stark messages like 'We're too centralised, we must de-centralise' or 'We're too de-centralised we must de-centralise'. They aren't told the detail and texture.

When they hear these messages, they feel they must act decisively.

It may also be that the hardest thing in business is the same as the hardest thing in politics: Be seen to do nothing. Perhaps the only way out of this cycle is for the bosses to make smaller, more considered changes but to do that takes better communications and finer filters.

I wonder whether Service Oriented Architecture is an example of a potential business architecture that can be used to break this cycle. It balances by centralising the control of the interfaces between business units and systems, but de-centralises the system implementations. The technologies of the interfaces are centralised and controlled but the technologies of the implementations can be the ones most suited to their requirements.

Of course the SOA model can be centralised when someone imposes the implementation technologies on the business units.

I think that there is a balance that must be struck between centralisation and de-centralisation. When I find a large company that manages it I'll be fascinated to see how they achieve it.

April 14, 2009

I just thought that I would share with the world a little bit of my family history. I am a descendent of Samuel Boote (1844-1921) a reasonably well known Argentine photographer. One of our cousins kindly loaned me a copy of the genealogy tracing the descendents of Samuel's grandfather, also Samuel Boote (1788-1854). I've scanned the pages as images and felt that it would be a good idea to make them more widely available.

To that end I just want to post on my blog that I have these images available and publish the 3 pages that I feel that I can show without violating data protection laws as they only refer to the long-dead.

If anyone wishes to contact me regarding access to more of the genealogy please leave a comment.



February 10, 2009

Pair Programming Interviews

My current project has undergone an highly protracted round of interviews due in part to the hiring freeze that for some strange reason started in September.

This was my first set of interviews where I used pair programming as part of the interview process.

It was extremely effective. We used a highly simplified 'story' to test the candidate's design and coding skills. The story defined an UserManager middleware component that was used to register new users given a login and an e-mail address. The inputs were to be validated and an e-mail sent to the e-mail address containing an auto-generated password.

An empty Eclipse or IntelliJ project was provided with Spring, JUnit and all the mocking frameworks available as dependencies. A working Maven 2 build was implemented on the command line. The candidate was asked to provide an implementation of the middleware component and provide interfaces for all the dependencies that this component required.

There were two interviewers with the candidate, one acting as pair and the other as the product owner.

We were looking for a clear understanding of interface-implementation separation, understanding of how unit testing and dependency injection fitted together and a clear understanding of the IDE tooling and the various refactorings.

It was certainly effective. It was surprising how many candidates that seemed technically competent verbally were unable to make any headway with this exercise. A lot of candidates just couldn't handle the unfamiliar interview situation; others really had not learnt the tools of their trade and were unable to use the IDE to speed up their coding; other candidates had claimed TDD experience and were unable to write a decent unit test or use mocks; many candidates were unable to handle the design element, defining interfaces and making clear, logical decisions about contracts and responsibilities.

The successful candidates usually sketched out the domain either as interfaces in code or on paper, they asked good questions of their pair and the product owner and they always were very clear about what testing was needed. They also picked up the fact that the story was incomplete and asked about details of the validation.

A pair programming interview highlights the candidates technical skills but provides great insight into their characters and how they will fit into a team.

December 12, 2008

Thoughts on Scrum

The project that I'm on is being run as a pure Scrum project. I'm still forming my thoughts, but I think it's still worth sharing my first impressions.

I'm something of a reactionary old architect; I still instinctively like to design a lot up front. I'm fully aware that design still has a part to play in Scrum, but I do feel better with more up-front thought about the problems.

I've been surprised at how heavyweight the Sprint meetings feel (Planning, Review and Retrospective). We were running a 2 week sprint and it felt like we spent far too much time in these meetings, it felt a lot better when we moved to a 3 week sprint.

The Scrum Owner role is critical. He or she needs to keep all the meetings very well focussed. The meetings can become very wasteful if they are allowed to drift.

On this project I've been introduced to Behaviour Driven Development using JBehave and will use BDD on any project that I can as it has made developing and refactoring the system infinitely easier. Being able to form a contract in English with the customer for the functionality and automatically regression test it brings to the integrated system what Unit Testing did for individual classes.

Speaking of Unit Testing, I've been doing purer and purer Test Driven Development. I'm don't think I'll ever be a purist as I find that sometimes you need to do exploratory programming.

I do find it sensible to limit the scope of my firm estimating to just the upcoming Sprint; I'm fed up with being forced to give estimates for a poorly understood system 6 months or a year in advance. I also like the fact that Scrum works so well in the face of changing requirements. There has been no project I've ever worked on without chanign requirements and scope creep and Scrum keeps the effects of those changes very visible to the stakeholders.

As the project moves towards production I'll blog again on how well Scrum has worked.

December 09, 2008

Rules with Jess

As part of my recent contract with I've been working on a project that is likely to have a large rulebase requiring frequent change. As a result we've decided to use a rules engine to manage them. We won't necessarily be allowing real time changes to rules, but we may well allow some changes between releases.

Jess was selected as it is already used within the organisation. It's not open source but meets the JSR-94 specification. It's based on the Rete algorithm and uses a CLIPS based language to define the rules. (CLIPS is a derivative of LISP so be ready for a lot of brackets...).

I have to say that it has taken me a while to get to grips with Jess as, for whatever reason, I ended up with the work of integrating it with our codebase. Trying to write the rules was incredibly difficult until I had a key realisation: Jess is not object oriented.

In fact the data (facts in Jess-speak) needs to be presented in a relational manner. In effect instead of references you need to create Primary and Foreign keys. Then you can create rules that navigate graphs of facts relatively easily. In effect you can start to leverage your experience with relational databases, the syntax is different but the thinking abou relationships is the same.

Once I had that realisation I produced a set of 'Fact' classes that present the Object Oriented data model in a relational manner. In order to bind the facts I define Jess templates with meaningful names from the 'Fact' classes and then use the Rete.addAsTemplate method to add the objects representing the data as Facts to Jess.

Jess is now much easier to work with.

December 08, 2008

Winter Recipes

In this weather one's mind turns to good sustaining food so here I present a couple of favourite recipes for this kind of weather. One is a proper recipe, the other is an out and out cheat...

The first recipe is derived from a baked mushroom recipe that I first saw on 'The Victorian Kitchen' cooked by Ruth Mott. Her recipe was simply Portobello mushrooms baked with butter salt and pepper for about 20 or 30 minutes depending on the size of the mushrooms.

Delicious though the recipe was, I thought of a few things that I could add to it.

Robert's Portobello Mushrooms.
Ingredients:
  • 4 or 5 large Portobello Mushrooms.
  • A cup of pearl barley.
  • 2 rashers of back bacon per mushroom.
  • A large nob of butter per mushroom plus one.
  • Salt & Pepper.
First cook the pearl barley in boiling water for about 45 minutes to an hour (untill cooked and lightly fluffy). Drain and keep the water if you want to make barley water. Pearl Barley once cooked can be kept in the fridge for a few days or even frozen for a few weeks.

Place a baking tray in an oven at 155 - 175 degrees centigrade to pre-heat.

Remove the stalks from the mushrooms trim them and add to a stock pot if you have one available.

Place the mushrooms with their bottoms up so that they then can be filled with the cooked pearl barley. Place a nob of butter on top of the pearl barley filling the mushrooms and season to taste. Be light with the salt as the bacon will add a lot of flavour.

Allowing two rashers of bacon per mushroom, halve the rashers so that you end up with 4 pieces per mushroom. Lay the first two pieces next to each on top of the mushroom. Then at ninety degrees to the first two pieces lay the next two pieces next to each other. You should end up with the pearl barley mostly covered by the bacon. Of course if you have massive mushrooms you may need to use whole rashers to do the covering.

Take the baking tray out and place the extra nob of butter in the bottom, allowing it to melt and cover the base. Place the filled and topped mushrooms carefully in the baking tray and return it to the oven.

Allow to bake for about 40-45 minutes.

Serve and eat...

I like to put the mushrooms in the oven before I go out for a brisk walk outside so I have a hot snack when I get back.

Cheat's Luxury Scotch Broth
This recipe depends entirely on the availability of certain products in the supermarket. Baxter's Scotch Broth is excellent, but is identifiably not a 'home made' Scotch Broth. Waitrose and a few other supermarkets have been selling a range of foods called 'Look What We've Found'. Basically stews, and casseroles in pouches that just need re-heating; the quality is excellent. One of the recipes that they do is 'Herdwick Mutton Stew'. It is delicious, but I find it a little heavy sometimes.

What I do when I'm feeling extremely lazy and want to have a delicious Scotch Broth or to impress someone is mix a pouch of the mutton stew with a can of the Baxter's soup and heat.

You end up what tastes like a brilliant home made Scotch Broth that will serve 2-3 people.

December 02, 2008

Are We Building a Biplane or a Jumbo Jet?

This is a question that I nowadays find myself asking very frequently; of myself and of my end clients. It's an important question as it frames the scope of the piece of work. The very technical versions of this question ('What percent Disaster Recovery do we need?' or 'What percent up time do we want to guaranteer?') tend to cause non-technical people to boggle and not come up with any useful answers.

My technique nowadays tends to lead people gently into the whole topic.

The analogy of building a Biplane or a Jumbo Jet is a good one and can be extended quite a long way. You can get across the English Channel safely in both, but you could only take a few people in a Biplane and the safety levels are not as good as in a Jumbo. A Jumbo tends to be faster but limited in where it can land, it tends to be more robust but costs much more to build and maintain. I'd happily cross the Atlantic in a Jumbo, but would be much less happy in a Biplane.

Once you've used this analogy to lead the client gently into thinking about the robustness and availability of the system, I then tend to get them thinking about the cost implications of the various levels of Service that the system could be offering. Point out that 99.999% availability would cost several tens of millions and do they really want to spend that much to build a system that would guarantee only a few minutes of downtime in a year? When you frame it that way they tend to become more amenable to 99.9% or less availability. It is also worth coming prepared with the staffing costs for a 24/7 system, they often realise that the system is only needed 5 days a week and for only 12 hours a day...

Until you know the Disaster Recovery and Availability requirements for a system you can't architect or design it. However you must be careful in asking the right questions so that your clients think thoroughly about what they need to be. 

November 26, 2008

Futurama: Bender's Game...

Last night Henry, Neil and Dimple joined me to watch the newest feature length Futurama: Bender's Game.

We all had similarly mixed feelings about it: We laughed at various points during the episode but on balance felt that it was not vintage Futurama.

Afterwards we looked on IMDB and saw a mix of reviews that reflected our thoughts. The review that best expressed my feelings was by 'howTVshouldbe'. His summary was 'It starts as a refreshing character comedy but when the "Game" portion kicks in the movie just curls up and dies under it's own nerdy indulgences'.

The writers evidently had so many jokes for the subject matter that they were unable to self-edit. Good comedy needs pacing so that the audience can react and digest the jokes. Bender's Game threw the jokes in so thick and fast that I barely had time to react to a joke before the next one was cracked. There didn't seem to be any attempt to create any recurring jokes to give the show structure and rhythm either.

I wonder whether the writers were trying to create a layered show that would bear re-watching. Unfortunately the gags were too obviously telegraphed.

Bender's Game for me was a disappointment. The Futurama writers may be better off under the time constraints of a series based episodic format.

September 22, 2008

Recycling Factory Pattern.

I've been thinking a little about patterns recently as a result of having to interview far too many people in the past week. I've been asking them to tell me about patterns and their relationships.

It's been interesting hearing people talk about the 'evil Singleton' pattern, get muddled over the Factory pattern and refer to the Pool pattern.

The thing that these patterns all have in common is that they provide a mechanism for accessing an object for immediate use,

These patterns all feel like special cases of a more general pattern. More specifically they are implementation patterns of that general pattern. What makes the Singleton evil is not that one instance exists in the JVM but that dependent code knows that a single instance exists in the JVM.

I'm thinking that there is a more general pattern for managing objects that includes a get instance method and a return instance method. The dependent code on this pattern becomes responsible for getting an object, using it and returning it (though a callback could be used too). The dependent code then becomes less aware of whether it is working with an object from a pool, a factory or a singleton and just gets on with using the object. The implementation of the pattern can be switched to any implementation pattern at any time according to need.

This pattern could be called the Recycling Factory Pattern.

On a more general basis this leads me to think that patterns need to be classified into whether they are implementation patterns or something else (can't think of a name yet).

August 29, 2008

Motormouths Anonymous

Hello, my name is Robert and I am a recovering motormouth...

Spare a thought for those of use who are born with verbal diarrhea, we have to work very hard not to monopolize any conversation and are typically very uncomfortable with silence.

My whole family seem to be motormouths, some recovering like me, other just keep talking. My father was famous in the family, he was late in learning to talk and my paternal grandmother was convinced that he was spending the rest of his life making up for lost time.

I have it largely under control. Every now and then I catch myself talking too much and deliberately stop talking. I can have actual conversations with people and learn things from them. A few things make me go off the wagon - I had a concussion recently and a friend had to put up with me talking non-stop for 4 hours straight. More everyday, I find myself having to avoid caffeine, a cup of coffee and I am off and 'talking for England'.

Being a motormouth has its advantages, to be one you have to have a brain capable of keeping up but on balance the disadvantages are too great.

August 26, 2008

Distributed Transactions

One of the hardest things to design in distributed systems is the transactional behaviour.

I am perpetually surprised at the naivety with which people approach this set of problems. There does not seem to be any formalism to approaching this kind of design. What I do is deliberately take as step back and design the 'transactional architecture':
  1. Identify parts of the architecture that may participate in distributed transactions; components with storage that will be changed irrevocably as a result of a transaction. The obvious ones are databases and message queues which are themselves transactionally aware but one must also take into account non-transactional components such as the file systems and LDAP servers that may be changed in the scope of a transaction.
  2. Document the expected transactional behaviours involving these components.
  3. Design transactional schemes that meet the expected behaviours.
At each step I revisit the designs of the system based.
  1. Should I really be changing this component? I may stop using the file system for storage and instead move it into a database table.
  2. Does the expected transactional behaviour introduce circular dependencies between components? Is ACID-ity expected between two widely divergent transactional domains?  I may move whole entities from one component to another.
  3. If there is no way to implement the behaviour can I make some change to produce the same business behaviour but avoid the implementation problems.
In general I believe that using two-phase commits (XA and the like) is the sign of a poorly designed solution architecture as they are only necessary if the system requires 'immediate' consistency between two or more transactional domains. I will use other transactional management techniques such as idempotency and compensating transactions which provide 'eventual' consistency not 'immediate'.

August 03, 2008

Carbon-neutral foods

The biggest problem with modern food production is that it is entirely dependent on oil. We've already seen food costs rise as a direct response to rising oil prices.

Many farmers have profitably moved their food production over to organic methods, I wonder how long it will be before we see certified carbon-neutral foods on our supermarket shelves?

I'd certainly be happy to pay a premium initially in the sure and certain knowledge that as oil runs low my food prices will remain stable.

I can see that there would be degrees of carbon-neutrality. The foods that still use oil, but are carbon-offset, the foods that use no oil in production and the farms that use no oil in production or distribution.

All plants should be fertilised using non-oil derived fertilisers. The by-products of plants such as the unused stems and leaves should be put into bio-reactors to produce fuel or fertiliser. All farm animals should be fed on carbon neutral food stuffs and their by-products should be recycled for fuel or fertiliser.

A collective of farms could work together: One specialising in producing plants for food and fertiliser, another producing fuels and yet another producing meat, fuels and fertiliser. Each farm would work to offset the others' needs.

I don't know when this will happen, I only know that it will have to happen.

July 30, 2008

An Abstract View of Documentation

I've been thinking a little about the documentation of a system.

I think that there are two sorts:
  • Descriptive
  • Transformative
Descriptive documentation describes a system at a given state of time.

Transformative documentation describes the transformation of a system as a result of change.

Transformative documentation is only maintained for the duration of the change and is thereafter left as a record of the change. The work defined by transformative documentation results in change to the descriptive documentation.

Descriptive documentation is maintained for the lifetime of the system and is changed as the system is changed.

I think that by classifying documentation in this way one can start to define what documentation is required.

Ideally descriptive documentation should provide a clear, succinct and easily maintainable view of the system in question. Transformative documentation needs to clearly define the necessary changes and only be maintained to correct mistakes.

July 21, 2008

Prototyping

I've been working on defining development processes for NHSBT and have been discussing useful forms of prototypes.

The way I see it there are two main classifications:
  • User Interface Prototype
  • Technology Prototype
One typically prototypes to communicate concepts to a user or to prove a technology.

I tend to use two types of User Interface Prototype:
  • Wireframe
  • Ghost Town
A wireframe prototype is used in most projects that have a UI. It is typically a set of drawings (electronic or paper) that show the layout and flow of the UI screens. It is great when you have experienced users who are familiar with a broadly similar application and can visualise the final functionality. It falls down badly when the users are unfamiliar with the type of functionality.

A ghost town prototype fills in the gap. The UI is implemented in actual code as a shell with no actual business logic or persistence. It is called a 'ghost town' because it is like the towns in western shows and films where the entire street is just a set of thin facades held up by beams and boxes. The UI is fully interactive and it makes it very easy for the user to grasp the functionality. Traditionally ghost towns were considered 'throw away'; nowadays with the Model View Controller pattern it is very easy to implement ghost towns where the majority of code can be reused. The only real drawback is that they can give a false impression to the user that the application is finished and ready.

There are also only two technology prototype forms that I use:
  • Proof of Concept
  • Tracer Bullet
The proof of concept is a purely exploratory prototype which is done to investigate a new technology. It follows a simple process of writing code and seeing if it works in the expected way. This prototype can be very dangerous if clear goals and criteria are not established early on as it can become very undirected and any decisions made as a result can become matters of opinion that may not be in the best interests of the wider project or programme. The proof of concept should always be written expecting to throw the code away as the technology being investigated may not be suitable; that does not mean that the code should be written as if it were expected to be thrown away. A poorly written proof of concept can be less than useless in evaluating a technology, also with modern refactoring a successful proof of concept can be turned into production code.

The tracer bullet is used when the choice of technology is well established but the particular technology stack has never been used before. A very thin vertical slice of functionality is selected that touches all the novel integrations and the prototype is written to implement it. This prototype should always be of production quality as the code it produces will define the patterns and standards for the finished application and will normally be the first check in to the development repository.

These are the types of prototype that I use on a regular basis, are there any that I should be using that I have missed?

July 17, 2008

I've got to be careful not to let it grow out of control...

I've been working and thinking about my encrypted zip framework and in the process of refactoring and abstracting I find myself teetering on the brink of creating a grand unified storage framework for Java, where one defines 'entries' and those can be stored to filesystem, database, zip file, etc.

I think that a Grand Unified Storage Framework would not be a bad thing as I have long been frustrated at the Java API differences in writing data to the different storage media, but I really should focus on delivering the encrypted zip framework. I think that I need to keep the GUSF in mind so that whatever I produce for encrypted zip can be rolled in.

July 16, 2008

Mobile Broadband

Well I've taken the plunge and have got mobile broadband from T-Mobile.

I'm still trying to work out how to get access the the wireless hotspots as I need to send a text using my SIM card to get a username and password.

I think that I'm just going to have to stick the SIM card in my normal phone, send the text and see what happens.

I've been working on the Encrypted Zip on the train to and from work and having access to the internet has been a godsend. Especially when negotiating the deep, dark waters of Java Cryptography. Why are the APIs so overcomplicated? I'm facade-ing a lot away and so may end up with some classes that will be of wider use.

I'm also swearing at the wider Cryptography community. There is a dearth of decent keystore standards and implementations. For those who don't know what a keystore is, it is a type of file where you keep the keys and certificates that you use to encrypt things and to prove your identity online. The only broadly portable 'standard' keystore is PKCS#12 which is widely held up as an example of how not to create a cryptography standard. In my implementation I'll probably have to default to simple password based encryption and allow users to specify better levels of encryption at the cost of reduced cross-platform portability.

On the other hand I have working code and am beginning to see my way clear to a working encrypted zip.

June 30, 2008

Working with OutputStreams

I'm a big fan of the implementations of OutputStreams in Java - a very good example of the decorator pattern. It is great to be able to add capabilities by simple composition.

I'm using various OutputStream implementations to create an API to encrypt files within a Zip file. Unfortunately there are some maddening inconsistencies.

The one that is giving me the biggest headache is the fact that the CipherOutputStream close method closes the underlying output stream. With any other OutputStream implementation one would call the flush method, but for block ciphers, the block is not flushed until close is called. If one is encrypting multiple files to the underlying stream one has to work out some way to close the CipherOutputStream without closing the underlying stream.

What is even more maddening is the fact that DeflaterOutputStream has a method called finish that does exactly what I would need, it completes the deflation without closing the underlying stream.

Why can't CipherOutputStream have a similar method?

April 01, 2008

Tumultuous Times

Been a bit busy of late, landing a role as the Enterprise Architect for NHSBT. As usual I intend to work myself out of the role, but for now my day is filled with learning a whole new business domain and fleshing out my skills with a few new ones.

I'm having to pinch myself, realising that Enterprise Architect is only one step down from board level. My self-belief and arrogance will really be put to the test.