A New Combination of Flavours

Well, my time off has given me back my leisure time and I have been pursuing a number of projects. All of which will be blogged here some time soon. The first and simplest one is a new combination of flavours that I have found that works really, really well.

• Fennel Seed
• Szechuan Pepper
• Garlic (smoked if available)
• Avocado Oil
• Lime Juice

It work brilliantly well either as a marinade for fish, poultry or quorn or as a salad dressing.

First grind about a teaspoons worth of fennel seed with the same amount of the pepper in a pestle and mortar Then chop a large clove of the garlic roughly and grind together with the fennel seeds and pepper. Scrape the micture out in a bowl and add the juice of one lime before pouring in about three times as much avocado oil.

If you're going to use it as a marinade; it can be a good to add a tablespoon of corn flour (corn starch as the USA has it) and beat thoroughly to help it coat better.

As a salad dressing use sparingly as it packs a punch.

The TT has landed.

Well, my car arrived last Friday and in between the fun of running it in I've taken the time to take a couple of photos...

Is Writing Software an Engineering Discipline?

The answer is a very and extremely definite - no.

There have been many learned and wise discussions on this topic. Some people believing that it is an art and other people people believing that it is engineering. I vociferously straddle the fence and state clearly that we are all artisans. We, like the medieval cathedral builders, partake equally of art and science to build our towering edifices.

The simple fact is that for the majority of projects we still can't properly predict the time and effort required to complete them. We are not methodical enough and have yet to build up the corpus of knowledge and expertise to bring true science to our discipline. We are at the most creative phase, where new problems and new solutions are being devised all the time. We are slowly going the engineering path, using patterns and other tools to start to define best practices, but we are not there yet.

There is one major thing that I believe that people working in software can learn from engineers: The fact that all true engineering is a methodical approach to compromise.

Too many people fail to understand and manage the fact that every piece of software is a balance between competing technical, business and personal concerns. We are still artisans because personal opinion still is the best tool in determining the compromise that is reached and so many of the so called 'methodologies' are nothing more than snake oil.

Twice Cooked Lamb

I decided to cook up some leg of lamb the other day and inspired by the way that hams are made I did a two part cooking process: Poaching the lamb before roasting it.

I poached the leg of lamb for about 2 hours in a liquor made up of:

• Enough lamb and vegetable stock to cover the leg.
• 2 cloves of garlic, crushed.
• A cup of good quality red wine vinegar
• A bouquet garni, made up of thyme, bay, marjoram and rosemary
• 10 or so whole black peppercorns

I turned the lamb over about half way.

I then took the lamb out of the poaching liquor and placed it in a baking tray. I poured a little teriyaki and sunflower oil over the lamb before putting it in an hot oven (200c) for about half an hour.

Skim and boil the poaching liquor down to about a third to one half of it's original volume, allow to cool a little before tossing in some chopped mint.

Serve sliced with steamed vegetables and the starch of your choice - I like brown rice or quinoa - pouring the liquor and mint combination over.

The lamb will not be 'pink' in the modern idiom, but will be tender and very delicious.

Binary Manipulation in a String Friendly Language

Well, it's been a while since I blogged; and there's been a good reason since I had nothing worthwhile to say.

I've now got two things that I want to talk about; one is technical and the other is about cooking.

On to the technical. I'll blog the cooking in another entry.

Just recently at work I had cause to revisit an old technique that I hadn't used in years and I thought that it would be worth recording here.

We've been working with small binary fields in the database containing information that needs to be parsed. So far within the system we have kept much of the data-centric heavy lifting within the database but we were having difficulty with the binary information. The database provided a set of procedures for manipulating the binary information, but they were not very efficient. They were designed to work with extremely large pieces of binary information and tended to keep the binary information out of memory and in the database.

In general the database was far more designed to work with strings.

The work around was one that had been used when binary information needed to be manipulated by string-centric languages in the past. We converted the binary into a hexadecimal string. This allowed us to make use of in-memory string datatypes, and all the very efficient string manipulation tools that the database made available to us (but with the offsets doubled!). In this case we were fortunate as the database was able to do direct conversions from hex into numbers and then perform AND operations on the numbers as needed.

If that capability hadn't been there we could have used a further technique: to do a fast AND from hex you just build a look up table. The table provides a look up between the two hex values that you want to AND. For example a look up table that does just one hex digit at at time would look like:

A hexadecimal AND lookup table

HEX - AND	0	1	2	3	4	5	6	7	8	9	A	B	C	D	E	F
0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
1	0	1	0	1	0	1	0	1	0	1	0	1	0	1	0	1
2	0	0	2	2	0	0	2	2	0	0	2	2	0	0	2	2
3	0	1	2	3	0	1	2	3	0	1	2	3	0	1	2	3
4	0	0	0	4	0	0	0	4	0	0	0	4	0	0	0	4
5	0	1	0	1	4	5	4	5	0	1	0	1	4	5	4	5
6	0	0	2	2	4	4	6	6	0	0	2	2	4	4	6	6
7	0	1	2	3	4	5	6	7	0	1	2	3	4	5	6	7
8	0	0	0	0	0	0	0	0	8	8	8	8	8	8	8	8
9	0	1	0	1	0	1	0	1	8	9	8	9	8	9	8	9
A	0	0	2	2	0	0	2	2	8	8	A	A	8	8	A	A
B	0	1	2	3	0	1	2	3	8	9	A	B	8	9	A	B
C	0	0	0	4	0	0	0	4	8	8	8	C	8	8	8	C
D	0	1	0	1	4	5	4	5	8	9	8	9	C	D	C	D
E	0	0	2	2	4	4	6	6	8	8	A	A	C	C	E	E
F	0	1	2	3	4	5	6	7	8	9	A	B	C	D	E	F

If you don't mind sacrificing more memory a 2 hex digit or greater lookup table can be created and of course a lot of the time you can distill the lookup further as you only need to do a single bit mask operation and so you would only need on one axis 1,2,4,8 etcetera.

By turning the binary manipulation into a hexadecimal one, we managed to increase the performance by 500%. Of course for really heavy binary manipulation you can't beat a binary friendly language but this technique can be an useful 'good enough' solution.

Software Mass Production

Every few years I come across a company where some managers come up with the idea of a 'software factory'. The thinking is that mass production techniques and their economies of scale could bring some major cost savings.

Let us look in detail at the mass production of cars. Immediately a vision of a production line comes to mind. Cars being bolted together from thousands of mass produced pieces for very little money.

Unfortunately this vision is where most people's understanding of mass production begins and ends. The reality is far more complex. Reality uses prototyping.

When a manufacturer decides on a new line of cars, many prototypes are produced. Each prototype serving a different purpose. Some are mechanical prototypes used to design and test key new components. Some are aerodynamic prototypes. Others are ergonomic and styling prototypes.

There is one final prototype - the culmination of all that has gone before. This prototype costs tens or hundreds of millions as this has specific jigs, robots, templates and all kinds of other machines made expressly for it.

This prototype is the 'Production Prototype'. Once the millions have been spent and the prototype is complete and tested, you suddenly find that you have a production line ready to make thousands of copies of it for very little money.

Most people when thinking about applying mass production techniques to software do not understand the production prototype.

Every new piece of software is a production prototype. Software's production lines are the scripts and the disk copier.

We have already learnt most of the lessons we need from mass production in industry. In the car industry the production prototype often saves money by using an established platform and reusing existing components. In mass production every effort is made to reduce the cost of the production prototype, in software production we do the same.

Anyone who thinks that they can revolutionise software production by copying the mass production in industry are about twenty years too late. It has already happened.

The Boothby Fogcutter

My Boothby grandfather was an interesting man aside from his exotic pets he was a bit of an inventor. He came up with all kind of things, mainly aeronautic despite being a naval man.

His one invention that has directly impacted my life was the 'Boothby Fogcutter'. Described as a 'catch-up drink', he invented it so that when he arrived at parties late where everyone was several drinks merrier, he could be as merry as them within 10 minutes.

The recipe is as simple as it is fiendish:

• Take a half pint glass.
• Add 1-4 shots of dark naval rum depending on how much catching up you need to do.
• Fill the rest of the glass with cider. I preferred dry but with a sweet cider it is described as tasting like an inferior Tokay.

The real problem with this drink is gauging how much to catch up. It is all too easy to overshoot. It is also worth noting that under no circumstances should this be drunk at any time other than at the start of a session of drinking as I have learnt to my cost.

I introduced the 'Boothby Fogcutter' to my college at Cambridge in my second year and I don't think that Corpus Christi ever recovered. A number of deplorable incidents ensued, usually because the drinkers did not judge the shots of rum correctly or drank it later in the evening after already getting merry.

The incident that sticks most in my memory was at one of Corpus' annual Rugby Club dinners. These were alway boozy affairs.

The routine of the evening was extremely well defined. The club members would meet up in the college bar for a few pints at six o'clock. At around half past seven we would head up to a reception room to make a start on the sherry. At 8 o'clock we would be seated to eat, accompanied by the mandatory fellow of the college who would have to sit in on any function making use of the college facilities. There were invariably three courses, each accompanied by a different wine. The main course was alway a posh variant on steak and chips. After dessert we would start in on the port and after that was finished various people would be sent to the college bar for further supplies of alcohol. At about 10:30 we would pour ourselves out and adjourn to the bar for a few more convivial drinks and, if any of us were still standing, we would see what else would happen.

The year that I introduced the 'Boothby Fogcutter' proceeded along the lines of any other I was pretty well lubricated by the time that the port was finished and when volunteered to get further supplies I decided that this was an opportune moment to introduce the rugby club to a new drink.

I staggered into the bar and somehow managed to convince the bar manager that what I really wanted was 22 'level 2' Boothby Fogcutters. I felt that 2 shots of rum each would be sufficient. Carrying a heavily laden tray I made my way back to the dinner.

Looking back, the major warning sign came when I tried to mount a flight of steps carrying the tray and lost my footing. I fell, but was in that peculiar elevated state that the very drunk have that allows them to keep their containers of alcohol intact. There was a little sloshing but I managed to keep all 22 glasses on the tray and largely full.

Arriving back at the table of drunken rugby players I dispensed my largesse. A couple of people declined and despite my suggestions to the contrary the responsible, senior fellow decided to have two. My last recollections of the evening were of the start of a round of the 'I have never' game where you stand up and drink if you have.

I woke up the following day in a terrible state. I had mixed grain and grape and had had a 'Boothby Fogcutter' halfway through a nights drinking. I had lost my glasses, I was pretty sure that I had been sick given the state of my mouth and chin and had an enormous bruise on my left shoulder.

Over the course of the next couple of days I reconstructed what I had done. I had made it back down to the bar and then gone into the JCR (Junior Combination Room - basically a common room with seating) where I had decided to join in a game of rugby using a rolled up newspaper. My fellow prop forward Tom and a guy named Ed had tackled me and I had gone shoulder first through the closed double doors of the neighbouring television room, taking one of them off their hinges. I then staggered forwards through the assembled viewers to the front of the television room and stood being sick out of the window behind the television for about 5 minutes. At the time I had not realised that there was a bicyle under the window and with uncanny accuracy I had targeted the seat.

After that night I never drank heavily again. The 'Boothby Fogcutter' continued to cause mischief around college and I'm pretty sure continues to cause havoc wherever Corpus people go.

I leave my final thoughts with the fellow of the college who became the first to be up before the Dean for discipline in several decades. I still don't know what he got up to after the second fogcutter.

Smoked Cod and Chips

When I lived in Ireland one of the great pleasures in life was to adjourn to a local chippy called Caffola's and devour a beatifully cooked smoked cod and chips. The cod was only lightly smoked, more to give flavour than preserve and added a delightful high note to a wonderful dish. It was a truly great way to prepare for an evening out at Rocky's followed by Badger Brown's.

I've always loved smoked fish; salmon, mackerel, haddock etc.

I've recently found smoked cod becoming available more widely in the UK so I decided to attempt to recreate the experience.

In consulting various cookery books and looking online I discovered an argumentative group of people who either create or collect frying batter recipes and my goodness there are a lot of them.

In the end I settled on a batter recipe that was a synthesis of several of the others and so here it is:

• 2/3 cup of strong flour
• 1/3 cup of cornflour (For Americans that is cornstarch)
• 1 generous pinch of salt
• 1 spoonful of castor sugar
• 1 cup of good water (mineral or filtered)
• 2 egg whites whipped to soft peaks

Sift the dry ingredient together in a mixing bowl and add the water gradually, mixing until a smooth paste is formed - I would normally expect to have some water left over. Fold a little of the egg whites into the paste to loosen it, fold about half the remaining egg whites in and finally fold the remaining egg whites in.

To batter and cook the smoked cod I cut it into nice fillets, dried them with kitchen towel and coated in a little flour.

I had a wok with combined sunflower and avocado oil ready heated - I tested the temperature by dropping a small piece of batter in and checking that it puffed and crisped within a few seconds.

I passed the prepared cod through the batter and slowly lowered it into the wok so that the batter in the oil had a chance to seal and puff before it came into contact with the base or sides of the wok or other cooking pieces. For a large piece of cod I allowed 12 minutes cooking time and for a smaller one I allowed 9 minutes. The cod floated as it cooked so I turned it over every few minutes to ensure that both sides cooked evenly.

For a first pass I didn't do any chips, I just served it with a herb salad and a large piece of lemon.

It was really quite good. In fact it was bloody delicious.

I'm going to take a little more time on the batter to get it absolutely perfect but so far, so good.

Techniques for working with Temporal Databases.

I've been working recently with a temporal database and have gone back over my blog as a result to see what I've written on the subject.

All that I've written so far is a simple diatribe on temporal modelling.

I think that it is about time that I share a hard won lesson about working with temporal databases in multi-user environments: It is alarming how frequently concurrency can be messed up.

The problem begins with the fact that classical key based uniqueness checks just don't work. In a temporal database you must be able to support many rows referring to the same entity, each row covering a period in the entity's life. This means that any time that you wish to retrieve a view of an entity you either retrieve a set of rows defining the whole life of the entity or you choose a point in time and use a BETWEEN ... AND clause to select a single row indicating the state of the entity at that point in time. Effectively any key has a temporal element. The full key being the identity of the entity, a start date and an end date for the period of time for which the attached information was/is/will be correct.

So far so good, however databases are just not set up for checking this kind of key. They can trivially check the uniqueness of a key - the fact that a given row representing an entity does not have the same combination of identity key and start and end dates. Can you sport the flaw though? What is needed is the ability to check that the row does not have start and end dates that OVERLAP with those of another row for the same identity key. If they do overlap then we end up with two valid sets of data for an entity on a given day - obviously no good.

Keeping up? Then on with the complexity! The first pass attempt at fixing this is to add application or database level programmatic checks that check whether there exists a row that overlaps. That seems fine as far as it goes, however there is a problem - if two users are performing an update at the same time then standard 'transaction isolation' causes a problem. In most databases the default behaviour is that a user can only see committed work done by other users this means that if both users are updating the same entity then the programmatic checks will not see the changes being done by the other user until they commit it and the overlapping rows can still occur.

I've seen a number of bizarre solutions that attempt to solve this, two of the least satisfactory were to either relax transaction isolation so that each user can see what the other is working on (doesn't work in the end) or to have processes that find overlaps and manually resolve them (oh the work!).

The proven solution is to use a sensible locking strategy. The locking strategy can be used to ensure that the programmatic checks are run correctly despite multiple users maintaining the data. You can't use the temporal rows to do this useful locking. Typically I would analyse the data model and either identify non-temporal entities that can be used to lock sets of temporal entities or create my own non-temporal entities for the locking. This kind of non-temporal entity used to lock temporal entities can be referred to as a lifeline/timeline lock. When the lock is obtained you are locking out the entire lifeline of the entity being maintained and preventing other users/processes working on it simultaneously.

A New Car

A couple of weekends ago I finally bit the bullet and put the deposit down on a new car - an Audi TT. I went with the 3.2 litre quattro in Ibis White with a Mineral Grey interior. The salesman (the third that I had dealt with - the other two had left the industry soon after I saw them) was quite enthusiastic as he had been trying to convince his boss that the TT in white would make a nice show car and he would finally have the chance to show him.

I went moderately wild on the extras. iPod attachment, decent stereo, Audi magnetic ride etc. As this is my first ever brand new car I have to admit to lessened self-control. I did avoid things like the electric seats and the self-dipping xenon headlights though. I'm still torn as to which was the most useless option that I chose though: The luggage nets or the extended leather trim.

I'm still debating whether to get the car debadged as I'm not particularly interested in advertising the fact that I've gone for the most expensive model to all and sundry.

What I'm most irritated about is the fact that I may have a fight on my hands when it comes to getting the licence plate that I want BO07 HBY. I checked the DVLA website and it is up for auction 'soon'. I'm not absolutely desperate to get it, but I would be upset if it ended up not with a Boothby but with some obsessed Star Trek fan.

Balancing Ease of Use in Designs

I've long held that in software design and development that the only wrong way of solving a problem is one that does not work. I do however believe that some ways of solving problems are better than others.

In various entries I've talked about the ways I approach trying to arrive at better solutions, some of them were quite abstract. Most of them are ways of evaluating whether one solution is better than another.

In conversation yesterday with Neil, one evaluation came up that I have found very useful: which solution makes the everyday usage easiest?

This evaluation is probably quite familar to consumers of Application Programming Interfaces (APIs) but often does not seem to be familiar to producers of APIs.

When designing something always consider adding a bit of complexity to doing the unusual things in order to make it easier to do the usual things.

Albert the Mongoose

Time for another family history blog entry. Today I want to tell some stories about Albert the mongoose.

Albert was ridiculously tame, he was a particular pet of my father's who had a poachers pocket sewn into many of his coats so that Albert could travel with him. Often, when my father was having a chat with an acquaintance or talking to a shopkeeper, the conversation would grind to a sudden, staring halt as Albert stuck his head out of the coat to see what was going on.

My grandfather was aware that Albert needed plenty of exercise. The favoured form was to tie a brick to a piece of rope and swing it around. Albert would bounce over the brick on each pass with evident enjoyment.

Albert loved people as they were always doing something interesting and would often wander off to see what they were up to. Once shrieking was heard from one of the neighbouring houses and on investigation it was found that Albert had introduced himself to a new cook. The cook did not appreciate the introduction and proceeded to defend herself with what was to hand: A basket of eggs. Albert thought this was wonderful as he not only got to dodge each egg as it was hurled but also had a race to see how much he could eat before the next egg arrived.

Possibly Albert's finest hour was one afternoon as he rested in a sunny spot on the large back lawn. A local tom cat spotted him. This cat was well known, he had caused no end of havoc among the local feline population, had despoiled birds nest for miles around and had caused much angst to the local humans too. The tom cat went into the full stalking mode; body low to the ground, ears pointed and slow, careful movements. Apparently it was a masterpiece of stalking, no noise, barely a shiver of the grasses and always downwind.

After nearly half an hour of stalking, the cat was perfectly positioned to pounce. What the cat had failed to notice was that thoughout all his zig-zag stalking from cover to cover, Albert had managed to always keep his back square to the cat.

With a tail twitch and a wriggle the attack was launched, but when he landed on the patch of ground that was Albert's resting place, the mongoose was no longer there - he was three feet up and dropping perfectly onto the cat's back. In went all of Albert's claws and with evident satisfaction a firm bite was made to the fold of loose skin on the back of the cat's neck.

The cat took off, squalling round and around the garden. As time went by the cat got more and more tired, more and more quiet until finally, panting heavily, he slowed to a staggering walk and stopped by where Albert had been resting. The mongoose let go and daintily jumped down.

The cat staggered from the garden and was apparently never seen again.

Latin is a language....

Much of my programming is informed by my mother's career as a linguist.

She grew up part of an Anglo-Scots family in Argentina and is fluent in both Spanish and English. From an early age she worked hard to expose me to a range of languages through friends. Not just European languages but also any others (including Nepalese, Urdu and others from an old Gurkha officer).

At school I was made to study French, Latin and Ancient Greek.

I did not turn out as she expected - she taught me the analysis and understanding of languages in general but I instead focussed it on English and computer languages.

When dealing with customers it is essential to understand what they are trying to communicate. I became better at this when I fully realised that English is not just one language but a family of dialects and jargons. The way that different groups of people use English can lead to all kinds of misunderstandings. Just think of the different uses of the word 'Monitor':- A computer display, a thread monitor, a school prefect/official, the verb etc.

In many cases I have found that disagreements arise not because of differences about how a system should work but instead because of differences in how the words describing the system are used.

When it comes to computer languages the awareness of language can be very revealing. One of my favourite examples of this is a brilliant speech by Guy Steele: Growing a Language.

I'd recommend that any software developer, designer or architect pay a little more attention to linguistics.

The Quantum Boolean Anti-Pattern in XML

One of my pet peeves is the Quantum Boolean anti-pattern.

'What the heck is that?' I hear you cry!

Well it is where some bright spark defines an element or attribute as boolean (true or false) but it may or may not be present (minOccurs="0").

What then happens is that you never get to see false. Either the element or attribute is present and set to true or it is absent and you have to guess that it is implicitly false.

Please decide to set it to boolean and make it mandatory or make it an optional marker element with no sub-elements or attributes.

George the Rock Python.

After some positive responses to the family anecdotes about Dulce de Leche, I though that I might share another one.

My father's family certainly accumulated a fair share of these anecdotes. My grandfather kept some quite interesting pets: George an Indian Rock Python and his old friend Albert the Mongoose.

Albert and George were good friends despite their natural instincts. Many was the time that George would be seen asleep in a sunlit place with Albert asleep on top of him.

Next door to my grandfather lived a retired British India Army colonel (my father always referred to him as a 'Boom-Puna' British India Army Colonel). He and his wife had returned to England and had settled into a contented if liverish life filled mainly by complaining how much worse off they were in England. He was a bit of a drinker; my father at the age of 12 delivered a message to him and was offered a gin and tonic.

Soon after moving in, my grandfather (who had an evil sense of humour) bribed my father to take George and deposit him on the colonel's front lawn. He was told to hide in the hedge to keep an eye on George and to report the reaction back faithfully.

With a certain amount of effort (as my father was only a boy at the time and George was a big snake - 12 feet or so), George was deposited on the lawn. My father retired to the nearby privet hedge and, having made himself comfortable, awaited the colonel's arrival with interest.

Soon after the colonel strode out of his front door to take his morning constitutional and found himself confronted by George who was just stirring sluggishly in the morning sun.

He froze for nearly a minute and paled dramatically before rushing back into the house crying 'Maud! Maud draw a cold bath! I've got the DTs again!'

One Way to Reduce the Risk of Social Engineering Attacks

In my day to day job I work with a number of secure systems. A number of them are entirely internal, but the others are customer facing. There are also certain operational procedures which are instigated at a customer's behest (the customer phones up and asks - I've been taken to task for my sesquipedalian posts...).

The problem comes when the customer is asked to provide identification. The classical approach is via some form of shared secret - you know the kind of thing - a maiden name or some other personal piece of information.

I've never been comfortable with this kind of security scheme as it relies on security through obscurity in the first instance and on the honesty of the company staff in the second. Anyone can now trawl through public records to find out the kind of personal information used for this security scheme and increasing numbers of cases of personal information being stolen at call centers are coming to light.

Many of the systems I work with have dual channels for customer interaction:

• A secure online channel - for day to day customer use of the system.
• A phone number - for out of the ordinary requests.

To reduce the risk of Social Engineering attacks I advocate using a scheme that provides authentication for the phone calls through the secure online channel.

The steps of the scheme work like this:

1. The customer decides that he or she needs to make a call.
2. The customer accesses a function on the secure online channel to enter a password for that call.
3. The customer enters the password.
4. The system digests the password and stores it.
5. The customer makes a phone call.
6. The customer representative requests the password and enters it on his system.
7. The system digests this password and compares it with the stored one.
8. If they do not match then the customer has not been authenticated and the representative politely handles this.
9. If they do match then the customer is authenticated and the call proceeds.
10. The system clears the digest preparatory for a new support call.

This scheme does rely on a couple of things:

• The presence of the online channel
• The fact that the customer does not mind the added complexity versus the increased security (this can be handled by clear public relations explaining how this scheme reduces the risk of identity theft etc.)

We are building a chain of trust here. We base the authentication of the customer over the phone on the authentication of the customer over the online channel.

The security of this system can be fine tuned in a number of ways:

• A more complex password (length, character types etc.) can increase security but at the cost of increasing the complexity of the customer interaction with the system and the representative - try explaining !B00thbY over the phone. The fact that the password is intended to be only used once should allow for a simpler password.
• Store the digests of previous passwords and introduce a clear reuse policy. This introduces a marginal increase in customer complexity for a good increase in security.
• Allow (limited) reuse of passwords. This reduces the security but can be very useful in a scenario where support is needed when the online channel is down. The user can generate a password in advance for, say, 5 uses after which time one would hope that the online channel is back and a new password can be generated. Further modifications can come in the event of an extended outage where the password use could be extended.

This security scheme produces a limited shared secret in a relatively simple and secure way and neatly eliminates the use of most personal information from customer interactions. I hope that I have illustrated not only the scheme but the compromises involved.

Mock Gravy

I think that it's time for another culinary discussion.

I love gravy. To me gravy is one of the great culinary delights and adds immeasurably to the right dishes.

The trouble is that I hate all instant gravies and have never felt that gravy recipes that did not involve the juices from roasting were really gravy.

In the end I resolved to come up with a way of making a good 'mock' gravy. It turns out that with the right tweaks it can make a really superior vegetarian gravy.

The fundamental idea is simple. To emulate the roasting process and produce the same flavours.

To do this I put oil (as a substitute for fat) in the bottom of a pan. Depending on whether I am trying to make a chickem, pork or beef 'mock' gravy, I use more or less oil.

To the oil I add a tablespoon of vegetable bouillon and two tablespoons of the appropriate other bouillon. I add a tablespoon of soy sauce, a drop of mushroom ketchup and a couple of tablespoons of lemon or lime juice.

You can optionally add some of the herbs that you would use for roasting, a little garlic and/or onion powder. If I'm making a beef gravy I will often also add a teaspoon of beef Bovril.

The next job is to heat the ingredients in the pan over a medium heat, stirring frequently. The intent is to recreate the process that in the bottom of a roasting tin.

The oil will stay somewhat separate from the other ingredients. You are looking to end up with the oil separated from a thick, glutinous liquor after cooking the ingredients. This should take a few minutes only. If you end up with a gritty mixture that will generally mean that the ingredients have been overcooked.

Now add enough flour to absorb the oil, mixing thoroughly.

Cook the flour and oil mixture for a minute or two. You want the flour cooked but not burnt so stir it continuously. The pan should now look to contain very dark brown bread crumbs.

Add 100-150 ml of cold water to the pan and stir hard. The flour will absorb the water and start to come together.

Now start adding the stock to the mixture a 50 ml or so at a time. Each time stir until the mixture (roux) has become smooth again and there are no more large lumps. The pan remains on the heat throughout this process.

When enough stock has been added the roux will have become runny and the rest of the stock can be addded.

Bring the pan back to the boil and taste. Season if desired. One can either add gravy granules/cubes or boil the gravy down (stirring frequently) until the desired intensity of flavour is achieved. Other flavours such as redcurrant jelly can be added at this time.

The gravy is now ready for serving. Pour it over a large helping of bangers & mash, a bowl of couscous or anything else that takes your fancy.

To make the vegetarian version simply use all vegetable bouillon and stock. This recipe scales up very well.

Ingredients

• Oil (vegetable, olive or something similar)
• Bouillon (I tend to use the Knorr 'Touch of Taste' bouillons)
• Soy Sauce
• Mushroom Ketchup
• Lemon or Lime Juice
• Roasting Herbs (Optional - Rosemary, sage, thyme etc.)
• Garlic Powder (Optional)
• Onion Powder (Optional)
• Beef Bovril (Optional)
• Flour
• 100 - 150 ml Cold Water
• 1 Litre of Good Stock
• Salt and Pepper

Good Threaded Code 2

I've been writing concurrent code in Java for some years now and like to believe that I've learnt a few things in the process.

In Java all objects have a mutex and monitor associated with them and so any object can be used to provide concurrency control and signalling. Java allows code to synchronize on any objects to which it has access, this includes the object declaring the code. In fact Java positively encourages you to synchronize on the declaring object by providing synchronized methods and allowing synchronized(this).
For example:

public class ConcurrentClass
{
    public void doSomething() 
    {
        //do something thread safe...
        ...
        //do something not thread safe...
        synchronized(this)
        {
            ...
        }
    }

    public synchronized void doSomethingElse() 
    {
        //do something not thread safe...
        ...
    }
}

When you call doSomething() or doSomethingElse() the mutex of the ConcurrentClass instance is only held for the duration of the synchronized block or the synchronized method.

I believe that this approach violates encapsulation. The mutex used to control the internal concurrency issues of the ConcurrentClass instance is available to other classes and so other classes can participate in the internal concurrency concerns of the ConcurrentClass instance. At first sight this might not seem a bad thing but just stop and think for a second. This means that concurrency control for a specific resource can be spread across several classes. Rather than debugging one class you may have to debug many.

I would far rather write something like:

public class ConcurrentClass 
{
    private Object mutex = new Object();
    ...
        
    public void doSomething() 
    {
        //do something thread safe...
        ...
        //do something not thread safe...
        synchronized(mutex)
        {
            ...
        }
    }

    public void doSomethingElse() 
    {
        //do something not thread safe...
        synchronized(mutex)
        {
            ...
        }
    }

}

This means that the mutex and thus the concurrency control is fully encapsulated within the class.

An example where the lack of encapsulation can cause a problem is in java.lang.Thread. I have seen application code that sub-classes java.lang.Thread and uses synchronized methods or synchronize on the thread object. If the application code deadlocks it becomes impossible to use any of the normal methods to control the thread as they are all synchronized on the thread themselves.

Idempotence? What's that?

When I'm architecting enterprise systems I often get that response when I talk about my preferred alternative to using 2 phase commits and XA.

Idempotence is not a particularly difficult concept. I define it as a behaviour that has the same outcome on the second, third, fourth etc. time of trying as it had on the first.

A lot of things in our day to day life are idempotent. Say you are going to the car with some bags. On reaching the car you put the bags down and go to unlock the car; you are then interrupted by a friend calling to you and you have a chat. When you turn back to the car you can't remember if you unlocked it or not so you go to unlock it again. If you had actually already unlocked the car, it will remain unlocked. Unlocking a car is normally idempotent - I'm not answerable for odd car designs or for people that actually lock their cars again when they do this.

This can be extremely useful when you are trying to integrate two disparate transactional systems. Lets call one system A and the other B. If we are doing something in A that needs a change to be applied in B we can make the change to B idempotent.

The process would often work like this.

1. Open a transaction in A.
2. Do the work in A.
3. Open a transaction in B.
4. Do the work in B. Idempotent
   
5. Close the transaction in B.
6. Close the transaction in A.

If a problem occurred in steps 1 to 5, systems A and B would be in a consistent state as both transactions would be rolled back. If a problem occurs at stage 6 then B would have the change while A doesn't.

When the process is re-run successfully A and B will end up in a consistent state as long as the work done at B is idempotent.

This is a much lighter weight mechanism that the 2 phase commit / XA one but it does mean that there is a window in which A and B can be inconsistent. For many systems this window may not be a problem in which case I'd recommend the use of idempotence.

Another month between posts.

A lot has happened to me in the past month. My cousin James came to visit, I faced eviction due to my landlord's non-payment of mortgage, I moved to a lovely new place in a converted chapel, I went speed dating and have spent an awful lot of time trying to get broadband into my new place so that I can work from home.

This morning I had a good breakthrough with JOGL - working out how to initiate an offscreen pixel buffer without needing a full blown AWT or Swing GL canvas.

With my life calming down a little (off to see the Two Gallants tomorrow in Brighton), I'll hopefully blog a little more.

Test Driving a New Model Audi TT

I currently drive an X-reg Ford Puma and am now looking for a replacement. I've really enjoyed the Puma, it handles well, has nice pick up and as a 2+2 can carry 4 people over moderate distances. It has swallowed enough camping gear to take me and a friend to music festivals and has regularly slogged around the M25.

However the time has come to think about something else as the Puma is getting on.

What I've been looking has been some kind of successor to the Puma; another sporty coupe that fits well with my life. The trouble is that Ford no longer seems to have cars of that kind available to buy. I'd be forced to choose between one of the Ka or Focus range or end up with a barge.

I ended up casting my eyes further afield and there seemed to be three major choices:
BMW z4 Coupe
Nissan 350Z
Audi TT (New model)

By process of elimination I ended up at the Audi TT, allegedly the handling was massively improved in the new model, it was a 2+2 and can definitely shift. The reviewers had suggested that the 2.0T was nicer handling than the 3.2 quattro, so that was the one that I asked to test drive.

An old friend of mine had asked to come along on the test drive and so, having picked him up a little late, I roared on to the garage.

On arrival we spotted the new and old TT models sat side by side. It was very interesting to compare them. At first blush they looked very similar. The face of the car had obviously changed. The headlights and grille were definitely 'new Audi' and had given it a more aggressive look. The two other obvious changes had occurred at the back of the car with the introduction of a pop-up rear spoiler (my friend and I were both a little dubious - something else to go wrong) and the roofline had shifted back where it met the boot lid.

On further inspection it became clear that the new model was a little wider and a little longer. While most of the distinctive creases and curves remained, they had been subtly altered.

On entering the dealership, I was greeted by our friendly salesman Matt who had cheerfully ignored my request to test the new 2.0T and had lined me up with the 3.2 quattro. At least it had the S-Tronic gearbox so I could happily play with flappy-paddles.

The salesman took us out of the showroom and then out of town with me in the passenger seat and my friend in the back. It was obvious that the +2 seats were not much use at all except for headless people and contortionists. The roofline meant that any adult sitting in the seats had to bend their neck very uncomfortably. It was possible to provide survivable leg room.

Sat in the front passenger seat, I was able to look around the cabin. In basic trim there was quite a lot of basic plastic and, while the build quality was good, I did not much like it. The controls were very well laid out and the seats were extremely comfortable.

The noise in the interior was very pleasant - a nice V6 rumble, pleasantly audible but not overpowering.

Once we had left town I exchanged places with the salesman and prepared myself to drive the car. Fortunately the test drive took in several roads that I had often driven before in a number of cars so I was able to concentrate on the feel of the car and not on the route.

I started off with the S-Tronic box in automatic mode and attempted to pull away. Not having driven an automatic for a while I flubbed the basics - I forgot to put it in drive. Having sorted that out (and remembering to put my foot on the brake) I gently lifted my foot off the brake to see if the car would creep in automatic mode - it did. The indicator stalk on the left was used and I went to pull away. Carefully keeping my left foot out of play I manage a relatively clean pull away and quickly accelerated up to 60. The gearbox was almost unnoticeable, the only sign that gears were being changed was the change in exhaust note.

Rapidly reaching the first corner - a long right hander suitable for 60 miles an hour I simply steered around it. The steering was light but surprisingly communicative. A few hundred yards later I was coming up behind another road user and the road was about to drop away a long way down to a sharp left turn. I deliberately slowed to open up a gap between me and the car in front and let engine braking open up a good gap so that I could come down the hill at speed and take the left hander quite fast. The left hander was dispatched with no fuss. Once again I was coming up behind a small queue of cars so I eased back and followed them. I noticed that it was possible to occasionally wrong-foot the automatic and end up in a less than perfect gear on exiting the corner.

After a little pleasant gentle driving we reached a round about and fortunately the drive took us away from the other cars and onto a lovely half mile of road where I was able to plant my foot and see what would happen. The car took off and I felt that my kidneys were making intimate conversation with the seat. Half way down the road is a right hand bend which rises to a crest half way. The car handled this with ease and dropped down to join the dual carriageway back into town. Joining was no hassle, the only thing that I had to do was make sure that I did not accelerate too much.

On the way back into town I got the salesman to talk me through the gearbox and paddles and I had a pleasant time trying them out. Very neat, very easy. Move the gear stick left to be able to use it to push up and down through the gears, back right for automatic. The paddles worked at any time and could override the choice of the automatic. If I were to get the car I would probably use the paddles almost exclusively.

The final part of the test drive involved me coming up a long hill with a patch of road at the bottom that always manages to unsettle two wheel drive cars. I used the accelerator aggressively and the quattro system worked perfectly. There was no wriggle or squirm and the car accelerated up the hill again. At the top of the hill I got out and sat in the +2 seats, of that the less said the better.

Climbing thankfully out at the showroom I sat down with the salesman and went through the options and colours. The deposit was only to be £1000 and I was sorely tempted to buy then and there. The car was very handsome, the handling impeccable, the engine intoxicating and the gearbox nearly perfect.

I think that I probably shall get a TT despite the cramped rear passenger accomodation. The boot is large and the car is a very good compromise.

I may just need to invest in a guillotine.

Dulce de Leche

Despite my grandparents living in Argentina, my mother was born in Uruguay as they did not want her to be conscripted if she were a boy. As I understand it the Argentines are not always well liked in South America so my mother is often quite firm on the fact that she was born Uruguayan.

Despite largely growing up in England I was exposed to a fair chunk of another culture for which I will always be grateful. A significant part of that is South American cuisine. Dulce de Leche (Milk Jam) is one of the elements of South American cooking that plays no small part in a number of my family's stories.

The recipe for Dulce de Leche is deceptively simple:

• Two cups of milk.
• One cup of sugar.
• A pinch of cream of Tartar.
• Vanilla.

One can use vanilla sugar, good vanilla essence or infuse the vanilla into the milk before cooking the rest.

Simply place all the ingredients into a thick pan over a low heat and cook until the mixture turns a medium caramel brown and thickens, stirring occasionally. This can take anything up to an hour and a half.

The resulting concoction can be eaten with a spoon, used in cake and pastry fillings or just spread on bread.

A rather more simple way of preparing it (not without its own risks) is to boil a can of sweetened condensed milk for 2 and a half hours. Remember to keep an eye on it and top up the water if necessary. Do not open the can in any circumstance until it has had a chance to cool. If the can starts bulging while cooking take it off the heat immediately and stand well back.

The Dulce de Leche produced in this way tends to be a little coarser and less flavoursome.

An english cousin was shown the simple recipe once but unfortunately forgot that it was cooking. 6 hours later the pan had boiled dry and the can had exploded spraying hot metal and hot overcooked Dulce de Leche everywhere. As it was my mother's kitchen and it still smelled of Dulce de Leche 3 months later she was less than pleased.

Another time a very old friend of my mother's was living in the States and started to crave real Dulce de Leche - just like grandmother used to make. She had been using the condensed milk recipe but this time it just wasn't good enough.

She called her Argentine friend in the next town, who after some deliberation said 'No I don't have a recipe for it, but if you take a tin of sweetened condensed milk...'.

My mother's friend then called an Argentine friend in the next county, who said, after some thought, 'Well if you take a tin of sweetened condensed milk...'.

Calls were placed to assorted Argentine friends throughout the Eastern Seaboard. Who without exception replied: 'I don't know the proper recipe but if you take a tin of...'.

My mother's friend not being one to give up without a fight said to herself 'Right, I'll call up the Argentine Embassy in Washington D.C.'.

The operator receiving her call was more than a little surprised by her request and had to reply that she didn't know but if my mother's friend would hold then the operator would ask around in the embassy.

Twenty minutes passed and the operator returned to the call.

'Well I asked everybody, even the Ambassador's chef and the Ambassador's wife and none of them had the recipe, but if you take a tin of...'.

A few weeks after, my mother received a letter from her friend telling this story and my mother was able to send back her own grandmother's recipe as I have recorded above.

Extension Points

This entry is dedicated to Hardev who came up and ever so politely reminded me that it has been some weeks since my last one.

I've been accumulating topics and do have a few things that I want to talk about so here is one.

A classic refrain in the IT business is 'future proofing', a dangerous and terrible thing. All too often I have sat through requirements and design meetings where people sit and try to anticipate all future ways in which the software could be used. If you ever find yourself in such a meeting, either take it over and talk about 'extension points' or run (don't walk) for the nearest exit and wait for the bloated mess that results to collapse in on itself.

An extension point is an abstraction that allows the easy extension of application behaviour by using patterns such as decorator, factory, strategy etc. Unfortunately as I've already discussed many moons ago, abstractions do make it more difficult to comprehend the code; this means that just adding extension points do have a cost. I find that the extension points that are likely to be used in the future are the ones that are 'sympathetic' to the thrust of the functionality.

In the end extension points should fall naturally out of the design process, however they can be used as a good argument to use against the 'future proofers'.

A meeting with an old friend...

I was pleasantly surprised the other day to hear from Ian Hannaford. He wanted to benefit from what little wisdom I could muster. We had a good chat about his new role (thrown in the deep end as usual) and generally had a good old chat. Ian boosted my ego by telling me that he had directed other people to my blog (Hi other people!) and we gossipped about old friends and talked techie.

In the course of our chat, Ian told me about his pattern use but that he found the Gang of Four book a touch dry and academic. He suggested that I could at some stage write something to make it a little more approachable and relevant.

I'm going to think on this a bit and may start writing entries on situations where I have used GOF patterns and why I used them.

I'm going to toddle now and have me a bite of dinner (Teryaki beef, sticky rice and pak choi.).

P.S. I got LocoRoco on the PSP this weekend - at last I am using the PSP as a games machine instead of a mobile video player. LocoRoco is great!

GPGPU with JOGL part 1.

Well I've started tinkering with the GPGPU concept that I mentioned in an earlier post. I've landed on JOGL as it seems to provide me with all the basic functionality that I am going to need and is reasonably cross-platform.

I need to be able to bind a texture (representing an input array of data) into a buffer against which I will run a shader program (the function that I want to perform on the data) and place the outcome of the shader program into a buffer from which I can extract the output array.

I'm struggling with a few handicaps -

• I'm using my PowerBook for this and support for OSX by JOGL is still a bit flakey.
• I've never programmed using GL APIs before so I'm having to learn them from scratch.
• Once I've grasped enought of the APIs, I still have to master at least one shader language.

It's certainly a fun learning curve.

I'm going to struggle on and hopefully I'll have a bitonic sort algorithm implemented before too long.

Apple and Bacon Omelette

I was visiting with some friends the other day, enjoying a very pleasant evening catching up. As I was just heading out the door to go home and cook myself supper, Henry jokingly asked me whether there was anything that he could do with some apples that he had in front of him for his own supper. In short order we had determined that there was some bacon in the fridge and in front of the bemused eyes of his mother, we had rolled up our sleeves and proceeded to cook a new omelette.

We used two to three rashers of salted rindless back bacon, two to three eggs, half a cup of milk and a quarter of a Pink Lady apple per person. The bacon was cut into lardons and the apple was finely diced. The eggs and milk were beaten together with a generous pinch of black pepper. A nob of butter was placed in the bottom of a very large pan and gently melted, we then added the bacon and cooked it through slowly, avoiding frying it to crunchiness, the apples were cooked with the bacon until soft and the egg mixture was added. The low heat was kept on until the eggs were nearly cooked through, the omelette was folded over with a palette knife and then cut into portions and served.

It was extremely delicious.

Facets

I first started thinking about facets after reading an article on java.net nearly two years ago.

I kind of liked the idea that object can gain attributes and behaviours and indeed gain and lose types through it's life cycle.

In normal OO practices where an object is defined having all the types that it is ever going to have (A String is an Object and a CharArray and that is all it will ever be). Facets would allow an object to gain and lose types (A panel in Swing could become a type of Window or a scrollable viewport in another Window). In the real world when a person joins a footbal team they gain the facet of being a footbal player and when they leave the team they lose it. The facet of being a football player brings a team association and a shirt number with it plus a lot of interesting behaviours involving a ball and a pitch.

A number of similar patterns exist such as Mixins, but mixins only really bring behaviour not type information. Unfortunately such a useful word has already been used here.

In fact the idea has already been had by this bunch, and unfortunately they have patented it as US Patent 6,513,157. I did nothing more about facets for quite some time after finding this.

A couple of weeks ago I came across some of the code that I had knocked together to play with facets and decided to bite the bullet and see if I could do something with it. I went through the patent again and realised that the patent covered an implementation of facets that did not need changes to source code. As long as I made sure that what I implemented required changes to existing objects to add the functionality then I would be fine.

I'm finally getting around to completing my implementation and when I've had a chance to explore it I'm going to make it available and blog its uses.

One thing has come out of my work on Facets - Abstract super classes are a snare and a delusion. Take an interface 'Faceted', implement it with a concrete class called 'FacetedSupport'. 'FacetedSupport' can be used as an ancestor of 'Faceted' objects OR it can be used as a support class for objects that are implementing the 'Faceted' interface. If you made it an abstract super class then you would be forced to create a sub-class to make use of its functionality.

Water, Water Everywhere and not a Drop to Drink

I never fail to be irritated by short-sighted stupidity.

I live in the South of England and have been watching all the green spaces where I live being relentlessly developed. I often feel that in order to eliminate the North-South divide, the government is trying to get the entire population to live in the South. Every few months there seems to be a new diktat that several thousand new homes be built in one of the Southern Counties.

One of the problems is that we are in the middle of one of the worst droughts to ever hit the Southern Counties. Why are we moving thousands of people to an area that does not have the infrastructure to support them? It seems that someone is failing their basic planning course - ensure that you have sufficient resource available before you start.

Something that adds heavily to the irony is that, as the South is already very densely populated, most of the decent land has already been used. This means that many of the new developments are occurring on less-suitable land such as flood-plains. Already people are being flooded out of their homes while they can't use their sprinklers.

Will the time come when people are forced to use stand-pipes while their downstairs is under water?

More on Threads and Parallelisation

Two posts in as many days - a brand new record for me.

I talked a little while ago about the future of programming being parallel / multi-core: Good Threaded Code.

Trawling through the one of my favourite web sites I came upon this piece: The Register: Deconstructing databases with Jim Gray. The title is a little misleading, as Mr. Gray actually spends more time talking about the use of GPUs for massively parallel processing and the future of programming in such an environment. He mentions that a common language called 'Accelerator' is already being defined for programming GPUs for parallel processing within a Microsoft environment. Now what I want to know is whether there is similar functionality available for Java? If there isn't I suppose that we'll need to start something in this area.

I want to be able to implement this algorithm and see how well we can get it accelerated: Bitonic Sort.

Process Singleton or Cluster Mutex.

My friend Neil and I have been discussing this pattern for some time. I finally came up with a couple of sober yet pithy names for it as to date we've been calling it the 'Talking Stick Pattern'. This pattern is intended to be used primarily in clustered applications though there is nothing to prevent it being used elsewhere.

The singleton pattern is well known and various attempts have been made to implement it within a cluster. For most instances a node-singleton is sufficient as the singleton pattern is used to prevent excessive memory usage - effectively a pool of one. However on occasions the singleton pattern is used to provide access control to a shared resource that cannot support multiple concurrent accesses or to control the running of a business process that could have issues if run concurrently. Failover is the weakness with implementing a cluster-singleton: You are either dependent on a vendor-specific approach or you have to come up with a solution of your own.

The solution that Neil and I have arrived at is derived from the 'Talking Stick' idea used in group discussions. Rather than have a pell-mell of competing voices, the talking stick is handed to an individual who can then talk, when he is done the talking stick is handed on to the next individual who is allowed to talk. When you do not have the talking stick you are not allowed to talk.

We apply this to a set of objects. Before the object can act, it must get hold of the 'talking stick'. This means that only one object can act at a time. The implementation of the talking stick must be robust in the event of a failure and we have two tried and tested implementations that work.

The first talking stick implementation is a little database dependent. The use of a row-lock in a database that supports either a read-past semantic or like Oracle supports SELECT...FOR UPDATE NOWAIT. When the transaction commits, the lock is released for the next access. If something goes wrong the database transaction rolls back and the row lock becomes available for another transaction.

The second implementation relies on a transactional JMS implementation and a message on a queue is used as the talking stick. An object waits on the queue for a message and when it has it, it is allowed to act. When it has finished it places the message back on the queue for the next object to access. Again, when a failure occurs, the transaction tolls back and the talking stick becomes available again.

Depending on whether it is being used to manage processes or manage access control I would call it a Process Singleton or a Cluster Mutex.

Edit - Gil, a colleague working with me, pointed out that the talking stick is a form of Token.

Are They Running the Country or Running for Re-Election?

This is an huge question, but one which has a simple answer.

I believe that all political parties in Britain are about running for re-election and running the country is a side issue.

This has not always been the case, but the temptation to run for election is one of the great weaknesses of democracy. It is much harder to get re-elected by doing a good job of running the country than by standing around for photo opportunities and telling us what we want to hear. Especially when you consider that your opponents only have to do the photo opportunities. Of course the machiavelllian solution is to get your opponents deeply enmeshed in running the country 'in the interest of non-partisan politics'. The opposition would get credit for not being out of practice in running the country and for any good work they do. You'd increase the availability of competent people for key positions, tie your opponents up and level the field when it came time for elections. With more competent people, the country would benefit overall and you'd get the credit for that (and of course for the good work that your opposition do...).

The trouble is exacerbated by the press. We are told that a strong fifth estate is one of the great pillars of a strong democracy. Unfortunately at the moment our press is lazy and lets the politicians set the agenda. Many stories are handed to them by the current political leadership and by their opposition in their struggle to be elected. It is also in the press' interest to keep politics partisan as the latest bickering is an easy story that sells papers and airtime.

I wonder whether the press could ever be convinced that the real story is in the fact that politicians are running for re-election and not running the country. What would happen if the press were banned from naming individual politicians except when they were doing something wrong? Can democracy ever produce a leadership that truly focusses on running the country and is able to work together with their opponents for the common good? Any other ideas?

Learning: By Rote Vs, Asking Why.

I had a very pleasant lunch yesterday with two Voca colleagues, Roger and Peter. As my lunches tend to always go, we ended up discussing a number of interesting topics, this lunch steered entirely clear of technology.

We discussed counselling, consciousness, upbringing and education. I told Roger and Peter in my typical bombastic manner about a very interesting piece of research I read recently in New Scientist. The article touched on the manners in which humans and our cousins great apes learn from our parents. If you were asked which species is more likely to learn by simple, exact copying of parental activities, which species would you pick?

It turns out that humans are far more likely than any other great ape to shortcut learning by simple imitation. The example that is often quoted is the Mother making a 'pot roast', she cuts off one end of the joint and puts it in the pot beside the main part. Her daughter asks her why she does that and she is forced to reply that she doesn't know, her mother did it that way. So the Mother asks here mother who replies that her own mother always did it that way. The Mother then calls her grandmother who replies that she started doing that because she didn't have a big enough pot.

It appears that humans are extremely likely to learn by rote. It seems to be a shortcut that we have evolved. I believe that this shortcut emerged because of the volume of information that we are forced to learn in order to survive in our culture. If we questioned every single tiny fact, we would never learn enough to sustain the complexity of our culture before we were 50 years old.

That's not to say that questioning is not important. Without 'Why?' we would never have progressed our culture.

I would suggest that the best education a child can get is one that is primarily by rote that does not suppress the desire to question. In effect there is a balance: sufficient rote learning to provide a basic body of knowledge to survive in the world coupled with time spent teaching how to question this body of knowledge.

I believe that you actually have to have a basic level of knowledge before you can decide which questions are worth asking.

When I say 'Culture' I mean it in the more scientific sense of the set of learnt behaviours and knowledge that we as a species learn rather than are born knowing.

Driving Concentration

When I first started driving I had an old MGB Roadster (known as Summer). It was a spartan driving experience the height of luxury being a heater that had only two settings off and BURN YOU TO DEATH!

My driving was probably at its best when I drove Summer, I kept my temper better, drove more defensively and was generally more considerate.

When I started racking up the motorway miles I got a new car. It was a lovely little Ford Puma (known as Sue). It was immensely more reliable and more luxurious.

However it was with Sue that my driving started to disimprove.

My concentration became worse, I failed to anticipate and, when something happened unexpectedly, I was more inclined to lose my temper. I came to realise that something needed to be done, so I thought about it and finally came to a conclusion.

It was the CD/Radio.

In dear old rackety Summer, there was no point to putting a radio in as I wouldn't have been able to hear it above the engine and the noises of the world outside. I was forced to keep all my concentration on my driving and the environment around me. With Sue the radio almost automatically went on and my concentration drifted. I've now taken to turning the radio off and my driving is beginning to return to the quality that it enjoyed before my fall from grace.

The Real Questions.

I've been watching the various manoeuverings going on in parliament to change various aspects of the balance between the legal system and the political system in this country.

When misgivings are raised, the politicians always seem to justify their actions on two bases:

• It is necessary because the current system is unwieldy/expensive.
• Don't you trust us/me?

The real questions that should be asked of the politicians are:

• Whether I trust you or not is irrelevant. Can I trust your successors? Can you guarantee that the 2nd, 3rd or 4th set of politicians to be voted in won't abuse the powers that you are giving them?
• Don't you think it is worth the inconvenience/expense in order to guarantee a reasonable level of freedom from oppression?

I wonder whether we will ever see these questions posed clearly, well and in a situation where the politician is forced to answer.

My Preferred try...finally Semantic

Carrying on a JDBC motif from my previous entry, I want to talk about the use of try...finally blocks in managing resources.

All too often I see null-checking used in finally blocks because this kind of construct is used:

Connection conn = null;
try
{
    conn = ds.getConnection();
    //do some work with the connection.
    ...
}
catch(SQLException sqle)
{
    LOG.error(sqle);
}
finally
{
    if(conn!=null)
    {
        try
        {
            conn.close();
        }
        catch(SQLException sqle)
        {
            LOG.error(sqle);
        }
    }
}

I personally prefer this kind of construct which eliminates the null check at the expense of adding another try..catch block:

try
{
    Connection conn = ds.getConnection();
    try
    {
        //do some work with the connection.
        ...
    }
    finally
    {
        try
        {
            conn.close();
        }
        catch(SQLException sqle)
        {
            LOG.error(sqle);
        }
    }
}
catch(SQLException sqle)
{
    LOG.error(sqle);
}

This is purely stylistic, but I prefer not to have the null check and I do like having the exceptions all towards the end of the unit of code. Do not be tempted to remove the catch from the connection closure as you will mask the original cause of the error. This semantic gets even better if you need to propagate the core SQL exception up to a common handler as it then looks even tidier:

Connection conn = ds.getConnection();
try
{
    //do some work with the connection.
    ...
}
finally
{
    try
    {
        conn.close();
    }
    catch(SQLException sqle)
    {
        LOG.error(sqle);
    }
}
//SQLException gets propagated out to another block of code handling it.

A Connection Pool That Satisfies a Previous Rant

A little while ago I was talking about how many connection pools do not properly satisfy the JDBC specifications (Here.). Many home rolled connection pools do not close the resources associated with a connection before returning it to the pool.

I finally pulled my finger out and decided to have a look for a connection pool that behaved properly. Having pulled the source code for a number of connection pools (and shuddered once or twice when reading), I found a connection pool that actually works properly: Apache Commons-DBCP. The connection pool keeps track of all Statements that are created and closes them and their associated resources when the connection is returned to the pool.

As long as this pool is used, we can avoid all the JDBC clutter in our finally blocks and just close the connection when we're done.

A Persuasive Theory on the Origin of Consciousness

I've been reading a very interesting book these past few days, Julian Jaynes' "The Origin of Consciousness in the Breakdown of the Bicameral Mind".

I've often thought about how we came to be as we are and read upon the subject, this book presents the best theory that I have yet seen.

According to the theory, consciousness is a late comer to the party.

Humanity's evolution of speech and reason did not need consciousness. Speech allowed far larger social groups to be coherent and reason allowed us to cope with the increased complexity of our environments.

As groups increased in size and settlements were formed, humanity needed some means of "sustaining" activities that have no immediate apparent reward.

The mechanism postulated is that of part of a brain that evolved to store the admonitions and advice given by parents and those in authority and to play it back. To a modern mind this playback would appear to be an auditory hallucination. Thus a non-conscious human could persist in preparing a field for planting despite the lack of immediate reward. A voice would be heard, possibly of a parent or of a leader continually reminding them to 'prepare the field for planting'.

Initially this mechanism would have been a simple playback mechanism, but driven by societal advances, it would have increased in complexity and would have been able to synthesise original commands.

The potential ramifications are interesting, the auditory hallucinations based on the voice of a person would have persisted long after the person had died, leading to a belief in life after death and even to worship of ancestors. As the complexity of the mechanism increased, there is no reason not to assume that the voices were limited to known people, and instead could have been interpreted as coming from gods.

It is interesting to note how much of the early writings available to us, talk about how people acted on the promptings of gods or goddesses and how there is very little about personal motivation. Indeed a case can be made that the introspection is a later addition to the text.

In effect early humanity were hallucinating schizophrenics (Bicameral) and much of the structure of ancient societies can be explained by this.

Consciousness only emerged when language gained enough complexity to support a concept of "I" and humanity was forced to evolve mentally by the breakdown of their bicameral societies. Consciousness emerged as an outcome of the integration of the hallucinatory aspects of our minds with the logical/active portions and the ability to conceptualise a self.

I've summarised very briefly here my understanding of a much more complete argument and have only touched on its consequences. I came across this theory first in fiction through the works of Neal Stepenson and other authors, now I can see where they took there inspiration.

I'm certain that this theory is not totally correct, but I would argue that it needs to be considered seriously as we continue to try to arrive at a full understanding of consciousness. It addresses for me how consciousness could arise without a significant physiological change.

The 'self' may be just a construct and this speaks interestingly for future human social and mental evolution.

Change and Stability

I've been giving some thought of late to how groups of people are run, not just at a team level but at an organisational or even a national level.

I've been trying to draw lessons from out politicians past and present. Not many of the lessons have been how to do it well...

One of the major things that have come out of this thought is the dynamic tension between change and stability. Notice the words that I have used, another way of expressing it is the tension between chaos and stasis.

An example of this is the NHS. It is obvious to all politicians that something must be done. The trouble is they never seem to get the something quite right. The old NHS worked as well as it did before the politicians started fiddling because the patients, nurses, doctors and administrators had worked out a modus vivendi. They had discovered ways to work around the grossesr of flaws and it more or less worked. Unfortunately for the NHS it has become a political football, a month doesn't go by without some new announcement. This change is done with the best of intentions but it never gives the participants a chance to settle into the new practices and so the grossest of flaws never get worked around as new ones are introduced the whole time.

Change is essential, especially in this day and age. New technologies, new ideas and new social groups all mean that many of the old ways of doing things do not remain correct. You can't stand still.

But neither do you have to keep running.

I think that the lesson that we and our politicians need to learn is to moderate the rate of change. The real skill comes in introducing changes that only do what is needed and no more. The necessity is to fine-tune the structures we have and only do significant restructuring when absolutely necessary.

This even extends into my domain. It is very tempting to completely re-write systems from the ground up with no regard for the havoc that those changes will cause. All in pursuit of some perception of perfection.

Lasting perfection is unattainable in a dynamic world. All we can ever do is approach it by making sensible, minor changes to proven, stable systems.

The real skill may be in knowing when not to do something...

Good Threaded Code.

I've recently upgraded my main x86 development machine. After a lot of to-ing and fro-ing I decided on an AMD Athlon 64 X2 processor. It has become increasingly clear that the wave of the future is multi-processor / multi-core. Intel and AMD are both talking about 8 core processors before the end of the decade, Sun have just release their Niagara core, Microsoft/IBM are using a 3 core Power PC CPU in the new XBox and Sony/IBM have produced the Cell processor initially going multi-core in the PlayStation 3.

With the advent of consumer level multi-core processors in both home computers and consoles it is becoming clear that we are all going to write more in the way of threaded and concurrent code.

I for one am looking forward to having all that processor power, but as usual the question is how are we, as developers, going to make best use of it?

A part of the solution will be the increasing rise of APIs and frameworks such as that by Doug Lea http://gee.cs.oswego.edu/dl/classes/EDU/oswego/cs/dl/util/concurrent/intro.html for Java. By providing well designed and well tested concurrency code, programming multi-threaded applications will be simplified.

Application Servers will take up part of the burden, but at the usual price of lowered performance and increased cost.

I suspect that certain diagrams will become worth their weight in gold such as UML Activity diagrams.

Lesser known concurreny paradigms such as spin-locks will become more widely understood.

The major expense in using these multi-core systems will be in the synchronization points when different threads / processes exchange data. The skill will be in minimising the impact of the synchronization points.

I'd recommend all developers / designers / architects intending on making any money over the next decade to get their hands on this new generation of SMP machines as soon as possible and start understanding the complexities and opportunities.

Neil negotiating with Meeraj...

Poor Neil, another day in the Voca offices and another day trying to get Meeraj's approval for a code change.

The First Post-Industrial Technology?

I was sat with Neil Ellis one lunchtime last week, we were discussing how and where mass-production techniques could be applied to software.

I explained to Neil why I belived that this couldn't be done.

I feel that the point of mass-production is to reduce the cost of producing copies of a prototype.

In car manufacturing, for example, even the simplest of prototypes for the cheapest of cars cost upwards of £100,000. Millions of pounds will be spent fitting out a production line. After applying mass-production techniques, the copies will be sold for a twentieth of the cost of the original.

With software we are in a very different space, the cost of mass-production is essentially zero. All the cost is in the prototype.

Building a car prototype involves designing components, testing them, putting them together and testing the whole. Engineer will often design a car prototype re-using components from other vehicles and will design components to be re-used. Does this sound familiar?

The advent of devices that will print components and eventually nanofactories mean that the cost of mass-production will shrink down to purely that of the raw materials and the energy.

This means that we may find that trends in software development may be pointers to the future based on these new devices. I wonder what an open-source washing machine will look like?

OptimalJ - My Review.

I've been holding off on this entry for some time. I wanted to wait until I left the place where I was using it so that I could feel able to be completely honest.

I'll be giving an user's eye view of it both from the perspectives of an architect and a developer, I'll talk about designing using it and developing with the artifacts. I've been using OptimalJ 3.2 which is a relatively old version, newer versions will have addressed some of the problems that I'll be raising. I'll mention any fixes that I am aware of.

Model Driven Architecture is yet another attempt to increase the application complexity while reducing development difficulty by drawing picures (usually UML) to define the components and code generation to produce them.

OJ contains a set of UML modelling tools, which are separate and distinct from the MDA tools. It did not seem to be possible to move the analysis and design done using OJ's own UML tools directly into the MDA tools. On the project that I was working on we used Rational Rose to do the analysis and design, before using OJ's MDA tools. The inbuilt UML tools were inferior to those in Rose.

Using OJ to produce the MDA models is little different from using a class modeller in an UML tool, defining classes, attributes and methods. Unfortunately aside from the class modelling most of the rest of the process is about walking through wizards or setting properties. A lot more thought could have gone into using more UML diagram types. For example when one wants to define dependencies between services, one has to add to the 'UsedComponent' property values. These kind of dependencies could be easily defined using collaboration or sequence diagrams.

The three major 'models' that OJ works with are the Domain Model (used to define the domain objects and services), Application Model (fleshing out the domain model and getting quite platform specific) and lastly the code model which is the generated code.

The following layers are defined by OJ by default in the Application Model:

• DBMS - The physical data model.
• Common - cross layer objects such as DTOs (in OJ speak UpdateObjects and DataObjects), OJ enumerations and structs.
• BusinessLogic - if I need to explain this one to you, perhaps you shouldn't be reading this... Well maybe I should explain one thing; BusinessLogic lumps Entity and Session Beans together and does not attempt to guide one down more structured approaches such as using the Session Facade pattern.
• BusinessFacade - a curious set of auto-generated facades that will try to use UserTransactions if you don't watch them very carefully. Mainly useful.
• Presentation - auto-generated struts forms and actions that are only really useful for data entry and prodding the services. In more recent versions of OJ a workflow designer has been added which will hopefully make this a lot more useful.

Out of the box one can only really architect J2EE/EJB applications, persistence only uses entity beans.

In practical use, OJ gets increasingly sluggish as the application increases in size. For what I would call a medium-small application we needed to wait anything up to a minute while OJ digested simple property changes; essential tools that check and update models that needs to be run frequently needed 10 minutes or more. It is also incredibly memory hungry, architects' machines were running with 2 gigabytes of RAM.

These problems can be managed by splitting your application into independent subsystems.

The code generation produces code that is split up into free and guarded blocks. Developers are expected to place business and application logic into the free blocks. I personally believe that weaving hand-cranked and generated code together in this way is a bad idea:

• It makes re-designing and re-factoring unnecessarily difficult as OJ is not properly joined up. If you change a class name or package, the developer code gets put into a 'recycle bin' and needs to be retrieved.
• The quality of code generated by the default patterns is more than a little suspect and makes it very difficult to get useful data out of reporting tools such as findbugs and checkstyle.
• It means that the model is not the thing. Not only do you have to check the model into your repository, you have to check in large swathes of generated code.

The code produced by the default patterns does not inspire confidence in the Java abilities of the people producing the application. For example a form of dirty marker pattern in provided to support the 'UpdateObject's (DTOs) produced in the common layer. Unfortunately it is one of the worst implementations that I have ever come across. The authors seem to have so little understanding of encapsulation that they require a developer to actually call a method to set a changed flag for each field manipulated. When one looks at an UpdateObject's interface, one sees nearly the entire workings of the OJ dirty marker pattern marked public for all to see and it is not a pretty sight. I was so outraged that I was very tempted to go and find the authors and slap them till they promised never to produce something like that ever again.

The version that I was using had one other significant failing, the model-merge functionality that allows multiple modellers to work simultaneously was broken and this introduced a major bottle nect.

Out of the box OJ falls into negative ROI. Any modern project using open source code generation can out-produce it.

Reading over what I have written it would seem that I hate it with a passion. That would be untrue. The potential is huge especially when one realises that one can rewrite the meta-models and patterns.

I would only recommend OJ to an organisation that had significant up-front time to invest:

• Take the time to really understand OJ's capabilities
• Improving the meta-model to provide a better breakdown of the business logic tier,
• Throw out many of the patterns and introduce new ones to use callback or dependency injection to move developer code out of the generated code.
• Produce patterns that make use of a wider range of technology.
• Write a decent dirty marker pattern.
• Wait for Compuware to make the generation steps full scriptable in ant.
• In monster computers for its architects.

Once all this had been done, then a positive ROI should materialise.

Another time to use dependency injection?

I was talking to some friends this lunchtime, explaining IOC containers and what dependency injection could mean for them.

A though came to me. What if IOC containers could also inject dependencies when objects were deserialized? Objects could behave completely differently across tiers. Say an object has a dependency on a persistence interface. On the middle tier the object would get stored to the database buton the client the object would get serialised to the middle tier for storage.

Chilidish Men? Or Childish Women?

I'm sure that the women in my life will give me a lot of grief for this entry, but here goes...

I'm a contentedly childish man, I like my toys, I like cartoons; generally I like having the time to play from time to time. When I have to be I can be as adult as necessary, dealing with difficult things in personal or professional life.

What I have noticed over the years is a certain undercurrent in our society where certain of the feminine elements belittle men because of the very form of childishness that I acknowledge in myself. Expressions such as 'He's such a big kid.' or 'Boys and their toys...' are a mild form of this.

I have started to wonder however whether women are any more grown up than men. I think that they are just better at public relations.

When a boy is playing with toy cars and guns he is being childish, when a girl is playing with dolls, makeup and dressing up they are preparing for adulthood.

This transfers forward to later life, when women spends hours trying on different outfits or being pampered at a beauticians - that is adult. I wonder whether are really just reverting to childhood.

Oracle OCI JDBC Driver problems

Yesterday, I spent a couple of hours trying to work out why we were getting an UnsatisfiedLinkError on the OCI8 libraries when we were running the 9.2 JDBC OCI drivers against an Oracle 9.2 database on Windows Server 2003.

After a lot of head scratching it turned out that the application's ojdbc14.jar and the OCI libraries were mutually incompatible despite appearing to be for the same point release of Oracle.

The application's driver jar was the one that was checked into CVS, whereas the libraries came as part of a new install of Oracle. When we used the driver jar from the new install, our problems went away.

It is definitely worth remembering to always use the driver jars that came with the libraries that you are using.

Fast Web Applications - Design thoughts

Whenever I design web applications there are a number of steps that I go through in order to achieve decent performance. Most of these you will all know about, however there is one thing that I do that does not seem to be well recorded in any book or online article.

Most of you will already have thought about static and dynamic content, in a java web application that will probably mean that there will be a set of HTML pages and images on the apache server and JSPs or Servlets on a Servlet engine.

I have a third classification of content that is very useful to think about: Semi-static.

Semi-static content is content that is data-driven (like dynamic content), but the underlying data changes relatively infrequently compared to the number of times that the content is viewed. Good examples would be a daily graph of stocks or a airport flight information page that is displayed across hundreds of displays.

There are various ways of dealing with this type of content, but they all come down to one thing - caching. Many people when they come across this type of content end up creating a jack-of-all-trades servlet that manages the caching and invalidation of this content. I tend to favour a different approach. I like to generate this content using a template engine (or equivalent for images) to the file system behind my web-server and let the web-server and the downstream caches do their work. The template engine should be event driven, I would normally use a lightweight JMS system to do this.

With a little careful configuration you can leverage the strengths of HTTP to allow an HTTP HEAD request to be issued, checking whether the browser cache, proxy cache or web-server in-memory cache has the up to date version and serve that directly from the cache or retrieve the new version from the web server file system accordingly.