IntelliJ and General Development on a Windows 7 Corporate Laptop...

The more development I do with IntelliJ, the more attached I become to it. It works well with my development style and I don't mind the licence costs as a result.

I recently started working on a new project with a company and discovered that they were in the middle of a project to move all its developers over to using a new laptop build based on Windows 7.

They have a great team working on this but they were relatively inexperienced at dealing with developers requirements so they needed some guinea pigs and our team fitted the bill nicely.

When the proposed laptop was first trialled a misconfiguration of the anti-virus meant that they were very slow, as a result our boss was able to demand SSDs... Which was nice. I've been running SSDs on my home development machines to provide the best possible build performance though I always make sure my machines and repositories are backed up to spinning rust.

The laptops with SSDs finally arrived and it turned out that I was the first of the development focused guys to get our hands on it.

Some clever tooling meant that while none of us developers had admin privileges, there was a clear and sensible route for installation of any software that we required. The technique involved meant that temporary admin privileges were given to the install process which meant that our normal users were unable to alter the installed files in any way. This had several implications for the installation of software, IntelliJ in particular.

I encountered a number of teething problems - the first one was that our user profiles were configured to be read from a network share. Fine as long as we were plugged into our head office networks, less so when we were on the road. The most visible problem was the corporate software provisioning tools having a snit fit and not wing able to launch various items of software if we were connecting using a VPN. The VPN could only be launched after login for various good reasons.

Fixing this problem was easy as it only required using the Offline Sync capabilities in Windows 7 on the necessary parts of our profile.

This however pointed to a more significant problem for developers. By default IntelliJ, Maven and other tools keep a lot of their configuration and caches in the profile. This means a large volume of data needs to be sync-ed initially and as the IntelliJ caches are updated and new Maven dependencies are added even more sync-ing needs to happen. Not great across a VPN over ADSL.

The solution is to reconfigure your tools to store these high entropy folders in the local drive away for your profile. This also makes the support team happy as it stops the developer profiles getting too large and clogging up the networks.

I created a 'development' folder specifically for these folders and my projects.

Configuring Maven is easy: create a settings.xml file in your profile fold if you haven't already and override the default local repository location. At the same time you may as well be sorting out your proxies or any corporate Nexus repositories. Set the Maven environment properties on the user environment variables and you're done.

IntelliJ is not too hard as well. I copied the idea.properties file from the IntelliJ bin directory (I couldn't edit it directly as it had been written with admin privileges) and edited it to move my .IntellijIDEA folder to my development folder. I replaced the ${user.home} entry in all the relevant properties with a hardcoded value as replacing it with a self declared property like ${configuration.home} caused the logging to end up in a directory under IntelliJ called ${configuration.home}.

I then needed to tell IntelliJ to use the new idea.properties file so I had to set up the IDEA_PROPERTIES environment variable. This could not be done as a system environment variable but fortunately user level environment variables do the job. It is worth noting that you should be able override the Path environment variable that is set in the system environment variables by setting one up in the user ones. Anecdotally this can be a bit temperamental if you don't match the case of the system one and you may have problems if the admins have locked it down too.

The same process of configuration needs to be repeated for any other tools that like to put things in the profile.

The next issue encountered was when starting IntelliJ. IntelliJ 11 would start but not be able to index the project whereas IntelliJ 12 would fail completely, complaining about being unable to start webserver processes. It turned out that the McAfee firewall had been configured too aggressively and local loop back calls (localhost and 127.0.0.1) were blocked. A quick chat with the support team sorted this one out.

The last major issue was with performance due to McAfee Data Loss Prevention (DLP). I'd encountered this before on other corporate sites. You need to ensure that your development folder is excluded. This is due to the number of files typically deleted due to a Maven clean install cycle; DLP just eats CPU and slows disk access to a crawl when this happens. Once more the support team came to the rescue.

Once these issues were sorted I was delighted to have a corporate laptop that was entirely pleasant to develop on. As a bonus we were given widescreen 27" monitors, I was able to rotate mine to landscape mode and view entire classes at a time!

- Posted using BlogPress from my iPad

Towards a richer GPGPU Mandelbrot Sets and OpenCL example.

Nearly 3 years ago I started playing with OpenCL on the GPGPU front and posted about my experiences which excited some interest. Since then I've worked on a number of different technologies but I do keep coming back to OpenCL.

I've decided to work on enhancing my Mandelbrot generator and thought I'd share some of my experience doing so. My intention is to create a richer and more capable Mandelbrot Set generator that properly leverages the capabilities of OpenCL and allows someone to more interactively explore the set.

I'm having to dust off my Swing skills to do this as I want to create a richer UI experience. So first off lets talk about current features in my Mandelbrot Generator:

Current Features

• Single, Double and Fixed Precision (128bit) calculation modes.
• Simple zooming using the mouse left button to zoom in and the mouse right button to zoom out.
• Re-sizable window.
• Simple Benchmarking application using the different calculation modes to demonstrate GPU performance using Single, Double and Integer mathematics.
• Automatic detection of Work Group size and appropriate buffering.

Future Features

• Configurable colour palettes.
• Julia Set generation - for the current location in the Mandelbrot set..
• UI components to be able to configure the calculation mode and a few other options.
• Further, richer Fixed Precision modes (256 and 512 bit seem a natural fit for uint8 and uint16)
• Interactive Benchmarking
• Ability to save specific 'views' and 'journeys' through the Mandelbrot Set and interact with them.
• Rendering of ultra high resolution images, either directly rendered in one pass on the GPU or rendered as tiles and stitched together.
• Render to OpenGL contexts and real-time fly throughs of the set.

Today I'm going to talk about some key components of my Mandelbrot Generator:

• JavaCL
• MandelbrotView Interface
• Fixed Precision algorithm

JavaCL

I've been looking at the various Java to OpenCL bindings and having played with a number of them I keep on coming back to Olivier Chafik's JavaCL for the cleanest and best experience. I have yet to use the Maven bindings generator but even without it, the code is very clean.

My MandelbrotGenerator class which handles all the generation modes looks like:

package co.uk.fvdl.mandelbrot;

import co.uk.fvdl.mathematics.FixedPrecision128;
import co.uk.fvdl.util.TwoTuple;
import com.nativelibs4java.opencl.*;

import java.io.IOException;
import java.io.InputStreamReader;

/**
 * Generic class for generating Mandelbrot set using OpenCL. May be able to use as the basis for generating other
 * fractals such as Julia sets by making it possible to choose different OpenCL programs as parameters.
 */
public class MandelbrotGenerator {

    private static final int ESCAPE_THRESHOLD = 4;
    private final CLKernel mandelbrotFloatKernel;
    private final CLKernel mandelbrotDoubleKernel;
    private final CLKernel mandelbrotFixedPrecision128Kernel;
    private final CLQueue queue;
    private final CLContext context;
    private int workgroupWidth;
    private int workgroupHeight;
    private boolean profiling;


    public MandelbrotGenerator(CLContext context, boolean profiling) throws IOException {
        this(context, Utility.detectLargest2DWorkGroupSize(context), profiling);
    }

    public MandelbrotGenerator(CLContext context, TwoTuple<integer, integer> workgroupSize, boolean profiling) throws IOException {
        this.profiling = profiling;
        System.out.println("Workgroup size: " + workgroupSize);
        this.context = context;
        queue = context.createDefaultQueue();
        queue.setProperty(CLDevice.QueueProperties.ProfilingEnable, profiling);

        //Will always support integers...
        String fixedPrecision128Source = MandelbrotViewer.readFully(
                new InputStreamReader(this.getClass().getResourceAsStream("opencl/MandelbrotFixedPrecision128.cl")),
                new StringBuilder()
        ).toString();
        String fixedPoint128LibrarySource = MandelbrotViewer.readFully(
                new InputStreamReader(this.getClass().getResourceAsStream("opencl/FixedPrecision128.cl")),
                new StringBuilder()
        ).toString();
        CLProgram fixedPrecision128Program = context.createProgram(fixedPoint128LibrarySource, fixedPrecision128Source);
        mandelbrotFixedPrecision128Kernel = fixedPrecision128Program.createKernel("MandelbrotFixedPrecision128");

        //Will always support floats...
        String floatSource = MandelbrotViewer.readFully(
                new InputStreamReader(this.getClass().getResourceAsStream("opencl/MandelbrotFloat.cl")),
                new StringBuilder()
        ).toString();
        CLProgram floatProgram = context.createProgram(floatSource);
        mandelbrotFloatKernel = floatProgram.createKernel("MandelbrotFloat");

        //May not always support doubles...
        if(context.isDoubleSupported()){
            //Read the source file.
            String doubleSource = MandelbrotViewer.readFully(
                    new InputStreamReader(this.getClass().getResourceAsStream("opencl/MandelbrotDouble.cl")),
                    new StringBuilder()
            ).toString();
            CLProgram doubleProgram = context.createProgram(doubleSource);
            mandelbrotDoubleKernel = doubleProgram.createKernel("MandelbrotDouble");
        } else {
            mandelbrotDoubleKernel = null;
        }

        this.workgroupWidth = workgroupSize.getFirstElement();
        this.workgroupHeight = workgroupSize.getSecondElement();
    }

    public GenerationResult generateMandelbrot(MandelbrotView mandelbrotView) throws IOException {

        Padded2DBuffer padded2DBuffer = new Padded2DBuffer(mandelbrotView.getRealAxisPixelCount(), mandelbrotView.getImaginaryAxisPixelCount(), workgroupWidth, workgroupHeight);
        CLBuffer<integer> outputBuffer =  context.createBuffer(CLMem.Usage.Output, padded2DBuffer.getPointer());

        CLEvent event;
        CLKernel mandelbrotKernel = null;
        switch (mandelbrotView.getMandelbrotGenerationStrategy()){
            case DOUBLE:
                if(context.isDoubleSupported()){
                    mandelbrotKernel = mandelbrotDoubleKernel;
                    mandelbrotKernel.setArgs(
                            new double[]{mandelbrotView.getRealAxisPixelSize().doubleValue(), mandelbrotView.getImaginaryAxisPixelSize().doubleValue()},
                            new double[]{mandelbrotView.getRealAxisMinimum().doubleValue(), mandelbrotView.getImaginaryAxisMinimum().doubleValue()},
                            mandelbrotView.getMaxIter(), ESCAPE_THRESHOLD, padded2DBuffer.getActualWidth(), outputBuffer);
                    break;
                }
            case SINGLE:
                mandelbrotKernel = mandelbrotFloatKernel;
                mandelbrotKernel.setArgs(
                        new float[]{mandelbrotView.getRealAxisPixelSize().floatValue(), mandelbrotView.getImaginaryAxisPixelSize().floatValue()},
                        new float[]{mandelbrotView.getRealAxisMinimum().floatValue(), mandelbrotView.getImaginaryAxisMinimum().floatValue()},
                        mandelbrotView.getMaxIter(), ESCAPE_THRESHOLD, padded2DBuffer.getActualWidth(), outputBuffer);
                break;
            case FP128:
                mandelbrotKernel = mandelbrotFixedPrecision128Kernel;
                mandelbrotKernel.setArgs(
                        FixedPrecision128.convertToFixedPrecision128(mandelbrotView.getRealAxisPixelSize()), FixedPrecision128.convertToFixedPrecision128(mandelbrotView.getImaginaryAxisPixelSize()),
                        FixedPrecision128.convertToFixedPrecision128(mandelbrotView.getRealAxisMinimum()), FixedPrecision128.convertToFixedPrecision128(mandelbrotView.getImaginaryAxisMinimum()),
                        mandelbrotView.getMaxIter(), ESCAPE_THRESHOLD, padded2DBuffer.getActualWidth(), outputBuffer);
                break;
        }
        event = mandelbrotKernel.enqueueNDRange(queue, new int[]{padded2DBuffer.getActualWidth(), padded2DBuffer.getActualHeight()}, new int[]{workgroupWidth, workgroupHeight});

        outputBuffer.read(queue, padded2DBuffer.getPointer(), true, event);

        if(profiling){
            return new GenerationResult(padded2DBuffer, event.getProfilingCommandEnd() - event.getProfilingCommandStart());
        } else {
            return new GenerationResult(padded2DBuffer, Long.MIN_VALUE);
        }
    }

    public boolean isDoubleAvailable() {
        return context.isDoubleSupported();
    }
}

I've pushed all the JavaCL code into this class and am able to reuse it in both my interactive and benchmarking applications.

MandelbrotView

To provide a coherent view of the set to my Java code I've created a standard interface that defines the place in the Mandelbrot Set that we are looking at and how we are looking at it:

package co.uk.fvdl.mandelbrot;

import java.math.BigDecimal;

/**
 * Classes implementing this interface represent views onto the Mandelbrot set coordinate space. It is fundamentally
 * represented in terms of the centre of the view on the set and in terms of the zoom level represented by the span of a
 * pixel in that coordinate space. This representation of the view makes it easier to handle re-sizing of the containing
 * window, treating it as a viewport on to the Mandelbrot set.
 */
public interface MandelbrotView {

    /**
     * Get the size of pixels on the Real Axis.
     * @return the size of pixels on the Real Axis.
     */
    BigDecimal getRealAxisPixelSize();

    /**
     * Get the size of pixels on the Imaginary Axis.
     * @return the size of pixels on the Imaginary Axis.
     */
    BigDecimal getImaginaryAxisPixelSize();

    /**
     * Get the number of pixels currently used in the view on the real axis.
     * @return the number of pixels currently used in the view on the real axis.
     */
    int getRealAxisPixelCount();

    /**
     * Get the number of pixels currently used in the view on the imaginary axis.
     * @return the number of pixels currently used in the view on the imaginary axis.
     */
    int getImaginaryAxisPixelCount();

    /**
     * Set the number of pixels to be used in the view on the real axis.
     * @param realAxisPixelCount the number of pixels currently used in the view on the real axis.
     */
    void setRealAxisPixelCount(int realAxisPixelCount);

    /**
     * Set the number of pixels to be used in the view on the imaginary axis.
     * @param imaginaryAxisPixelCount the number of pixels currently used in the view on the imaginary axis.
     */
    void setImaginaryAxisPixelCount(int imaginaryAxisPixelCount);

    /**
     * Calculates the minimum value that the current view has on the real axis.
     * @return the minimum value that the current view has on the real axis.
     */
    BigDecimal getRealAxisMinimum();

    /**
     * Calculates the minimum value that the current view has on the imaginary axis.
     * @return the minimum value that the current view has on the imaginary axis.
     */
    BigDecimal getImaginaryAxisMinimum();

    /**
     * Create a new MandelbrotView for a sub-rectangle of the current view. It will preserve the same pixel dimensions
     * as the original and so will choose the scaling axis to ensure that the whole of the selected rectangle is
     * visible within the new MandelbrotViewImpl instance.
     * @param realAxisPosition The pixel position on the real axis within the current view of the top left corner of the
     *                         rectangle.
     * @param imaginaryAxisPosition  The pixel position on the imaginary axis within the current view of the top left
     *                               corner of the rectangle.
     * @param realAxisPixels The pixel width on the real acis of the rectangle
     * @param imaginaryAxisPixels The pixel width on the imaginary axis of the rectangle
     * @return a new MandelbrotViewImpl representing the sub-rectangle of the current view.
     */
    MandelbrotView selectSubRectangle(
            int realAxisPosition, int imaginaryAxisPosition,
            int realAxisPixels, int imaginaryAxisPixels);

    /**
     * Create a new MandelbrotView for a super-rectangle of the current view where the current view will be contained
     * within the selected rectangle. It will preserve the same pixel dimensions as the original.
     * @param realAxisPosition The pixel position on the real axis within the new view of the top left corner of the
     *                         rectangle containing the current view.
     * @param imaginaryAxisPosition  The pixel position on the imaginary axis within the new view of the top left
     *                               corner of the rectangle containing the current view.
     * @param realAxisPixels The pixel width on the real axis of the rectangle
     * @param imaginaryAxisPixels The pixel width on the imaginary axis of the rectangle
     * @return a new MandelbrotViewImpl representing the super-rectangle of the current view.
     */
    MandelbrotView selectSuperRectangle(
            int realAxisPosition, int imaginaryAxisPosition,
            int realAxisPixels, int imaginaryAxisPixels);

    /**
     * Get the preferred generation strategy.
     * @return the preferred generation strategy.
     */
    MandelbrotGenerationStrategy getMandelbrotGenerationStrategy();

    /**
     * The maximum number if iterations to use before considering that we have escaped the set.
     * @return the maximum number of iterations to use.
     */
    int getMaxIter();

    /**
     * Set the maximum number if iterations to use before considering that we have escaped the set.
     * @param maxIter the maximum number of iterations to use.
     */
    void setMaxIter(int maxIter);
}

While the view interface talks about the 'minimums' and the number of pixels which maps well to what we need to render, the underlying implementation defines the view in terms of the centre of the view and the pixel size. This means that when we resize the window containing the view the view remains centred on what we intend:

package co.uk.fvdl.mandelbrot;

import co.uk.fvdl.mathematics.BigDecimalArithmetic;

import java.math.BigDecimal;

import static co.uk.fvdl.util.Preconditions.checkNotNull;

/**
 * This class represents a view onto the Mandelbrot set coordinate space. It is fundamentally represented in terms of
 * the centre of the view on the set and in terms of the zoom level represented by the span of a pixel in that coordinate
 * space. This representation of the view makes it easier to handle re-sizing of the containing window, treating it as a
 * viewport on to the Mandelbrot set.
 */
public class MandelbrotViewImpl implements MandelbrotView {
    private final BigDecimal realAxisCentre;
    private final BigDecimal imaginaryAxisCentre;
    private final BigDecimal realAxisPixelSize;
    private final BigDecimal imaginaryAxisPixelSize;
    private final BigDecimalArithmetic numberArithmetic;
    private final MandelbrotGenerationStrategy mandelbrotGenerationStrategy;

    private int realAxisPixelCount;
    private int imaginaryAxisPixelCount;
    private int maxIter;

    /**
     * Construct an instance of the Mandelbrot view for the following parameters.
     * @param realAxisCentre the centre of the view on the Real Axis.
     * @param imaginaryAxisCentre the centre of the view on the Imaginary Axis.
     * @param realAxisPixelSize the size of pixels on the Real Axis.
     * @param imaginaryAxisPixelSize the size of pixels on the Imaginary Axis.
     * @param realAxisPixelCount the number of pixels to be used in the view on the real axis.
     * @param imaginaryAxisPixelCount the number of pixels currently used in the view on the imaginary axis.
     * @param mandelbrotGenerationStrategy the prefered strategy to use when generating the Mandelbrot set.
     * @param maxIter The maximum iteration to use for generating the set.
     */
    @SuppressWarnings("unchecked")
    public MandelbrotViewImpl(BigDecimal realAxisCentre, BigDecimal imaginaryAxisCentre,
                              BigDecimal realAxisPixelSize, BigDecimal imaginaryAxisPixelSize,
                              int realAxisPixelCount, int imaginaryAxisPixelCount, MandelbrotGenerationStrategy mandelbrotGenerationStrategy, int maxIter) {
        this.mandelbrotGenerationStrategy = mandelbrotGenerationStrategy;
        this.maxIter = maxIter;
        this.numberArithmetic = new BigDecimalArithmetic();
        this.realAxisCentre = checkNotNull(realAxisCentre, "Real Axis centre is null.");
        this.imaginaryAxisCentre = checkNotNull(imaginaryAxisCentre, "Imaginary Axis centre is null.");
        this.realAxisPixelSize = checkNotNull(realAxisPixelSize, "Real Axis pixel size is null.");
        this.imaginaryAxisPixelSize = checkNotNull(imaginaryAxisPixelSize, "Imaginary Axis pixel size is null.");
        this.realAxisPixelCount = realAxisPixelCount;
        this.imaginaryAxisPixelCount = imaginaryAxisPixelCount;
    }

    /**
     * Get the centre of the view on the Real Axis.
     * @return the centre of the view on the Real Axis.
     */
    public BigDecimal getRealAxisCentre() {
        return realAxisCentre;
    }

    /**
     * Get the centre of the view on the Imaginary Axis.
     * @return the centre of the view on the Imaginary Axis.
     */
    public BigDecimal getImaginaryAxisCentre() {
        return imaginaryAxisCentre;
    }

    @Override
    public BigDecimal getRealAxisPixelSize() {
        return realAxisPixelSize;
    }

    @Override
    public BigDecimal getImaginaryAxisPixelSize() {
        return imaginaryAxisPixelSize;
    }

    @Override
    public int getRealAxisPixelCount() {
        return realAxisPixelCount;
    }

    @Override
    public int getImaginaryAxisPixelCount() {
        return imaginaryAxisPixelCount;
    }

    @Override
    public void setRealAxisPixelCount(int realAxisPixelCount) {
        this.realAxisPixelCount = realAxisPixelCount;
    }

    @Override
    public void setImaginaryAxisPixelCount(int imaginaryAxisPixelCount) {
        this.imaginaryAxisPixelCount = imaginaryAxisPixelCount;
    }

    @Override
    public BigDecimal getRealAxisMinimum() {
        return numberArithmetic.subtract(
                realAxisCentre, 
                numberArithmetic.divide(
                        numberArithmetic.multiply(
                                realAxisPixelSize, 
                                numberArithmetic.fromInteger(realAxisPixelCount)
                        ),
                        numberArithmetic.fromInteger(2)));
    }

    @Override
    public BigDecimal getImaginaryAxisMinimum() {
        return numberArithmetic.subtract(
                imaginaryAxisCentre, 
                numberArithmetic.divide( 
                        numberArithmetic.multiply(
                                imaginaryAxisPixelSize, 
                                numberArithmetic.fromInteger(imaginaryAxisPixelCount)
                        ),numberArithmetic.fromInteger(2)));
    }

    @Override
    public MandelbrotViewImpl selectSubRectangle(
            int realAxisPosition, int imaginaryAxisPosition,
            int realAxisPixels, int imaginaryAxisPixels) {
        BigDecimal scale;
        if((realAxisPixelCount / imaginaryAxisPixelCount) > (realAxisPixels/imaginaryAxisPixels)){ //use the imaginary axis to provide the scaling
            scale = numberArithmetic.divide(
                    numberArithmetic.fromInteger(imaginaryAxisPixels),
                    numberArithmetic.fromInteger(imaginaryAxisPixelCount)
            );
        } else { //use the real axis to provide scaling.
            scale = numberArithmetic.divide(
                    numberArithmetic.fromInteger(realAxisPixels),
                    numberArithmetic.fromInteger(realAxisPixelCount)
            );
        }

        float newRealPixelCentre = (float)realAxisPosition + (float)realAxisPixels / 2.0F;
        float newImaginaryPixelCentre = (float)imaginaryAxisPosition + (float)imaginaryAxisPixels / 2.0F;
        float currentRealPixelCentre = (float)realAxisPixelCount / 2.0F;
        Float currentImaginaryPixelCentre = (float)imaginaryAxisPixelCount / 2.0F;

        BigDecimal deltaRealPixels = numberArithmetic.fromFloat(newRealPixelCentre - currentRealPixelCentre);
        BigDecimal deltaImaginaryPixels = numberArithmetic.fromFloat(newImaginaryPixelCentre - currentImaginaryPixelCentre);
        
        BigDecimal newRealAxisCentre = numberArithmetic.add(
                realAxisCentre,
                numberArithmetic.multiply(
                        deltaRealPixels,
                        realAxisPixelSize));
        
        BigDecimal newImaginaryAxisCentre = numberArithmetic.add(
                imaginaryAxisCentre,
                numberArithmetic.multiply(
                        deltaImaginaryPixels,
                        imaginaryAxisPixelSize));

        BigDecimal newRealAxisPixelSize = numberArithmetic.multiply(realAxisPixelSize, scale);
        BigDecimal newImaginaryAxisPixelSize = numberArithmetic.multiply(imaginaryAxisPixelSize, scale);

        return new MandelbrotViewImpl(
                newRealAxisCentre, newImaginaryAxisCentre,
                newRealAxisPixelSize, newImaginaryAxisPixelSize,
                realAxisPixelCount, imaginaryAxisPixelCount, mandelbrotGenerationStrategy, maxIter);
    }

    @Override
    public MandelbrotViewImpl selectSuperRectangle(
            int realAxisPosition, int imaginaryAxisPosition,
            int realAxisPixels, int imaginaryAxisPixels) {
        BigDecimal scale;
        if((realAxisPixelCount / imaginaryAxisPixelCount) > (realAxisPixels/imaginaryAxisPixels)){ //use the imaginary axis to provide the scaling
            scale = numberArithmetic.divide(
                    numberArithmetic.fromInteger(imaginaryAxisPixelCount),
                    numberArithmetic.fromInteger(imaginaryAxisPixels)
            );
        } else { //use the real axis to provide scaling.
            scale = numberArithmetic.divide(
                    numberArithmetic.fromInteger(realAxisPixelCount),
                    numberArithmetic.fromInteger(realAxisPixels)
            );
        }

        int newRealPixelCentre = realAxisPixelCount / 2;
        int newImaginaryPixelCentre = imaginaryAxisPixelCount / 2;
        int currentRealPixelCentre = realAxisPosition + realAxisPixels / 2;
        int currentImaginaryPixelCentre = imaginaryAxisPosition + imaginaryAxisPixels / 2;

        BigDecimal deltaRealPixels = numberArithmetic.fromInteger(newRealPixelCentre - currentRealPixelCentre);
        BigDecimal deltaImaginaryPixels = numberArithmetic.fromInteger(newImaginaryPixelCentre - currentImaginaryPixelCentre);

        BigDecimal newRealAxisPixelSize = numberArithmetic.multiply(realAxisPixelSize, scale);
        BigDecimal newImaginaryAxisPixelSize = numberArithmetic.multiply(imaginaryAxisPixelSize, scale);

        BigDecimal newRealAxisCentre = numberArithmetic.add(
                realAxisCentre,
                numberArithmetic.multiply(
                        deltaRealPixels,
                        newRealAxisPixelSize));

        BigDecimal newImaginaryAxisCentre = numberArithmetic.add(
                imaginaryAxisCentre,
                numberArithmetic.multiply(
                        deltaImaginaryPixels,
                        newImaginaryAxisPixelSize));


        return new MandelbrotViewImpl(
                newRealAxisCentre, newImaginaryAxisCentre,
                newRealAxisPixelSize, newImaginaryAxisPixelSize,
                realAxisPixelCount, imaginaryAxisPixelCount, mandelbrotGenerationStrategy, maxIter);

    }

    @Override
    public String toString() {
        final StringBuffer sb = new StringBuffer();
        sb.append("MandelbrotViewImpl");
        sb.append("{maxIter=").append(maxIter);
        sb.append(", realAxisCentre=").append(realAxisCentre);
        sb.append(", imaginaryAxisCentre=").append(imaginaryAxisCentre);
        sb.append(", realAxisPixelCount=").append(realAxisPixelCount);
        sb.append(", imaginaryAxisPixelCount=").append(imaginaryAxisPixelCount);
        sb.append(", realAxisPixelSize=").append(realAxisPixelSize);
        sb.append(", imaginaryAxisPixelSize=").append(imaginaryAxisPixelSize);
        sb.append(", mandelbrotGenerationStrategy=").append(mandelbrotGenerationStrategy);
        sb.append('}');
        return sb.toString();
    }

    @Override
    public MandelbrotGenerationStrategy getMandelbrotGenerationStrategy() {
        return mandelbrotGenerationStrategy;
    }

    @Override
    public int getMaxIter() {
        return maxIter;
    }

    @Override
    public void setMaxIter(int maxIter) {
        this.maxIter = maxIter;
    }
}

The final component of the MandelbrotView is the MandelbrotViewHolder which acts as a delegate for the multiple MandelbrotView instances that we progress through as we zoom. The intent is that eventually the MandelbrotViewHolder will morph into something that allows us to trace the history of a session exploring the MandelbrotSet.

package co.uk.fvdl.mandelbrot;

import java.math.BigDecimal;

import static co.uk.fvdl.util.Preconditions.checkNotNull;

/**
 * This class provides a level of indirection that allows us to work with immutable implementations of the
 * MandelbrotView interface without having to deal with a large amount of reference passing in UI components as the
 * view is modified.
 */
public class MandelbrotViewHolder implements MandelbrotView {

    private MandelbrotView wrappedInstance;

    /**
     * Construct and instance using the wrapped instance.
     * @param wrappedInstance the instance of MandelbrotView to wrap.
     */
    public MandelbrotViewHolder(MandelbrotView wrappedInstance) {
        this.wrappedInstance = checkNotNull(wrappedInstance, "The MandelbrotView instance to be wrapped may not be null");
    }

    @Override
    public BigDecimal getRealAxisPixelSize() {
        return wrappedInstance.getRealAxisPixelSize();
    }

    @Override
    public BigDecimal getImaginaryAxisPixelSize() {
        return wrappedInstance.getImaginaryAxisPixelSize();
    }

    @Override
    public int getRealAxisPixelCount() {
        return wrappedInstance.getRealAxisPixelCount();
    }

    @Override
    public int getImaginaryAxisPixelCount() {
        return wrappedInstance.getImaginaryAxisPixelCount();
    }

    @Override
    public void setRealAxisPixelCount(int realAxisPixelCount) {
        wrappedInstance.setRealAxisPixelCount(realAxisPixelCount);
    }

    @Override
    public void setImaginaryAxisPixelCount(int imaginaryAxisPixelCount) {
        wrappedInstance.setImaginaryAxisPixelCount(imaginaryAxisPixelCount);
    }

    @Override
    public BigDecimal getRealAxisMinimum() {
        return wrappedInstance.getRealAxisMinimum();
    }

    @Override
    public BigDecimal getImaginaryAxisMinimum() {
        return wrappedInstance.getImaginaryAxisMinimum();
    }

    /**
     * Calls the underlying wrapped instance to create a new view on the Mandelbrot set for the sub-rectangle, replacing
     * it with the new one.
     * @param realAxisPosition The pixel position on the real axis within the current view of the top left corner of the
     *                         rectangle.
     * @param imaginaryAxisPosition  The pixel position on the imaginary axis within the current view of the top left
     *                               corner of the rectangle.
     * @param realAxisPixels The pixel width on the real acis of the rectangle
     * @param imaginaryAxisPixels The pixel width on the imaginary axis of the rectangle
     * @return this instance of the MandelbrotViewHolder which now wraps the new instance of the view.
     */
    @Override
    public MandelbrotView selectSubRectangle(int realAxisPosition, int imaginaryAxisPosition,
                                                int realAxisPixels, int imaginaryAxisPixels) {
        wrappedInstance = wrappedInstance.selectSubRectangle(
                realAxisPosition, imaginaryAxisPosition,
                realAxisPixels, imaginaryAxisPixels);
        return this;
    }

    /**
     * Calls the underlying wrapped instance to create a new view on the Mandelbrot set for the super-rectangle, replacing
     * it with the new one.
     * @param realAxisPosition The pixel position on the real axis within the new view of the top left corner of the
     *                         rectangle containing the current view.
     * @param imaginaryAxisPosition  The pixel position on the imaginary axis within the new view of the top left
     *                               corner of the rectangle containing the current view.
     * @param realAxisPixels The pixel width on the real axis of the rectangle
     * @param imaginaryAxisPixels The pixel width on the imaginary axis of the rectangle
     * @return a new MandelbrotViewImpl representing the super-rectangle of the current view.
     */
    @Override
    public MandelbrotView selectSuperRectangle(int realAxisPosition, int imaginaryAxisPosition, int realAxisPixels, int imaginaryAxisPixels) {
        wrappedInstance = wrappedInstance.selectSuperRectangle(
                realAxisPosition, imaginaryAxisPosition,
                realAxisPixels, imaginaryAxisPixels);
        return this;
    }

    @Override
    public MandelbrotGenerationStrategy getMandelbrotGenerationStrategy() {
        return wrappedInstance.getMandelbrotGenerationStrategy();
    }

    @Override
    public int getMaxIter() {
        return wrappedInstance.getMaxIter();
    }

    @Override
    public void setMaxIter(int maxIter) {
        wrappedInstance.setMaxIter(maxIter);
    }
}

Fixed Precision Mathematics

Having delved deep into the Mandelbrot set using both single and double precision implementations I discovered that with the raw performance of OpenCL on a decent graphic card it is entirely too easy to run into the limits of precision of the numeric representation. Having read around the problem I came upon the idea of Fixed Precision Mathematics to go deeper and came across an excellent piece of work by a gentleman called Eric Bainville at this site.

Using his work as a jumping off point I started seeing if I could integrate Fixed Precision calculations into my Mandelbrot Generator. The only problem I had was that I didn't fully understand the code that I was using and wanted to be able to extend it to greater precisions so I decided to write my own implementation from first principles. I did reuse a useful hack that he used replacing all the insignificant digit carry calculations with a '3'.

In the end I created a stripped down library to service my own Fixed Precision Mandelbrot generation.

It was interesting to compare his and my Fixed Precision libraries to see the similarities and differences - I deliberately didn't take some of the optimisation paths (e.g. dedicated squaring logic) - but I did end up with a mixture of similar and different methods. The largest difference was in the Positive / Positive multiplication where I decided to try to take advantage of the vector methods:

#define MAX_UINT4 (uint4)(0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF)
#define MOST_SIGNIFICANT_BIT_MASK 0x80000000U
uint4 addFP128(uint4 firstAddend, uint4 secondAddend)
{
    uint4 naiveAddition = firstAddend + secondAddend;
    uint4 propagatableVector = convert_uint4(naiveAddition == MAX_UINT4);
    uint4 overflowCarries = (convert_uint4(naiveAddition < firstAddend) & (uint4)(0,1,1,1)).yzwx;
    uint4 propagatedCarries = (uint4)(
                    ( (overflowCarries.z & propagatableVector.z) | overflowCarries.y ) & propagatableVector.y,
                    overflowCarries.z & propagatableVector.z,
                    0,
                    0);

    return naiveAddition + overflowCarries + propagatedCarries;
}

uint4 doubleFP128(uint4 operand)
{
    uint4 propagated = (operand & (uint4)(0,MOST_SIGNIFICANT_BIT_MASK, MOST_SIGNIFICANT_BIT_MASK, MOST_SIGNIFICANT_BIT_MASK)).yzwx  >>(uint4)(31U);
    return (operand<<(uint4)(1U)) + propagated;
}

uint4 incrementFP128(uint4 operand)
{
    uint4 propagatableVector = convert_uint4(operand == MAX_UINT4); // Determine if any of the operand elements are equivalent to 0xFFFFFFFF
    uint4 carriedIncrement = (uint4)(
                    propagatableVector.y & propagatableVector.z & propagatableVector.w & 1, //Carry up to x
                    propagatableVector.z & propagatableVector.w & 1, //Carry up to y
                    propagatableVector.w&1, //Carry up to z
                    1); //increment to be added to w
    return operand + carriedIncrement;
}

uint4 negateFP128(uint4 operand)
{
    uint4 complement = operand ^ MAX_UINT4;
    return incrementFP128(complement);
}

uint4 absoluteFP128(uint4 value) {
    return ((value.x & MOST_SIGNIFICANT_BIT_MASK)?negateFP128(value):value);
}

uint4 multiplyFP128PositivePositive(uint4 firstMultiplicand, uint4 secondMultiplicand)
{
    uint4 wXYZW = firstMultiplicand * (uint4)(secondMultiplicand.w);
    uint4 wXYZWHigh = mul_hi(firstMultiplicand, (uint4)(secondMultiplicand.w));
    uint4 zXYZW = firstMultiplicand * (uint4)(secondMultiplicand.z);
    uint4 zXYZWHigh = mul_hi(firstMultiplicand, (uint4)(secondMultiplicand.z));
    uint4 yXYZW = firstMultiplicand * (uint4)(secondMultiplicand.y);
    uint4 yXYZWHigh = mul_hi(firstMultiplicand, (uint4)(secondMultiplicand.y));
    uint4 xXYZW = firstMultiplicand * (uint4)(secondMultiplicand.x);
    uint4 xXYZWHigh = mul_hi(firstMultiplicand, (uint4)(secondMultiplicand.x));

    uint4 resultPart1 = addFP128((uint4)(yXYZWHigh.x, zXYZWHigh.x, wXYZWHigh.x, 3), (uint4)(xXYZWHigh.y, yXYZWHigh.y, zXYZWHigh.y, wXYZWHigh.y));
    uint4 resultPart2 = addFP128((uint4)(0, xXYZWHigh.z, yXYZWHigh.z, zXYZWHigh.z),   (uint4)(0, 0, xXYZWHigh.w, yXYZWHigh.w));
    uint4 resultPart3 = addFP128((uint4)(0, 0, 0, wXYZW.x), (uint4)(0, 0, zXYZW.x, zXYZW.y));
    uint4 resultPart4 = addFP128((uint4)(0, yXYZW.x, yXYZW.y, yXYZW.z), (uint4)(xXYZW.x, xXYZW.y, xXYZW.z, xXYZW.w));
    return addFP128(addFP128(resultPart1, resultPart2), addFP128(resultPart3, resultPart4));
}

//TODO consider if a dedicated squaring function would be a good idea.

uint4 multiplyFP128AnyAny(uint4 firstMultiplicand, uint4 secondMultiplicand)
{
      // Get the absolute product;
      uint4 p = multiplyFP128PositivePositive(absoluteFP128(firstMultiplicand), absoluteFP128(secondMultiplicand));
      //Return it with the appropriate sign
      return ((firstMultiplicand.x & MOST_SIGNIFICANT_BIT_MASK)^(secondMultiplicand.x & MOST_SIGNIFICANT_BIT_MASK))?negateFP128(p):p;
}

I implemented my Fixed Precision 128 Mandelbrot Kernel using it looking like:

__kernel void MandelbrotFixedPrecision128(
    const uint4 realDelta,
    const uint4 imaginaryDelta,
    const uint4 realMinimum,
    const uint4 imaginaryMinimum,
 const unsigned int maxIter,
 const unsigned int escapeNumber,
 const unsigned int hRes,
 __global int* outputi
)
{
 int realId = get_global_id(0);
 int imaginaryId = get_global_id(1);

    uint4 realPosition = addFP128(realMinimum, multiplyFP128AnyAny(realDelta, realId));
    uint4 squaredRealValue = multiplyFP128AnyAny(realPosition, realPosition);
    uint4 realValue = realPosition;

    uint4 imaginaryPosition = addFP128(imaginaryMinimum, multiplyFP128AnyAny(imaginaryDelta, imaginaryId));
    uint4 squaredImaginaryValue = multiplyFP128AnyAny(imaginaryPosition, imaginaryPosition);
    uint4 imaginaryValue = imaginaryPosition;

 int iter = 0;
 while ( (iter < maxIter) && (addFP128(squaredRealValue,squaredImaginaryValue).x < escapeNumber) )
 {
     imaginaryValue = addFP128(doubleFP128(multiplyFP128AnyAny(realValue, imaginaryValue)), imaginaryPosition);
     realValue = addFP128(addFP128(squaredRealValue, negateFP128(squaredImaginaryValue)), realPosition);

        squaredRealValue = multiplyFP128AnyAny(realValue, realValue);
        squaredImaginaryValue = multiplyFP128AnyAny(imaginaryValue, imaginaryValue);

  iter++;
 }
 if(iter >= maxIter)
  iter = maxIter + 1;

 outputi[imaginaryId * hRes + realId] = iter;
}

This new implementation gives me the ability to delve much deeper into the Mandelbrot Set but at the cost of significantly reduced performance.

On my AMD 6970 Radeon graphics card:

Workgroup size: TwoTuple{firstElement=16, secondElement=16}
10 runs of MandelbrotViewImpl{maxIter=512, realAxisCentre=-0.75, imaginaryAxisCentre=0, realAxisPixelCount=1280, imaginaryAxisPixelCount=1280, realAxisPixelSize=0.00234375, imaginaryAxisPixelSize=0.00234375, mandelbrotGenerationStrategy=FP128} took on average 157.7323556 milliseconds
10 runs of MandelbrotViewImpl{maxIter=512, realAxisCentre=-0.75, imaginaryAxisCentre=0, realAxisPixelCount=1280, imaginaryAxisPixelCount=1280, realAxisPixelSize=0.00234375, imaginaryAxisPixelSize=0.00234375, mandelbrotGenerationStrategy=DOUBLE} took on average 7.4757887 milliseconds
10 runs of MandelbrotViewImpl{maxIter=512, realAxisCentre=-0.75, imaginaryAxisCentre=0, realAxisPixelCount=1280, imaginaryAxisPixelCount=1280, realAxisPixelSize=0.00234375, imaginaryAxisPixelSize=0.00234375, mandelbrotGenerationStrategy=SINGLE} took on average 6.429966899999999 milliseconds

Selenium, HtmlUnit, JavaScript and JQuery - Writing Performant Tests

I'm working on a project that uses a certain amount of JavaScript to build a dynamic UI, the project team were of the opinion that we used barely any JavaScript and were flummoxed as to why the integration tests using Selenium with the HtmlUnit Driver were so slow.

I finally got time a couple of weeks ago to investigate the problem and because I tend to avoid JavaScript like the plague, it took me a little while to garner the necessary skills and utilities to diagnose the problems.

I thought I would share the lessons I learnt in case anyone else comes across similar problems.

The first thing I did was to upgrade all the libraries to their latest versions this often fixes performance bugs and can highlight problems with the code base. In my case it only garnered a minor speed up but highlighted the fact that the old version of Selenium we were using allowed interaction with hidden and disabled HTML elements and a number of our tests were badly written.

The second thing I discovered is that the Selenium / HtmlUnit event model means that SELECT element change events were only fired when focus changed to another element in the page.

Seeing no major speed increase I instead reverted to profiling the Java processes. We don't have access to a commercial profiler but a remarkable amount can be achieved using VisualVM which is included as part of the Sun JDK since about version 6.012.

The first thing you should always do when installing this utility is to increase the memory available using the VisualVM options file in the JDK configuration folders.

I was very careful to limit what I profiled to only a certain subset of the classes involved as even with the tweaked memory settings VisualVM is a little flaky when profiling large, long-running processes.

I used the CPU profiling capability and focused in on the projects test classes, the Selenium classes and the HtmlUnit classes.

After a certain amount of faffing around I discovered that a large amount of time was spent in the JavaScript and particularly around event bubbling and DOM change listening.

Unfortunately there is a certain level of indirection involved in the JavaScript implementation and could not discover precisely which JavaScript elements were involved even when using Java debugging.

I instead reverted to using FireBug and FireQuery to profile and navigate through the JavaScript within Firefox. I quickly discovered some obvious JavaScript performance problems and fixed them. Equally rapidly I discovered that the performance characteristics of the Firefox JavaScript engine is very different to those of the HtmlUnit JavaScript engine - Mozilla Rhino.

FireBug and FireQuery were excellent for helping me to understand what was executing and would be perfect for profiling browser performance issues, I would have to look elsewhere for the root cause of the HtmlUnit performance problems.

I resorted to old-school - comment stuff out until it broke or performed faster.

The problem was rapidly resolved to being the use of JQuery live() methods. They are designed to handle events highly dynamic DOMs by listening to events that have bubbled all the way up to the root and then matching the source of the event to a selector before calling the method intended to handle the event.

I replaced them with JQuery bind() and delegate() methods and saw a massive performance increase: A UI test suite that took 26 minutes to run completed in 10.

I learnt that highly dynamic JQuery based scripting could cause severe test performance problems and learnt a lot about how it worked. Longer term I'm going to move to statically bound JavaScript as much as possible particularly when I need to be able to test it.

A framework like JQuery is a little like an iceberg, it may look like you're barely using any JavaScript but under the surface...

- Posted using BlogPress from my iPad

On the Insanity of Expecting Nothing to Happen...

On my current project we're building an asynchronous message based system with various outputs, in particular a file for consumption by a mainframe.

On a number of occasions tests have been written that assert that something has not happened.

When you're testing synchronous processes this is a perfectly reasonable assertion to make, when the process returns all the work will be done.

For asynchronous processes?When the process returns then you have no direct way of knowing if it has completed.

For many people the most obvious way of checking that something has not happened in an asynchronous process is the counterpart for checking that something has happened asynchronously. You put in a timeout.

The problem with this is the likelihood of a given outcome. When expecting something to happen asynchronously you only have to wait until it has happened which means that you only ever wait until the timeout when something has gone wrong. When expecting something not to happen, you end up waiting for the time out most of the time. This means that for every assertion that nothing will happen you end up adding a fixed timeout that causes your test phase to increase in duration. 

When you use a time out to wait for something not to happen, you risk setting the timeout too short and the test becoming invalid because the thing you asserted wouldn't happen occurs after the test completes. This can lead to further increases in the timeouts to ensure that the tests remain valid.

There is a related problem with testing asynchronous processes where the test only checks an intervening step in the process before moving on the next test. This leaves you open to side effects from the processes started by the preceding tests causing hard to diagnose problems with your later tests. This is particularly likely with tests that assert that something doesn't happen as they have a greater chance of leaving the test before the process under test has completed. 

How do we avoid these problems and still be able to make assertions that something hasn't happened?

The 'Sentinel' pattern is a good place to start. 

Send a second request after the first that will produce a result after the first has completed, in this way you can assert that if you only see the result of the second request then you can state that the first request hasn't produced anything. Maybe you can ask the asynchronous process to do two things in one request and can then assert that only the effects of the additional work are visible. 

Alternatively you can look at other ways of knowing that the process under test has completed. Check the logging for a statement that is only produced when the process has completed or maybe introduce an event or message that is produced when the process completes regardless of whether the other output is produced. 

It is always a bad idea to expect 'nothing' to happen asynchronously, always expect something. 

Debugging Neo4J Spatial - Update 3 on Neo4J Spatial and British Isles OSM Data import.

I have been having a long discussion with Craig Taverner over an issue that I think that I have identified in the Neo4J spatial codebase with particular reference to the OSM code. Unfortunately I have completely failed to communicate the issue that I have perceived. Craig has variously believed that the issue is in my code or my misperception of the Neo4J spatial unit test code.

The actual issue is buried deep in private and protected methods within the Neo4J Spatial codebase in particular in the interaction between the spatial search classes and the OSM layer and index. I have yet think of a way to write a simple and direct unit test that will illustrate the problem so instead I am going to write this blog entry to explain what I found and how I found it.

I actually found the issue when I was attempting to debug why some of my unit test code was failing to return the results that I expected. It turned out that my problem was because I had swapped latitude and longitude on the point that I passed in. While debugging I spotted another issue in the way that the spatial search classes (in particular the org.neo4j.gis.spatial.AbstractSearch and org.neo4j.gis.spatial.query.AbstractSearchIntersection classes) interact with the OSM Layer.

The actual issue is that the AbstractSearch class implements its own getEnvelope(Node geomNode) method that takes a node as a parameter and uses the related layer's GeometryEncoder to parse the node and create the envelope. This method is used in AbstractSearchIntersection's needsToVisit(Node geomNode) method to determine whether to visit the node or any of its children.

When I execute the following code which I believe is correct:

SearchContain searchContain = new SearchContain(point);

osmLayer.getIndex().executeSearch(searchContain);

The OSMLayer's GeometryEncoder is used by AbstractSearch to decode the node's envelope and this produces an envelope with the bounding attributes mixed up.

For many searches this is not a huge issue and the search will appear to work. In the case of the Buckinghamshire (Bucks) search the Bucks envelope no longer subtended fractions of a degree but instead subtended 50 odd degrees of latitude and longitude which included Bucks' true envelope.

As far as I can make out the correct results are returned because within the OSM layer the nodes coordinates and bounding boxes are handled differently so even if the spatial search classes visit unnecessary nodes, the correct answers are returned. This means however that there may be performance issues due to unnecessary visits. Also I think that there may be situations where index nodes that should be visited aren't.

At a fundamental level I think that there is some pretty bad schizophrenia between the way that OSMGeometryEncoder handles envelopes in the Envelope decodeEnvelope(PropertyContainer container) method and the way that the RTreeIndex handles envelopes in the private Envelope bboxToEnvelope(double[] bbox) method.

My gut feeling is that all envelopes should be handled by the relevant GeometryEncoder rather than by a private method on an index, However the OSMGeometryEncoder just plain seems to get it wrong.

Update 2 on Neo4J Spatial and British Isles OSM Data import.

Well even 6GB of heap space isn't enough to fully import and re-index the British Isles Open Street Map Data set. I'm thinking that there will have to be some work on memory management somewhere. First thing I'm going to need to do is profile the import and work out where the memory is being used (I'm assuming it isn't a memory leak per se). Basically I want to determine whether it is because of a misconfiguration by me or there is a need to look at the memory usage in the OSMImporter class.

I'm also thinking that for reverse geo and local searches I really don't need the full OSM data set so a customised version of the OSM importer would be a good idea. I really don't need ways or the node data associated with them. I suspect a multi-pass approach would be a good idea. The first pass through the OSM data set determines which nodes are needed for the features that I want to import, the second pass imports the required nodes and their associated features, followed by a final re-indexing.

I've been in discussions with one of the people behind Neo4J Spatial, Craig Taverner, on the Neo4J User List. I found an issue with the bounding box used in the spatial search index visitor pattern. It's been quite illuminating to dig into this. I'd been thinking for a while that a number of the techniques that I encountered working with 3D graphics and scene graphs would be relevant and it definitely seems to be the case. I'm beginning to think that OpenGL or OpenCL backed indices could prove very useful in high throughput micro-batched situations where the bus latency can be hidden.

Update 1 on Neo4J Spatial and British Isles OSM Data import.

Successes and failures...

I managed to shepherd the import all the way through to import completion with 4096 MB of memory but unfortunately fell short when it came to the re-indexing. As I commented on my post I upped the memory to 6144 MB and restarted and it has again successfully imported and has been re-indexing overnight without falling over. I'm waiting to see how this goes.

During the past import I did a little more characterisation of the system utilisation and came up with the following: As reported before the node import process is single threaded and does not saturate my disk IO, what is more interesting is the way import which is heavily reliant on the already imported nodes. The way import saturated neither IO nor a single core on my CPU (again it appeared largely single threaded), I'm not sure what the problem is, it could be that the limit is the disk seek time / latency as the ways are composed of nodes already imported. I'm guessing that there would be a need to look at how the nodes are stored and whether any useful caching can be done to accelerate the way import.

The 'relation' import phase was too short for me to characterise.

The indexing phase does appear to be multi-threaded during a brief glance this morning before I left. using 2 cores quite consistently (4 cores are available).

Aside from the importing I've started playing with the API, using the Buckinhamshire data set that I imported on my laptop. So far not too impressed by performance, I used the code from some of the Neo4J spatial unit tests to do a way (highway) search from a given point. I chose coordinate in the middle of a lake (51.808721,-0.689735) and gradually expanded the search from 10m, 100m, 1000m, 10000m. quite slow but at least not an exponential increase in time - 14.4s, 85.8s, 88.7s, 86.1s.

Interesting to note that the OSM parts of the Neo4J Spatial codebase so far does not seem to have any convenience methods for area and boundary searches, only for way searches.

Here is my current (very rough) code below:

package com.presynt.neo4j;

import static org.junit.Assert.*;

import com.vividsolutions.jts.geom.Coordinate;
import com.vividsolutions.jts.geom.GeometryFactory;
import com.vividsolutions.jts.geom.Point;
import org.geotools.referencing.CRS;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Ignore;
import org.junit.Test;
import org.neo4j.gis.spatial.Layer;
import org.neo4j.gis.spatial.SpatialDatabaseService;
import org.neo4j.gis.spatial.SpatialTopologyUtils;
import org.neo4j.gis.spatial.osm.OSMImporter;
import org.neo4j.gis.spatial.osm.OSMLayer;
import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.kernel.EmbeddedGraphDatabase;
import org.neo4j.kernel.impl.batchinsert.BatchInserterImpl;

import javax.xml.stream.XMLStreamException;
import java.io.IOException;


public class TestNeo4JSpatial {

   private static GraphDatabaseService graphDB;
   private static SpatialDatabaseService spatialDB;
  
   @Test
   @Ignore
   public void createSpatialDB() throws XMLStreamException, IOException {
       OSMImporter importer = new OSMImporter("OSM-BUCKS");
       BatchInserterImpl inserter = new BatchInserterImpl("/testDB");
       importer.importFile(inserter ,"/OSMData/buckinghamshire.osm");//"british_isles.osm"); // "C:\\british_isles.osm");
       inserter.getGraphDbService().shutdown();
       graphDB = new EmbeddedGraphDatabase("/testDB");
       importer.reIndex(graphDB, 100000);
       graphDB.shutdown();
   }

   @Test
   public void retrieveLayer(){
       final Layer layer =
               spatialDB.getLayer("OSM-BUCKS");
       assertNotNull(layer);
       assertEquals(OSMLayer.class, layer.getClass());
   }

   @Test
   public void useLayer() {
       final OSMLayer osmLayer = (OSMLayer)spatialDB.getLayer("OSM-BUCKS");
       final GeometryFactory factory = osmLayer.getGeometryFactory();
       System.out.println("Unit of measure: " + CRS.getEllipsoid(osmLayer.getCoordinateReferenceSystem()).getAxisUnit().toString());

       final Point point = factory.createPoint(new Coordinate(51.808721,-0.689735));

//        final Layer boundaryLayer = osmLayer.addSimpleDynamicLayer("boundary", "administrative");
//        SearchContain searchContain = new SearchContain(point);
//        boundaryLayer.getIndex().executeSearch(searchContain);
//        for(SpatialDatabaseRecord record: searchContain.getResults()){
//            System.out.println("Container:" + record);
//        }

       final Layer highwayLayer = osmLayer.addSimpleDynamicLayer("highway", null);
       long startTime = System.currentTimeMillis();
       SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 10.0);
       System.out.println("10m took:" + ((System.currentTimeMillis() - startTime) / 1000d) +"s");
       startTime = System.currentTimeMillis();
       SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 10.0);
       System.out.println("10m took:" + ((System.currentTimeMillis() - startTime) / 1000d) +"s");

       startTime = System.currentTimeMillis();
       SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 100.0);
       System.out.println("100m took:" + ((System.currentTimeMillis() - startTime) / 1000d) +"s");
       startTime = System.currentTimeMillis();
       SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 100.0);
       System.out.println("100m took:" + ((System.currentTimeMillis() - startTime) / 1000d) +"s");

       startTime = System.currentTimeMillis();
       SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 1000.0);
       System.out.println("1000m took:" + ((System.currentTimeMillis() - startTime) / 1000d) +"s");
       startTime = System.currentTimeMillis();
       startTime = System.currentTimeMillis();
       SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 1000.0);
       System.out.println("1000m took:" + ((System.currentTimeMillis() - startTime) / 1000d) +"s");

       SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 10000.0);
       System.out.println("10000m took:" + ((System.currentTimeMillis() - startTime) / 1000d) +"s");
       startTime = System.currentTimeMillis();
       SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 10000.0);
       System.out.println("10000m took:" + ((System.currentTimeMillis() - startTime) / 1000d) +"s");
       startTime = System.currentTimeMillis();
//        for(SpatialTopologyUtils.PointResult result : SpatialTopologyUtils.findClosestEdges(point, highwayLayer, 10000.0)){
//            System.out.println(result);
//        }
   }

   @BeforeClass
   public static void initialiseDatabase() {
       graphDB = new EmbeddedGraphDatabase("target/test-classes/spatialTestDB");
       spatialDB = new SpatialDatabaseService(graphDB);
   }

   @AfterClass
   public static void shutdownDatabase() {
       graphDB.shutdown();
       spatialDB = null;
       graphDB = null;
   }
}

Neo4J Spatial and Importing the British Isles Open Street Map Data

I know, I know. Overly ambitious and all that.

While we've been working on Presynt we've been having fun and games with Geo Data; both for Local Search and Reverse Geo lookups. We've also got a number of ambitious new Geo features planned for future versions that needs needs some form of Geo store.

So with that in mind I started playing with Neo4J Spatial and the Open Street Map data set.

It's been an interesting experience with some gotchas as Neo4J Spatial is still not fully ready for prime time, so I thought I would share my experience with those who are interested.

First off I thought I would try to import the full global data set from Open Street Map. Not necessarily a mistake, but a huge undertaking as we are talking about over 200GB of map data in raw OSM format, billions of data points and relationships.

The first gotcha is that if you wish to import the full OpenStreetMap data set I'd recommend starting with at least Neo4j version 1.3 Milestone 4 as that version was the first one to hugely expand the number of nodes supported from 4 billion to 128 billion.

Of course attempting this type of import I also started running into memory problems. I ended up running with -Xmx4096m and -XX:+UseConcMarkSweepGC. The concurrent mark and sweep garbage collector proved most resilient to the loads placed on it. The default parallel GC would fall over and cry quite regularly due to an issue with consuming too much CPU to too little effect.

In the end I discovered that the full global import would take too long to run and set my sights a little lower on just the British Isles data set which is all that Presynt needs right now.

In order to test my set up I decided to use the Buckinghamshire data set which is the smallest county data set in the UK and this highlighted another gotcha: Not all the dependencies required are actually included in the Neo4J POMs.

In the end I used the following dependencies:

• org.neo4j:neo4j:1.3.M04
• org.neo4j:neo4j-spatial:0.5-SNAPSHOT
• org.geotools:gt-referencing:2.6.5
• org.geotools:gt-main:2.6.5
• org.geotools:gt-cql:2.6.5
• org.geotools:gt-epsg-hsql:2.6.5
• com.vividsolutions:jts:1.11

They may not all be necessary but they certainly work...

Once I had resolved these issues then I was able to start the full British Isles import. I was running it on a quad-core machine and found that the OSMImporter/BatchInserter combination (all default configuration mind) was using just one CPU. Looking at the OSMImporter codebase it became obvious that it was written to stream read the OSM xml data and synchronously write it out to Neo4J. I think that there may be some room to parallelise it as large amounts of the GPS node data is utterly independent and the XML data set is structured in such a way that it could be broken down into independent units of work. The only points where serialised behaviour are required are during the parsing of the XML, the construction of the Way and Relation data and possibly the writing to Neo4J.

The process seems to be largely CPU bound as I ran it on SATA II RAID 0 SSDs and their read and write capabilities were barely touched while the single CPU in use was pegged at 100% almost continuously.

The import of the British Isles dataset seems to take about 48 hours.

I used the vanilla OSMImporter and BatchInserter configurations as I am still a Neo4J Neophyte (bad pun intended ;-)) I will be reading up on alternative configurations to see if I can improve the performance. I will also look at the import of changesets so that I shouldn't need to do a full inport again.

My Web Comics...

I've been lifelong comic reader, starting with Peanuts, Giles and Thelwell but then graduating to the delights of The Far Side, Calvin and Hobbes and Bloom County. In many ways they were as influential to me as all the conventional books I read as among their humour was much wisdom.

With such strong early influences it's little wonder that I've kept on reading cartoons though I've transferred my allegiance from the funny pages to the web comic. I think that this is because syndicated cartoons have lost their way - both as a result of lack of courage and the pressures of the modern newspaper business.

I now regularly read a number of web comics and I would recommend these whole heartedly to anyone:

• Butternutquash - Telling the tales of a group of friends that hang around a comic strip. It is far more cynical than you would expect but always funny. I love the Narcoleptic Dog.
• Ctrl+Alt+Del - A delightfully odd ball comic strip following the personal and imaginary life of a dedicated gamer.
• Diesel Sweeties - The life and loves of an assorted group of people, including a fading porn actress her sometime boyfriend - a robot, a chronic hipster, a Canadian, a very disturbing furry and the list goes on. This comic strip has resulted in some of my favourite T-Shirts.
• Girl Genius - A gloriously drawn story of an orphaned. young woman who discovers that she has inherited the 'Spark' from her parents; the ability to build amazing steam-punk devices. This is a complex and involving tale set in a richly realised world.
• Gunnerkrigg Court - I only discovered this recently - A comic that becomes more beautiful as the artist develops his skill. telling the story of a conflict between science and magic through the eyes of a young girl thrown into the middle.
• Kukuburi - By the artist of Butternutsquash. Again a fabulously beautiful comic strip set in a seemingly dream world. Now sadly on hiatus but well worth reading.
• Orneryboy - A richly humorous gothic story about a couple who live with Brian the Zombie and three cats, one of whom is a ghost.
• Penny Arcade - A pure geek gamers comic strip but highly entertaining if you get the jokes. I'm a particular fan of Catsby and Twisp who crop up from time to time.
• PvP - The Godfather of web comics. Many web cartoonists cite Scott Kurtz as their influence both artistically and professionally. He is credited with pioneering making web cartooning a viable career. The comic itself follows the adventures of a bunch of journalists working at a games magazine.
• Scary-Go-Round - A very British comic strip and all the better for it. Over the years it has mutated but always centres around ordinary people living very un-ordinary lives.
• Sluggy Freelance - For a comic strip that started as a little story about a group of house mates this has developed into one of the richest and most perpetually rewarding comic strips that I read. Dimension hopping, big guns and humour make this a firm favourite. One character in particular should always be mentioned: Bun-Bun a psychotic dwarf lop rabbit with a flick-knife.
• The Abominable Charles Christopher - Tough to describe as it follows the perambulations of a perpetually innocent hairy beast that is largely man-shaped. He says nothing but still converses freely with the animals of the forest. Just before the summer break he has found himself face to face with a human town. I love it for the art and the crazy animal characters.
• User Friendly - One of the oldest web comics. I was introduced to this many years ago by my friend Neil. Gloriously anarchic tales told of an Internet Service Provider's support team with occasional visits by characters of the Cthulhu pantheon.

I have read many more over the years but these are particular favourites. What web comics do you read and if not why not?

Further Forays Into the HTML5 Stack - Animation

Short update this one.

I've been doing some simple animations replacing one image with another every 5 seconds using the SVG SMIL animation framework. I'm using the 'defs' element to define a number of image elements that will not be immediately rendered. I then use an 'use' element to refer to the first element in the sequence before using the 'animate' element to alter the 'xlink:href' attribute of the 'use' element every 5 seconds from a set of values in the 'animate' element.

Unfortunately this approach seems to be very CPU intensive even between changes to the 'use' element's 'xlink:href' attribute.

I'm wondering whether the animation timer does not take into account the down time between the changes and just spins its wheels, wasting CPU, all the time.

I think that I need to do some rough benchmarking to work out whether to use CSS3 or SVG SMIL animation and to work out which techniques are more CPU friendly.

Forays into the HTML5 stack.

While working on Cazcade with Neil, Dan C., Jon et al I'm also working on my own little project - DiarWise. It's still pretty stealth so I'm not going to discuss it in detail on an open forum but I think that now is a good time to discuss some of the technology that I've been using.

For DiarWise I've been looking at a much more rich UI experience highly interactive and completely resolution independent. I wanted to be able to create a UI that would be equally renderable on a smartphone, a tablet and a PC without extensive customisation for each. I also wanted a UI that would would be able to interact well with the browser event models for mouse and multi-touch.

I've been focussing on SVG embedded in HTML over the HTML 5 Canvas element for a number of reasons:

• It is significantly more mature as a specification than the HTML 5 Canvas element.
• The HTML 5 canvas element is procedural in its rendering so with a command it renders pixels directly to screen and so does not have a screen graph; the SVG xml elements are the scene graph and are embedded directly in the DOM and so are manipulable using all the tricks that we are familiar with. This also means that a lot of the heavy lifting for re-drawing is done for us.
• Text is more of a first class citizen in SVG and so is easier to make available to Accessible browsers.
• SVG graphics elements are style-able using CSS.
• HTML can also be embedded back into SVG (and manipulated via the DOM) using the foreignObject element

There are a few weaknesses as I've discovered as I've worked with it.

• The default SVG animation framework is built around the SMIL standard which is pretty low level and is not always the easiest to work with though it is very powerful. I'm intending to see how well SVG interacts with the CSS3 animation effects which are much nicer to work with.
• SVG support is present in the latest iteration of all the browsers but is patchy to say the least. The best of the moment is the WebKit family, Firefox is the next best though it has big issues with the text rendering. IE9 claims to be a huge step forwards over IE8 but I have yet to try out the beta and there are question marks over how well it will support SVG animation.
• While its text support is good there is one glaring omission - word wrap. The neatest work-around is to embed an appropriate HTML element in the SVG using foreignObject.

The best trick I've found so far with SVG is the simplicity of decoupling the screen resolution from the render resolution and this cascades down to any embedded HTML elements. The 'svg' element defines a local coordinate space for all its contents and you can override the coordinate system using the viewBox attribute. This means that all embedded elements can be positioned and sized relative to each other in terms of local 'pixels' and the SVG renderer will scale according to the screen resolution. The handles aspect ratios elegantly using the preserveAspectRatio attribute. This gives you the ability to present a consistent interface in a wide range of resolutions and can handle pan and zoom exceedingly smoothly for people with limited eyesight.

So far my grand experiment with SVG is going well and it seems to be well worth using as one of the technologies that make up the HTML 5 stack.

My life with BDD...

This is a blog post that's been very long overdue, but I finally acknowledged that I really need to capture my experiences after a chat with an old friend at Twitter (shameless name drop...).

I first came across Behaviour Driven Development thanks to Mauro Talevi, my first hire on the team that I set up at HSBC back in 2008. He's a major contributor on the JBehave project that is one of the major BDD frameworks. I hope that I'm not going to embarrass him too much by saying that he was an invaluable influence on the project.

I'm not going to talk about the technical implementation as that is well covered elsewhere but instead in this blog I will talk about the general principles and practical experience of working with BDDs.

The basic idea is simple. Any story or feature being implemented in a project should have automated integration tests that proves that the work has been completed successfully. It is important the the tests be comprehensible to the stakeholders requesting the story / feature so that they can confirm that it is complete and done. This provides a very clear goal for the developers and works to build up a suite of regression tests that give a healthy confidence in the application. The tests are defined in terms of business scenarios that represent various paths through the business process.

BDD is complementary to Unit Testing and is in no way a replacement.

The key to building an effective suite of BDD test scenarios is the derivation of a clearly understood Domain Specific Language for the tests. This language bridges the gap between the business concerns and the services and components created by the development team. In one sense this should not be a difficult task as the solution domain being developed should reflect the problem domain defined by the customer. Thus the effort needed to bridge the gap between the DSL and the technical solution is relatively small.

We had a lot of difficulty at first working with the stakeholders to agree and define the DSL as they were used to more traditional development practices and we had to educate them to understand that the scenarios that they were producing were testable. What they had to understand was that the scenarios were the contract between the development team and the project stakeholders. If the scenario was incorrect as signed of by the stakeholders and the analysts then the work produced by the development team would be incorrect too. The scenarios provided a very important structure to the SCRUM methodology we used. The analysis producing the scenarios was typically 1 to 2 sprints ahead of the implementation work.

JBehave provides a set of tools for building the elements of the DSL and for running the scenarios defined in it. Any scenarios in JBehave are written in simplified plain English (or the language of choice for the organisation). A scenario is structured as a list of sentences defining steps in the scenario. These sentences all begin with one of three key words:

• Given: The remainder of the sentence defines pre-conditions for the next steps of the scenario.
• When: Defines activity that the system under test is carrying out as part of the test.
• Then: The assertions that the test should be checking.

These sentences are the reusable elements of your BDD scenarios; the Givens, Whens and Thens can be put together to build different scenarios. The sentences are of course parameterised so that different values can be used in different tests.

What proved very important is the granularity of the concerns addressed by the DSL. Too coarse and you have very short (but deep) scenarios made of elements which are almost impossible to reuse; too fine and the size of the scenarios balloon and they become unmaintainable because of the sheer volume of change required as the system evolves.

In the end on the project at HSBC the appropriate level emerged organically, we paid close attention to the tests and worked hard to refactor and leverage common elements that emerged.

Another critical element is the data representation. When we started we bootstrapped the BDD test data management by using XStream to dump the objects help in memory to be compared with saved text files. This proved very helpful in the early stages but proved an increasing burden as the suite of tests broadened and deepened. The problem was that the test data was acutely sensitive to the domain data model and as that evolved we found ourselves expending major effort to correct tests that were not testing the changes but were affected because of our dumping the whole domain model.

The solution was to move to a factory and builder based mechanism. All tests used the same basic data sets generated by factories which then used builders to alter them to match the test requirements. When we came to make assertions we found that it was best to focus on the data elements required by the specific test and allow other tests to check their own data elements. This dramatically reduced the amount of data that needed incidental maintenance.

As we worked we found that after every major release it was worth spending some time on consolidating the BDD scenarios. The earlier scenarios for new functionality tended to be supplanted by scenarios testing more elaborate versions of the functionality. We worked to actively identify situations where this occurred and reduced the volume of tests while keeping their coverage.

JBehave integrated very effectively with Selenium and proved very effective at automating Web UI testing but rapidly exposed shortcomings in Selenium's implementation. By the time I left work was underway to use the WebDriver integration to address this.

By working to continually polish the DSL and the data representation we started seeing great reuse and efficiencies in developing provably tested new functionality. Where we originally spent anything up to 50% of the development time on building the BDD scenarios we saw that fall to as little as 5% of the development time.

Where this really began to pay dividends was that we had the tools that the test team needed to build out a much wider suite of tests and as they developed them we had probably the most thoroughly tested system I have ever worked on. The regression tests were run on our continuous build server producing near real time feedback as we developed which meant that problems were fixed as they occurred rather than after weeks of formal testing. This meant a much smaller testing team was required to thoroughly test the system at release. Fewer regressions were found and we were much more confident that the acceptance testing cycle would run on budget.

BDD is a key technique that I will use on future projects. It provides real benefits for the work invested and contributes materially to successful delivery.

Some New Social Networking Infrastructure

I'm a long time Facebook and LinkedIn user but as part of my involvement in developing Cazcade I've had to get my feet wet with Twitter.

I've been running FlipBoard for a while and I've found an outlet for sharing what I find on it by adding my Twitter account. I've now joined my blog to my Twitter account by using TwitterFeed and this post will test the whole process.

Twittering is very addictive and I suspect that I will begin to make more and more use of Twitter both to publish and consume content. Certainly Twitter together with FlipBoard makes consumption entirely easy and pleasurable. I'm very keen to bring together a number of my favourite RSS feeds in one place and read them in FlipBoard which has yet to have direct RSS support so TwitterFeed will be further utilised to aggregate them into one Twitter account.

What's interesting to me is how Cazcade fits into all of this. FlipBoard provides a brilliant mechanism for consuming and re-tweeting individual pieces of information. The limitation is that this is a piecemeal mechanism, you can only present one piece of information at a time. Cazcade's analogy by contrast allows you to accumulate various sources (web pages, videos, images and Twitter) into pools and present the information as a wider vision or argument and then pass them on via Twitter.

Cazcade is attempting to provide a mechanism above and beyond the current transient social network conversations.

The question I am asking myself is whether this will meet an unsatisfied need. Only time will tell.

Lyrical Ambiguity in the Kinks' Lola

I greatly enjoy a number of the Kinks songs, they were a brilliant group that created a number of classics that still resonate today.

One thing that always makes me stop and wonder is how many people believe firmly that they know the gender of Lola.

To me it seems that they have missed the brilliance of the lyrical ambiguity. The lines that really seem to be misunderstood are: "But I know what I am and I'm glad I'm a man and so is Lola.". Most people I talk to who have yet to get it believe firmly that the lines indicate that Lola is a man.

This interpretation arises from the belief that "so is Lola" refers to the fragment "I'm a man". This is a perfectly valid interpretation of the lyric but ignores another equally valid interpretation.

There is another fragment that could be referred to by "so is Lola"; "I'm glad I'm a man". If this is the reference then the interpretation would be that Lola is also glad that the protagonist is a man thus leaving Lola's gender entirely unstated.

In many ways this is the aural equivalent of the face/vase illusion where you can see either the faces or the vase but not both.

Whatever else it is, it is a brilliant song with brilliant lyrics.

- Posted using BlogPress from my iPad

iPhone 4 has arrived...

Faced a rather unique problem... I got two iPhones in my package rather than the one I expected. The nice young customer service rep nearly fell off her seat when I called and told her about it. The first time they had heard of that happening.

I think that I slightly restored her faith in humanity as she thought that no one would have called up and would have just stayed quiet about it. She asked me to drop off the extra to a store and gave me a nice in-store credit.

I'm now on my way to London waiting for the new SIM to activate and just using the iPhone as an iPod. I'm hoping that the rest of the experience is as painless as the iPad was.

- Posted using BlogPress from my iPad

Location:Stoats Nest Rd,Croydon,United Kingdom

My First Track day...

On Monday I took my life into my hands and went on a track day at Bedford Autodrome with EasyTrack.co.uk.

I had a blast! The track was ideal for a novice and my car (an Audi TT RS) proved more than capable. I had asked for a session with an instructor and I got a great guy named Steve. In the end I took a second session in the afternoon. As a result I went from being slowest on the track to not being the slowest by a decent margin. There were some fabulous drivers on the track that day, the two sticking in my mind included one in a GT3 RS and an utter genius in a bog standard MX5.

I've included a photo that was taken by TrackPhoto.co.uk.

- Posted using BlogPress from my iPad

The Height of Geekery

Well, here I am at my favourite cafe for breakfast with my iPad. I'm trying the whole blogging from my iPad thing with an app called BlogPress.

It's really interesting to see how modern integrated technology is making it feasible to do things that would have been awkward if not impossible a few years ago. The opportunities in this space are endless.

The excitement about the potential for location based services is well known. Unfortunately the excitement can obscure the real win. Location-less Services. The fact that I can start doing this kind of thing truly independently of my location is what's exciting. There have been some early mis-fires with the excitement of being able to take a laptop with you on holiday. Lugging a laptop though is really pretty hard work. We're on the cusp of being able to access and interact with the "datasphere" anywhere in the world.

This is where reading science fiction is a great advantage. Writers have been exploring how to use this for years. I feel that I am ready to make use of this and build on their thinking to go places that people who have not been reading will never think of.

How about you?

- Posted using BlogPress from my iPad

Location:Cliffe High St,Lewes,United Kingdom

Constructor Versus Setter Based Dependency Injection

There seems to be two extreme camps here and no middle ground.

I, as usual, agree with neither extreme.

The simple fact is that my preferred approach is to design classes that meet the original intent of object orientation. That the class is fully usable after construction.

That is not to say that I believe that Constructor injection is the one, true way. Rather, I believe that all mandatory dependencies should be configured in the constructor and optional ones should be satisfied by setter methods. This makes the class much easier to understand. It also means that the constructor does not end up completely overloaded with parameters but neither are you left looking at a a sea of setter methods wondering which ones have to be used.

By using this convention the code becomes more self-documenting.

The problem with modern 3D films

What are the problems and limitations of modern 3D? I'm not going to claim any particular expertise in this subject past a decent familiarity with 3D computer graphics. However I do think that there are a couple of key points that are passing people by in the recent excitement about 3D cinema. The pedant in me wants to go on record ;-).

The main thing that is not being said is that the 3D we are experiencing is solely 'Stereoscopic 3D' this means that everyone in the audience is seeing the same 2 images for every frame of the movie.

The best way of highlighting the limitations is to point out the differences in experience between real 3D (i.e. the world around us) and the 3D we experience in the cinemas.

While we do experience the world through two eyes those eyes do have to a degree the ability to sense distance individually through their focus - we do not completely rely on the stereoscopic effect to determine distance and so never lose the ability to determine how far away something is even if we were to lose an eye. This means that the illusion can never be complete as the focus in a 3D movie is determined by the director and camera departments. If something catches your eye that is not in the plane of focus then all you can ever see is a blur though your eyes may strain to focus it. This is both a blessing and a curse as it provides another tool to the director to direct your attention within the frame but can also wreck the illusion.

Another issue is that the stereoscopic effect is achieved by filming the scene from a fixed viewpoint. This means that the frames captured look best from a specific place in the cinema. The further away from that position the less effective the 3D effect is.

A related issue is that the illusion requires one orientation of the spectator. If you lean your head to one side the illusion starts to dissolve as instead of seeing the frames for the top eye and the bottom eye you will keep on seeing the frames for the left and right eyes. This gets worse if you were to continue the rotation as the eyes would then be reversed... Not many people will see the movie standing on their heads but couples leaning their heads on each other's shoulders will have a problem.

The final issue is that of movement. In dealing with a 3D world we are used to moving around it and seeing different perspectives. Despite the fact that we are largely fixed in our seats in a cinema we still have the impulse to crane up or slump down, lean left or lean right just to see that little bit more. Unfortunately the Stereoscopic 3D effect cannot handle that behaviour.

It worries me a little that there is talk of bringing Stereoscopic 3D into the home as there may be an effect where people are trained to sit absolutely still in one orientation to maximise the effect. There are already concerns about how children sitting still in front of the traditional 2 D television and not fidgeting are contributing to the obesity epidemic, how much worse may it become with Stereoscopic 3D TV?

What do I think about Business Analysis?

At work we're currently looking for a new Business Analyst and it has given me pause to review my thoughts on Business Analysis. I originally trained as an Analyst/Programmer and have during the course of my career had cause to do rather a lot of Business and Systems Analysis even though you will not find a huge amount about it on my CV.

I started off using Entity Relationship Diagrams as an Oracle Specialist and, as I encountered more types of formal analysis, made a point of learning about all of them.

As a result of my Java work I became exposed to UML as a diagramming notation for OO programming concepts but over time came to appreciate it as a fairly complete set of diagrams for supporting Business and Systems Analysis.

The recent search for a Business Analyst has highlighted to me that an awful lot of people are described as Business Analysts who, to my mind, do not have the skills to perform the job.

A good Business Analyst can add a great deal of value to a project. They are the front runner in terms of understanding the business domain. They have the following key roles to perform:

• Working with the business to understand the business domain.
• Capturing and documenting business requirements.
• Analyse the business domain to highlight areas of inconsistency and identify areas of uncertainty.
• Distill and communicate the business requirements to the implementation team.

There is a continuum of competency between the parrot and the professional.

The parrot is the so-called Business Analyst that just sits between the implementation team and the business and just acts as a conduit for questions and answers - actively subtracting value from the exchange by slowing it down.

The professional is, for me, the Business Analyst that can work with a business that has yet to fully formulate a problem and work with them to produce a full and clear definition of the problem and the business solution that can be used by the development team. The professional actively helps the Business think about the problem space and gives them the tools to properly understand it themselves. The professional will be able to answer a very high percentage of the implementation team's questions without recourse to the business as the questions have already been addressed by the analysis.

As far as I can tell the vast majority of Business Analysts in the Investment Banking sector fall perilously close to being parrots. While I am most used to UML, any business analyst that can use a clear approach to formally analyse the business domain and its requirements adding value all the while is fine by me.

As many of my colleagues will know I firmly believe that 'code is cheap' and I have come to realise that the same holds true of analysis documentation. What is truly valuable is the knowledge it embodies. While the concept of Bit-Rot is well understood in code; the same concept applies to analysis documentation (Word-Rot?). The real value is held in the analyst's head and in the heads of the business and implementation team at the end of the whole process.

I no longer believe that huge repositories of analysis have any real value - what is valuable is the process of producing the analysis and the knowledge that process imparts. A certain level of high level analysis documentation for invariant concepts can have longer term value but that will only be a small fraction of all that is produced.

I believe that there is no value to analysis artefacts that cannot be formally tested. Once an unit of analysis is complete it should be handed over to a tester for formal testing and then be used as the basis for User Acceptance Tests for the delivered system.

When a business analyst is required then you had better find yourself a professional and once the analysis is done you had better hold onto that professional. Inevitably team members move on, but perhaps the best hand over will be for the outgoing analyst to guide the replacement through a complete analysis of the domain.

I have worked with excellent analysts who have added immense value to a project and I have worked with some complete incompetents. Guess which ones I actively seek to work with again.