Distributed Transactions

One of the hardest things to design in distributed systems is the transactional behaviour.

I am perpetually surprised at the naivety with which people approach this set of problems. There does not seem to be any formalism to approaching this kind of design. What I do is deliberately take as step back and design the 'transactional architecture':

1. Identify parts of the architecture that may participate in distributed transactions; components with storage that will be changed irrevocably as a result of a transaction. The obvious ones are databases and message queues which are themselves transactionally aware but one must also take into account non-transactional components such as the file systems and LDAP servers that may be changed in the scope of a transaction.
2. Document the expected transactional behaviours involving these components.
3. Design transactional schemes that meet the expected behaviours.

At each step I revisit the designs of the system based.

1. Should I really be changing this component? I may stop using the file system for storage and instead move it into a database table.
2. Does the expected transactional behaviour introduce circular dependencies between components? Is ACID-ity expected between two widely divergent transactional domains?  I may move whole entities from one component to another.
3. If there is no way to implement the behaviour can I make some change to produce the same business behaviour but avoid the implementation problems.

In general I believe that using two-phase commits (XA and the like) is the sign of a poorly designed solution architecture as they are only necessary if the system requires 'immediate' consistency between two or more transactional domains. I will use other transactional management techniques such as idempotency and compensating transactions which provide 'eventual' consistency not 'immediate'.