Getting TransactionScope to play nice with NHibernate / by Matt Wrock

My team is beginning to integrate NHibernate into a fairly large code base that makes frequent use of TransactionScope to ensure that the saving of related entities succeed or fail together. My own NHibernate experience has been primarily on green field projects and on existing projects where the NHibernate work was well isolated from existing plain ADO calls. Furthermore, my use of transactions has typically been from within stored procedures and I have never used any of the transactional functionality exposed by the ADO APIs. So this strange thing called TransactionScope was an entirely new concept for me to understand.

Now before I elaborate on the drama and adventure that accompanied this learning experience and some of the failures that ensued, I’ll quickly jump to the moral of this story for those of you that simply want to know what is the right way to get NHibernate transactions to participate in a transaction defined by TransactionScope. Simply stated: its all in the nesting. Instantiate a TransactionScope on the outside and and begin and commit/rollback an NHibernate transaction on the inside.

using(var scope = new TransactionScope()){    SaveEtitiesNotManagedByNH();    using (var transaction = session.BeginTransaction())    {        session.SaveOrUpdate(entity);        transaction.Commit();    }    scope.Complete();}

What is happening here? Instantiating a new TransactionScope creates an “ambient” transaction.  What the heck is an ambient Transaction? An ambient transaction is part of the implicit programming model provided by the Systems.Transactions namespace. The msdn documentation advises developers to use this model instead of creating their own explicit transactions. The TransactionScope is ThreadStatic and therefore all ADO operations on that thread will participate in this ambient transaction.

There are some optional constructor arguments that can be used when creating a new TransactionScope that controls whether to create a brand new ambient transaction or to participate in one that already exists. For example, in the above block, it is possible that SaveEtitiesNotManagedByNH calls into methods that also have using blocks around a new TransactionScope. If new TransactionScopes are created with the default constructor, those scopes will participate in the same transaction as the one created by the outermost  TransactionScope. See the MSDN documentation for details on creating new TransactionScopes.

Since NHibernate 2.1 onward, the NHibernate transactions will participate in an ambient transaction if one exists. So in the code block above, all NHibernate  persistence logic is a part of the same transaction as any non NHibernate persistence code since it all occurs within the same TransactionScope.

When scope.Complete() is called, it simply sets a bit on the TransactionScope that claims that everything within the scope succeeded. When the TransactionScope is disposed at the end of the using block (or the outermost block if there are nested scopes), the transaction is commited as long as Completed was called. If completed was never called, the transaction is rolled back.

TransactionScope/NHibernate Pitfalls

So now that we have looked at the right way to manage the relationship between the  TransactionScope and NHibernate Transaction, lets look at some other techniques that might seem like a good idea but I assure you are bad.

Following a Single Transaction per Request Unit of Work pattern

Our original NHibernate implementation was following a Unit of Work pattern. This is a common and very nice pattern that typically involves opening an NHibernate session upon the first call into NHibernate and then closing the session in the EndRequest method of global.asax or a custom HttpModule. Our infrastructure took this one step further and opened a transaction after creating the session and committed or rolled it back in EndRequest. For now lets kindly suspend our disbelief in bad transaction management and pretend that this is a good idea because it supports a simple model where the developer can assume that all activity in a single request will succeed or fail together.

Following this technique, it is likely that an NHibernate explicit transaction will be created before an Ambient transaction exists.

session.GetEntityById(id);

using(var scope = new TransactionScope()){    SaveEtitiesNotManagedByNH();     session.SaveOrUpdate(entity);     scope.Complete();}

Here, NHibernate is used to retrieve an entity which, in our framework, causes a Transaction to begin. Therefore the non NHibernate activity is in a completely separate transaction. When the ambient transaction is created, it will look for the presence of an existing ambient transaction. There will be none and a completely new and separate transaction will be created.  As a result, operations in these two contexts will have no visibility of uncommited data in the other.

To remedy this, we created a TransactionScope in BeginRequest and stored it in HttpContext. On EndRequest, we would retrieve that TransactionScope and call Completed and dispose of it. This way we are guaranteed that the ambient transaction  exists before BeginTransaction is called on the NHibernate transaction and we can be assured that the NHibernate operations will enlist in that transactions.

This would fail randomly because in ASP.NET, there is no guarantee that BeginRequest and EndRequest will occur on the same thread. A Transaction scope will throw an exception if you try to dispose of it on a different thread than the one where it was created. So in the occasional event that EndRequest executed on a different thread then BeginRequest, this exception was thrown.

I then tried using PreRequestHandlerExecute and PostRequestHandlerExecute instead of BeginRequest and EndRequest. Those will always occur on the same thread. It appeared that this was a working solution and worked in our dev environments. However, when we moved this to an integration environment, we began seeing database timeout errors. Reviewing the active connections in the database, spids were blocking and the spid at the top of the blocking chain was holding an open transaction. What was particularly odd was that this spid was locking a resource that we thought was being inserted outside of a TransactionScope or any explicit transaction.

It ended up that the reason this was happening is that there are certain circumstances where PostRequestHandlerExecute is never called. For example on a Response.Redirect where endRequest is false. Under such circumstances, the TransactionScope is not disposed. Since the TransactionScope is in Thread Local Storage, it remains tied to the request’s thread even after the request ends. The thread returns to the worker pool and later gets pulled by another thread. In our case, this was a background thread that typically never used explicit transactions and simply inserted a row into a commonly queried table. After the background thread finished its work, since it never calls Commit, the table remains locked and simple queries by other requests on that table time out. To summarize this unfortunate series of events:

  1. Web request begins and creates a new TransactionScope.
  2. The Request returns a 302 redirect and PostRequestHandlerExecute does not execute and therefore the TransactionScope is not disposed and remains in thread al storage.
  3. Thread returns to the worker pool.
  4. Background thread is launched from the worker pool and thread with undisposed TransactionScope is used.
  5. Thread inserts row into Table A.
  6. Thread exits and still does not call dispose and the transaction remains open.
  7. A new web request queries Table A and waits for the open transaction to release its exclusive lock.
  8. The new request times out since the lock is never released as the thread that owns the locking transaction is sitting idle in the thread pool.

This is a nasty bug that thankfully never reached production. These kinds of bugs are always difficult to to troubleshoot, occur in nondeterministic patterns and affect not only the user who came in on that thread but can bring down the entire application.

Reflecting on this scenario, it became apparent that:

  1. TransactionScope should always be limited to the confines of a using or Try/Finally block. Risking the fact that Dispose may never be called is too great.
  2. Keeping a transaction open during the entirety of a web request is extremely dangerous. It is unwise to assume that something, perhaps completely non database related, will never trigger a long running request and therefore lock up database resources that  again could bring down an entire site. This is one of the cardinal rules of ACID transactions to keep them as short as possible.

So I decided to eliminate the one transaction per request technique. We continue to use Unit of Work for Sessions, but developers will be responsible for defining their own transactions according to the needs of the request.

Failing to use explicit NHibernate Transactions within the TransactionScope

Having resolved to abandon a single master TransactionScope and continue to use the TransactionScopes currently sprinkled throughout the code, it seemed rational that we could simply invoke our NHibernate persistence calls within a TransactionScope and all would be well. Something like the following seemed innocent enough:

using(var scope = new TransactionScope()){    SaveEtitiesNotManagedByNH();    session.SaveOrUpdate(entity);     scope.Complete();}

Well this quickly started throwing an error I had never seen before when disposing the TransactionScope. The message was something like “The state of this transaction is in doubt” with some methods in the stack trace calling something like PhaseOneCommit and an inner timeout exception. I’ll admit that  I have not dug into the NHibernate source to see exactly what is happening here, but it sounds like a distributed transaction gone bad. My guess is that the NHibernate activity is treated like a distributed transaction even though it is operating on the same database as the on NHibernate code. When NHibernate saves the data, that operation enlists in the TransactionScope but NHibernate has no awareness that it is involved and therefore it will never Commits its end of the distributed transaction and causes the TransactionScope to timeout.

The Final Solution

So the final solution involved a nesting pattern like the one shown at the beginning of this post. However, to make it easier to implemented I created a wrapper to wrap both the TransactionScopoe and the NHibernate transaction:

public interface IOrmOuterTransactionScope : IDisposable{    void Complete();}

public class NhOuterTransactionScope : IOrmOuterTransactionScope{    private readonly ISession session;    private readonly TransactionScope scope;    private readonly ITransaction transaction;

    public NhOuterTransactionScope(ISession session)        : this(session, TransactionScopeOption.Required, new TransactionOptions {            IsolationLevel = IsolationLevel.RepeatableRead})    {    }

    public NhOuterTransactionScope( ISession session,                                     TransactionScopeOption transactionScopeOption,                                     TransactionOptions transactionOptions)    {        scope = new TransactionScope(transactionScopeOption, transactionOptions);        this.session = session;        transaction = session.BeginTransaction();    }

    public void Complete()    {        session.Flush();        transaction.Commit();        scope.Complete();    }

    public void Dispose()    {        try        {            transaction.Dispose();        }        finally        {            scope.Dispose();        }    }}

Using an NHOuterTransactionScope, avoids the need for developers to create and commit the separate NHibernate transaction and more importantly it enforces the appropriate nesting order.