Automatic retries using the Transient Fault Handling from Enterprise libraries (EntLib)


Hello,

I was reading about the Transient Fault Handling at http://msdn.microsoft.com/en-us/library/hh680901(v=pandp.50).aspx This is a block of the Enterprise Libraries which is part of the Enterprise Integration Pack for Windows Azure. I haven’t played much with Azure yet, but one part of the documentation took my attention:

When Should You Use the Transient Fault Handling Application Block?
…… (some explanation about when you are using Azure services) ……
You Are Using a Custom Service
If your application uses a custom service, it can still benefit from using the Transient Fault Handling Application Block. You can author a custom detection strategy for your service that encapsulates your knowledge of which transient exceptions may result from a service invocation. The Transient Fault Handling Application Block then provides you with the framework for defining retry policies and for wrapping your method calls so that the application block applies the retry logic.Source: http://msdn.microsoft.com/en-us/library/hh680901(v=pandp.50).aspx#sec10

So, I thought why do I have to write always the retry logic if this is already done ;-).

The problem

We want to write a generic retry mechanism for our application. Moreover we need to support some level of customization thru configuration files.

The solution
Simple let set up the Transient Fault Handling Block (from now on FHB). First of fall you will have to download the libraries, you could do it using NuGet – which by default adds also references to support Windows Azure -. You can remove the Windows Azure DLL if you’re not going to be using Windows Azure in your app.

The FHB lets you specify a error detection strategy as well as retry strategy when you construct the retry policy. Also, a Retry policy support the execution of Action, Funcdelegates or Actionfor async retries.

As part of the configuration process for FHB, it’s required to indicate which errors should be handled by FHB. A mechanism for doing this is by implementing the interface: ITransientErrorDetectionStrategy. In our sample, we only want to retry when a FileNotFoundException is thrown. Thus we have coded the FileSystemTransientErrorDetectionStrategy:

using System;
using System.IO;
using Microsoft.Practices.TransientFaultHandling;

namespace SampleConsoleApplication.TransientErrors.Strategies
{
    public class FileSystemTransientErrorDetectionStrategy : ITransientErrorDetectionStrategy
    {
        #region ITransientErrorDetectionStrategy Members

        public bool IsTransient(Exception ex)
        {
            return ex is FileNotFoundException;
        }

        #endregion
    }
}

Certainly you could implement it by handling any particular exception.
Now we need to define a retry policy. This is done by specifying a retry strategy. The following code example configures a retry policy by providing a retry strategy incremental. Additionally in our sample service.DoSlowAndImportantTask will always fail, so the FHB will retry automatically based on the retry policy.

 private static void RetryPolityUsingCode(IUnityContainer container, IService service, OutputWriterService writer)
        {
            writer.WriteLine("Begin sample: RetryPolityUsingCode");
            // Define your retry strategy: retry 5 times, starting 1 second apart
            // and adding 2 seconds to the interval each retry.
            var retryStrategy = new Incremental(5, TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(2));

            // Define your retry policy using the retry strategy and the Windows Azure storage
            // transient fault detection strategy.
            var retryPolicy = new RetryPolicy(retryStrategy);

            try
            {
                // Do some work that may result in a transient fault.
                retryPolicy.ExecuteAction(service.DoSlowAndImportantTask);
            }
            catch (Exception exception)
            {
                // All the retries failed.
                writer.WriteLine("An Exception has been thrown:\n{0}", exception);
            }
            writer.WriteLine("End sample: RetryPolityUsingCode");
        }

Additionally it’s possible to specify the retry policy in the configuration file, for example:

<!--?xml version="1.0" encoding="utf-8" ?-->

<section name="RetryPolicyConfiguration" type="Microsoft.Practices.EnterpriseLibrary.WindowsAzure.TransientFaultHandling.Configuration.RetryPolicyConfigurationSettings, Microsoft.Practices.EnterpriseLibrary.WindowsAzure.TransientFaultHandling" requirepermission="true"></section><section name="typeRegistrationProvidersConfiguration" type="Microsoft.Practices.EnterpriseLibrary.Common.Configuration.TypeRegistrationProvidersConfigurationSection, Microsoft.Practices.EnterpriseLibrary.Common"></section>

Please notice that there is a default Retry Strategy indicating that will use “Fixed Interval Retry Strategy”. Our C# code will ask for the RetryManager instance in order to retrieve the Retry Policy and then perform the task.

private static void RetryPolityUsingConfigFile(IUnityContainer container, IService service, OutputWriterService writer)
        {
            writer.WriteLine("Begin sample: RetryPolityUsingConfigFile");
            // Gets the current Retry Manager configuration
            var retryManager = EnterpriseLibraryContainer.Current.GetInstance();

            // Asks for the default Retry Policy. Keep on mind that it's possible to ask for an specific one.
            var retryPolicy = retryManager.GetRetryPolicy();
            try
            {
                // Do some work that may result in a transient fault.
                retryPolicy.ExecuteAction(service.DoSlowAndImportantTask);
            }
            catch (Exception exception)
            {
                // All the retries failed.
                writer.WriteLine("An Exception has been thrown:\n{0}", exception);
            }
            writer.WriteLine("End sample: RetryPolityUsingConfigFile");
        }

The Transient Fault Handling Block a simple and flexible alternative for a well-known task. A automatic retry system, moreover this supports configuration file, so we should be able to change the retry policy just by modifying our configuration file.

Here is an screen shot of the app, which automatically retries calling our mischievous method.

Transient Fault Handling Tries Output Capture

As usual the code sample is at https://github.com/hmadrigal/playground-dotnet/tree/master/MsEntLib.TransientErrorHandling

Advertisements

7 comments

  1. Good write. I’m trying out the retrypolicy when I perform a putblock.
    Below is the code

    var retryManager = EnterpriseLibraryContainer.Current.GetInstance();
    var retryPolicy = retryManager.GetRetryPolicy
    (“Fixed Interval Retry Strategy”);
    retryPolicy.ExecuteAction(
    () =>
    {
    Parallel.For(0, mySet.Count(), options, i =>
    {
    blob.PutBlock(mySet.ElementAt(i).Id, new MemoryStream(mySet.ElementAt(i).Content, true), null);
    });
    });

    Configuration file:

    —————————————————————-

    I would like to know on how to simulate a transient fault on putblock.

    Thanks,
    Shweta

    1. Hi Shweta

      I think you could have some options for producing a transient faults.

      – You could wrap the method into a class, and hide the implementation thru an interface. Thus, using dependency injection you could get a implementation that produce transient faults.
      – Alternatively you could use a mock framework (such as RhinoMocks) which kind of produce the same pattern letting you customize a given behavior for producing transient faults.
      – Something kind of simpler it is using DEBUG symbols to product code that fails. So, the code will be only compiled when the symbol is present, enough for testing.

      I’ll try to get back with a code sample!
      Regards,
      Herber

      1. Thanks Herber for the quick reply. Please provide me with the sample code.

        Regards,
        Shweta

      2. Sorry for the late response. Here is a sample:
        https://github.com/hmadrigal/CodeSamples/tree/master/MockingTransientFaultHandling
        Basically I tried to create two approaches:

        1- A cool down time, e.g. you call your PutBlock method, but next call should happen after 200ms. If not then an exception is thrown
        2- The second approach is simpler, it uses a random number to define whether or not an exception should be thrown. E.g. if Randon.NextDouble() < 0.2 (it mean that there is a 20% chance of getting an exception).

        I hope you find then useful, something that I noticed it's that you must be careful about what method you would like to be retried automatically. In your code sample, it's using a Parallel.For so, do you want to retry all the Parallel calls or do you want to retry only the failed call. Then you must organize your code well in order to achieve this.

        Kind regards,
        Herb

  2. Hello Herber,

    Thanks a lot for the sample code. I will try it out.
    I would like to retry only the failed putblock calls within the Parallel.For. The main intension is perform putblock operation in parallel as well as retry transient faults if any with minimal time to upload data to azure blob.

    Thanks,
    Shweta

  3. Heya i am for the first time here. I came across this board and I find It
    really useful & it helped me out a lot. I hope to give something back
    and help others like you helped me.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s