Dynamics CRM 2015 Online and Azure Service Bus

In the previous post I talked about sending messages to another system via a message queue.  This is the design pattern Microsoft recommends when you want to use Dynamics CRM Online to update systems that are on-premise. They recommend using Azure Service Bus and the integration is built into the Online version by creating a Service Endpoint. This is also available for the on-premise version.

The plugin registration tool for Dynamics CRM  offers the ability to register a Service EndPoint when you are connected to CRM Online.  What is not clear though is that there are two ways you can configure it.
First though there are two gotchas when setting up the Azure Service Bus.

GOTCHA #1: You set up the Azure Service Bus Namespace using the Portal.  Wrong.
Doing it that way you no longer have the option to use ACS authentication and that is what the Plugin Registration tool uses.  Delete it. Download the PowerShell Azure Commands add-in and run this command:
New-AzureSBNamespace -Name yourservice -Location "West Europe" -CreateACSNamespace $true -NamespaceType Messaging

The response you get returned is

You need to use the DefaultKey when the Plugin Registration tool prompts you for the Management Key!

GOTCHA #2: You create the Queue (or whatever) using the Portal or PowerShell. Wrong.
You need to leave this for the Plugin Registration tool to do.

I won't give the rest of the details for configuring the endpoint because that is covered in other blogs.

Once you have the Service EndPoint registered there are two ways forward.

The first and seemingly the most attractive option is you can register steps and images right there under the endpoint just as you would do with a plugin.  The advantage is that this is a zero code solution. Just by configuring an Entity with the appropriate step and image you can get messages in your queue (or whatever). The thing is though is this method only supports Asynchronous operations.  That may be fine if you have a very simple CRM solution and want to configure only one or two entities. In more real world scenarios this is not going to work for you because it won't guarantee ordered delivery.  That is what I covered in my previous post.  To maintain Ordered Delivery you must use synchronous plugins steps.

The second route is to create an Azure-aware plugin. There is sample code in the SDK for doing this and out there in blogosphere.  In this case you just create the service endpoint and copy the Id that it creates. Create your Azure-aware plugin and paste the Id into the Unsecure Configuration section.  Register your plugin steps and images as usual. The plugin uses an instance of the IServiceEndpointNotificationService and essentially posts the context (using the Execute method) to the Service Bus endpoint.  The point here though is that you have full choice over how to register your steps, so if you need Ordered Delivery you can choose Synchronous.

Personally I find the whole method of configuring a Service EndPoint sucks. What about when I want to deploy this to other environments?  I am going to have to repeat the manual steps for each environment and when I deploy my Azure aware plugin I am going to have to amend the Id each time. Now you might argue this will be a one off process and its no big deal.  But I prefer my deployments not to involve manual steps so I'm inclined to post messages to the Azure Service Bus using code and have the connection string stored in a configuration entity along with other environment settings.  Remember though that you have to use the REST API to post messages because the plugin runs in Sandbox mode. 

Dynamics CRM, Plugins, Ordered Delivery and Queues

This post applies to both Dynamics CRM Online and On-Premise. The scenario is where you need to keep another system synchronized with changes to Dynamics CRM entities.  To make this loosely coupled you can write messages to a queue. If your target system is unavailable, the queue can store messages until it comes back online. The same design pattern is recommended for Dynamics CRM Online where messages are written to Azure Service Bus queue. You then have a process on-premise (it my be an ESB) that reads messages from the queue and sends then on to the target system.

This post stems from work I did on a previous project where we used CRM on-premise to write messages to MSMQ.  If you are reading this far I assume you already know about Ordered Delivery but here is the bottom line:

If you want to maintain Ordered Delivery you must use Synchronous Plugins.

If you use a plugin registered for asynchronous it may appear to give you Ordered Delivery 4 out of 5 times, but you cannot guarantee it for all messages.

You can spend the time proving it for yourself or read this explanation.

We had a custom entity for address that meant you could create an address that was the primary address or the regulatory address or both. The business rule was that you could only have one active address for primary and regulatory. To achieve this we created a plugin on that fires on Create of an address and as a Pre-Operation, if you set the both primary and regulatory flags on the address to true it checks if any existing addresses are primary or regulatory, sets them to false and then deactivates the address(es). Now the target system has to obey the same logic so we need to send any messages to it n the correct order, i.e. in ordered delivery.
So I created a plugin that was generic and would write a message out to MSMQ. I registered it to run as a Post Operation on Create and Update of an Address and set it to run Asynchronously.
In one test case we have two existing addresses one set as primary, the other set as regulatory  (lets call them 'Primary'  and 'Regulatory'),  We create a new address ('New') and set it to both primary and regulatory.  That creates 5 messages.
1. Update of Primary to set the primary flag to false
2. Update of Primary when status is set to deactivate
3. Update of Regulatory to set the regulatory flag to false
4. Update of Regulatory when status is set to deactivate
5. Create of the New address

Now you want to maintain the order that the addresses were written to the database. The Create must come last or you've broken the business rule about only have one active primary or regulatory address. With the on-premise CRM I could examine the Asynchronous table and could see the 5 messages there. They were flagged as belonging to the same transaction but when you looked at the processed time they were all identical.  All five records are executed simultaneously and its a matter of chance which message gets in the queue first.  There is an order, but its not consistent.

BizTalk works in a similar way to the CRM Asynchronous Service and its architected that way for performance reasons. 

When I changed the plugin to work synchronously then it does maintain the correct order of the messages.  You do need to pay attention though to the Rank when you have multiple plugins registered on the same entity for the same stage. By default, Rank is zero but you can put any integer up to 99 into it, and this will set the order that the plugins fire in.  I wanted my message to be the last plugin to execute so I set it to 99.  Remember though that it affects the order
of the plugins within the same stage. The plugin pipeline always executes as
1. Pre-Validation
2. Pre-Operation (before the database write)
3. Post-Operation(after the database write)

Here is the bottom line again

If you want to maintain Ordered Delivery you must use Synchronous Plugins.