Dynamics CRM 2015 Online and Azure Service Bus Messages

If you've configured Dynamics CRM 2015 Online with Azure Service Bus you will know that the "message" it puts on the queue is the plugin execution context. This is the same context that you use within a plugin.  You will know that to do anything with the context you have to use the Microsoft.Xrm.Sdk and extract the Pre or Post Image and the so called Target image. "Target" is s stupid name I always think. It contains the delta - the attributes that were changed during the operation.  The Post Image of say an Update event will contain all the attributes that have values in it and excludes any attributes with null values. 

The reason for using the Azure Service Bus is usually to get data from CRM to your on-premise systems and more often than not you may be using an ESB to read messages from the Azure Service Bus.  BizTalk for example has an adaptor that you just need to configure to read messages from Azure Service Bus.  However any BizTalk developer is going to be very disappointed if you provide them with just a Plugin Execution Context because they will have to use the CRM SDK to convert it into an XML message.  Even then the entity objects when serialized will result in a collection of KeyValuePairs.  The biggest problem with that is trying to map it to a strongly typed schema particularly because the structure varies so much when attributes that are null are set to a value and vice versa.  I've tried using the BizTalk mapper and failed.

UPDATE: CRM Online restricts plugins to Sandbox mode and you cannot serialize objects to XML because this is prohibited. I have recently stumbled on a way of getting round this. This may provide a much simpler solution then described in the rest of this article.

One solution is to have custom code that will read the context, convert it to a strongly typed schema and then put it back in another queue.  This needs to be done serially to ensure that Ordered Delivery is maintained but at least it gets to a schema that is worthy of the name.

Now you have to be careful when using CRM Online because if your plugins consume a lot of CPU they will be terminated with extreme prejudice.  It is best to keep the code within your plugin as minimal as possible and then have the bulk of your code execute on an Azure VM where you don't need to worry about CPU usage because you can size the VM to suit.

The best way to architect this is
a) Have a Plugin that sends the Plugin Execution Context to a Azure Service Bus Queue
b) Create an Azure Worker Process that reads messages from the Queue, creates an XML message and puts it into another Azure Service Bus Queue (be sure to maintain the order)
c) If using BizTalk create a Receive Port that will read XML messages from the second Queue

The Azure Worker Process is a lot like a Windows Service (in Azure its called a Cloud Service) and will automatically restart if it is stopped.  When you deploy the Azure Worker Process it will create its own VM.  You can supply configuration settings which in our case would include the EndPoint of the Queue we are reading from and the EndPoint of the Queue we are writing to.

While you messages should be placed on the first queue using a plugin running synchronously, the Azure Worker Process is running asynchronously.  If you want to post messages back to CRM because of a failure then you need to setup another asynchronous process to send the messages back to CRM. 

Dynamics CRM 2015 Online and Azure Service Bus

In the previous post I talked about sending messages to another system via a message queue.  This is the design pattern Microsoft recommends when you want to use Dynamics CRM Online to update systems that are on-premise. They recommend using Azure Service Bus and the integration is built into the Online version by creating a Service Endpoint. This is also available for the on-premise version.

The plugin registration tool for Dynamics CRM  offers the ability to register a Service EndPoint when you are connected to CRM Online.  What is not clear though is that there are two ways you can configure it.
First though there are two gotchas when setting up the Azure Service Bus.

GOTCHA #1: You set up the Azure Service Bus Namespace using the Portal.  Wrong.
Doing it that way you no longer have the option to use ACS authentication and that is what the Plugin Registration tool uses.  Delete it. Download the PowerShell Azure Commands add-in and run this command:
New-AzureSBNamespace -Name yourservice -Location "West Europe" -CreateACSNamespace $true -NamespaceType Messaging

The response you get returned is

You need to use the DefaultKey when the Plugin Registration tool prompts you for the Management Key!

GOTCHA #2: You create the Queue (or whatever) using the Portal or PowerShell. Wrong.
You need to leave this for the Plugin Registration tool to do.

I won't give the rest of the details for configuring the endpoint because that is covered in other blogs.

Once you have the Service EndPoint registered there are two ways forward.

The first and seemingly the most attractive option is you can register steps and images right there under the endpoint just as you would do with a plugin.  The advantage is that this is a zero code solution. Just by configuring an Entity with the appropriate step and image you can get messages in your queue (or whatever). The thing is though is this method only supports Asynchronous operations.  That may be fine if you have a very simple CRM solution and want to configure only one or two entities. In more real world scenarios this is not going to work for you because it won't guarantee ordered delivery.  That is what I covered in my previous post.  To maintain Ordered Delivery you must use synchronous plugins steps.

The second route is to create an Azure-aware plugin. There is sample code in the SDK for doing this and out there in blogosphere.  In this case you just create the service endpoint and copy the Id that it creates. Create your Azure-aware plugin and paste the Id into the Unsecure Configuration section.  Register your plugin steps and images as usual. The plugin uses an instance of the IServiceEndpointNotificationService and essentially posts the context (using the Execute method) to the Service Bus endpoint.  The point here though is that you have full choice over how to register your steps, so if you need Ordered Delivery you can choose Synchronous.

Personally I find the whole method of configuring a Service EndPoint sucks. What about when I want to deploy this to other environments?  I am going to have to repeat the manual steps for each environment and when I deploy my Azure aware plugin I am going to have to amend the Id each time. Now you might argue this will be a one off process and its no big deal.  But I prefer my deployments not to involve manual steps so I'm inclined to post messages to the Azure Service Bus using code and have the connection string stored in a configuration entity along with other environment settings.  Remember though that you have to use the REST API to post messages because the plugin runs in Sandbox mode. 

Dynamics CRM, Plugins, Ordered Delivery and Queues

This post applies to both Dynamics CRM Online and On-Premise. The scenario is where you need to keep another system synchronized with changes to Dynamics CRM entities.  To make this loosely coupled you can write messages to a queue. If your target system is unavailable, the queue can store messages until it comes back online. The same design pattern is recommended for Dynamics CRM Online where messages are written to Azure Service Bus queue. You then have a process on-premise (it my be an ESB) that reads messages from the queue and sends then on to the target system.

This post stems from work I did on a previous project where we used CRM on-premise to write messages to MSMQ.  If you are reading this far I assume you already know about Ordered Delivery but here is the bottom line:

If you want to maintain Ordered Delivery you must use Synchronous Plugins.

If you use a plugin registered for asynchronous it may appear to give you Ordered Delivery 4 out of 5 times, but you cannot guarantee it for all messages.

You can spend the time proving it for yourself or read this explanation.

We had a custom entity for address that meant you could create an address that was the primary address or the regulatory address or both. The business rule was that you could only have one active address for primary and regulatory. To achieve this we created a plugin on that fires on Create of an address and as a Pre-Operation, if you set the both primary and regulatory flags on the address to true it checks if any existing addresses are primary or regulatory, sets them to false and then deactivates the address(es). Now the target system has to obey the same logic so we need to send any messages to it n the correct order, i.e. in ordered delivery.
So I created a plugin that was generic and would write a message out to MSMQ. I registered it to run as a Post Operation on Create and Update of an Address and set it to run Asynchronously.
In one test case we have two existing addresses one set as primary, the other set as regulatory  (lets call them 'Primary'  and 'Regulatory'),  We create a new address ('New') and set it to both primary and regulatory.  That creates 5 messages.
1. Update of Primary to set the primary flag to false
2. Update of Primary when status is set to deactivate
3. Update of Regulatory to set the regulatory flag to false
4. Update of Regulatory when status is set to deactivate
5. Create of the New address

Now you want to maintain the order that the addresses were written to the database. The Create must come last or you've broken the business rule about only have one active primary or regulatory address. With the on-premise CRM I could examine the Asynchronous table and could see the 5 messages there. They were flagged as belonging to the same transaction but when you looked at the processed time they were all identical.  All five records are executed simultaneously and its a matter of chance which message gets in the queue first.  There is an order, but its not consistent.

BizTalk works in a similar way to the CRM Asynchronous Service and its architected that way for performance reasons. 

When I changed the plugin to work synchronously then it does maintain the correct order of the messages.  You do need to pay attention though to the Rank when you have multiple plugins registered on the same entity for the same stage. By default, Rank is zero but you can put any integer up to 99 into it, and this will set the order that the plugins fire in.  I wanted my message to be the last plugin to execute so I set it to 99.  Remember though that it affects the order
of the plugins within the same stage. The plugin pipeline always executes as
1. Pre-Validation
2. Pre-Operation (before the database write)
3. Post-Operation(after the database write)

Here is the bottom line again

If you want to maintain Ordered Delivery you must use Synchronous Plugins.

Calling a WCF Web Service over HTTPS (SSL)

I was recently trying to access a web service that I wanted to secure over HTTPS. I got it working as an HTTP service, as you do, and made sure that I had a certificate on the server and enabled the https protocol.
I was using basic Http binding and here are the changes that need to be made

          <security mode="Transport">
            <transport clientCredentialType="None" />

That now worked in the browser if I prefixed the url with https://

Next step was that I needed to call the web service where I was not able to access a web.config or an app.config.  Without the service reference being available you have to do this in code. First thing is to make sure you have a copy of the interface class accessible in the client.  It doesn't need to be the same name but it does need to specify the operation contract exactly.

    [ServiceContract]
    public interface IFormDefinition
    {
        [OperationContract]
        [FaultContract(typeof(CRMSoapFault))]
        void PublishFormMetaData(string crmEndPoint, string formId, string webResource, string token);
    }
    [DataContract]
    public class CRMSoapFault
    {
        public CRMSoapFault(string errorMsg)
        {
            this.ErrorMsg = errorMsg;
        }
        ///

        /// This property is used to pass the custom error information
        /// from service to client.
        ///

        [DataMember]
        public string ErrorMsg { get; set; }
    }

To call the web service and set the binding information through code to match this you need to add:

 BasicHttpBinding myBinding = new BasicHttpBinding();
            myBinding.Security.Mode = BasicHttpSecurityMode.Transport;
            myBinding.Security.Transport.ClientCredentialType = HttpClientCredentialType.None;
            EndpointAddress myEndpoint = new EndpointAddress(endPointUrl);
            ChannelFactory myChannelFactory = new ChannelFactory(myBinding, myEndpoint);
            try
            {
                IFormDefinition wcfClient1 = myChannelFactory.CreateChannel();
                // call the web service method
                wcfClient1.PublishFormMetaData(crmEndPoint, formId, webResource, token);

            }
            catch(FaultException faultEx)
            {
             }

Accessing SharePoint Online with Web Client

Misleading title really because if you try and access SharePoint Online using the WebClient it will fail to authenticate.  What you need to do is use the CookieContainer and SharePointOnlineCredentials. 
I got the basics of this from this post. and also from this post which uses a class inherits from WebClient. It mentions that you need the SharePoint Client Components SDK which will install Microsoft.SharePoint.Client.DLL and Microsoft.SharePoint.Client.RunTime.DLL. Add references to both DLLs in your project and add these two using statements

using Microsoft.SharePoint.Client;
using System.Security;


 Add this class to your project
 
public class ClaimsWebClient : WebClient
{ private CookieContainer cookieContainer;

public ClaimsWebClient(Uri host, string userName, string password)
 
{

cookieContainer = GetAuthCookies(host, userName, password);

}
  protected override WebRequest GetWebRequest(Uri address)
{
    WebRequest request = base.GetWebRequest(address);
    if (request is HttpWebRequest)
 
    {         (request as HttpWebRequest).CookieContainer = cookieContainer;
   }
     return request;
 
}
  private static CookieContainer GetAuthCookies(Uri webUri, string userName, string password)
{    var securePassword = new SecureString();
    foreach (var c in password) { securePassword.AppendChar(c); }
    var credentials = new SharePointOnlineCredentials(userName, securePassword);
    var authCookie = credentials.GetAuthenticationCookie(webUri);
    var cookieContainer = new CookieContainer();
    cookieContainer.SetCookies(webUri, authCookie);      return cookieContainer;
}

} 

Then call the ClaimWebClient class in the same way as you would the WebClient.  Note that you do not need to set the credentials because it is done within the ClaimWebClient class.

ClaimsWebClient wc = new ClaimsWebClient(new Uri(sharePointSiteUrl), userName, password);

byte[] response = wc.DownloadData(sourceUrl);
 

Raising SoapFaults on a Web Service

I have used SoapFaults (FaultExceptions) on Web Services before so here is  quick recap. In your Interface class add this declaration

[DataContract]
public class SPSoapFault
{
   public SPSoapFault(string errorMsg)
   {
       this.ErrorMsg = errorMsg;
   }
   [DataMember]
    public string ErrorMsg { get; set; }
}

Beneath the OperationContract declaration of the method you want to use this on addthe FaultContract attribute

[OperationContract]
[FaultContract(typeof(SPSoapFault))]

Now in the service you can throw FaultExceptions of this type

throw new FaultException<SPSoapFault>(new SPSoapFault("The byte array is null"), new FaultReason("Required parameter"));

Note that you must add a Fault Reason.

When calling this from a client application make sure you declare the faultexception that is in the web service references.

SPService.SPSoapFault spfault = new SPService.SPSoapFault();

the catch block of your try catch should include this

catch (FaultException<SPSoapFault> faultEx)
{

  spfault = faultEx.Detail;
  string error = spfault.ErrorMsg;
  string reason = faultEx.Reason.ToString();
}

Dynamics CRM 2013 Microsoft Web Page Dialog error

If you use Dynamics CRM 2013 for any length of time then you will have noticed the Microsoft dialog box that pops up intermittently.  It seems to occur randomly and is hard to reproduce.  As far as I can see it doesn't result in any data loss so I just ignore it.

But the screen is irritating and doesn't give a good impression to first time users.

It can be disabled by logging on to CRM as the System Administrator and going to Administration and Privacy Preferences. On the Privacy Preferences dialog click the Error Reporting tab. Select "Specify the Web application error notification preferences on behalf of users" checkbox and then select the radio button "Never send an error report to Microsoft".

The errors are still happening but at least the screen isn't popping up.