Zapier and Microsoft Dynamics 365 CRM
Have you ever used Zapier to integrate between another service and Microsoft Dynamics 365 CRM? I have, while Zapier is a very useful tool to link processes together that have no direct integration path, there are some interesting quirks and behaviours with Microsoft Dynamics 365 CRM which might be challenging or potential barriers to automation tasks. As a web developer working with Microsoft Dynamics 365 CRM regularly, I wanted to document some of these quirks for others to help make you Zapier process smoother when using Microsoft Dynamics 365 CRM.
Zapier may not handle some custom fields for entities correctly
At the time of writing, the latest version of the Microsoft Dynamics 365 CRM Zapier app is version 1.18. This version has potential bugs with custom fields. Depending on your CRM environment, you may trigger a similar error to the below with create actions for certain fields.
This error is because the field name referenced ccl1000_EnquiryTypeId, is actually wrong, it should be ccl1000_enquirytypeid. Casing of field names is important with Dynamics 365 CRM. This is because the Zapier app has used the SchemaName for the field and not the LogicalName. With Dynamics 365 CRM, fields have two versions of their handle a SchemaName and LogicalName, using the right handle is important, any incorrect handles will cause a 400 Bad Request and the payload rejected. The Zapier app unfortunately triggers this, because of it’s payload handling, but this is a more recent bug.
Interestingly the previous version 1.17 version of the app, seems to handle custom fields correctly and works, but why? It would appear a logic change occurred between 1.17 and 1.18 occurred. It is an interesting scenario, I have seen conflicting information on what the rules are correct with SchemaName vs LogicalName, some resources say you should use SchemaName where as others suggest LogicalName. Regardless, only one will work for a CRM environment. The real source of truth is the CSDL for the CRM environment. This document serves the correct property/entity names that are always correct. The problem? It’s a XML only document with no filtering and can be quite large (testing the environment of my company, 17 MB). I personally dislike XML with a passion and parsing a 17 MB XML document is resource intensive, even with implementing caching. However this is the one reliable method of making sure you always use the correct entity names/properties for any given request.
A workaround currently, if you ask Zapier support nicely, they’ll give you access to the 1.17 version of the Dynamics 365 CRM app to your Zapier account, when you accept the invite, you’ll get two Microsoft Dynamics 365 CRM apps available to select in Zaps, the latest public version and the 1.17 version (this appears as under the Custom Integrations tab in My apps).
Do bear in mind, any support for a previous version of an app is not likely forthcoming from Zapier even if you quote the famous SchemaName vs LogicalName dileemma which is documented internally, but you can use a mixture of versions of the same in app in a single zap with no issues. Do bear in mind that because they are two different Zapier apps, you’ll have to authenicate both and they are separate.
The Microsoft Dynamics 365 CRM apps uses literal single quotes in find queries
This one is a bit niche, but it can still cause some issues. The actual use of single quotes is valid, you can use double or literal quotes in Dynamics 365 Web API queries to represent values.
Scenario: I want to perform a find query and lookup for the existence of a contact record matching a rule of same First Name, Last Name and E-mail value. At the Dynamics 365 Web API level it would look something like this:
{{webapiurl}}contacts?$filter=firstname eq 'James' and lastname eq 'White' and emailaddress1 eq 'email@example.com'Nothing complicated, just a find query that will return one or more contact records matching the criteria. Now let’s throw in an edge case. Let’s pretend my last name is something O’White (it’s not, but pretend). The query might become something like this:
{{webapiurl}}contacts?$filter=firstname eq 'James' and lastname eq 'O'White' and emailaddress1 eq 'email@example.com'Can you see a problem? With the introduction of a single quote in the lastname value, this now has caused a syntax error, because the single quote is not escaped and therefore now looks like it’s part of the API syntax not the value of lastname. This is further complicated because of variable data, which is often provided from triggers used in Zaps. Now technically a literal single quote is the wrong character in the context of this name example, it should be an apostrophe, but not all systems handle this or convert to the correct character (a user can and will potentially use a single quote character directly), so if a single quote makes it through, this is a problem when being used in a find query. There’s a few solutions:
- Use double quotes around values in the API call to allow single quotes to be used (bear in mind you would now have the reverse problem with values with double quotes).
- Escape quotes when being used in the API call as values e.g.
O'Whitewould becomeO''Whitewhen sent as a query but not change the original value. - Parse and replace any field with such characters with alternatives before sending to the API.
Interestingly when creating data, it appears double quotes are used (probably because it’s JSON format) so this problem does not happen and ironically makes escaping not reliable either, because in the find logic the single quote is treated as an escape sequence, in a create it gets treated as the literal value so you’d have the value created as O''White. As there is no consistency, the only way around that is split the Find and Create actions up, use the escaped value for the Find action and then use the original values for a Create action, this does however now duplicate the amount of steps, given the convenience of the Find with optional create action.
For name data you can likely replace any single quotes with the apostrophe character (as that’s likely the correct character that should have been used anyway), rather than escape them. Email addresses can also have single quotes before the @ symbol and is valid according to RFC 5322, which is a bit trickier with the above in play. You cannot just strip them, given that changes the email data and it would be either invalid or the wrong recipient. How common single quotes in email addresses are is likely low, so you may be able to ignore this, and ensure name data is handled.
Because you are likely to come across this in multiple Zaps, you might consider implementing a sub-task, so you don’t have to repeat the same logic and can simple call the sub-zap to return the parsed values each time.
Zapier do have this noted as a bug, but with no commitment to a fix currently. The best fix would be for escape logic to be direct performed within the app specifically for the find query, rather than the data being manipulated in a Zap itself.
“The app did not respond in-time. It may or may not have completed successfully”
Timeouts are something that occur with many Zapier tasks from time to time. Sometimes API calls don’t respond in time or the API itself could have problems, it does happen, these are after all public apps. More recently Zapier has smarter automatic replay functionality that if a Zap task run triggers an error it gets automatically replayed to be retried. This can help the situation but depending on the CRM actions being performed it can also be a potential problem.
It’s important to first understand Zapier’s timeout handling and restrictions. Based on the information available it is believed that most Zapier actions have a maximum of 30 seconds before it is deemed a timeout. While we can argue this could be too short in some cases, this cannot be increased, it is a hard limit and Zapier do not seem interested in increasing it or making it available to be changed. In reality, the timeout value is just a single integer passed to a HTTP client making a request, but exposing this to be customised is something Zapier aren’t looking to do.
Unfortunately, Microsoft Dynamics 365 CRM web API request times can be wildly variable based on the environment, action performed and other processes involved. There are a few factors to this which influence the response time:
- Workflows and processes — A CRM environment may trigger other workflows to run on data when created, this can slow the API response down which Zapier needs to receive to deem the action successful.
- Other tasks and process — If there is a queue of other tasks occurring at the same time this can slow down the response too.
- Real-time processes — These happen immediately (synchronously) on certain events rather than in the background (asynchronously) and can also slow responses from API requests.
Based on my own testing, it is mainly when creating records in CRM that this is most likely to be an issue and because no two CRM environments are the same, results vary, but the general principles apply.
The issue here is any of those scenarios could be happening at any given time and therefore you have the problem of timeouts occurring at anytime and you can’t predict it. Which leads onto another interesting point related to this factor.
Timeout responses could lead to duplicated data with automatic replay
Automatic replay itself isn’t the problem but with timeouts being possible, the timeout issue can turn automatic replay into a bit of a monster and coerce the feature to start causing a mess, I’ve personally seen it happen, but the feature itself isn’t to blame.
Scenario: A create lead action responds with a timeout and deems the task as “errored”. Was the data still created?
The answer is: Probably. Because of the timeout error the task would have been marked as “errored” but there’s actually a good chance the data was created, just that a “201 Created” response was not received in time, deeming Zapier to be unsure of what has happened. Even the timeout error message text, somewhat admits “not sure, you’re on your own”.
Without handling, automatic replay can cause havok and duplicate data frequently, leading to messy data in your production CRM. Not good!
Because of this, for safety and defensive logic, any Microsoft Dynamics 365 CRM action related to creating data, should use the Find action with the optional create logic and not the single Create action. This advice/solution comes from the Zapier itself from one of their community managers and it is actually quite elegant (Thanks SamB). Even if there is absolutely zero chance of the data existing beforehand and performing the find query is essentially wasteful, the beauty of this process is the outcome will return an GUID/ID of the entity in question no matter what. If a record exists a GUID is returned, if one doesn’t exist, a record is created and a GUID returned (of the created record).
From experience find actions will take significantly less time than a create action, so if a record does exist, that will likely have minimal risk of timing out, compared to a create action.
For contacts, you can use a match rule of something like First name, Last name and Email address. Providing enough data points to reduce false positive results, given a contact is intended to be a singular record.
Note: The find contact action does not provide access to all fields on a contact to search against (mainly limited to String type fields).
For leads, you need to be a bit more careful. By their very design leads are designed to be many. Using a database analogy, one contact can have one or more leads meaning the relationship is one-to-many. This means that personal details alone on a lead should not be just the factor to consider.
Scenario: You have multiple lines of business (two different leads for the same contact).
You’d need to be able to differentiate this lead outside of personal information.
Fortunately, you can utilise the Subject/Topic field on a lead to provide a further check ensure you can preventing duplicate data while not matching too widely. As the Subject/Topic field is a string, you can insert your own value here. What I tend to do is send some form of value like:
“Zapier [Trigger Name/Service] ID: [Unique ID]”.
Zapier Formstack ID: 122354645For the ID, often the trigger provides such value in the payload. You could also use a timestamp if no unique ID is available. I use this to ensure if a task is replayed, it would be safe to compare with. You can’t use a truly random value e.g. random number, because it would change on each task execution and defeat the matching criteria purpose on a replay.
This means now that we’ve implemented what would be deemed as defensive logic, in the event the task gets auto replayed because of timeout (remember it’s still possible this could happen we cannot 100% defend against it) the find action will prevent duplicate data being created and simply return an GUID value and even better this would also be treated as a success from a task execution perspective, so it covers all scenarios.
CustomerId fields are not available in the Zapier mapping
Zapier to be fair, somewhat warns you of this on the mapping screen when you map fields under the action step, it makes reference to certain fields not being available.
While not specifically stated in the note. A CustomerId field is a Customer AttributeType which is also excluded in the mapping. This could be a problem for certain CRM environments which use this field, for relations between entities e.g. linking a lead to a contact.
The CustomerId field is a field which allows you to link different entities under a single field, for example on a lead, you can use the customerid to link a contact or an account, this is done with the payload values of:
- customerid_contact@odata.bind — Contact GUID
- customerid_account@odata.bind — Account GUID
Because this field can accept multiple entities, I believe this is why Zapier doesn’t allow it to be mapped, because it can’t reliably handle the dynamic scenario of two different entities, given the prefix needed to specify this along with the GUID value needing to align to either contacts or accounts.
Instead the fields that are available in the mapping is Parent contact for lead (linking a contact) and Parent account for lead (linking an account).
If your CRM environment uses the customerid field. You might need to consider implementing a workflow to help with copying the value into the required customerid field for data sent by Zapier.
The ownerid field is not available in the Zapier mapping
Similar to the customerid field, the ownerid also appears to fall under the excluded fields category. An owner field has an AttributeType of “Owner” and interestingly ownerid is a SystemRequired field, so it has to actually be mapped, you just can’t customise it and based on my own experiences there’s likely a few reasons why:
- The owner/assigned to values used on entities needs to have a minimum set of permissions in order to be used on the target entity. This usually involves read/create privileges.
- Zapier does not directly know if a system user/team provided has such permissions and therefore likely avoids letting you select a value which won’t work or could cause errors due to permissions.
- The user context in which the Zapier app was authenticated under, is the ownerid value used in create actions. This is likely because by authenticating this proves the user used has the required access.
You can confirm this by performing a request to the WhoAmI endpoint
{{webapiurl}}WhoAmIWhich when authenticated, would return something like:
{
"@odata.context": "https://examplecrm.crm11.dynamics.com/api/data/v9.2/$metadata#Microsoft.Dynamics.CRM.WhoAmIResponse",
"BusinessUnitId": "ce319713-1082-ea11-a811-000d3a7f2195",
"UserId": "8b8e7759-a283-ea11-a812-00224801c2d0",
"OrganizationId": "e3fe131e-03df-ed11-a809-6045bdcfc9af"
}The UserId is the key part, this is the user the token authentication was authenticated under and which Zapier uses. Best practice would be to have some form of service account with a non-expiring password which is used for authentication. This still however doesn’t solve being able to change the owner, only set a specific owner for all Zapier related create actions.
To control the owner value, you are once again looking at a workflow solution that would query certain data with various conditions that match Zapier data to handle accordingly. There are various ways this can be achieved, you’d be mainly querying the current owner value and then looking to build further conditions off any other field property as required.
Zapier notes this as a feature request but thus far it has not been implemented.
App Requests/Extensions Beta!
Zapier more recently have released two newer BETA features which can help with more bespoke requirements for certain apps including Microsoft Dynamics 365 CRM, these are “App Requests” and “App Extensions”. Both are similar in their purpose but offer different solutions and customisation.
App Requests — Allows you to make custom HTTP requests within Zapier. Ever been blocked by the fact the Zapier app you use doesn’t allow you to do something i.e. no option in the action menu? This feature is now your new best friend. If an API endpoint exists for what you want to do, you can now implement it yourself, rather than submitting a feature request and sitting in limbo for months or possibly years if it ever makes it into the Zapier app updates.
App Extensions — Similar to App Requests but extensions go further and provides the ability to make a custom HTTP request, but also allows you to provide a customised interface in a Zap, accept input variables and share across teams, making them re-usable and more dynamic.
While using something in BETA is risky for production scenarios, I have personally implemented a couple of Zapier App Extensions in live Zaps with great success. There are some rough parts currently (it is a BETA after all) but the ability to create custom HTTP requests while also providing input variables and passing these down to said API calls is a game changer for more advanced requirements.
The major benefit of these features is that the authentication layer is still linked to the main public app configured, so you don’t have to implement this yourself. Prior to these features if you wanted to make a request that wasn't supported, you’d literally have to create a private app and implement everything yourself, which is a massive burden for most and unrealistic even for developers, given the purpose of Zapier.
Given Microsoft Dynamics 365 CRM can be quite complicated and customisations of said environments create further complexities or bespoke handling, the ability to create custom HTTP requests can workaround some of the issues outlined. For advanced CRM requirements, you certainly want to read up on these newer features.
Note: These features are more advanced and developer orientated, so not necessarily suited to non-developers. You will need to have knowledge and understanding of APIs and how to work with them.
It is interesting that while Zapier remains focused on being a no code solution, they have overtime hit a balance of offering advanced options should developers want to use them e.g. Code by Zapier (JavaScript/Python) and now having the ability to create your own HTTP requests. Amazing stuff.
Wrapping up
These are the main quirks and challenges I have found when working with Zapier and Microsoft Dynamics 365 CRM. Zapier is an important tool and I hope it helps others are using Zapier with Microsoft Dynamics 365 CRM and helps improves your processes.
