Using Dataverse Service Client to connect to OnPrem Dynamics 365 CRM (From .NET 6+)

Can you use new Dataverse ServiceClient
(Microsoft.PowerPlatform.Dataverse.Client) with .NET 6+ for for your console app to connect to on-premises instance of Dynamics 365 CRM? Yes you can! Here is the connection string:

AuthType=OAuth;Url=https://org.crm.example.com/;Username=[e-mail];Password=<pw>;RedirectUri=http://localhost:54321;AppId=174697c7-4ec8-401a-88ce-e6af4d05b6dd;LoginPrompt=Auto

Well, after you appropriately configure ADFS and CRM, that is.

Intro and Supportability

There are 2 Client SDKs that abstract API calls:

• Long time CRM Service Client SDK (Microsoft.CrmSdk.XrmTooling.CoreAssembly) windows-only, supports .NET Framework 4.6.2+, but not .NET Core world, thus not cross-platform. Still supported.
• Dataverse ServiceClient (Microsoft.PowerPlatform.Dataverse.Client (DVSC)) - .NET 6, cross platform. Moving away from SOAP, but still uses it. No guidance from Microsoft to use for on-premises, so at the time of writing can be considered unsupported. Transition apps to Dataverse ServiceClient article specifically has a note for on-premises clients that says: > Leave your application projects and code as is. Continue using the Microsoft.CrmSdk.CoreAssemblies NuGet package and CrmServiceClientclass. However, plan to update your projects from using any custom service clients to instead use the CrmServiceClient or ServiceClient in the near future. See the planned timeline for 2011 SOAP endpoint shutdown below.

So are we stuck with .NET Framework for on-premises client applications?

DVSC source has bits and pieces, logic and variables like isOnPrem that indicates that maybe there is light at the end of the tunnel? And the note you just read actually says "... plan to update ... to instead use the CrmServiceClient or ServiceClient in the near future.". But maybe that "plan to update" means only plan to update the respective package, when they will get rid of all the SOAP requests in both and use WebAPI purely.

However it turns out, it is possible to connect to on-premises instance with DVSC if you are using ADFS2019. And in a different way, if you are using ADFS2016.

I may note that whatever is not officially documented may be considered an unsupported scenario. Things to consider:

• For Online, Microsoft has so tightly integrated into Azure... I personally don't expect much changes for on-premises. And we really haven't seen much love there lately.
• With these changes we are actually moving in a direction where Microsoft stands today: MSAL. So would be actually easier to move to online afterwards.
• Nothing unsupported is being done for CRM/ADFS side configuration. Sans 1 thing to overcome current bug in DVSC that will hopefully be fixed.
• Worst that could happen is some new version breaks something for on-premises and no one cares to fix it and we are stuck with older version of DVSC. It is a valid concern, of course, as we may be left stranded with some known vulnerabilities. But for me this is a low probability because of the plan to update note. So your decision whether to use .NET 6/DVSC or not in a production environment.

Configuring CRM/ADFS 2019

1. You must have: Windows Server 2019+/ADFS 2019+
2. KB4490481 must be installed. Required for MSAL support.
3. Configure the Microsoft Dynamics 365 Server for claims-based authentication and also for IFD. Don't forget to do the appropriate configuration on AD FS server for claims and IFD auth.

So I will assume the following:

• ADFS is at https://adfs.example.com
• CRM non-IFD URL for particular org is at https://crm.example.com/ORG
• IFD URL for org is at: https://org.crm.example.com

See IFD configuration notes at the end of the article.

Regarding non-IFD, Claims-only configuration for internal access - I tested it with DVSC and as of writing, it didn't work. It did get the access token, but then failed to send requests to right URL - it didn't append the /org after the URL host part. There is an open issue 397: non-IFD URLs not handled correctly for on-Prem.

1. On ADFS Server you have to tell that you have to register your application. You will use ClientId in connection string. And you have to give permissions for this application to get access token for CRM IFD relying party.

RedirectUri must contain port and must use http protocol (for localhost). If port will not be specified, random will be used by MSAL when you connect via DVSC, but ADFS just won't accept it.

   # MSAL will listen on localhost this particular port to get answer from ADFS (authorization code), but that is only true for authorization code grant flow: https://learn.microsoft.com/en-us/windows-server/identity/ad-fs/overview/ad-fs-openid-connect-oauth-flows-scenarios#authorization-code-grant-flow
   # As for name, in real world, that would be something like: "Accounting Integration" or "Postman" or whatever that communicates the software or intent for the application that is connecting to CRM
   > Add-AdfsClient -ClientId ([guid]::NewGuid()) -Name "DVSC Client" -RedirectUri "http://localhost:54321" -Description "Dataverse Service Client will connect to CRM" -ClientType Public
   > $ClientId = (Get-AdfsClient -Name "DVSC Client").ClientId
   # If you don't grant permissions, you will get error along with an event id 1021 in AD FS event log: MSIS9605: The client is not allowed to access the requested resource. when trying to issue token at /adfs/oauth2/token.
   # For ServerRoleIdentifier, "external domain" URL must be used you chose when you configured IFD. You inputted 4 domains/urls, this would be the 4th one.
   # Another way to find out - go to ADFS, open Relying parties, open the IFD relying party (auth.crm.example.com), open Identities, and pick the first one: https://auth.crm.example.com/
   > Grant-AdfsApplicationPermission -ClientRoleIdentifier $ClientId -ServerRoleIdentifier "https://auth.crm.example.com/" -ScopeNames "openid"
   > $ClientId

   1260534b-00ca-4663-870f-d77b8d4ad6d3

What does openid has to do anything with this? It means that you allow your client application to use OpenID Connect (OIDC) authentication protocol.

1. OAuth must be enabled for CRM. Execute this on CRM server. Changes will be applied immediately:

   Add-PSSnapin Microsoft.Crm.PowerShell  
   $ClaimsSettings = Get-CrmSetting -SettingType OAuthClaimsSettings  
   $ClaimsSettings.Enabled = $true  
   Set-CrmSetting -Setting $ClaimsSettings  

If you want to validate OAuth is enabled, you must open: https://org.crm.example.com/XRMServices/2011/Organization.svc/web and see with, say, Fiddler, whether you get WWW-Authenticate header that is like: Bearer redirect_uri=https://adfs.example.com/adfs/ls/

You may get multiple headers, like WWW-Authenticate: Negotiate and WWW-Authenticate: NTLM, but one with Bearer must be present. If you don't want to configure fiddler to catch TLS requests, you can check what headers that request returns like this:

   > Invoke-WebRequest -Uri "https://org.crm.exmple.com/XRMServices/2011/Organization.svc/web" -UseBasicParsing
   > $error[0].exception.response.headers["WWW-Authenticate"]

   Bearer authorization_uri=https://adfs.example.com/adfs/oauth2/authorize, resource_id=https://org.crm.example.com/

Don't worry, the Invoke-WebRequest returns 401 error - it is expected.

1. EDIT: There are actually some required steps documented after enabling OAuth. Particularly removing windows auth from 3 folders. Read Required steps after enabling OAuth for Dynamics 365 Server

The previous note:
As of writing, the DVSC will fail to connect if you get multiple WWW-Authenticate headers. It is due to a bug in the client itself and hopefully it gets fixed. isOnPrem not passed down correctly when invoking GetAuthorityFromTargetServiceAsync · Issue #396 · microsoft/PowerPlatform-DataverseServiceClient (github.com).

Thus, at the time of writing, we need additional configuration:

1. Open C:\windows\system32\inetsrv\config\applicationHost.config on CRM server.
2. Find this XML element:

       <location path="Microsoft Dynamics CRM/XRMServices/2011/Organization.svc">
        <system.webServer>
            <security>
                <authentication>
                    <digestAuthentication enabled="false" />
                    <basicAuthentication enabled="false" />
                    <anonymousAuthentication enabled="true" />
                    <windowsAuthentication enabled="true" />
                </authentication>
            </security>
        </system.webServer>
    </location>

1. Change windowsAuthentication to false. After save, changes are applied immediatelly, no IIS restart was needed.

       <location path="Microsoft Dynamics CRM/XRMServices/2011/Organization.svc">
        <system.webServer>
            <security>
                <authentication>
                    <digestAuthentication enabled="false" />
                    <basicAuthentication enabled="false" />
                    <anonymousAuthentication enabled="true" />
                    <windowsAuthentication enabled="false" />
                </authentication>
            </security>
        </system.webServer>
    </location>

Connect with DVSC

I actually uploaded various scenarios to GitHub: DataverseServiceClientOnPremSamples: Examples on how to use Dataverse Service Client to connect to Dynamics 365 CRM on-premises (github.com)

appsettings.json:

{
  "ConnectionStrings": {
    "default": "AuthType=OAuth;Url=https://org.crm.example.com/;Username=[e-mail];Password=<pw>;AppId=174697c7-4ec8-401a-88ce-e6af4d05b6dd;LoginPrompt=Never"
  },
  "Logging": {
    "LogLevel": {
      "Default": "Trace"
    }
  }
}

Program.cs:

using Microsoft.Crm.Sdk.Messages;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
using Microsoft.PowerPlatform.Dataverse.Client;
using Microsoft.Xrm.Sdk.Query;

namespace ADFS2019ConnectionString
{
    class Program
    {
        IConfiguration Configuration { get; }
        static ILogger Logger { get; } = LoggerFactory.Create(builder => builder.AddConsole()).CreateLogger<Program>();

        Program()
        {
            var path = Environment.GetEnvironmentVariable("DATAVERSE_APPSETTINGS") ?? "appsettings.json";
            Configuration = new ConfigurationBuilder().AddJsonFile(path, optional: false).Build();
        }

        static void Main(string[] args)
        {
            Program app = new();
            var connectionString = app.Configuration.GetConnectionString("default");
            ServiceClient serviceClient = new(connectionString, logger: Logger);
            WhoAmIResponse resp = (WhoAmIResponse)serviceClient.Execute(new WhoAmIRequest());
            Console.WriteLine("User ID is {0}.", resp.UserId);
            var systemuser = serviceClient.Retrieve("systemuser", resp.UserId, new ColumnSet("fullname"));
            Console.WriteLine($"Hello, {systemuser["fullname"]}, where do you want to go today?");

            // Pause program execution before resource cleanup.
            Console.WriteLine("Press any key to continue.");
            Console.ReadKey();
            serviceClient.Dispose();
        }
    }
}

Packages:

• Microsoft.Extensions.Configuration
• Microsoft.Extensions.Configuration.Json
• Microsoft.Extensions.Logging.Console
• Microsoft.PowerPlatform.Dataverse.Client

Result:

User ID is e0f638f8-e3a0-44df-a242-8557ed7e0e33.
Hello, John Doe, where do you want to go today?

Lets look at 2 ways we can use the connection string:

• Authorization code grant: AuthType=OAuth;Url=https://org.crm.example.com/;Username=[e-mail];Password=<pw>;RedirectUri=http://localhost:54321;AppId=174697c7-4ec8-401a-88ce-e6af4d05b6dd;LoginPrompt=Always

UI required (browser). Takes 2 requests to get access token. MSAL library takes care of listening to particular port specified in RedirectUri, so that when first request goes out to ADFS and you get authorization code, browser on your computer where the app runs can get the response, extract authorization code and issue another request to ADFS to finally get the access token.

• Resource Owner Password Credentials grant: AuthType=OAuth;Url=https://org.crm.example.com/;Username=[e-mail];Password=<pw>;AppId=174697c7-4ec8-401a-88ce-e6af4d05b6dd;LoginPrompt=Never

You authenticate as particular user. No UI required. Takes 1 request to get access token.

This flow is considered for Desktop and Mobile application types, however I am going to use it for Service (Daemon) application as it requires no user interaction and I'm not so sure if Client credentials flow (which is officially meant for daemon applications) can be supported by on-premises instance.

Has drawbacks - no MFA support, passwords with leading/trailing whitespace not supported etc. See notes and consider what applies to on-premises.

ADFS 2016

Well that was for fancy ADFS 2019 which is in friends with MSAL. But there is another option to authenticate against ADFS 2016. Or in this case, we can use more OAuth authentication flows if we want.

Luckily, there is ServiceClient constructor available that takes tokenProviderFunction - we can just feed in access token that we got by any means we want.

public ServiceClient(Uri instanceUrl, Func<string, Task<string>> tokenProviderFunction, bool useUniqueInstance = true, ILogger logger = null)

Will use a plain HttpClient request to ADFS to get access_token. As MSAL doesn't support ADFS 2016, one can maybe use ADAL (Microsoft.IdentityModel.Clients.ActiveDirectory) - however it is deprecated.

Beware this is a quick and dirty example - no refresh tokens, no checking for slash suffixes (i.e for adfsUrl variable), no correct naming for TokenResponse class, bare exceptions, no error handling.

using Microsoft.Crm.Sdk.Messages;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
using Microsoft.PowerPlatform.Dataverse.Client;
using Microsoft.Xrm.Sdk.Query;
using System.Net.Http.Json;

namespace ADFS2016
{
    class Program
    {
        IConfiguration Configuration { get; }
        static ILogger Logger { get; } = LoggerFactory.Create(builder => builder.AddConsole()).CreateLogger<Program>();

        Program()
        {
            var path = Environment.GetEnvironmentVariable("DATAVERSE_APPSETTINGS") ?? "appsettings.json";
            Configuration = new ConfigurationBuilder().AddJsonFile(path, optional: false).Build();
        }

        static void Main(string[] args)
        {
            Program app = new();
            var crmUrl = app.Configuration["CrmUrl"] ?? throw new ArgumentNullException("CrmUrl");
            ServiceClient serviceClient = new(new Uri(crmUrl), app.TokenProviderAdfs2016, logger: Logger);
            WhoAmIResponse resp = (WhoAmIResponse)serviceClient.Execute(new WhoAmIRequest());
            Console.WriteLine("User ID is {0}.", resp.UserId);
            var systemuser = serviceClient.Retrieve("systemuser", resp.UserId, new ColumnSet("fullname"));
            Console.WriteLine($"Hello, {systemuser["fullname"]}, where do you want to go today?");

            Console.WriteLine("Press any key to continue.");
            Console.ReadKey();
            serviceClient.Dispose();
        }

        /// <summary>
        /// Cache access_token for subsequent requests
        /// </summary>
        string? AccessToken { get; set; }
        async Task<string> TokenProviderAdfs2016(string instanceUri)
        {
            if (AccessToken != null) return AccessToken;
            var resource = new Uri(Configuration["CrmUrl"] ?? throw new ArgumentNullException("CrmUrl")).GetLeftPart(UriPartial.Authority) + "/";
            ///Construct Resource owner password credentials grant request - username/password enought for auth
            HttpClient http = new HttpClient();
            var adfsUrl = Configuration["AdfsUrl"] ?? throw new ArgumentNullException("AdfsUrl");
            var request = new HttpRequestMessage(HttpMethod.Get, adfsUrl + "/oauth2/token");
            request.Headers.Add("Accept", "application/json");
            request.Content = new FormUrlEncodedContent(new Dictionary<string, string>() {
                { "grant_type", "password" },
                { "client_id", Configuration["AppId"] ?? throw new ArgumentNullException("AppId") },
                //If CrmUrl is https://crm.example.com/org, resource will be https://crm.example.com/. This must match with one of identifiers for relying party for CRM in ADFS when configured either claims auth or IFD.
                //ADFS 2016 requires resurce parameter. ADFS 2019 allows resource within scope parameter. MSAL will use scope parameter.
                { "resource", resource },
                { "scope", "openid" },
                { "username", Configuration["Username"] ?? throw new ArgumentNullException("Username") },
                { "password", Configuration["Password"] ?? throw new ArgumentNullException("Password") },
            });

            var response = await http.SendAsync(request);
            var token = await response.Content.ReadFromJsonAsync<TokenResponse>();
            AccessToken = token?.access_token ?? throw new Exception("Couldn't get access_token");
            return AccessToken;
        }

        public record TokenResponse
        {
            public string access_token { get; set; }
            public string refresh_token { get; set; }
        }
    }
}

I already commited ADFS2019DeviceFlow project example which would implement device code flow. Ever seen this message for some tools? To sign in, use a web browser to open the page https://adfs.example.com/adfs/oauth2/deviceauth and enter the code PLSSCZGKY to authenticate - you got covered.

IFD Configuration notes

When I configured IFD, I did use slightly different URLs than what is at the documentation. Enabling IFD means you will get URL for each organization:

• org1.<web application server domain> (org1.contoso.com)
• org2.<web application server domain> (org2.contoso.com)

If you choose contoso.com, or example.com and if it is the same as your AD domain... well, lets say I'm not a fan of "polluting" DNS. I like to separate things. If any of your org names happen to have a same name as any of your hosts in AD domain or more correctly - if DNS already has such entry that doesn't point to CRM server - well, that URL wont work and no DVSC for you.

Another reason would be if you have SharePoint and it also uses ADFS, you may get additional issues like MSIS7065: There are no registered protocol handlers on path /adfs/ls/ to process the incoming request.: Passive federation request fails when accessing an application using AD FS and Forms Authentication after previously connecting to Microsoft Dynamics CRM also using AD FS - Microsoft Support

So you may do yourself a favor and choose crm.example.com as web application server domain. In my case I did the following configuration:

• Web Application Server Domain: crm.example.com (You will get org.crm.example.com URLs for organizations)
• Organization Web Service Domain: crm.example.com
• Discovery Web Service Domain (Err, hostname): dev.crm.example.com
• (On the next page) Enter the external domain where your internet-facing servers are located: auth.crm.example.com (This URL will be used later when giving ADFS permissions)

Regarding certificates, you have 2 options. The DNS is on two levels, but wildcard can only cover one domain level. So you have 2 options now:

1. You now need additional wildcard cert for *.crm.example.com
2. You need all-in-one cert with these SANs:
   • org.crm.example.com
   • <any other org names>.crm.example.com
   • auth.crm.example.com
   • dev.crm.example.com
   • crm.example.com (non-IFD URL, maybe it is even your CRM hostname - wildcard cert doesn't cover this cert so it must be added to SAN anyways)

If you go with the first option by already having a cert for crm.example.com and would like to add just another certificate for *.crm.example.com:

• Add it to local computer certificate store.
• Tell your webserver to use this cert when accessing *crm.example.com:443. If you execute following command with PowerShell, escape curly braces {} with backticks: `{ and `}

   netsh http add sslcert appid={4dc3e181-e14b-4a21-b022-59fc669b0914} hostnameport=*.crm.example.com:443 certstorename=MY certhash=FF9688D828EEFEA33696CEB61153947623EBF2E1

I copied the appid from netsh http show sslcert ipport=0.0.0.0:443 - but it can be whatever, even [guid]::NewGuid() if you will.

• Also make sure in IIS, Microsoft Dynamics CRM app bindings have empty "Host Name" for https

About OAuth

Aren't you confused about different grants/flows/terminology? What to use when? How it works? How can I construct those requests myselff, etc. I was/am. Things start to make sense when I dig deeper. Not so much when I want to just get past authentication. These things helped me in a way:

1. Luckily the first thing that helped me orient is which flows are suitable for which case: Authentication flow support in the Microsoft Authentication Library (MSAL) - Microsoft Entra | Microsoft Learn - there you can click links and follow how exactly they work and how the request must be constructed.
2. Now the ClientId or apps or relying parties which I see in ADFS. Think of a ClientId like identifier for your app (doh). A thing that enables you to use OAuth. And there you can give access to some resources (relying parties or applications) - if permission is granted, the application you are trying to access will be satisfied with your token. Otherwise not. Remember Grant-AdfsApplicationPermission

💡 Aah, that's what registering an app in Azure AD is about - it is to enable OAuth! Ahh, this is what New-AdfsClient does, kind of the same!

Why is Microsoft doing this to us?

Why don't we get WS-Trust or windows integrated authentication? If we consider the following, all of this kind of makes sense:

• Microsoft.CrmSdk.XrmTooling.CoreAssembly being windows only has access to underlying windows security libs and ADFS via .NET Framework to construct Kerberos / WS-Trust token formats.
• ADAL is deprecated without backward support for ADFS 2016 which is still supported.
• Dynamics 365 CRM on-premises uses WS-Trust protocol which is deprecated by Microsoft, due to various vulnerabilities.
• All in all, DVSC being designed to be cross platform, has lost many authentication scenarios.

And then ADFS 2016 lacking necessary support for MSAL, because:

• Still relies on resource parameter (whereas these libraries only use scope)
• Doesn't support Proof Key for Code Exchange (PKCE)

So on-premises along with number of authentication requirements and support for ADFS 2016 is supported with Microsoft.CrmSdk.XrmTooling.CoreAssembly package. And Microsoft.PowerPlatform.Dataverse.Client supporting modern authentication requirements for cloud environment.

Helpful literature:

• Differences between ADAL.NET and MSAL.NET apps
• Migrate to the Microsoft Authentication Library (MSAL)

But the on-premise dinosaurs 🦖 can be modern too, right?

Other libraries

You may also want to check out Data8 .NET Core Client SDK for On-Premise Dynamics 365/CRM - Haven't tried it, but from what I read it supports WS-Trust and Windows Integrated Authentication.

P.S. 🦖 use the legacy vocabulary. Please don't be bothered too much when I call CRM what is in one way or another a subset/superset for Dataverse, PowerPlatform, Dynamics 365 Customer Engagement/Sales/Marketing, Common Data Service, XRM. 3 characters are the shortest way to reference CRM.

Comments

[Stepan Hakobyan]

Thanks Janis.