Moving Sharepoint access from Dynamics 365 OnPrem to Cloud with Graph API

I recently stumbled on a bug I completely underestimated, and lived to tell the story. It involved several unexpected issues, like version incompatibility, broken builds, technical debt, and all the classic problems legacy code brings.
I gained valuable insights along the way. Not only about the code but also dealing with clients and tight deadlines.
Let me take you through this journey tracking down this bug, and the lessons I learned.

Background

My problem started with the bug title, "Online Termination of employee not working". I completely misjudged the situation from the start. There were other bugs I had to fix for this client with very similar titles. Most issues were fixed by tracking down what is trying to access a null value or over-complicated logic that was falling on edge cases.

For the record, this client has been migrating their Dynamics 365 from OnPrem to the Cloud environment for 3 years, lost most of their staff in the process and was planning to go live just 1 week after that bug was assigned to me. I guess no one ever bothered to QA this, or it wasn't a core feature (but it was).

Diving into the code, I realized they were using Sharepoint to store termination letters and many other documents. I noticed that the code would fail at the ClientContext, part of the Microsoft.SharePoint.Client package.
Simple fix, I thought. Find out why it fails and close the bug. Regrettably, Microsoft decided to deprecate parts of the Sharepoint Client API.
Here's a quick peek at the OnPrem code:

private SharePointDocumentLocation CreateSharepointFolderForApplication(IOrganizationService service, org_application application, Guid applicationId)
        {
            SharePointDocumentLocation spDocLoc = new SharePointDocumentLocation
            {
                Name = "Documents on org 1",
                ParentSiteOrLocation = new EntityReference(SharePointDocumentLocation.EntityLogicalName, Guid.Parse("00000000-0000-0000-0000-000000000000")),
                RelativeUrl = application.org_name + "_" + applicationId.ToString(),
                RegardingObjectId = new EntityReference(org_application.EntityLogicalName, applicationId)
            };
            Guid _spDocLocId = service.Create(spDocLoc);

            SecureString securePassword = new SecureString();
            char[] arrPassword = SharePointPWD.ToCharArray();
            foreach (char c in arrPassword)
            {
                securePassword.AppendChar(c);
            }
            ClientContext context = new ClientContext("https://sharepoint.org.com/org/")
            {
                Credentials = new NetworkCredential("s_mywebdoc", securePassword)
            };
            Web web = context.Web;
            context.Load(web);
            CreteFolder(context, "Application", application.org_name + "_" + applicationId.ToString());
            xLog.AddLog("Done creating sharepoint folder", 1);

            return spDocLoc;
        }

At this point I was starting to realize it would not be that simple, but I was still in denial.

Since ClientContext wasn't an option anymore, or anything from Microsoft.SharePoint.Client for that matter, I had to find another package that would replace it.

I quickly found out about Microsoft Graph client and started to like it. Again, I thought, "simple fix". Replace the deprecated client with the new and close the bug.

I could not fathom my naivety. One does not simply introduce new packages to legacy code without paying the price.

To venture into legacy code is to tread lightly, for even the smallest change may awaken the wrath of unforeseen version incompatibility. -Tolkien J.R.R

Since the birth of .NET Core 1.0 in 2016, Microsoft has continuously emphasized its compatibility but has not figured out a way to allow Dynamics 365 plugins to be written using any version other than 4.6.2 (or 4.8.1 for the daring).

Long story short, there were conflicts with some packages, like CrmSdk.CoreAssemblies and CrmSdk.Workflow. And when everything seemed to work, either the build would not complete without errors or the dll could not be deployed to Dataverse. Trying to debug it was a huge waste of time. The use of ILMerge probably caused some of the issues, but the outcome would be the same. And of course I wasn't going to introduce Package Assemblies to this legacy code base one week away from go live (as if I didn't make a similar mistake before).

Scraping the whole idea of adding another package to the project, I did a quick git reset --hard HEAD and decided to go with the good ol' System.Net.Http.HttpClient. And there's no other way to start other than getting that sweet auth token and start playing with the API on Postman.

After going through the trouble of setting up an App Registration in Azure for Sharepoint, and start hitting the endpoint with unexpected responses, I noticed the deprecated feature also apply in this case.
Fortunately, I could just add the Graph API permissions and change my request. I'll add all the relevant requests in here:

The How To Guide

• Set up an App Registration:

You probably don't need to add all of those. For a more step by step guide I recommend following this blog post.

• Get your Auth Token. You should be ready to start making your requests now:

After getting all your variables in place, I recommend setting another variable for the token and adding this script. Use {{token}} for simplicity, but you may need a more specific name.

// Parse the JSON response body
const jsonData = pm.response.json();

if (jsonData.access_token) {
    pm.globals.set("token", jsonData.access_token);
}

• Create a request. Once you got a token, create a new request and add it to your header value, while the key will be Authorization:

Notice the URL is different from the one we used to get the token. Use https://graph.microsoft.com/v1.0/ as base URL.
We are also using parameters to search for the sitename. You can replace that with your site of choice, or get all of them. Later, we will see how this is relevant for the code we are writing.

I don't have a script for you, but you might want to copy the id of the chosen SharePoint site and also add to a variable. I decided to use {{QA-SiteId}}.

• Get the Drive id. By the way, I don't understand how these names make sense since Microsoft uses these terms interchangeably sometimes (site, drive, list, folder). Also I'm not a Sharepoint developer, so take my view of the names with a grain of salt.

What I recommend is getting all your Drive IDs first:
{{BaseUrl}}sites/{{QA-SiteId}}/drives/

Response:

Then copy the id from line 5 from the response and change/duplicate your request to search for the id every time. You might want to create another variable:

Having the specific ID also helps with request URL. You can make a bit shorter if you have the drive ID. It would look like this: {{BaseUrl}}drives/{{ApplicationsDriveId}}.

• Get folder id. This is necessary for this implementation, but may not be for you.

We need the parent folder for that drive.

The folder ID should look like this:

• Copy documents into another folder: Documentation for this can be found here.

How the code should look like

1. Get you access token: string accessToken = MyWebBusinessLogic.GetSharePointAccessToken(service).GetAwaiter().GetResult();

public static async Task<string> GetSharePointAccessToken(IOrganizationService service)
{
    var tenantId = GetEnvironmentVariable(service, "org_sharePointAppRegTenantId");
    var clientId = GetEnvironmentVariable(service, "org_sharePointAppRegClientId");
    var clientSecret = GetEnvironmentVariable(service, "org_sharePointAppRegClientSecret");
    string authority = $"https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token";

    using (HttpClient client = new HttpClient())
    {
        var requestData = new FormUrlEncodedContent(new[]
        {
            new KeyValuePair<string, string>("grant_type", "client_credentials"),
            new KeyValuePair<string, string>("client_id", clientId),
            new KeyValuePair<string, string>("client_secret", clientSecret),
            new KeyValuePair<string, string>("scope", "https://graph.microsoft.com/.default")
        });
        HttpResponseMessage response = await client.PostAsync(authority, requestData);

        if (!response.IsSuccessStatusCode)
        {
            throw new HttpRequestException($"Error fetching token: {response.ReasonPhrase}");
        }

        string responseContent = await response.Content.ReadAsStringAsync();
        dynamic tokenData = JsonConvert.DeserializeObject(responseContent);
        return tokenData.access_token;
    }

}

The environment variable can be fetched this way:

public static string GetEnvironmentVariable(IOrganizationService service, string schemaName)
        {
            // Query the EnvironmentVariableDefinition entity by Schema Name
            QueryExpression query = new QueryExpression("environmentvariabledefinition");
            query.ColumnSet = new ColumnSet("environmentvariabledefinitionid");
            query.Criteria.AddCondition("schemaname", ConditionOperator.Equal, schemaName);

            var envVarDef = service.RetrieveMultiple(query).Entities.FirstOrDefault();

            if (envVarDef == null)
            {
                throw new InvalidOperationException($"Environment Variable '{schemaName}' not found.");
            }

            var envVarDefId = envVarDef.Id;

            // Query the EnvironmentVariableValue entity for the value
            QueryExpression valueQuery = new QueryExpression("environmentvariablevalue");
            valueQuery.ColumnSet = new ColumnSet("value");
            valueQuery.Criteria.AddCondition("environmentvariabledefinitionid", ConditionOperator.Equal, envVarDefId);

            var envVarValue = service.RetrieveMultiple(valueQuery).Entities.FirstOrDefault();

            return envVarValue?.GetAttributeValue<string>("value") ?? string.Empty;
        }

1. Get the folder ids

public static async Task<string> GetFolderIdAsync(string accessToken, string driveId, string folderName)
{
    using (var client = new HttpClient())
    {
        client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);

        var url = $"https://graph.microsoft.com/v1.0/drives/{driveId}/root:/{folderName}";
        var response = await client.GetAsync(url);

        if (response.IsSuccessStatusCode)
        {
            var jsonResponse = await response.Content.ReadAsStringAsync();

            using (JsonDocument document = JsonDocument.Parse(jsonResponse))
            {
                if (document.RootElement.TryGetProperty("id", out JsonElement idElement))
                {
                    return idElement.GetString(); //Return the ID
                }
                throw new InvalidOperationException("ID not found in the response.");
            }
        }
        else
        {
            throw new HttpRequestException($"Request failed with status code: {response.StatusCode} - {response.ReasonPhrase}");
        }
    }
}

public static async Task<string> GetParentFolderIdAsync(string accessToken, string driveId)
{
    using (var client = new HttpClient())
    {
        client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);

        var folderUrl = $"https://graph.microsoft.com/v1.0/drives/{driveId}/root/children?$top=1";
        var response = await client.GetAsync(folderUrl);

        if (response.IsSuccessStatusCode)
        {
            var jsonResponse = await response.Content.ReadAsStringAsync();

            using (JsonDocument document = JsonDocument.Parse(jsonResponse))
            {
                if (document.RootElement.TryGetProperty("value", out JsonElement valueArray) && valueArray.GetArrayLength() > 0)
                {
                    if (valueArray[0].TryGetProperty("parentReference", out JsonElement parentReference) &&
                        parentReference.TryGetProperty("id", out JsonElement parentId))
                    {
                        return parentId.GetString(); // Return the parent folder ID
                    }
                }

                throw new InvalidOperationException("Parent folder ID not found in the response.");
            }
        }
        else
        {
            throw new HttpRequestException($"Request failed with status code: {response.StatusCode} - {response.ReasonPhrase}");
        }
    }
}

1. Copy the folder content

public async Task<HttpResponseMessage> CopyFolderAsync(ITracingService tracingService, string accessToken, string srcFolderDriveId, string srcFolderId, string desFolderId, string desFolderName, string applicationDriveQAId)
{
    tracingService.Trace($"Source Folder ID: {srcFolderDriveId}, Destination Folder ID: {desFolderId}, Destination Folder Name: {desFolderName}");

    try
    {
        using (HttpClient client = new HttpClient())
        {
            tracingService.Trace("Setting Authorization header with access token.");
            client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);

            tracingService.Trace($"Preparing to copy source folder (ID: {srcFolderDriveId}) to destination folder (ID: {desFolderId}).");
            //E.g. srcFolderDriveId = b!3ek... , srcFolderName
            string copyUrl = $"https://graph.microsoft.com/v1.0/drives/{srcFolderDriveId}/items/{srcFolderId}/copy";

            var copyPayload = new JObject
            {
                { "parentReference", new JObject {
                    { "driveId", applicationDriveQAId }, 
                    { "id", desFolderId } } 
                },
                { "name", desFolderName }
            };

            var copyContent = new StringContent(copyPayload.ToString(), Encoding.UTF8, "application/json");

            tracingService.Trace($"Sending POST request to copy folder. URL: {copyUrl}");
            HttpResponseMessage copyResponse = await client.PostAsync(copyUrl, copyContent);

            if (!copyResponse.IsSuccessStatusCode)
            {
                string copyErrorMsg = $"Error copying folder: {copyResponse.StatusCode} - {copyResponse.ReasonPhrase}";
                tracingService.Trace(copyErrorMsg);
                throw new Exception(copyErrorMsg);
            }

            return copyResponse;
        }
    }
    catch (Exception ex)
    {
        tracingService.Trace($"An error occurred in CopyFolderAsync: {ex.Message}");
        throw;
    }
}

1. Now we are ready to put everything together:

HttpResponseMessage copyResponse = null;
try
{
    var myWebSubmissionDriveId = MyWebBusinessLogic.GetEnvironmentVariable(service, "org_myWebSubmissionDriveId");
    var applicationDriveId = MyWebBusinessLogic.GetEnvironmentVariable(service, "org_applicationDriveId");

    string srcFolderName = results.org_name;
    string desFolderName = response.GetAttributeValue<string>("RelativeUrl");

    string srcFolderId = MyWebBusinessLogic.GetFolderIdAsync(
        accessToken,
        myWebSubmissionDriveId,
        srcFolderName).GetAwaiter().GetResult();

    string desFolderId = MyWebBusinessLogic.GetParentFolderIdAsync(
        accessToken,
        applicationDriveId).GetAwaiter().GetResult();

    copyResponse = myWebLogic.CopyFolderAsync(tracingService,
        accessToken,
        myWebSubmissionDriveId,
        srcFolderId,
        desFolderId,
        desFolderName,
        applicationDriveId).GetAwaiter().GetResult();

}
catch (Exception spEx)
{
    xLog.AddLog("ProcessSubmission Sharepoint Exception: " + spEx.Message, 3);
}
finally
{
    tracingService.Trace($"Copy folder reponse: {JsonConvert.SerializeObject(copyResponse)}");

}

Final remarks

This code shows the implementation I came up with for my client's needs. Moving away from On-Prem to Cloud in Dynamics 365 can be a lengthy process, but tackling issues like hardcoded IDs, even for non-sensitive data, and tightly coupling is important. Going forward, I guarantee my client will have a much better pattern for the next developers to base their code on.

There's still room for improvement, like Dependency Injection for the HTTP Client and better error handling. But overall, I'm proud of the result. The goal was achieved with the minimum required effort while respecting the client's budget and also providing an elegant and easy-to-maintain solution.