Dataverse Git integration (Preview) - How are solution components saved?

This is the last article (for now) of the series about Dataverse Git integration native capability. We have already seen:

• How to set up the integration between Dataverse and an Azure DevOps Git repository
• How to properly manage the commit+push and pull operations, and resolve conflicts
• What kind of ALM approach arises from the bottom up analysis of the capabilities provided by the feature

Now we'll deep dive on how solution components are actually saved in the repository, and compare the solution folder structure with the one generated by PAC CLI.

👋🏻 The manual approach - pac solution clone

As of today, if you want to version your solution content, the best way is to use the PAC CLI command

pac solution clone --name sampleSolution

It will:

1. download the sampleSolution from your environment
2. create a subfolder called sampleSolution in your current folder
3. unpack the solution contents into sampleSolution folder
4. create a .cdsproj file that can be used by msbuild or dotnet build to recreate the original solution, in both managed or unmanaged way
5. add also a .gitignore file in the same folder, to avoid committing in the repo the outputs of the build process

Once cloned, one can use

pac solution sync

to sync locally any change made to the solution in the Dev dataverse environment, and use git commit and git push to save the changes to his preferred git repo. Just the delta will actually be included in the commit.

The structure of the unpacked folder is pretty strikeforward. You have:

• 📂 The root folder which name matches the unique name of the solution you exported
  • 📄 the solution_unique_name.cdsproj file
  • 📄 the .gitignore file``
  • 📂 an src folder containing the actual contents of the solution. This folder contains a subfolder for each solution component type, such as:
    • 📂 AppModules
    • 📂 CustomApis
    • 📂 Entities
    • 📂 Other
    • 📂 PluginAssemblies
    • 📂 PluginPackages
    • 📂 Roles
    • 📂 WebResources
    • 📂 ...

Each solution component folder has his own subfolder, but we can identify a few common rules:

1. Actual component information is saved in XML format
2. Subfolders and file names reflects the component meaningful name (e.g. the xml file containing the role definition is saved as .xml)

The 📂 Other folder is a special folder that contains all solution specific content, such as:

• 📂 A subfolder called Relationships, where all the relationship metadata between tables of the current solution are saved
• 📄 customizations.xml: the index of the current solution contents, required to rebuild the solution from the repo
• 📄 solution.xml: the current solution metadata
• 📄 other generic xml files such as Relationships.xml (index of all relationships included in the solution), FieldSecurityProfile.xml (list of all field security profiles included in the solution), and so on.

🆕 Folder structure from Git integration

There are several differences between the way solution contents are unpacked by pac solution clone/sync, and the way they are unpacked and saved by native Git integration. Let's take a look at them.

❌ No more .cdsproj file

Native git integration doesn't generates a .cdsproj file. This means that you cannot generate a solution package directly from the Git repository: you need to rely on a "consolidation" environment, as described in my previous article.

This is also clearly stated in the official docs:

Git integration is intended to be used with developer environments and not in your test or production environments where deployments can be made using builds to create solution artifacts and pipelines to deploy.

📚 A common structure for all solutions of your env

The native Git integration is designed to

allow(s) for multiple solutions to use a single root folder location and keep(s) track of which components belong to each solution in a separate file. It's no longer a requirement to use a unique root folder for each solution

This is a big change compared to the standard approach, where you need to have a separate folder for each solution. With native Git integration you can store several solutions under the same root, and shared components
(if any) will be written only once. Moreover, solution specific metadata will be saved under the 📂 solution folder:

📃 YAML replaces XML

One of the biggest differences is the language used for customization files. YAML is used instead of XML because:

it's easier to read, understand, and facilitates easier merges

Even if sometimes it looks like a plain YAML serialization of the old XML files, without specific parsing/transformation (see image below)

Personally, I'm not a big fan of YAML, the whole idea that "spacing matters" gives me headaches. I think JSON would have been easier to read, parse and validate as YAML, but... that's the choice, we'll have to stick with it.

📁 Folder/file redundancy

Sometimes the new approach generates redundant files/folders. Take as example the image below:

appmodules is the folder where all apps are stored. Under that folder there is a folder for each app, called like the app unique name, then under the app folder there is an appmodule.yml file. Same for appmodulesitemaps, attributeimageconfigs, optionsets, sdkmessageprocessingsteps, publishers and other solution components.

Wouldn't it be easier to put a simple appuniquename.yml under the appmodules folder?

🌳 Improved structure of the entity folder

PAC CLI unpacks entities separating:

• Formulas
• Forms
• SavedQueries
• RibbonDiffXml

From the main entity definition XML, that still contains all the entity flags, attributes and keys into a single, huge, xml file.

This has been improved a lot in the new Git integration. Now:

• each attribute definition is saved in a separate YAML file under the 📂 attributes folder

• even entity keys are stored separately, even if they suffer from the redundancy issue described previously

• for system forms you no longer have the difference between managed and unmanaged version (moreover, the new approach suffers of folder redundancy)

• saved queries are saved by ID, like in the pac solution clone/sync version. Personally I would have preferred they to be saved by localized name, to be more easily recognizable (roles have that approach, they are saved by name and not by ID).

😵‍💫 The bad: WebResources

If the way entities are exported improved a lot from the old version, WebResources suffered from a painful and unexplicable downgrade.

While PAC CLI saves WebResources using a developer designed, folder based, approach, that reflects the naming of the webresource in the actual environment:

The native Git integration saves each WebResource in a folder named after the WebResource internal ID. Unreadable and unbrowsable.

🚫 Some components are not exported

As of today not all solution components are properly exported. In my tests I've found that:

• Plugin Packages
• Field Security Profiles

Are not present in the git repo after the sync. I haven't tried all kind of solution components... let me know in the comments if you find any other missing item.

🤔 Conclusions

We have seen the main differences between the way PAC CLI and native Git Integration generates the solution folders in our repos.

🟢 I really like the way native Git integration works with entities, and also the usage of YAML instead of XML that makes everything more readable (even if JSON would have been better, IMHO).

⚠️ There's still room for improvements, e.g.

• on the way WebResources are saved (PAC CLI approach is WAAAAAY better)
• on removing redundant, useless folders
• on supporting the missing component types

On the other hand, the capability it's still in preview, and features evolve fast, so... let's stay tuned for changes on this topics!

What do you think about it? Drop a comment below if you found something that I've missed in my analysis, or to share any feedback on the capability itself!

Comments

[vkama]

What about connection references that are use in an unmanaged solution?

Tried to copy a solution that has many connection references to my dev environment and when i edited a cloud flow that was part of the solution i couldn't find any connection reference, plus that all of the actions that used these references were broken.