Unit Testing Project references & CI/CD Pipelines?

Unconventional approach, isn't it? Who has the time to unit test pipelines? Well, if you've ever had a deployment go sideways because someone forgot to update a pipeline file, you'll understand why I did this. Nobody wants to be the person who breaks the build right before the weekend.

Ensuring the integrity of a CI/CD pipeline is crucial, especially in a monorepo where multiple projects depend on each other. A simple update to one package can have a ripple effect, and before you know it, half the services are failing. Wouldn't it be nice to catch those issues before they land in production? That’s exactly what I set out to accomplish. Why Unit Test Pipelines?

- Catch Errors Early: Detect issues before they reach production.
- Check Dependencies: Ensure changes in one project don’t disrupt others.
- Early Feedback: Identify and fix problems quickly to reduce stress.
- Follow Best Practices: Avoid shortcuts to maintain quality.
- Prevent Costly Failures: Resolving issues early saves time and resources.

How I Did It
I used YamlDotNet for parsing YAML pipeline files and MSBuild API to scan project dependencies. By comparing what's in the pipeline with what projects actually need, I ensured everything stays aligned and prevented nasty surprises.

Parsing YAML with YamlDotNet
YamlDotNet is a fantastic .NET library for handling YAML, which is great because most CI/CD pipelines rely on YAML configurations. I used it to read and analyze pipeline definitions to extract relevant build steps, dependencies, and triggers.

public static IEnumerable<string> GetYamlPaths(string yamlFilePath)
{
    var yamlContent = File.ReadAllText(yamlFilePath);
    var deserializer = new DeserializerBuilder()
        .WithNamingConvention(CamelCaseNamingConvention.Instance)
        .IgnoreUnmatchedProperties()
        .Build();

    var config = deserializer.Deserialize<Yaml>(yamlContent);

    return (config.Trigger.Paths.Include ?? [])
        .Where(path => path.Contains("sdk", StringComparison.OrdinalIgnoreCase));
}

Necessary classes represent Yaml structure.

public class Yaml
{
    public Trigger Trigger { get; set; } = null!;
}

public class Trigger
{
    public Paths Paths { get; set; } = null!;
}

public class Paths
{
    public List<string>? Include { get; set; }
}

Scanning Dependencies with MSBuild API
The below code shows scanning for our needs. It is limited only by your imagination! You can check external packages as well, or even prevent specific packages from being updated if necessary. The possibilities are endless, and it all depends on how you choose to apply it.

The MSBuild API helped me dynamically analyze project dependencies. This was especially useful in a Monorepo where projects reference each other and changes can cascade.

public static List<string> GetProjectDependencies(string path)
{
    var solution = SolutionFile.Parse(path);
    var list = new List<string>();

    foreach (var projectInSolution in solution.ProjectsInOrder)
    {
        if (projectInSolution.ProjectType == SolutionProjectType.KnownToBeMSBuildFormat)
        {
            var projectPath = projectInSolution.AbsolutePath;
            var project = new Project(projectPath);

            var dependencies = project.Items
                .Where(item => item.ItemType == "ProjectReference" &&
                    Path.GetFileNameWithoutExtension(item.EvaluatedInclude)
                        .EndsWith("_SDK", StringComparison.OrdinalIgnoreCase))
                .Select(item => Path.GetFileNameWithoutExtension(item.EvaluatedInclude))
                .ToList();

            list.AddRange(dependencies);

            ProjectCollection.GlobalProjectCollection.UnloadProject(project);
        }
    }

    return list.Distinct().ToList();
}

What does the following test do?
Load the pipeline YAML file.
Get project dependencies.
Compare both lists and verify they match.

Sample Test Case

[Fact]
public void VerifyPipelineTriggers_MatchProjectDependencies()
{
    MSBuildLocator.RegisterDefaults();

    var baseDirectory = AppContext.BaseDirectory;
    var testsFolder = Directory.GetParent(baseDirectory)?.Parent?.Parent?.Parent?.FullName!;
    var rootDirectory = Directory.GetParent(testsFolder)?.FullName!;
    var yamlFilePath = Path.Combine(rootDirectory, "azure-pipelines.yml");
    var sln = Path.Combine(rootDirectory, "Service.Users.sln");

    var yamlPaths = PipelineTestHelper.GetYamlPaths(yamlFilePath)
        .Select(c => c.Split("services")[1])
        .Select(c => c.Replace('/', '.').TrimStart('.'))
        .ToList();

    var projectDependencies = PipelineTestHelper.GetProjectDependencies(sln);

    Assert.True(yamlPaths.SequenceEqual(projectDependencies, StringComparer.OrdinalIgnoreCase));
}

The test checks only paths in the YAML file and compares them with the suffix .Sdk to ensure that the SDK projects are included in the pipeline. You can extend it further based on your needs—your imagination is the only limit! For instance, you could enhance it to prevent certain vulnerable packages from being introduced into the project. This is especially beneficial when working with a large team of 40+ developers, ensuring that the project remains secure and aligned with best practices.

Conclusion
A well-tested pipeline leads to fewer disruptions, reduced manual intervention, and a more predictable release process. Investing in pipeline validation ultimately results in long-term software stability and operational efficiency.

Next Steps:
Planning Test app.settings Files: Many of our microservices follow the same structure, making it crucial to ensure that we do not overlook any common configurations. Implementing validation checks across these settings can prevent misconfigurations and inconsistencies across environments.