Creating a TRX File from an API for Azure DevOps CI/CD Pipeline Integration

The purpose of this article is to provide you with an example of a TRX file structure and how our team took a TRX formatted string response from a web service and injected it into our CI/CD pipeline on Azure DevOps.

This web service is called Palladium and is an open source solution for executing asynchronous, long-running tests in Orleans and produces TRX results via a HTTP web API endpoint.

A TRX File, Rawr! 🦖

A TRX file (or T-REX file as I like to pronounce it) is a test result file created typically by Visual Studio when outputting the results of unit tests. It has a .trx file extension and is formatted the same as xml. It can be viewed and edited in most text editors or IDE's such Visual Studio or Jetbrains Rider.

The Belly of the Beast 😱

Below is an example of the TRX output from Palladium /AsynchronousTests. We made some slight alterations from the default file structure to better serve our needs. For example we removed the deployment node from TestSettings and made it self closing. We also did away with the contents of the Output node.

<?xml version="1.0" encoding="UTF-8"?>
<TestRun xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010" id="a5e9e312-66fe-4f2d-b0c9-ce0fbc57cd9b" name="10ed7f37-b86d-4b36-a378-3c3fa724757e">
   <Times creation="2022-02-24T15:14:59.6966860Z" queueing="2022-02-24T15:14:59.6981900Z" start="2022-02-24T15:14:59.6981900Z" finish="2022-02-24T15:15:00.0311280Z" />
   <TestSettings id="fe569017-6430-466c-b8b8-2360549b5576" name="39bc153c-86b0-404a-b93f-67113b5cca75" />
   <Results>
      <UnitTestResult executionId="09b849b9-2e9d-42ec-808e-a530ac272082" testId="2a3b891d-a24b-4e16-b730-a73607835817" testName="Test1" computerName="352509f7-e206-465e-966c-b7574939feb5" duration="0:00:00.129834" startTime="2022-02-24T15:14:59.7007960Z" endTime="2022-02-24T15:14:59.8306300Z" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="2c0cd20b-ad9a-4ade-b491-479e8d9d4a0d" relativeResultsDirectory="09b849b9-2e9d-42ec-808e-a530ac272082" />
      <UnitTestResult executionId="a27998f9-7b79-4b6f-86f9-6b569f46685b" testId="1ece5274-6d1e-4dff-b314-dcee7b751358" testName="Test2" computerName="352509f7-e206-465e-966c-b7574939feb5" duration="0:00:00.227086" startTime="2022-02-24T15:14:59.7030020Z" endTime="2022-02-24T15:14:59.9300880Z" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="2c0cd20b-ad9a-4ade-b491-479e8d9d4a0d" relativeResultsDirectory="a27998f9-7b79-4b6f-86f9-6b569f46685b" />
      <UnitTestResult executionId="99fb1cb4-8ad8-48d8-97f2-97805f3d0746" testId="546e9edb-7b96-4ada-9f45-9f947860839e" testName="Test3" computerName="352509f7-e206-465e-966c-b7574939feb5" duration="0:00:00.327032" startTime="2022-02-24T15:14:59.7032380Z" endTime="2022-02-24T15:15:00.0302700Z" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="2c0cd20b-ad9a-4ade-b491-479e8d9d4a0d" relativeResultsDirectory="99fb1cb4-8ad8-48d8-97f2-97805f3d0746" />
   </Results>
   <TestDefinitions>
      <UnitTest id="2a3b891d-a24b-4e16-b730-a73607835817" name="Test1" storage="AsynchronousTests.dll">
         <Execution id="09b849b9-2e9d-42ec-808e-a530ac272082" />
         <TestMethod codeBase="AsynchronousTests.dll" className="Grains.AsynchronousTests" name="Test1" adapterTypeName="orleans" />
      </UnitTest>
      <UnitTest id="1ece5274-6d1e-4dff-b314-dcee7b751358" name="Test2" storage="AsynchronousTests.dll">
         <Execution id="a27998f9-7b79-4b6f-86f9-6b569f46685b" />
         <TestMethod codeBase="AsynchronousTests.dll" className="Grains.AsynchronousTests" name="Test2" adapterTypeName="orleans" />
      </UnitTest>
      <UnitTest id="546e9edb-7b96-4ada-9f45-9f947860839e" name="Test3" storage="AsynchronousTests.dll">
         <Execution id="99fb1cb4-8ad8-48d8-97f2-97805f3d0746" />
         <TestMethod codeBase="AsynchronousTests.dll" className="Grains.AsynchronousTests" name="Test3" adapterTypeName="orleans" />
      </UnitTest>
   </TestDefinitions>
   <TestEntries>
      <TestEntry testId="2a3b891d-a24b-4e16-b730-a73607835817" executionId="09b849b9-2e9d-42ec-808e-a530ac272082" testListId="2c0cd20b-ad9a-4ade-b491-479e8d9d4a0d" />
      <TestEntry testId="1ece5274-6d1e-4dff-b314-dcee7b751358" executionId="a27998f9-7b79-4b6f-86f9-6b569f46685b" testListId="2c0cd20b-ad9a-4ade-b491-479e8d9d4a0d" />
      <TestEntry testId="546e9edb-7b96-4ada-9f45-9f947860839e" executionId="99fb1cb4-8ad8-48d8-97f2-97805f3d0746" testListId="2c0cd20b-ad9a-4ade-b491-479e8d9d4a0d" />
   </TestEntries>
   <TestLists>
      <TestList id="2c0cd20b-ad9a-4ade-b491-479e8d9d4a0d" name="All Loaded Results" />
   </TestLists>
   <ResultSummary outcome="Complete">
      <Counters total="3" executed="3" passed="3" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
      <Output StdOut="" />
   </ResultSummary>
</TestRun>

VSTS Integration 🤜🤛

Below is an extract of two steps from our CI/CD pipeline which occur after we deploy our application to our test environment. We configured our Palladium unit tests to run on that same environment.

We first used the Bash@3 task to call the internally hosted instance of the Palladium solution and save the resulting string as a .trx file in the DefaultWorkingDirectory of the VSTS server.

We then configured a PublishTestResult@2 task to VSTest and searched explicitly for the palladium-output.trx file in the working directory. This step publishes the contents of the TRX file to the UI on the Azure pipeline build.

- task: Bash@3
  displayName: Calling Palladium API Tests
  inputs:
    targetType: 'inline'
    script: |
      curl -s -m 5 $(PALLADIUM_API) -o palladium-output.trx

- task: PublishTestResults@2
  displayName: Printing Test Result File
  inputs:
    testResultsFormat: 'VSTest'
    testResultsFiles: 'palladium-output.trx'
    searchFolder: '$(System.DefaultWorkingDirectory)'
    testRunTitle: 'Test Palladium TRX'
    publishRunAttachments: true
    failTaskOnFailedTests: true

The $(PALLADIUM_API) environment variable holds our URL to the test result endpoint of our Palladium instance.

Conclusion

The reason for the above implementation is because we as a team wanted to abstract out our unit tests into an independent API solution to call our various environments. If you decide to implement a similar design you can integrate it into your CI/CD with the above solution.

Co-Author: Piotr Justyna