Measuring Performance in C#: Benchmarking with BenchmarkDotNet

Learn how to measure and optimize code performance in C# using BenchmarkDotNet. This detailed guide covers setup, benchmarking techniques, and best practices for comparing sorting methods, ensuring efficient and high-performing software

Performance is a critical aspect of software development, especially when working with large datasets and complex operations. In this article, we’ll explore how to measure the performance of code that operates on collections using BenchmarkDotNet, a powerful and lightweight library for benchmarking in C#.

---

Why Measure Performance?

Performance measurement helps identify bottlenecks and compare alternative solutions. Without an objective way to measure execution time, you’re essentially guessing whether a change improves performance. Benchmarking provides:

• Precise Measurements: Accurate execution time and resource usage.
• Comparison: Evaluate different implementations side by side.
• Guidance: Determine if optimization efforts are worthwhile.

---

Introducing BenchmarkDotNet

BenchmarkDotNet is a robust benchmarking library that:

• Eliminates noise from external factors (e.g., OS scheduling, JIT compilation).
• Provides detailed results, including mean execution time and relative performance.

Setting Up BenchmarkDotNet

1. Install BenchmarkDotNet: Add the NuGet package to your project:

   dotnet add package BenchmarkDotNet

1. Create a Benchmark Class: To keep the Program class clean, create a separate class, e.g., SortingBenchmark, where you’ll define benchmark methods.

---

Step-by-Step Example

Here’s a practical example of benchmarking a sorting operation on a list of 1 million objects:

1. Define the Benchmark Class

using BenchmarkDotNet.Attributes;
using System.Collections.Generic;
using System.Linq;

public class SortingBenchmark
{
    private List<Worker> workers;

    [GlobalSetup]
    public void Setup()
    {
        workers = Enumerable.Range(1, 1_000_000)
                            .Select(i => new Worker { Rate = i })
                            .ToList();
    }

    [Benchmark(Baseline = true)]
    public void SortList()
    {
        workers.Sort(Worker.RateComparer);
    }

    [Benchmark]
    public void SortWithLinq()
    {
        var sorted = workers.OrderBy(w => w.Rate).ToList();
    }
}

public class Worker
{
    public int Rate { get; set; }
    public static IComparer<Worker> RateComparer => Comparer<Worker>.Create((x, y) => x.Rate.CompareTo(y.Rate));
}

• Setup: The [GlobalSetup] attribute initializes data before running benchmarks.
• Benchmarks: The [Benchmark] attribute marks methods for measurement.
• Baseline: The Baseline = true property compares other methods to a reference implementation.

---

2. Run the Benchmark

Use the BenchmarkRunner in the Program class:

using BenchmarkDotNet.Running;

class Program
{
    static void Main(string[] args)
    {
        BenchmarkRunner.Run<SortingBenchmark>();
    }
}

---

Enhancing Benchmarks

Parameterized Benchmarks

Use the [Params] attribute to test with varying data sizes:

[Params(1000, 10000, 100000)]
public int Count;

[GlobalSetup]
public void Setup()
{
    workers = Enumerable.Range(1, Count)
                        .Select(i => new Worker { Rate = i })
                        .ToList();
}

Measuring Infrastructure Overheads

Include benchmarks for unavoidable operations (e.g., creating lists) to identify how much time they consume:

[Benchmark]
public void CollectList()
{
    var list = workers.ToList();
}

---

Understanding the Results

After running the benchmarks, BenchmarkDotNet generates a detailed report:

Method	Mean (us)	Relative
SortList	100.0	1.00
SortWithLinq	120.0	1.20

• Mean (us): Average execution time in microseconds.
• Relative: Execution time relative to the baseline.

Insights

In our example, SortWithLinq was slower than SortList. This outcome is expected since LINQ’s OrderBy introduces overhead for deferred execution and result conversion.

---

Best Practices for Benchmarking

1. Use Release Mode:

   • Always benchmark in Release mode to avoid debug-related overheads.

2. Isolate Code:

   • Focus on small, critical operations to ensure accurate measurements.

3. Set a Baseline:

   • Compare alternatives to a baseline method for meaningful results.

4. Anticipate Discoveries:

   • Performance tests may reveal unexpected results, guiding you toward better designs.

---

Conclusion

Benchmarking is an essential tool for understanding and optimizing code performance. Using BenchmarkDotNet, you can easily measure execution time, compare implementations, and make informed decisions about your code. Whether you’re sorting large datasets or evaluating different algorithms, benchmarking ensures your optimizations are based on hard data.

By integrating benchmarking into your development process, you’ll write better-performing and more efficient software.