Getting Started With BenchmarkDotNet

BenchmarkDotNet is a great library for performance testing on the dotnet platform. On this blog I’ve used it to benchmark different lists, for loops, and the cost of synchronous async methods.

A well written benchmark will help clarify and educate you about the performance of the code in question. Helping you write more performant applications.

Lets see how easy it is to get started.

First Benchmark

However you prefer, cli or Visual Studio, create a new C# project.

Then install the BenchmarkDotNet nuget package, at time of writing 0.12.0 is the latest version:

Nuget: Install-Package BenchmarkDotNet -Version 0.12.0

CLI: dotnet add package BenchmarkDotNet --version 0.12.0

That’s the only setup required. For our first benchmark lets demonstrate the benefits of using the StringBuilder class.

Paste this into a new file in your project.

We’re just concatenating strings, one using regular string concatenation and the other using a StringBuilder.

Then change your Program.cs to the following:

Then we’ll need to run this from the command line in release mode.

Release mode is important because there are optimizations applied for release mode that aren’t for debug mode. Benchmarking in debug mode will lead to inconsistent results.

If your project is setup like a normal dotnet project, Issue the following command from the folder that the .sln is in.

dotnet run -p .\BenchmarkIntro\ -c Release

The -p indicates which project to run and the -c indicates which mode to run it in (release)

On my machine that took about two minutes to run and at the end output the following chart.

The important columns to note are Method, N, Mean and Allocated.

|-------------- |----- |-----------:|----------:|
| RegularString |  100 |   6.331 us |   23736 B |
| StringBuilder |  100 |   2.981 us |    1000 B |
| RegularString | 1000 | 292.629 us | 2849740 B |
| StringBuilder | 1000 |  28.689 us |    9096 B |```

  * Method – which benchmark this is
  * N – what the value of N was.
  * Mean – how long the benchmark took
  * Allocated – how much memory was allocated during this benchmark.

As you can see, the performance and memory usage of the Regular String benchmark is pretty terrible compared to the StringBuilder. 

The reason being is C# strings are immutable. Every time we add a new character to our string we have to reallocate the entire string again. StringBuilder is mutable so it avoids all those memory allocations. 

Generally the immutability of strings is beneficial, but if you're doing a lot of string operations it's often useful to use StringBuilder.

## Why Use BenchmarkDotNet

I'm sure everyone has at some point broken out the timer to compare how fast a particular method might complete, or to compare two different methods.

That is at best a rudimentary benchmark, at worst it will lead to inconsistent results. 

BenchmarkDotNet handles a lot of the important details that gives credibility to benchmark results. It will:

  1. Perform warmp ups benchmarks
  2. Run the benchmark multiple times to ensure consistent results. 
  3. Automatically remove outliers that don't conform with the rest of the results
  4. It can run the benchmark against different .NET versions
      1. .NET Framework v .NET Core 2.2 vs .NET Core 3.0 Mono
  5. Run against different garbage collectors

Basically a lot of ton of useful features.

Outside of those great features, one of the main benefits of benchmarking is really giving facts for or against a particular belief about programming.

I've picked up a lot of programming knowledge in my 10ish years since first learning Java. I've written code in a half dozen languages, at 6 different companies, picking up random bits from who knows where. 

How much of what I believe true? I'd like to say a lot, but I know that's not true. Benchmarking my beliefs is one of the easiest ways to know if it's true or not. I've honestly been surprised by a benchmark result not agreeing with what I "thought" was true.

The same goes for you. It's a great tool to aid in learning and self improvement.

It's also a great tool to protect against performance problems in your apps before users encounter them. Ideally each application would have some sort of automated benchmark in their CI or CD pipeline. 

How do you know whether your twenty user stories this sprint severely degraded app performance? You won't know unless you're measuring it. 

## Closing Thoughts

I strongly advocate that you benchmark more things. Good benchmarks give clarity to code and beliefs. 

Benchmarking is a vital tool in the programmers toolbox. I'm glad tools like BenchmarkDotNet exist to make that task much easier.

#### Links:

  * [Github Repo](https://github.com/MorganW09/BenchmarkIntro)
  * [BenchmarkDotNet Repo](https://github.com/dotnet/BenchmarkDotNet)
  * [Pro .NET Benchmarking: The Art of Performance Measurement](https://www.amazon.com/gp/product/1484249402/)
      * Book by the maintainer of BenchmarkDotNet

 [1]: https://unsplash.com/@ideasboom?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText
 [2]: https://unsplash.com/s/photos/nature?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText