JMH Benchmarking in Docker Containers

This may sound a little crazy but I wanted to see if there was any difference in performance when running JMH benchmarks on bare metal vs. a container. Know your tools.

Why would you want to do this?

A lot of workloads run on Docker.

After you've written a few JMH tests, your mind naturally turns to workflow. How many of us have probably wished there was such a thing as "JMH studio" that let's you write JMH tests, push them to Github, have them run automagically in the cloud and have the results uploaded to a document store?

Not sure if that would qualify as a data lake or a pond though ;-)

Either way, I wanted to know how much difference containerisation makes to JMH tests, so off I went.

What did you find out?

I only ran some pretty basic JMH tests that were around calculating sums of arrays (reused from some earlier work), but I basically found that there is almost no difference for the case I was testing. Not saying that there won't be variation for other things, but at least I found none.

Show me some graphs?

Sure, why not:

I had to change the x-axis to make things even remotely visible. Here's a graph of the difference in performance:

We see a spike in the difference at smaller array sizes, with an outperformance of the test being run in the Docker container but this is only of the order of about 3% and I am also going to blame a small sample size.