FilterHN

Show HN: Benchi – A benchmarking tool written in Go

51 points

19 hours ago

| 2 comments

Benchi is a CLI tool for running benchmarks and collecting metrics. It's using Docker Compose to orchestrate the infrastructure and tools being benchmarked, making it repeatable and runnable on different machines. It allows you to run the same benchmark for different tools and compare the collected results.

The repository contains a simple example. For a more elaborate example see how we use Benchi to compare data pipelines running on Conduit and Kafka Connect, two data streaming tools (still work in progress): https://github.com/ConduitIO/streaming-benchmarks

▲

mixologic

14 hours ago

[-]

How do you control for changes in the host os environment that would affect performance of the system being observed?

▲

spockz

12 hours ago

[-]

We solve that partially by running the base state and the delta version through concurrently. This way most types of impact will impact both at the same time. This gives us the relative delta between versions.

Other than that, just like always, run benchmark on a stable dedicated set of hardware.

▲

edoceo

9 hours ago

[-]

And run a bunch of iteration, and mash the results together maybe?

▲

spockz

6 hours ago

[-]

Yes. Or for a sufficiently long duration. Some things like allocation rate and significantly worse performance are obvious and can be seen in shorter runs.

▲

meling

11 hours ago

[-]

What’s the overhead of running in docker vs bare metal?