Really feel that this is a missing area in CI today and looking forward to what codspeed and https://bencher.dev/ are able to create.
Unfortunately, I'm not using either atm. Last I looked at them:
- codspeed's Rust integration involved stubbing out a specific benchmark library which seems non-ideal and I've moved away from that library
- bencher required manually registering your benches before using it. I have responsibility over a large number of Rust repos and try to get rid of as many per-repo steps as possible. If I can't put it in my template (https://github.com/epage/_rust) and then merge it into each of my repos to get the benefit, I'm unlikely to use it.
While I've not worked with the codspeed developers, I have been providing feedback to the bencher developers and have hope that I'll be able to eventually use a solution.
> Really feel that this is a missing area in CI today
Years ago I wanted a tool to continuously build and test some soft while measuring and saving the resources consumed by the tests to spot large variations.
Couldn't find one so I built my own: https://github.com/rixed/lurch
It does not collect any instrumentation though, just measures cpu/io ressources in the cgroup as a whole.
Really feel that this is a missing area in CI today and looking forward to what codspeed and https://bencher.dev/ are able to create.
Unfortunately, I'm not using either atm. Last I looked at them:
- codspeed's Rust integration involved stubbing out a specific benchmark library which seems non-ideal and I've moved away from that library
- bencher required manually registering your benches before using it. I have responsibility over a large number of Rust repos and try to get rid of as many per-repo steps as possible. If I can't put it in my template (https://github.com/epage/_rust) and then merge it into each of my repos to get the benefit, I'm unlikely to use it.
While I've not worked with the codspeed developers, I have been providing feedback to the bencher developers and have hope that I'll be able to eventually use a solution.
> Really feel that this is a missing area in CI today
Years ago I wanted a tool to continuously build and test some soft while measuring and saving the resources consumed by the tests to spot large variations. Couldn't find one so I built my own: https://github.com/rixed/lurch It does not collect any instrumentation though, just measures cpu/io ressources in the cgroup as a whole.
Sincere question : how is it different that something like K6 ? > https://github.com/grafana/k6
As far as I understand, Codspeed is a tool to perform continuous benchmarking. k6 is a specific implementation of http benchmarking