Voxxed Days Berlin has ended
Thursday, January 28 • 10:20 - 11:10
Make your tests fail

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

How randomisation adds a whole new dimension to finding bugs in your code

It's easy as pie: before checking in, your test suite should always be green. Or should it? What if your tests are all green but you forgot to check one important edge case? What if your underlying system environment lets you down, but only under rare conditions that you didn't cover in your tests?

This talk introduces randomised testing as used by projects like Apache Lucene and Elasticsearch based on the Carrotsearch Randomised Testing framework. It has helped uncover (and ultimately fix) a huge number of bugs not only in these project’s source code, but also in the JVM itself which those projects rely on.

Writing unit and integration tests can be tricky: assumptions about your code may not always be true as any number of "this should never happen" log entries in production systems show. When implementing a system that will be integrated in all sorts of expected, unexpected, and outright weird ways by downstream users, testing all possible code paths, configurations and deployment environments gets complicated.

With the Carrotsearch Randomised Testing framework, projects like Apache Lucene and Elasticsearch have introduced a new level to their unit and integration tests. Input values are no longer statically pre-defined but are generated based on developer defined constraints, meaning The test suite is no longer re-run with a static set of input data each time. Instead, every continuous integration run adds to the search space covered. Though generated at random, tests are still reproducible as all configurations are based on specific test seeds that can be used to re-run the test with the exact same configuration.

Add to this randomising the runtime environment by executing tests with various JVM versions and configurations,and you are bound to find cases where your application runs into limitations and bugs in the JVM.

This talk introduces randomised testing as a concept, shows examples of how the Carrotsearch Randomised Testing framework helps with making your test cases more interesting, and provides some insight into how randomising your execution environment can help save downstream users from surprises. All without putting too much strain on your continuous integration resources.

avatar for Isabel Drost-Fromm

Isabel Drost-Fromm

Open Source Strategist, Europace AG
Isabel Drost-Fromm is a member of the Apache Software Foundation, a co-founder and board member of the InnerSource Commons Foundation, and a member of the inaugural Open Source and Intellectual Property Advisory Group of the United Nations Technology Innovation Labs (UNTIL). She's... Read More →

Thursday January 28, 2016 10:20 - 11:10 CET
Kosmos - Room 06