Rust testing ecosystem

This post is part of an ongoing series about tests in Rust:

  1. Rust testing ecosystem

  2. Fluent assertions for numbers in Rust

  3. Equality assertions

  4. Assertion reporting

  5. Make it look like rainbows 🌈

Introduction

I’ve been working with Rust daily for a year now and generally, I have been very happy with this language and it’s ecosystem. Of course, the ecosystem is still pretty new, a lot of things are missing or incomplete but it’s only a five year’s language and I feel that for this age, its already pretty mature.

However, there is one domain in which I feel this language is still pretty poor in term of ecosystem: testing. And I love testing! No really, I know most people find the task of writing tests boring, annoying but this is a task I am generally enthusiast to do for multiple reasons:

  • I don’t trust my code, test is a fun way to automatically verify what I’ve done

  • I found out that in general, TDD helps me think better of a problem before implementing it

  • I believe that testable code is producing better code architecture

Testing in Rust language

Today in Rust, testing can be summarized in a few functionalities:

  • a #[test] macro

  • an assert!() macro

  • a test configuration option (used in #[cfg(test)])

  • a folder tests/ with cargo [1]

  • Documentation tests

Compared to other languages, it’s pretty similar: an assert command and the rest is to declare what and where is a test. The notable exception is Documentation Tests which is quite innovative and awesome.

In this post, I will focus on assertions. Indeed, assert!() is a necessary building block and a simple one, but it’s not really fun to use. Better API can be built on top of it.

In this post, I will not talk about mocking (there is a bunch of libraries around) or other types of testing like property testing (check quickcheck and proptest) or fuzzing.

Assertion

For asserting in tests, most languages provides libraries on top of the assert function in order to make it more fluent to write unit tests. I will cite two notable examples I’ve worked with in other languages (but there is much more libraries out there): AssertJ and ChaiJS.

Let’s take a look at some existing libraries in Rust ecosystem.

difference

This library is specialized in asserting equalities between &str. The quality is quite good and it also show a very nice diff output…​ but of course, it’s only about asserting equality of &str.

approx

This library is specialized in asserting floating numbers. Assertion on floating numbers is not as easy because of rounding errors. approx helps in asserting floating numbers that are almost equals!

totems

This library is the first with an intuitive yet simple syntax for asserting things like enum assertions (although only for Result and Option), tuples, and collections. It’s pretty simple but effective. However, a more fluent API would be nice.

There hasn’t been a release in a year and everything has been developed in a month by one developer. This is not promising in term of maintenance.

hamcrest-rust and hamcrest2

The former is not maintained anymore, the second being a fork. hamcrest2 is the effort of one developer and no release (no commit on the project either) has been done in the last 6 months. And also, but that might be a personal preference of me, writing assertions with Hamcrest is not pleasant: you have to wrap in parenthesis everything. See this example:

assert_that!(1, not(eq(2)));

expecttest

Another very nice and relatively complete library. However, it seems to me it has almost the same kind of problems than hamcrest2, you have to wrap most of things in parenthesis (although the syntax is slightly better), there is only one developer and last release has been 7 months ago. One thing to notice is that the developer explicitely flagged the project as passively maintained so maybe some contributions would revive the project?

spectral

This one provide a more fluent API. For example, you can write things like:

assert_that(&vec![1, 2, 3]).has_length(3);

On top of that, the output failure message is improved to show in a more structured way the expected and the produced value. The default printing is not always easy to read.

This library is actually really close to something I would look for but, there is few problems with it.

First of all, it’s far from complete, more assertions functions could be added. I actually tried to add some more in this PR but this leads me to my second point.

The library is not maintained anymore. Last release was three years ago and since this crate is owned by the only developer who implemented it, only a fork seems to be a way forward at this point.

Finally, I don’t see anything about chaining assertions in this library. For example, in AssertJ, you can do things like this.

assertThat("QXNzZXJ0Sg")
	.decodedAsBase64()
	.containsExactly("AssertJ".getBytes());
assertThat(throwable)
	.hasMessage("top level")
	.getCause()
	.hasMessage("cause message");

Conclusion

Ideally, something like spectral is close from what I would look for. Forking might be a solution. I actually started to improve some annoyments I had in a branch. Much more functionalities could be added to it. Here is a non-exhaustive list of things:

  • more APIs (take example on other libraries in other languages)

  • ability to chain calls

  • implement as features for other libraries (for example, a specific map implementation like vec_map could implement HashMapAssertions).

  • colorize output error messages

Of course, I might have missed other libraries. I mostly used crates.io to search with only a bunch of keywords.

Sources

  • crates.io, where you can find most of the libraries from this post (the only exception seems to be hamcrest-rust

  • Awesome Rust, which lists interesting project in Rust, especially the “Testing” section

1. There is also benches/ but most of what you need to use it is only available in nightly Rust, see test::bench

links

social