Separating End to End Tests in Rust with Cargo Features

I've recently solved a problem I was having in my Rust tests, and I wanted to share my solution.

I have a web app that I've been working on, written in Rust of course. The build toolchain isn't anything fancy, I'm mostly sticking to what Cargo has to offer.

The problem started when I wanted to start adding end to end tests. In the context of web applications like this one, end to end testing means starting up the web app, including any infrastructure it depends on, and interacting with it using a real web browser via Selenium WebDriver. Infrastructure-wise, I currently use a SQLite database for simplicity, but will probably need to more to a server like PostgreSQL as the project matures. The tests also require a Git server running, since my web app can sync user data using Git.

In short, these tests are much heavier than my unit tests. They require starting up some Docker containers and running real web browsers.

I don't want to run them automatically when I type cargo test. In local development, I'd like to run just the unit tests most of the time, and then run the end-to-end tests only before committing.

I would like to do the following:

• cargo test should run all of the unit tests in my workspace. Basically everything except the end-to-end tests.

• cargo test-end-to-end should run only the end to end tests.

This is actually possible using only Cargo and feature flags.

The new tests live in the tests subdirectory of my web app crate. This is the standard place for integration tests, which will run after the web app's binary is built.

The trick is to add a new feature in the web server's Cargo.toml. The feature should be disabled by default.

# Cargo.toml

[features]
end-to-end-tests = []

Once you have this feature, you can add a custom test target that requires that feature.

# Cargo.toml

[[test]]
name = "end-to-end"
path = "tests/end-to-end.rs"
required-features = ["end-to-end-tests"]

This is already most of the way there! These integration tests will now be skipped when you run cargo test, unless you enable the feature with cargo test -F end-to-end-tests.

The next thing to solve is how to run only these tests. You can target this test suite with cargo test --test end-to-end -F end-to-end-tests. The only downside is that it's a pain to type. Luckily, Cargo has alias support. You can add an alias in .cargo/config.toml.

# .cargo/config.toml

[alias]
test-end-to-end = "test --test end-to-end -F end-to-end-tests"

Just like that, when we run cargo test-end-to-end it will use our alias to only run our end to end tests, including the feature to enable them.

Cargo's normal test harness, libtest, is great for unit tests. However, when it comes to end to end tests, the lack of a 'before all' and 'after all' step for setup and tear-down becomes a problem. The tests all running in parallel also isn't ideal for these end to end tests.

Luckily you're not stuck with libtest. If you add harness = false to your test target config then Cargo will run your main function.

# Cargo.toml

[[test]]
name = "end-to-end"
path = "tests/end-to-end.rs"
required-features = ["end-to-end-tests"]
harness = false

I'm going to leave what to do with that main function as an exercise for the reader. I personally used a crate called libtest-mimic, so that it could do the parts of libtest that I still wanted, like bring able to filter to a specific test case on the command line, while still being able to start my Docker containers and web server before any tests ran, and shut them down at the end.

So that's my trick for having end to end tests that aren't included in my unit test runs.

To recap, the method is to:

• Add a new feature

• Add a test harness that requires the new feature

• Add a Cargo alias for running that one specific test

• If required, disable the default test harness for greater control over how tests are run

I hope that this helps you in your own Rust testing journey!