rust-book-cn/nostarch/chapter11.md

[TOC]

Testing

Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence.

Edsger W. Dijkstra, "The Humble Programmer" (1972)

Rust is a programming language that cares a lot about correctness, but correctness is a complex topic and isn't easy to prove. Rust places a lot of weight on its type system to help ensure that our programs do what we intend, but it cannot help with everything. As such, Rust also includes support for writing software tests in the language itself.

For example, we can write a function called add_two with a signature that accepts an integer as an argument and returns an integer as a result. We can implement and compile that function, and Rust can do all the type checking and borrow checking that we've seen it's capable of doing. What Rust can't check for us is that we've implemented this function to return the argument plus two and not the argument plus 10 or the argument minus 50! That's where tests come in. We can write tests that, for example, pass 3 to the add_two function and check that we get 5 back. We can run the tests whenever we make changes to our code to make sure we didn't change any existing behavior from what the tests specify it should be.

Testing is a skill, and we cannot hope to cover everything about how to write good tests in one chapter of a book. What we can discuss, however, are the mechanics of Rust's testing facilities. We'll talk about the annotations and macros available to you when writing your tests, the default behavior and options provided for running your tests, and how to organize tests into unit tests and integration tests.

Writing Tests

Tests are Rust functions that use particular features and are written in such a way as to verify that non-test code is functioning in the expected manner. Everything we've discussed about Rust code applies to Rust tests as well! Let's look at the features Rust provides specifically for writing tests: the test attribute, a few macros, and the should_panic attribute.

The test attribute

At its simplest, a test in Rust is a function that's annotated with the test attribute. Let's make a new library project with Cargo called adder:

$ cargo new adder
     Created library `adder` project
$ cd adder

Cargo will automatically generate a simple test when you make a new library project. Here's the contents of src/lib.rs:

Filename: src/lib.rs

#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
    }
}

For now, let's ignore the tests module and the #[cfg(test)] annotation in order to focus on just the function. Note the #[test] before it: this attribute indicates this is a test function. The function currently has no body; that's good enough to pass! We can run the tests with cargo test:

$ cargo test
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished debug [unoptimized + debuginfo] target(s) in 0.22 secs
     Running target/debug/deps/adder-ce99bcc2479f4607

running 1 test
test it_works ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured

   Doc-tests adder

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured

Cargo compiled and ran our tests. There are two sets of output here; we're going to focus on the first set in this chapter. The second set of output is for documentation tests, which we'll talk about in Chapter 14. For now, note this line:

test it_works ... ok

The it_works text comes from the name of our function.

We also get a summary line that tells us the aggregate results of all the tests that we have:

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured

The assert! macro

The empty test function passes because any test which doesn't panic! passes, and any test that does panic! fails. Let's make the test fail by using the assert! macro:

Filename: src/lib.rs

#[test]
fn it_works() {
    assert!(false);
}

The assert! macro is provided by the standard library, and it takes one argument. If the argument is true, nothing happens. If the argument is false, the macro will panic!. Let's run our tests again:

$ cargo test
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished debug [unoptimized + debuginfo] target(s) in 0.22 secs
     Running target/debug/deps/adder-ce99bcc2479f4607

running 1 test
test it_works ... FAILED

failures:

---- it_works stdout ----
	thread 'it_works' panicked at 'assertion failed: false', src/lib.rs:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.


failures:
    it_works

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured

error: test failed

Rust indicates that our test failed:

test it_works ... FAILED

And shows that the test failed because the assert! macro in src/lib.rs on line 5 got a false value:

thread 'it_works' panicked at 'assertion failed: false', src/lib.rs:5

The test failure is also reflected in the summary line:

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured

Testing equality with the assert_eq! and assert_ne! macros

A common way to test functionality is to compare the result of the code under test to the value you expect it to be, and check that they're equal. You can do this using the assert! macro by passing it an expression using the == macro. This is so common, though, that the standard library provides a pair of macros to do this for convenience: assert_eq! and assert_ne!. These macros compare two arguments for equality or inequality, respectively. The other advantage of using these macros is they will print out what the two values actually are if the assertion fails so that it's easier to see why the test failed, whereas the assert! macro would just print out that it got a false value for the == expression.

Here's an example test that uses each of these macros and will pass:

Filename: src/lib.rs

#[test]
fn it_works() {
    assert_eq!("Hello", "Hello");

    assert_ne!("Hello", "world");
}

You can also specify an optional third argument to each of these macros, which is a custom message that you'd like to be added to the failure message. The macros expand to logic similar to this:

// assert_eq! - panic if the values aren't equal
if left_val != right_val {
    panic!(
        "assertion failed: `(left == right)` (left: `{:?}`, right: `{:?}`): {}"
        left_val,
        right_val,
        optional_custom_message
    )
}

// assert_ne! - panic if the values are equal
if left_val == right_val {
    panic!(
        "assertion failed: `(left != right)` (left: `{:?}`, right: `{:?}`): {}"
        left_val,
        right_val,
        optional_custom_message
    )
}

Let's take a look at a test that will fail becasue hello is not equal to world. We've also added a custom error message, greeting operation failed:

Filename: src/lib.rs

#[test]
fn a_simple_case() {
    let result = "hello"; // this value would come from running your code
    assert_eq!(result, "world", "greeting operation failed");
}

Running this indeed fails, and the output we get explains why the test failed and includes the custom error message we specified:

---- a_simple_case stdout ----
	thread 'a_simple_case' panicked at 'assertion failed: `(left == right)`
    (left: `"hello"`, right: `"world"`): greeting operation failed',
    src/main.rs:4

The two arguments to assert_eq! are named "left" and "right" rather than "expected" and "actual"; the order of the value that comes from your code and the value hardcoded into your test isn't important.

Since these macros use the operators == and != and print the values using debug formatting, the values being compared must implement the PartialEq and Debug traits. Types provided by Rust implement these traits, but for structs and enums that you define, you'll need to add PartialEq in order to be able to assert that values of those types are equal or not equal and Debug in order to be able to print out the values in the case that the assertion fails. Because both of these traits are derivable traits that we mentioned in Chapter 5, usually this is as straightforward as adding the #[derive(PartialEq, Debug)] annotation to your struct or enum definition. See Appendix C for more details about these and other derivable traits.

Test for failure with should_panic

We can invert our test's failure with another attribute: should_panic. This is useful when we want to test that calling a particular function will cause an error. For example, let's test something that we know will panic from Chapter 8: attempting to create a slice using range syntax with byte indices that aren't on character boundaries. Add the #[should_panic] attribute before the function like the #[test] attribute, as shown in Listing 11-1:

Filename: src/lib.rs

#[test]
#[should_panic]
fn slice_not_on_char_boundaries() {
    let s = "Здравствуйте";
    &s[0..1];
}

Listing 11-1: A test expecting a panic!

This test will succeed, since the code panics and we said that it should. If this code happened to run and did not cause a panic!, this test would fail.

should_panic tests can be fragile, as it's hard to guarantee that the test didn't fail for a different reason than the one you were expecting. To help with this, an optional expected parameter can be added to the should_panic attribute. The test harness will make sure that the failure message contains the provided text. A more robust version of Listing 11-1 would be the following, in Listing 11-2:

Filename: src/lib.rs

#[test]
#[should_panic(expected = "do not lie on character boundary")]
fn slice_not_on_char_boundaries() {
    let s = "Здравствуйте";
    &s[0..1];
}

Listing 11-2: A test expecting a panic! with a particular message

Try on your own to see what happens when a should_panic test panics but doesn't match the expected message: cause a panic! that happens for a different reason in this test, or change the expected panic message to something that doesn't match the character boundary panic message.

Running tests

Just like cargo run compiles your code and then runs the resulting binary, cargo test compiles your code in test mode and runs the resulting test binary. The default behavior of the binary that cargo test produces is to run all the tests in parallel and to capture output generated during test runs so that it's easier to read the output about the test results.

The default behavior of running tests can be changed by specifying command line options. Some of these options can be passed to cargo test, and some need to be passed instead to the resulting test binary. The way to separate these arguments is with --: after cargo test, list the arguments that go to cargo test, then the separator --, and then the arguments that go to the test binary.

Tests Run in Parallel

Tests are run in parallel using threads. For this reason, you should take care that your tests are written in such a way as to not depend on each other or on any shared state. Shared state can also include the environment, such as the current working directory or environment variables.

If you don't want this behavior, or if you want more fine-grained control over the number of threads used, you can send the --test-threads flag and the number of threads to the test binary. Setting the number of test threads to 1 means to not use any parallelism:

$ cargo test -- --test-threads=1

Tests Capture Output

By default, Rust's test library captures and discards output to standard out and standard error, unless the test fails. For example, if you call println! in a test and the test passes, you won't see the println! output in your terminal. This behavior can be disabled by sending the --nocapture flag to the test binary:

$ cargo test -- --nocapture

Running a Subset of Tests by Name

Sometimes, running a full test suite can take a long time. If you're only working on code in a particular area, you might want to only run the tests having to do with that code. cargo test takes an argument that allows you to only run certain tests, specified by name.

Let's create three tests with the following names as shown in Listing 11-3:

Filename: src/lib.rs

#[test]
fn add_two_and_two() {
    assert_eq!(4, 2 + 2);
}

#[test]
fn add_three_and_two() {
    assert_eq!(5, 3 + 2);
}

#[test]
fn one_hundred() {
    assert_eq!(102, 100 + 2);
}

Listing 11-3: Three tests with a variety of names

Running with different arguments will run different subsets of the tests. No arguments, as we've already seen, runs all the tests:

$ cargo test
    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
     Running target/debug/deps/adder-06a75b4a1f2515e9

running 3 tests
test add_three_and_two ... ok
test one_hundred ... ok
test add_two_and_two ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured

We can pass the name of any test function to run only that test:

$ cargo test one_hundred
    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
     Running target/debug/deps/adder-06a75b4a1f2515e9

running 1 test
test one_hundred ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured

We can also pass part of a name, and cargo test will run all tests that match:

$ cargo test add
    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
     Running target/debug/deps/adder-06a75b4a1f2515e9

running 2 tests
test add_three_and_two ... ok
test add_two_and_two ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured

Module names become part of the test name, so module names can be used in a similar way to run just the tests for a particular module. For example, if our code was organized into a module named adding and a module named subtracting with tests in each, as in Listing 11-4:

Filename: src/lib.rs

mod adding {
    #[test]
    fn add_two_and_two() {
        assert_eq!(4, 2 + 2);
    }

    #[test]
    fn add_three_and_two() {
        assert_eq!(5, 3 + 2);
    }

    #[test]
    fn one_hundred() {
        assert_eq!(102, 100 + 2);
    }
}

mod subtracting {
    #[test]
    fn subtract_three_and_two() {
        assert_eq!(1, 3 - 2);
    }
}

Listing 11-4: Tests in two modules named adding and subtracting

Running cargo test will run all of the tests, and the module names will appear in the test names in the output:

$ cargo test
    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
     Running target/debug/deps/adder-d84f1c6cb24adeb4

running 4 tests
test adding::add_two_and_two ... ok
test adding::add_three_and_two ... ok
test subtracting::subtract_three_and_two ... ok
test adding::one_hundred ... ok

Running cargo test adding would run just the tests in that module and not any of the tests in the subtracting module:

$ cargo test adding
    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
     Running target/debug/deps/adder-d84f1c6cb24adeb4

running 3 tests
test adding::add_three_and_two ... ok
test adding::one_hundred ... ok
test adding::add_two_and_two ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured

Ignore Some Tests Unless Specifically Requested

Sometimes a few specific tests can be very time-consuming to execute, so during most runs of cargo test, we'd like to exclude them. Instead of having to construct an argument to cargo test to run all tests except these and remember to use that argument every time, we can annotate these tests with the ignore attribute:

Filename: src/lib.rs

#[test]
fn it_works() {
    assert!(true);
}

#[test]
#[ignore]
fn expensive_test() {
    // code that takes an hour to run
}

Now if we run our tests, we'll see it_works is run, but expensive_test is not:

$ cargo test
   Compiling adder v0.1.0 (file:///projects/adder)
    Finished debug [unoptimized + debuginfo] target(s) in 0.24 secs
     Running target/debug/deps/adder-ce99bcc2479f4607

running 2 tests
test expensive_test ... ignored
test it_works ... ok

test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured

   Doc-tests adder

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured

We can run only the expensive tests by explicitly asking to run them using cargo test -- --ignored:

$ cargo test -- --ignored
    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
     Running target/debug/deps/adder-ce99bcc2479f4607

running 1 test
test expensive_test ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured

This way, most of the time that you run cargo test the results would be fast. When you're at a point that it makes sense to check the results of the ignored tests and you have time to wait for the results, you can choose to run cargo test -- --ignored instead.

Test Organization

As mentioned before, testing is a large discipline, and different people sometimes use different terminology and organization. The Rust community tends to think about tests in terms of two main categories: unit tests and integration tests. Unit tests tend to be smaller and more focused, testing one module in isolation at a time. They can also test private interfaces. Integration tests are entirely external to your library. They use your code in the same way any other code would, using only the public interface and exercising multiple modules per test. Both kinds of tests are important to ensure that the pieces of your library are doing what you expect them to separately and together.

Unit Tests

The purpose of unit tests is to test each unit of code in isolation from the rest of the code, in order to be able to quickly pinpoint where code is working as expected or not. Unit tests live in the src directory, in the same files as the code they are testing. They are separated into their own tests module in each file.

The Tests Module and cfg(test)

By placing tests in their own module and using the cfg annotation on the module, we can tell Rust to only compile and run the test code when we run cargo test. This saves compile time when we only want to build the library code with cargo build, and saves space in the resulting compiled artifact since the tests are not included.

Remember when we generated the new adder project in the last section? Cargo generated this code for us:

Filename: src/lib.rs

#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
    }
}

We ignored the module stuff so we could concentrate on the mechanics of the test code inside the module, but now let's focus on the code surrounding our tests.

First of all, there's a new attribute, cfg. The cfg attribute lets us declare that something should only be included given a certain configuration. Rust provides the test configuration for compiling and running tests. By using this attribute, Cargo only compiles our test code if we're currently trying to run the tests.

Next, the tests module holds all of our test functions, while our code is outside of the tests module. The name of the tests module is a convention; otherwise this is a regular module that follows the usual visibility rules we covered in Chapter 7. Because we're in an inner module, we need to bring the code under test into scope. This can be annoying if you have a large module, so this is a common use of globs.

Up until now in this chapter, we've been writing tests in our adder project that don't actually call any code we've written. Let's change that now! In src/lib.rs, place this add_two function and tests module that has a test function to exercise the code, as shown in Listing 11-5:

Filename: src/lib.rs

pub fn add_two(a: i32) -> i32 {
    a + 2
}

#[cfg(test)]
mod tests {
    use add_two;

    #[test]
    fn it_works() {
        assert_eq!(4, add_two(2));
    }
}

Listing 11-5: Testing the function add_two in a child tests module

Notice in addition to the test function, we also added use add_two; within the tests module. This brings the code we want to test into the scope of the inner tests module, just like we'd need to do for any inner module. If we run this test now with cargo test, it will pass:

running 1 test
test tests::it_works ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured

If we had forgotten to bring the add_two function into scope, we would get an unresolved name error since the tests module wouldn't know anything about the add_two function:

error[E0425]: unresolved name `add_two`
 --> src/lib.rs:9:23
  |
9 |         assert_eq!(4, add_two(2));
  |                       ^^^^^^^ unresolved name

If this module contained lots of code we wanted to test, it would be annoying to list everything in the use statement in the tests. It's common instead to put use super::*; within a module's test submodule in order to bring everything into the test module scope at once.

Testing Private Functions

There's controversy within the testing community about whether you should write unit tests for private functions or not. Regardless of which testing ideology you adhere to, Rust does allow you to test private functions due to the way that the privacy rules work. Consider the code in Listing 11-6 with the private function internal_adder:

Filename: src/lib.rs

pub fn add_two(a: i32) -> i32 {
    internal_adder(a, 2)
}

fn internal_adder(a: i32, b: i32) -> i32 {
    a + b
}

#[cfg(test)]
mod tests {
    use internal_adder;

    #[test]
    fn internal() {
        assert_eq!(4, internal_adder(2, 2));
    }
}

Listing 11-6: Testing a private function

Because tests are just Rust code and the tests module is just another module, we can import and call internal_adder in a test just fine. If you don't think private functions should be tested, there's nothing in Rust that will compel you to do so.

Integration Tests

In Rust, integration tests are tests that are entirely external to your library. They use your library in the same way any other code would. Their purpose is to test that many parts of your library work correctly together. Units of code that work correctly by themselves could have problems when integrated, so test coverage of the integrated code is important as well.

The tests Directory

Cargo has support for integration tests in the tests directory. If you make one and put Rust files inside, Cargo will compile each of the files as an individual crate. Let's give it a try!

First, make a tests directory at the top level of your project directory, next to src. Then, make a new file, tests/integration_test.rs, and put the code in Listing 11-7 inside:

Filename: tests/integration_test.rs

extern crate adder;

#[test]
fn it_adds_two() {
    assert_eq!(4, adder::add_two(2));
}

Listing 11-7: An integration test of a function in the adder crate

We now have extern crate adder at the top, which we didn't need in the unit tests. Each test in the tests directory is an entirely separate crate, so we need to import our library into each of them. This is also why tests is a suitable place to write integration-style tests: they use the library like any other consumer of it would, by importing the crate and using only the public API.

We also don't need a tests module in this file. The whole directory won't be compiled unless we're running the tests, so we don't need to annotate any part of it with #[cfg(test)]. Also, each test file is already isolated into its own crate, so we don't need to separate the test code further.

Let's run the integration tests, which also get run when we run cargo test:

$ cargo test
   Compiling adder v0.1.0 (file:///projects/adder)
     Running target/debug/deps/adder-91b3e234d4ed382a

running 1 test
test tests::it_works ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured

     Running target/debug/integration_test-952a27e0126bb565

running 1 test
test it_adds_two ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured

   Doc-tests adder

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured

Now we have three sections of output: the unit tests, the integration test, and the doc tests. Note that adding more unit tests in any src file will add more lines to the unit tests section. Adding more test functions to the integration test file we created will add more lines to that section. If we add more integration test files in the tests directory, there will be more integration test sections: one for each file.

Specifying a test function name argument with cargo test will also match against test function names in any integration test file. To run all of the tests in only one particular integration test file, use the --test argument of cargo test:

$ cargo test --test integration_test
    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
     Running target/debug/integration_test-952a27e0126bb565

running 1 test
test it_adds_two ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured

Submodules in Integration Tests

As you add more integration tests, you may want to make more than one file in the tests directory in order to group the test functions by the functionality they're testing, for example. As we mentioned before, that will work fine, given that Cargo treats every file as its own crate.

Eventually, you may have a set of helper functions that are common to all integration tests, for example, functions that set up common scenarios. If you extract these into a file in the tests directory, like tests/common.rs for example, this file will be compiled into a separate crate just like the Rust files in this directory that contain test functions are. There will be a separate section in the test output for this file. Since this is probably not what you want, it's recommended to instead use a mod.rs file in a subdirectory, like tests/common/mod.rs, for helper functions. Files in subdirectories of the tests directory do not get compiled as separate crates or have sections in the test output.

Integration Tests for Binary Crates

If your project is a binary crate that only contains a src/main.rs and does not have a src/lib.rs, it is not possible to create integration tests in the tests directory and use extern crate to import the functions in src/main.rs. This is one of the reasons Rust projects that provide a binary have a straightforward src/main.rs that calls logic that lives in src/lib.rs. With that structure, integration tests can test the library crate by using extern crate to cover the important functionality, and if that works, the small amount of code in src/main.rs will work as well and does not need to be tested.

Summary

Rust's testing features provide a way to specify how code should function to ensure the code continues to work in the specified ways even as we make changes. Unit tests exercise different parts of a library separately and can test private implementation details. Integration tests cover the use of many parts of the library working together, and use the library's public API to test the code in the same way other code will use it. Rust's type system and ownership rules help prevent some kinds of bugs, but tests are an important part of reducing logic bugs having to do with how your code is expected to behave.

Let's put together the knowledge from this chapter and other previous chapters and work on a project in the next chapter!