initial import/re-write of testing chapter

src/chXX-00-testing.md

@@ -1 +1,280 @@
 # Testing
+
+> Program testing can be a very effective way to show the presence of bugs, but
+> it is hopelessly inadequate for showing their absence.
+>
+> Edsger W. Dijkstra, "The Humble Programmer" (1972)
+
+Rust is a programming language that cares a lot about correctness. But
+correctness is a complex topic, and isn't exactly easy to get right. Rust
+places a lot of weight on its type system to help ensure that our programs do
+what we intend, but it cannot help with everything. As such, Rust also includes
+support for writing software tests in the language itself.
+
+Testing is a skill, and we cannot hope to learn everything about how to write
+good tests in one chapter of a book. What we can learn, however, are the
+mechanics of Rust's testing facilities. That's what we'll focus on in this
+chapter.
+
+## The `test` attribute
+
+At its simplest, a test in Rust is a function that's annotated with the `test`
+attribute. Let's make a new project with Cargo called `adder`:
+
+```bash
+$ cargo new adder
+     Created library `adder` project
+$ cd adder
+```
+
+Cargo will automatically generate a simple test when you make a new project.
+Here's the contents of `src/lib.rs`:
+
+```rust
+#[cfg(test)]
+mod tests {
+    #[test]
+    fn it_works() {
+    }
+}
+```
+
+For now, let's remove the `mod` bit, and focus on just the function:
+
+```rust
+#[test]
+fn it_works() {
+}
+```
+
+Note the `#[test]`. This attribute indicates that this is a test function. It
+currently has no body. That's good enough to pass! We can run the tests with
+`cargo test`:
+
+```bash
+$ cargo test
+   Compiling adder v0.1.0 (file:///projects/adder)
+    Finished debug [unoptimized + debuginfo] target(s) in 0.22 secs
+     Running target/debug/deps/adder-ce99bcc2479f4607
+
+running 1 test
+test it_works ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
+
+   Doc-tests adder
+
+running 0 tests
+
+test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
+```
+
+Cargo compiled and ran our tests. There are two sets of output here: one
+for the test we wrote, and another for documentation tests. We'll talk about
+documentation tests later. For now, see this line:
+
+```text
+test it_works ... ok
+```
+
+Note the `it_works`. This comes from the name of our function:
+
+```rust
+fn it_works() {
+# }
+```
+
+We also get a summary line:
+
+```text
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
+```
+
+## The `assert!` macro
+
+So why does our do-nothing test pass? Any test which doesn't `panic!` passes,
+and any test that does `panic!` fails. Let's make our test fail:
+
+```rust
+#[test]
+fn it_works() {
+    assert!(false);
+}
+```
+
+`assert!` is a macro provided by Rust which takes one argument: if the argument
+is `true`, nothing happens. If the argument is `false`, it will `panic!`. Let's
+run our tests again:
+
+```bash
+$ cargo test
+   Compiling adder v0.1.0 (file:///projects/adder)
+    Finished debug [unoptimized + debuginfo] target(s) in 0.22 secs
+     Running target/debug/deps/adder-ce99bcc2479f4607
+
+running 1 test
+test it_works ... FAILED
+
+failures:
+
+---- it_works stdout ----
+	thread 'it_works' panicked at 'assertion failed: false', src/lib.rs:5
+note: Run with `RUST_BACKTRACE=1` for a backtrace.
+
+
+failures:
+    it_works
+
+test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
+
+error: test failed
+```
+
+Rust indicates that our test failed:
+
+```text
+test it_works ... FAILED
+```
+
+And that's reflected in the summary line:
+
+```text
+test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
+```
+
+## Inverting failure with `should_panic`
+
+We can invert our test's failure with another attribute: `should_panic`:
+
+```rust
+#[test]
+#[should_panic]
+fn it_works() {
+    assert!(false);
+}
+```
+
+This test will now succeed if we `panic!` and fail if we complete.
+
+`should_panic` tests can be fragile, as it's hard to guarantee that the test
+didn't fail for an unexpected reason. To help with this, an optional `expected`
+parameter can be added to the `should_panic` attribute. The test harness will
+make sure that the failure message contains the provided text. A safer version
+of the example above would be:
+
+```rust
+#[test]
+#[should_panic(expected = "assertion failed")]
+fn it_works() {
+    assert!(false);
+}
+```
+
+## Testing equality
+
+Rust provides a pair of macros, `assert_eq!` and `assert_ne!`, that compares
+two arguments for equality:
+
+```rust
+#[test]
+fn it_works() {
+    assert_eq!("Hello", "Hello");
+
+    assert_ne!("Hello", "world");
+}
+```
+
+These macros expand to something like this:
+
+```rust,ignore
+// assert_eq
+if left_val == right_val {
+    panic!("message goes here")
+}
+
+// assert_ne
+if left_val =! right_val {
+    panic!("message goes here")
+}
+```
+
+But they're a bit more convenient than writing this out by hand.  These macros
+are often used to call some function with some known arguments and compare it
+to the expected output, like this:
+
+```rust
+pub fn add_two(a: i32) -> i32 {
+    a + 2
+}
+
+#[test]
+fn it_works() {
+    assert_eq!(4, add_two(2));
+}
+```
+
+## The `ignore` attribute
+
+Sometimes a few specific tests can be very time-consuming to execute. These
+can be disabled by default by using the `ignore` attribute:
+
+```rust
+pub fn add_two(a: i32) -> i32 {
+    a + 2
+}
+
+#[test]
+fn it_works() {
+    assert_eq!(4, add_two(2));
+}
+
+#[test]
+#[ignore]
+fn expensive_test() {
+    // code that takes an hour to run
+}
+```
+
+Now we run our tests and see that `it_works` is run, but `expensive_test` is
+not:
+
+```bash
+$ cargo test
+   Compiling adder v0.1.0 (file:///projects/adder)
+    Finished debug [unoptimized + debuginfo] target(s) in 0.24 secs
+     Running target/debug/deps/adder-ce99bcc2479f4607
+
+running 2 tests
+test expensive_test ... ignored
+test it_works ... ok
+
+test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured
+
+   Doc-tests adder
+
+running 0 tests
+
+test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
+```
+
+The expensive tests can be run explicitly using `cargo test -- --ignored`:
+
+```bash
+$ cargo test -- --ignored
+    Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
+     Running target/debug/deps/adder-ce99bcc2479f4607
+
+running 1 test
+test expensive_test ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
+
+   Doc-tests adder
+
+running 0 tests
+
+test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
+```
+
+The `--ignored` argument is an argument to the test binary, and not to Cargo,
+which is why the command is `cargo test -- --ignored`.

src/chXX-01-testing.md

@@ -1 +1,131 @@
 # Unit testing
+
+As we mentioned before, testing is a large discipline, and so different people
+can sometimes use different terminology. For our purposes, we tend to place
+tests into two main categories: *unit tests* and *integration tests*. Unit
+tests tend to be smaller, and more focused. In Rust, they can also test
+non-public interfaces. Let's talk more about how to do unit testing in Rust.
+
+## The tests module and `cfg(test)`
+
+Remember when we generated our new project in the last section? Cargo had
+generated some stuff for us:
+
+```rust
+#[cfg(test)]
+mod tests {
+    #[test]
+    fn it_works() {
+    }
+}
+```
+
+We deleted the module stuff so we could learn more about the mechanics of
+tests. But there's a reason that Cargo generated this module for us: it's the
+idiomatic way to organize unit tests in Rust. That is, unit tests are:
+
+* Stored inside of the same tree as your source code.
+* Placed inside their own module.
+
+For a more realistic example of how this works, consider our `add_two` function
+from before:
+
+```rust
+pub fn add_two(a: i32) -> i32 {
+    a + 2
+}
+
+#[cfg(test)]
+mod tests {
+    use add_two;
+
+    #[test]
+    fn it_works() {
+        assert_eq!(4, add_two(2));
+    }
+}
+```
+
+First of all, there's a new attribute, `cfg`. The `cfg` attribute lets us
+declare that something should only be included given a certain configuration.
+Rust provides the `test` configuration when compiling and running tests. By
+using this attribute, Cargo only compiles our test code if we're currently
+trying to run the tests. Given that they're not compiled at all during a
+regular `cargo build`, this can save compile time. It also ensures that our
+tests are entirely left out of the binary, saving space in a non-testing
+context.
+
+You'll notice one more change: the `use` declaration. The `tests` module is
+only a convention, it's nothing that Rust understands directly. As such, we
+have to follow the usual visibility rules. Because we're in an inner module,
+we need to bring our test function into scope. This can be annoying if you have
+a large module, and so this is a common use of globs. Let's change our
+`src/lib.rs` to make use of it:
+
+```rust,ignore
+pub fn add_two(a: i32) -> i32 {
+    a + 2
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn it_works() {
+        assert_eq!(4, add_two(2));
+    }
+}
+```
+
+Note the different `use` line. Now we run our tests:
+
+```bash
+$ cargo test
+   Compiling adder v0.1.0 (file:///projects/adder)
+    Finished debug [unoptimized + debuginfo] target(s) in 0.27 secs
+     Running target/debug/deps/adder-ce99bcc2479f4607
+
+running 1 test
+test tests::it_works ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
+
+   Doc-tests adder
+
+running 0 tests
+
+test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
+```
+
+It works!
+
+## Testing internal functions
+
+There's controversy within the testing community about unit testing private
+functions. Regardless of which testing ideology you adhere to, Rust does allow
+you to test them, due to the way that the privacy rules work. Consider this:
+
+```rust
+pub fn add_two(a: i32) -> i32 {
+    internal_adder(a, 2)
+}
+
+fn internal_adder(a: i32, b: i32) -> i32 {
+    a + b
+}
+
+#[cfg(test)]
+mod tests {
+    use internal_adder;
+
+    #[test]
+    fn internal() {
+        assert_eq!(4, internal_adder(2, 2));
+    }
+}
+```
+
+In this scenario, we have a non-`pub` function, `internal_adder`. Because tests
+are just Rust code, and the `tests` module is just another module, we can
+import and call `internal_adder` in a test just fine.

src/chXX-02-integration-testing.md

@@ -1 +1,163 @@
 # Integration testing
+
+In the last section, we talked about unit tests. But there's still that other
+category: integration testing. In Rust, an integration test is a test that is
+entirely external to your library. It uses it in the same way any other code
+would.
+
+Cargo has support for integration tests through the `tests` directory. If you
+make one, and put `.rs` files inside, Cargo will compile each of them as an
+individual crate. Let's give it a try! First, make a `tests` directory at the
+top level of your project, next to `src`. Then, make a new file,
+`tests/integration_test.rs`, and put this inside:
+
+```rust,ignore
+extern crate adder;
+
+#[test]
+fn it_works() {
+    assert_eq!(4, adder::add_two(2));
+}
+```
+
+There's some small changes from our previous tests. We now have an `extern
+crate adder` at the top. This is because each test in the `tests` directory is
+an entirely separate crate, and so we need to import our library.  This is also
+why `tests` is a suitable place to write integration-style tests: they use the
+library like any other consumer of it would.
+
+Let's run them:
+
+```bash
+$ cargo test
+   Compiling adder v0.1.0 (file:///home/steve/tmp/adder)
+     Running target/adder-91b3e234d4ed382a
+
+running 1 test
+test tests::it_works ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
+
+     Running target/lib-c18e7d3494509e74
+
+running 1 test
+test it_works ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
+
+   Doc-tests adder
+
+running 0 tests
+
+test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
+```
+
+Now we have three sections: our previous test is also run, as well as our new
+one.
+
+That's all there is to the `tests` directory. The `tests` module isn't needed
+here, since the whole thing is focused on tests.
+
+## Submodules in integration tests
+
+As your integration tests grow, you may want to make more than one file in the
+`tests` directory. As we mentioned before, that works well, given that Cargo
+treats every file as its own crate. But there's one small trap that can happen.
+
+Imagine we wanted some common helper functions to be shared across our tests.
+So we change our test to have a `common` module:
+
+```rust,ignore
+extern crate adder;
+
+mod common;
+
+#[test]
+fn it_works() {
+    common::helper();
+
+    assert_eq!(4, adder::add_two(2));
+}
+```
+
+And then, we create a `tests/common.rs` file to hold our common helpers:
+
+```rust
+pub fn helper() {
+    // no implementation for now
+}
+```
+
+Let's try running this:
+
+```bash
+$ cargo test
+   Compiling adder v0.1.0 (file:///home/steve/tmp/adder)
+    Finished debug [unoptimized + debuginfo] target(s) in 0.25 secs
+     Running target/debug/deps/adder-ce99bcc2479f4607
+
+running 1 test
+test tests::internal ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
+
+     Running target/debug/common-c3635c69f3aeef92
+
+running 0 tests
+
+test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
+
+     Running target/debug/integration_tests-6d6e12b4680b0368
+
+running 1 test
+test it_works ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
+
+   Doc-tests adder
+
+running 0 tests
+
+test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
+```
+
+Wait a minute. Now we have four sections?
+
+```text
+     Running target/debug/common-c3635c69f3aeef92
+```
+
+Because `common.rs` is in our `tests` directory, Cargo is also compiling it as
+its own crate. Because `common.rs` is so simple, we didn't get an error, but
+with more complex code, this might not work. So what can we do?
+
+The key is, always use the `common/mod.rs` form over the `common.rs` form when
+making modules in integration tests. If we move `tests/common.rs` to
+`tests/common/mod.rs`, we'll go back to our expected output:
+
+```bash
+$ mkdir tests/common
+$ mv tests/common.rs tests/common/mod.rs
+$ cargo test
+   Compiling adder v0.1.0 (file:///home/steve/tmp/adder)
+    Finished debug [unoptimized + debuginfo] target(s) in 0.24 secs
+     Running target/debug/deps/adder-ce99bcc2479f4607
+
+running 1 test
+test tests::internal ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
+
+     Running target/debug/integration_tests-6d6e12b4680b0368
+
+running 1 test
+test it_works ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
+
+   Doc-tests adder
+
+running 0 tests
+
+test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
+```

src/chXX-03-documentation-tests.md

@@ -1 +1,302 @@
 # Documentation Tests
+
+Nothing is better than documentation with examples. Nothing is worse than
+examples that don't actually work, because the code has changed since the
+documentation has been written. To this end, Rust supports automatically
+running examples in your documentation for library crates. Here's a fleshed-out
+`src/lib.rs` with examples:
+
+```rust
+//! The `adder` crate provides functions that add numbers to other numbers.
+//!
+//! # Examples
+//!
+//! ```
+//! assert_eq!(4, adder::add_two(2));
+//! ```
+
+/// This function adds two to its argument.
+///
+/// # Examples
+///
+/// ```
+/// use adder::add_two;
+///
+/// assert_eq!(4, add_two(2));
+/// ```
+pub fn add_two(a: i32) -> i32 {
+    a + 2
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn it_works() {
+        assert_eq!(4, add_two(2));
+    }
+}
+```
+
+Note the module-level documentation with `//!` and the function-level
+documentation with `///`. Rust's documentation supports Markdown in comments,
+and so triple graves mark code blocks. It is conventional to include the
+`# Examples` section, exactly like that, with examples following.
+
+Let's run the tests again:
+
+```bash
+$ cargo test
+   Compiling adder v0.1.0 (file:///projects/adder)
+     Running target/adder-91b3e234d4ed382a
+
+running 1 test
+test tests::it_works ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
+
+     Running target/lib-c18e7d3494509e74
+
+running 1 test
+test it_works ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
+
+   Doc-tests adder
+
+running 2 tests
+test add_two_0 ... ok
+test _0 ... ok
+
+test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured
+```
+
+Now we have all three kinds of tests running! Note the names of the
+documentation tests: the `_0` is generated for the module test, and `add_two_0`
+for the function test. These will auto increment with names like `add_two_1` as
+you add more examples.
+
+## Automatic `main` insertion
+
+Let's discuss our sample example documentation:
+
+```rust
+/// ```
+/// println!("Hello, world");
+/// ```
+# fn foo() {}
+```
+
+You'll notice that you don't need a `fn main()` or anything here. `rustdoc`
+will automatically add a `main()` wrapper around your code, using heuristics to
+attempt to put it in the right place. For example:
+
+```rust
+/// ```
+/// use std::rc::Rc;
+///
+/// let five = Rc::new(5);
+/// ```
+# fn foo() {}
+```
+
+This will end up testing:
+
+```rust
+fn main() {
+    use std::rc::Rc;
+    let five = Rc::new(5);
+}
+```
+
+Here's the full algorithm rustdoc uses to preprocess examples:
+
+1. Any leading `#![foo]` attributes are left intact as crate attributes.
+2. Some common `allow` attributes are inserted, including
+   `unused_variables`, `unused_assignments`, `unused_mut`,
+   `unused_attributes`, and `dead_code`. Small examples often trigger
+   these lints.
+3. If the example does not contain `extern crate`, then `extern crate
+   <mycrate>;` is inserted (note the lack of `#[macro_use]`).
+4. Finally, if the example does not contain `fn main`, the remainder of the
+   text is wrapped in `fn main() { your_code }`.
+
+This generated `fn main` can be a problem! If you have `extern crate` or a
+`mod` statements in the example code that are referred to by `use` statements,
+they will fail to resolve unless you include at least `fn main() {}` to inhibit
+step 4. `#[macro_use] extern crate` also does not work except at the crate
+root, so when testing macros an explicit `main` is always required. It doesn't
+have to clutter up your docs, though — keep reading!
+
+## Hiding extraneous code with `#`
+
+Sometimes this algorithm isn't enough, though. For example, all of these code
+samples with `///` we've been talking about? The raw text:
+
+```text
+/// Some documentation.
+# fn foo() {}
+```
+
+looks different than the output:
+
+```rust
+/// Some documentation.
+# fn foo() {}
+```
+
+Yes, that's right: we can add lines that start with `# `, and they will be
+hidden from the output, but will be used when compiling our code. We can use
+this to our advantage. In this case, documentation comments need to apply to
+some kind of function, so if We want to show off a documentation comment, I
+need to add a little function definition below it. At the same time, it's only
+there to satisfy the compiler, so hiding it makes the example more clear. We
+can use this technique to explain longer examples in detail, while still
+preserving the testability of our documentation.
+
+For example, imagine that we wanted to document this code:
+
+```rust
+let x = 5;
+let y = 6;
+println!("{}", x + y);
+```
+
+We might want the documentation to end up looking like this:
+
+> First, we set `x` to five:
+>
+> ```rust
+> let x = 5;
+> # let y = 6;
+> # println!("{}", x + y);
+> ```
+>
+> Next, we set `y` to six:
+>
+> ```rust
+> # let x = 5;
+> let y = 6;
+> # println!("{}", x + y);
+> ```
+>
+> Finally, we print the sum of `x` and `y`:
+>
+> ```rust
+> # let x = 5;
+> # let y = 6;
+> println!("{}", x + y);
+> ```
+
+To keep each code block testable, we want the whole program in each block, but
+we don't want the reader to see every line every time.  Here's what we put in
+our source code:
+
+```text
+    First, we set `x` to five:
+
+    ```rust
+    let x = 5;
+    # let y = 6;
+    # println!("{}", x + y);
+    ```
+
+    Next, we set `y` to six:
+
+    ```rust
+    # let x = 5;
+    let y = 6;
+    # println!("{}", x + y);
+    ```
+
+    Finally, we print the sum of `x` and `y`:
+
+    ```rust
+    # let x = 5;
+    # let y = 6;
+    println!("{}", x + y);
+    ```
+```
+
+By repeating all parts of the example, we can ensure that our example still
+compiles, while only showing the parts that are relevant to that part of our
+explanation.
+
+Another case where the use of `#` is handy is when you want to ignore
+error handling. Lets say you want the following,
+
+```rust,ignore
+/// use std::io;
+/// let mut input = String::new();
+/// try!(io::stdin().read_line(&mut input));
+```
+
+The problem is that `try!` returns a `Result<T, E>` and test functions
+don't return anything so this will give a mismatched types error.
+
+```rust,ignore
+/// A doc test using try!
+///
+/// ```
+/// use std::io;
+/// # fn foo() -> io::Result<()> {
+/// let mut input = String::new();
+/// try!(io::stdin().read_line(&mut input));
+/// # Ok(())
+/// # }
+/// ```
+# fn foo() {}
+```
+
+You can get around this by wrapping the code in a function. This catches
+and swallows the `Result<T, E>` when running tests on the docs. This
+pattern appears regularly in the standard library.
+
+## Adding attributes to control documentation testing.
+
+In the first part of the chapter, we talked about attributes that help with
+testing:
+
+```rust
+#[test]
+#[ignore]
+fn it_works() {
+}
+
+#[should_panic]
+fn it_works() {
+    assert!(false);
+}
+```
+
+We can use these annotations in documentation tests as well:
+
+```rust
+/// ```rust,ignore
+/// fn foo() {
+/// ```
+fn foo() {}
+
+/// ```rust,should_panic
+/// assert!(false);
+/// ```
+fn bar() {}
+```
+
+## The `no_run` attribute
+
+There's one attribute that's specific to documentation tests:
+
+```rust
+/// ```rust,no_run
+/// loop {
+///     println!("Hello, world");
+/// }
+/// ```
+# fn foo() {}
+```
+
+The `no_run` attribute will compile your code, but not run it. This is
+important for examples such as "Here's how to start up a network service,"
+which you would want to make sure compile, but might run in an infinite loop!