Let's Build A Cargo Compatible Build Tool - Part 5
This is the fifth post in an educational multipart series about how build tools like Cargo work under the hood, by building our own from the ground up. If you haven’t read the previous posts it’s recommended that you do so.
Previous posts:
Last time we did a few cleanup tasks that needed to be done. This time we're back to adding features. Today we're going to add rustdoc support for tests. In the feature we'll actually add support for docs, but today we want doc tests. These are nice because we can test our docs are always up to date when it comes to examples. Let's start by adding a doc test we can use to verify things are working properly. Open up src/rustc.rs and add the following doc strings to the builder function:
impl Rustc {
/// Create a builder type to build up commands to then invoke rustc with.
/// ```
/// # use std::error::Error;
/// # use freight::rustc::Rustc;
/// # use freight::rustc::Edition;
/// # use freight::rustc::CrateType;
/// # fn main() -> Result<(), Box<dyn Error>> {
/// let builder = Rustc::builder()
/// .edition(Edition::E2021)
/// .crate_type(CrateType::Bin)
/// .crate_name("freight")
/// .out_dir(".")
/// .lib_dir(".");
/// # Ok(())
/// # }
/// ```
pub fn builder() -> RustcBuilder {
// Code removed for brevity
}
}If we were to run rustdoc on this then we would only see this in the actual documentation output:
You'll notice the import statements and the main function body aren't in the output. You can hide lines using # inside of code blocks in the docs. This means you can write out an example that you want users to see for how it's used, while also writing out a test to show it works. It's important to note we are using freight here as a library not as if it was part the library itself. This is similar to an integration test and it's how rustdoc works. It will treat each doc test similar to a binary that it can run and so it will need to have libraries like freight linked into it.
This makes sense as rustdoc will only show publicly documented items by default. As such, if you can read the docs for them, then you can import them from the library in order to write a test for it.
With this we can catch some changes in our API so that docs can be updated. If for instance we renamed crate_name as a function to krate_name then our doc test will fail now until we also update it. Just because we added the test and docs though nothing will happen currently with Freight as the functionality we need for it has not been made yet. We need to get rustdoc to read the source code and make the tests for us.
We should do this by adding a new module rustdoc to our codebase as we'll want to expand on it's feature set in the future and we want it to be organized like how we just did with rustc in part 4.
Let’s open up a new file src/rustdoc.rs and add the following code to it.
use super::Result;
use crate::rustc::Edition;
use std::path::Path;
use std::path::PathBuf;
use std::process::Command;
pub struct RustDoc {
edition: Edition,
crate_name: String,
lib_path: PathBuf,
}
impl RustDoc {
pub fn new(
edition: Edition,
crate_name: impl Into<String>,
lib_path: impl Into<PathBuf>,
) -> Self {
Self {
edition,
crate_name: crate_name.into(),
lib_path: lib_path.into(),
}
}
pub fn test(&self, path: impl AsRef<Path>) -> Result<()> {
let path = path.as_ref();
Command::new("rustdoc")
.arg("--test")
.arg(path)
.arg("--crate-name")
.arg(&self.crate_name)
.arg("--edition")
.arg(self.edition.to_string())
.arg("-L")
.arg(&self.lib_path)
.spawn()?
.wait()?;
Ok(())
}
}This is quite similar to our rustc module. The big difference is we’re not using the builder pattern here. For now we only need a few things in order to make a test work and so we’ll just use a new function to hold that information. The actual function to compile the test is also pretty straightforward with many of the same args that we’re used to from rustc. We invoke rustdoc with a test flag, the file to use as an entry point, the name of the crate, the edition, and where to find the libraries it might need to link too. We then spawn it and wait on the output. The difference with rustdoc is that it does not actually produce a binary, but just runs the tests it creates. This is why we have no output directory. We just ask rustdoc to run tests and call it a day.
Now we need to actually update src/lib.rs to actually know our module exists and then we need to import the RustDoc type. Let’s do that now:
mod config;
mod logger;
pub mod rustc;
pub mod rustdoc; // This is new
use crate::rustc::CrateType;
use crate::rustc::Edition;
use crate::rustc::Rustc;
use crate::rustdoc::RustDoc; // This is new
use config::Manifest;
use logger::Logger;
use std::env;And with that we can actually update our run_tests function. We now need to pass information from the manifest to our creation of a new RustDoc type so let’s first change the opening of the function to handle that:
pub fn run_tests(test_args: Vec<String>) -> Result<()> {
let root = root_dir()?;
let manifest = Manifest::parse_from_file(root.join("Freight.toml"))?;
for item in root.join("target").join("debug").join("tests").read_dir()? {
// Code omitted
}
}We now create a new Manifest to use and then we clean up the for loop to use root so that we don’t call root_dir()? twice in the same function. We now need to actually add the code to call rustdoc. Just below that for loop and before the Ok(()) return we’ll add the following:
pub fn run_tests(test_args: Vec<String>) -> Result<()> {
let root = root_dir()?;
let manifest = Manifest::parse_from_file(root.join("Freight.toml"))?;
for item in root.join("target").join("debug").join("tests").read_dir()? {
// Code omitted
}
// This is all new
let lib = root.join("src").join("lib.rs");
if lib.exists() {
RustDoc::new(
manifest.edition,
manifest.crate_name,
root.join("target").join("debug"),
)
.test(lib)?;
}
Ok(())
}With that we now have rustdoc test support! Let’s give it a shot to see what happens.
❯ just test
rm -rf target
mkdir -p target/bootstrap
# Build crate dependencies
rustc src/lib.rs --edition 2021 --crate-type=lib --crate-name=freight --out-dir=target/bootstrap
# Create the executable
rustc src/main.rs --edition 2021 --crate-type=bin --crate-name=freight --out-dir=target/bootstrap -L target/bootstrap --extern freight
./target/bootstrap/freight build
Compiling crate freight...Done
Compiling bin freight...Done
mkdir -p target/test
# Test that we can pass args to the tests
./target/debug/freight test ignored-arg -- --list
Compiling bin freight...Done
Compiling crate freight...Done
Compiling bin freight...Done
0 tests, 0 benchmarks
running 1 test
test src/rustc.rs - rustc::Rustc::builder (line 22) ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.15s
rustc::crate_type_from_str: test
rustc::edition_from_str: test
2 tests, 0 benchmarks
running 1 test
test src/rustc.rs - rustc::Rustc::builder (line 22) ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.15s
running 1 test
test src/rustc.rs - rustc::Rustc::builder (line 22) ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s
# Actually run the tests
./target/debug/freight test
Compiling bin freight...Done
Compiling crate freight...Done
Compiling bin freight...Done
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
running 1 test
test src/rustc.rs - rustc::Rustc::builder (line 22) ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.15s
running 2 tests
test rustc::crate_type_from_str ... ok
test rustc::edition_from_str ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
running 1 test
test src/rustc.rs - rustc::Rustc::builder (line 22) ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s
running 1 test
test src/rustc.rs - rustc::Rustc::builder (line 22) ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.15sAs you can see we have run the doc tests, but unfortunately it's hard to discern which test is what. Even worse the logging for what is being built is just jumbled and it's unclear which part of the build is causing the library to even be built. The only clue is the output from the justfile so we know what is happening. For users of Freight they won't have that, so we need to fix the logging output and overhaul our implementation of the Logger type.
Let’s start by changing how it works by opening up src/logger.rs and just rewriting the file:
use crate::Result;
use std::io;
use std::io::Write;
pub struct Logger {
out: io::StdoutLock<'static>,
}
impl Logger {
pub fn new() -> Self {
Self {
out: io::stdout().lock(),
}
}
pub fn compiling_crate(&mut self, crate_name: &str) -> Result<()> {
self.out
.write_all(format!(" Compiling lib {crate_name}\n").as_bytes())?;
self.out.flush()?;
Ok(())
}
pub fn compiling_bin(&mut self, crate_name: &str) -> Result<()> {
self.out
.write_all(format!(" Compiling bin {crate_name}\n").as_bytes())?;
self.out.flush()?;
Ok(())
}
pub fn done_compiling(&mut self) -> Result<()> {
self.out.write_all(b" Finished dev\n")?;
self.out.flush()?;
Ok(())
}
pub fn main_unit_test(&mut self) -> Result<()> {
self.unit_test("src/main.rs")?;
Ok(())
}
pub fn lib_unit_test(&mut self) -> Result<()> {
self.unit_test("src/lib.rs")?;
Ok(())
}
fn unit_test(&mut self, file: &str) -> Result<()> {
self.out.write_all(b" Running unittests ")?;
self.out.write_all(file.as_bytes())?;
self.out.write_all(b"\n")?;
self.out.flush()?;
Ok(())
}
pub fn doc_test(&mut self, crate_name: &str) -> Result<()> {
self.out.write_all(b" Doc-tests ")?;
self.out.write_all(crate_name.as_bytes())?;
self.out.write_all(b"\n")?;
self.out.flush()?;
Ok(())
}
}There are quite a few changes here:
- We’re using
Result<()>everywhere now and will error ifwrite_allever fails - We’re flushing the output now. Before we weren’t doing that and sometimes the buffer wouldn’t be printed to stdout and now we make sure it is all of the time.
compiling_binandcompiling_cratenow look more like how cargo outputs messages and won’t depend on adoneto be output. If we see something else compiling in the logging output, the previous item is done compiling.- We now add specific logging output for each type of test that looks like what cargo does
Now we just need to make a few changes in src/lib.rs that will let us be done with logging. First we need to remove any reference to done_compiling as the function does not exist anymore. We also need to add a ? to every single call to a Logger function.
We need to remove the logger passed into test_compile and any function calls inside of it as it does not make sense to show we’re compiling the crate as part of the tests. If the tests are output and shown then that implies it compiled everything successfully. This also means we need to remove the Logger passed to each invocation of test_compile.
If you want to see all of these changes you can check out the full diff linked at the end of the section, but they’re not that noteworthy to show each individual link.
Lastly we need to update our run_tests function again to use our new test logging functionality to look like this:
pub fn run_tests(test_args: Vec<String>) -> Result<()> {
let mut logger = Logger::new(); // This is new
let root = root_dir()?;
let manifest = Manifest::parse_from_file(root.join("Freight.toml"))?;
for item in root.join("target").join("debug").join("tests").read_dir()? {
let item = item?;
let path = item.path();
let is_test = path.extension().is_none();
if is_test {
// This is new
let file_name = path.file_name().unwrap().to_str().unwrap();
if file_name == "test_freight_main" {
logger.main_unit_test()?;
} else if file_name == "test_freight_lib" {
logger.lib_unit_test()?;
}
Command::new(path).args(&test_args).spawn()?.wait()?;
}
}
let lib = root.join("src").join("lib.rs");
if lib.exists() {
// This is new
logger.doc_test(&manifest.crate_name)?;
RustDoc::new(
manifest.edition,
manifest.crate_name,
root.join("target").join("debug"),
)
.test(lib)?;
}
Ok(())
}Finally let's try this again to see the output!
❯ just test
rm -rf target
mkdir -p target/bootstrap
# Build crate dependencies
rustc src/lib.rs --edition 2021 --crate-type=lib --crate-name=freight --out-dir=target/bootstrap
# Create the executable
rustc src/main.rs --edition 2021 --crate-type=bin --crate-name=freight --out-dir=target/bootstrap -L target/bootstrap --extern freight
./target/bootstrap/freight build
Compiling lib freight
Compiling bin freight
mkdir -p target/test
# Test that we can pass args to the tests
./target/debug/freight test ignored-arg -- --list
Compiling lib freight
Finished dev
Running unittests src/main.rs
0 tests, 0 benchmarks
Running unittests src/lib.rs
rustc::crate_type_from_str: test
rustc::edition_from_str: test
2 tests, 0 benchmarks
Doc-tests freight
running 1 test
test src/rustc.rs - rustc::Rustc::builder (line 22) ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.17s
# Actually run the tests
./target/debug/freight test
Compiling lib freight
Finished dev
Running unittests src/main.rs
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running unittests src/lib.rs
running 2 tests
test rustc::crate_type_from_str ... ok
test rustc::edition_from_str ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Doc-tests freight
running 1 test
test src/rustc.rs - rustc::Rustc::builder (line 22) ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14sWith that we have output that is easier to read and looks more like cargo’s output and we can clearly see which tests are being run! Let's commit our changes for today now that we have rustdoc support.
mgattozziConclusion
The easy part ironically, was adding the rustdoc support. The hard part was making it so that our output wasn’t impossible to understand what was actually going on. Next time we’ll add support for integration tests in the tests directory just like cargo does. With them we’ll be able to even more tests than we currently do and open up the possibilities of testing the CLI itself automatically. Till next time!