Let's Build a Cargo Compatible Build Tool - Part 3
This is the third post in an educational multipart series about how build tools like Cargo work under the hood, by building our own from the ground up. You can find part 1 here, and part 2 here.
Today's post will be a bit shorter, but will pack quite the punch. Today we're going to add the ability for creating and running tests to Freight. Up to this point we've had the attitude of "If we build it and it can build itself we're good". However, that's just not going to be the case as Freight gets bigger. We're going to create logic bugs when the code is not as small and narrowly focused on the happy path. For many of our error cases for instance we just chuck the error up the stack and let fn main() -> Result<(), Box<dyn Error>> handle it. We make a lot of assumptions and are not handling errors. Today we're going to focus on starting to test these assumptions by creating tests to double check our work and also create a way for us to make these tests using the inbuilt testing framework for rustc.
Rust isn't a panacea and types will only get us so far, especially at the boundary of where our program interfaces with the system. In the previous post we added this impl of FromStr:
impl FromStr for Edition {
type Err = BoxError;
fn from_str(input: &str) -> Result<Self> {
match input {
"2015" => Ok(Self::E2015),
"2018" => Ok(Self::E2018),
"2021" => Ok(Self::E2021),
edition => Err(format!("Edition {edition} is not supported").into()),
}
}
}We want to write a test for this just to make sure that only these 3 numbers will parse. It's a small thing and sure we can validate it ourselves just looking at it, but better to make sure that's always the case than not. Besides we have to start somewhere.
Let's first start by having our justfile in the root of the directory run tests for us. We'll start by changing this:
run: build
./target/bootstrap/freight build
./target/debug/freight help
build:
rm -rf target
mkdir -p target/bootstrap
# Build crate dependencies
rustc src/lib.rs --edition 2021 --crate-type=lib --crate-name=freight \
--out-dir=target/bootstrap
# Create the executable
rustc src/main.rs --edition 2021 --crate-type=bin --crate-name=freight \
--out-dir=target/bootstrap -L target/bootstrap --extern freightto this instead:
run: build
./target/debug/freight help
build:
rm -rf target
mkdir -p target/bootstrap
# Build crate dependencies
rustc src/lib.rs --edition 2021 --crate-type=lib --crate-name=freight \
--out-dir=target/bootstrap
# Create the executable
rustc src/main.rs --edition 2021 --crate-type=bin --crate-name=freight \
--out-dir=target/bootstrap -L target/bootstrap --extern freight
./target/bootstrap/freight build # This was moved from the run: build section
test: build # This is all new
mkdir -p target/test
./target/debug/freight testThis does two things we care about. First it moves our bootstrapped freight build to the build command which is probably where it should have been in the first place. Secondly we add a new test command to run all of the build steps and then run our tests using Freight itself. With this we can call just run to make sure it builds and prints out the help message like before and we can also call just test to do the build like we expect and then build and run tests!
Let's update our CI definition to make sure it will now also build the tests. Edit .github/workflows/ci.yaml by appending this block of yaml to the end:
- name: "Build and test Freight"
run: "just test"
shell: bashAnd we should also update the hook at hooks/pre-commit.sh to include a call to just test as well to mimic CI:
#!/bin/sh
just run
just testMost of the changes today will be in lib.rs but we have two changes to make to main.rs which are to add the test command and update the help section. Let's open it up and make those quick changes:
fn main() -> Result<(), Box<dyn Error>> {
// This help string has been updated with the test command
const HELP: &str = "\
Alternative for Cargo\n\n\
Usage: freight [COMMAND] [OPTIONS]\n\n\
Commands:\n \
build Build a Freight or Cargo project\n \
test Test a Freight or Cargo project\n \
help Print out this message
";
let mut args = env::args().skip(1);
match args.next().as_ref().map(String::as_str) {
Some("build") => freight::build()?,
// This block is also new
Some("test") => {
freight::build_tests()?;
freight::run_tests()?
}
Some("help") => println!("{HELP}"),
_ => {
println!("Unsupported command");
With that we've set up most of the infrastructure needed to build and run tests for us, we just now need to write some tests and the actual functions to run them. We'll start off by writing the tests since they're the easiest part for this. We'll start with testing FromStr for Edition and to do that we need to derive a few extra traits. Open up lib.rs and keep it open since we'll be there for the rest of the post.
Scroll down to the enum definition of Edition and add the PartialEq, Eq, and Debug impls since we will want to test for equality in our tests and it's just good practice to have a Debug impl for library users to use. It should now look like this:
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Edition {We're also going to add a test for CrateType and a FromStr impl for future proofing and to add a new test. We know we'll need this as an option for our configuration in the future so might as well get it out of the way now.
Let's add the same derives to CrateType just like Edition:
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum CrateType {Now we can add our FromStr impl just below it:
impl FromStr for CrateType {
type Err = BoxError;
fn from_str(input: &str) -> Result<Self> {
match input {
"bin" => Ok(Self::Bin),
"lib" => Ok(Self::Lib),
"rlib" => Ok(Self::RLib),
"dylib" => Ok(Self::DyLib),
"cdylib" => Ok(Self::CDyLib),
"staticlib" => Ok(Self::StaticLib),
"proc-macro" => Ok(Self::ProcMacro),
crate_type => Err(format!("Crate Type {crate_type} is not supported").into()),
}
}
}It's pretty much the same as Edition just with different inputs and more variants to cover. Now we can get to writing our tests! Finally!
Let's add them to the bottom of the file below what we just wrote:
#[test]
fn edition_from_str() -> Result<()> {
let e2015 = Edition::from_str("2015")?;
assert_eq!(e2015, Edition::E2015);
let e2018 = Edition::from_str("2018")?;
assert_eq!(e2018, Edition::E2018);
let e2021 = Edition::from_str("2021")?;
assert_eq!(e2021, Edition::E2021);
if !Edition::from_str("\"2015\"").is_err() {
panic!("bad string parsed correctly");
}
Ok(())
}
#[test]
fn crate_type_from_str() -> Result<()> {
let bin = CrateType::from_str("bin")?;
assert_eq!(bin, CrateType::Bin);
let lib = CrateType::from_str("lib")?;
assert_eq!(lib, CrateType::Lib);
let rlib = CrateType::from_str("rlib")?;
assert_eq!(rlib, CrateType::RLib);
let dylib = CrateType::from_str("dylib")?;
assert_eq!(dylib, CrateType::DyLib);
let cdylib = CrateType::from_str("cdylib")?;
assert_eq!(cdylib, CrateType::CDyLib);
let staticlib = CrateType::from_str("staticlib")?;
assert_eq!(staticlib, CrateType::StaticLib);
let proc_macro = CrateType::from_str("proc-macro")?;
assert_eq!(proc_macro, CrateType::ProcMacro);
if !CrateType::from_str("proc-marco").is_err() {
panic!("bad string parsed correctly");
}
Ok(())
}We call from_str for each variant and assert that it's equal to what we expect and panic in the case it ever parses into anything it shouldn't. If we wanted to make it really robust we could add multiple mistyped versions of the strings used, but for now this is enough to see all the ones we expect to parse will.
Before we get into building out the code that will actually run things it's important to see how we'll actually get rustc to build and run a test. Let's just make a file somewhere called scratch_file.rs and put this test in it:
#[test]
fn two_plus_two() {
assert_eq!(2 + 2, 4);
}We'll then call rustc with the test flag:
❯ rustc --test scratch_file.rsThis will create a binary called scratch_file that when called produces this output:
❯ ./scratch_file
running 1 test
test two_plus_two ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00sThat output looks quite similar to Cargo's! That's because that's all Cargo is doing! It's invoking rustc to build the tests with the --test flag and then running the created binaries. Cargo is also doing things like printing out where the tests are from, such as tests from the tests folder (which is a Cargo convention), but overall this is all that's happening. rustc does all the heavy lifting for us. With that in mind let's extend our Rustc type to handle this new flag. First let's add it to the struct:
pub struct Rustc {
edition: Edition,
crate_type: CrateType,
crate_name: String,
out_dir: PathBuf,
lib_dir: PathBuf,
cfg: Vec<String>,
externs: Vec<String>,
test: bool, // This is new
}Since we only care about whether the flag is used or not we'll use a bool for it. true means we will build tests with it, false meaning we just build the code like normal. We now need to add this to our run function for Rustc:
.arg("-L")
.arg(self.lib_dir)
.args(if self.test { vec!["--test"] } else { vec![] }) // This is new
.args(We're doing the necessary hack here just like with externs and cfg to handle that we don't want to add spaces or white space to our command if we have self.test set to false.
We also need to add the test field to RustcBuilder:
#[derive(Default)]
pub struct RustcBuilder {
edition: Option<Edition>,
crate_type: Option<CrateType>,
crate_name: Option<String>,
out_dir: Option<PathBuf>,
lib_dir: Option<PathBuf>,
cfg: Vec<String>,
externs: Vec<String>,
test: bool, // This is new
}Let's add the test function now just above fn done for RustcBuilder:
pub fn test(mut self, test: bool) -> Self {
self.test = test;
self
}
Finally let's modify fn done to contain the new test field as well:
pub fn done(self) -> Rustc {
Rustc {
edition: self.edition.unwrap_or(Edition::E2015),
crate_type: self.crate_type.expect("Crate type given"),
crate_name: self.crate_name.expect("Crate name given"),
out_dir: self.out_dir.expect("Out dir given"),
lib_dir: self.lib_dir.expect("Lib dir given"),
cfg: self.cfg,
externs: self.externs,
test: self.test, // This is new
}
}With that Rustc can handle being run to compile tests! We're now going to split out the bin_compile and lib_compile closures in fn build into their own functions and delete the closures in fn build. First let's clean up fn build to look like we expect it too:
pub fn build() -> Result<()> {
let mut logger = Logger::new();
let root_dir = root_dir()?;
let manifest = Manifest::parse_from_file(root_dir.join("Freight.toml"))?;
let lib_rs = root_dir.join("src").join("lib.rs");
let main_rs = root_dir.join("src").join("main.rs");
let target = root_dir.join("target");
let target_debug = target.join("debug");
fs::create_dir_all(&target_debug)?;
// bin_compile and lib_compile were deleted from here
// This is mostly the same except the function calls have more arguments
// passed into it
match (lib_rs.exists(), main_rs.exists()) {
(true, true) => {
lib_compile(&mut logger, &manifest, &lib_rs, &target_debug)?;
bin_compile(
&mut logger,
&manifest,
&main_rs,
&target_debug,
&[&manifest.crate_name],
)?;
}
(true, false) => {
lib_compile(&mut logger, &manifest, &lib_rs, &target_debug)?;
}
(false, true) => {
bin_compile(&mut logger, &manifest, &main_rs, &target_debug, &[])?;
}
(false, false) => return Err("There is nothing to compile".into()),
}
Ok(())
}It's a little bit more compact than before. It's important to note we've added more arguments to the function calls. We were implicitly capturing these variables from the environment, but since we split it out into actual functions we need to explicitly pass them into the function.
So let's actually look at what lib_compile and bin_compile look like all split out. We'll add these just below our BoxError type alias.
pub type BoxError = Box<dyn Error>;
fn lib_compile(
logger: &mut Logger,
manifest: &Manifest,
lib_path: &Path,
out_dir: &Path,
) -> Result<()> {
logger.compiling_crate(&manifest.crate_name);
Rustc::builder()
.edition(manifest.edition)
.crate_type(CrateType::Lib)
.crate_name(&manifest.crate_name)
.out_dir(out_dir.clone())
.lib_dir(out_dir.clone())
.done()
.run(lib_path.to_str().unwrap())?;
logger.done_compiling();
Ok(())
}
fn bin_compile(
logger: &mut Logger,
manifest: &Manifest,
bin_path: &Path,
out_dir: &Path,
externs: &[&str],
) -> Result<()> {
logger.compiling_bin(&manifest.crate_name);
let mut builder = Rustc::builder()
.edition(manifest.edition)
.crate_type(CrateType::Bin)
.crate_name(&manifest.crate_name)
.out_dir(out_dir.clone())
.lib_dir(out_dir.clone());
for ex in externs {
builder = builder.externs(*ex);
}
builder.done().run(bin_path.to_str().unwrap())?;
logger.done_compiling();
Ok(())
}It's almost the same code as before, except this time they take in more arguments. It's worth nothing that builder.externs(*ex) now contains a deref call in bin_compile whereas before it was just builder.externs(ex). This is due to us passing in an &[&str] instead of a Vec<&str> or Vec<String>.
Okay so just below bin_compile let's finally add our test_compile function! This is the meat of what we've been working on today and it'll actually let us invoke Rustc to build a test binary:
fn test_compile(
logger: &mut Logger,
manifest: &Manifest,
bin_path: &Path,
out_dir: &Path,
externs: &[&str],
) -> Result<()> {
logger.compiling_bin(&manifest.crate_name);
let mut builder = Rustc::builder()
.edition(manifest.edition)
.crate_type(CrateType::Bin)
.crate_name(format!(
"test_{}_{}",
&manifest.crate_name,
bin_path.file_stem().unwrap().to_str().unwrap()
))
.out_dir(out_dir.clone())
.lib_dir(out_dir.clone())
.test(true);
for ex in externs {
builder = builder.externs(*ex);
}
builder.done().run(bin_path.to_str().unwrap())?;
logger.done_compiling();
Ok(())
}This is pretty much the same as bin_compile except for two key diferences:
- We set
test(true)forRustc - The name used for
crate_nameis different. It's a combination of the crate name and file name that was used (i.e. if it'slib.rsormain.rswith the.rsstripped). This way when we invoke it we can print out where the test came from and which crate if we need that.
Now we can create our build_tests function! It's mostly the same as build, but we call test_compile instead of bin_compile or lib_compile (mostly). Let's add it just below build:
pub fn build_tests() -> Result<()> {
let mut logger = Logger::new();
let root_dir = root_dir()?;
let manifest = Manifest::parse_from_file(root_dir.join("Freight.toml"))?;
let lib_rs = root_dir.join("src").join("lib.rs");
let main_rs = root_dir.join("src").join("main.rs");
let target = root_dir.join("target");
let target_tests = target.join("debug").join("tests");
fs::create_dir_all(&target_tests)?;
match (lib_rs.exists(), main_rs.exists()) {
(true, true) => {
test_compile(&mut logger, &manifest, &lib_rs, &target_tests, &[])?;
lib_compile(&mut logger, &manifest, &lib_rs, &target_tests)?;
test_compile(
&mut logger,
&manifest,
&main_rs,
&target_tests,
&[&manifest.crate_name],
)?;
}
(true, false) => {
test_compile(&mut logger, &manifest, &lib_rs, &target_tests, &[])?;
}
(false, true) => {
test_compile(&mut logger, &manifest, &main_rs, &target_tests, &[])?;
}
(false, false) => return Err("There is nothing to compile".into()),
}
Ok(())
}The big difference is when both lib.rs and main.rs exist. We first compile lib.rs into a test, but then we call lib_compile to build it like a normal library! This is so we can link it into main.rs when we call test_compile on it. Besides that it's not much different besides where we output tests and some file paths.
We're in the homestretch! Let's add the final piece of the puzzle that will actually run the test binaries we create. Just below build_tests we'll add a new function run_tests:
pub fn run_tests() -> Result<()> {
for item in root_dir()?
.join("target")
.join("debug")
.join("tests")
.read_dir()?
{
let item = item?;
let path = item.path();
let is_test = path.extension().is_none();
if is_test {
Command::new(path).spawn()?.wait()?;
}
}
Ok(())
}It goes to where our tests are, only grabs the files that are binaries (since we had it output a file with out an extension to it's name), invokes it and waits for it to finish. Let's give it a whirl and see what happens!
❯ just test
rm -rf target
mkdir -p target/bootstrap
# Build crate dependencies
rustc src/lib.rs --edition 2021 --crate-type=lib --crate-name=freight --out-dir=target/bootstrap
# Create the executable
rustc src/main.rs --edition 2021 --crate-type=bin --crate-name=freight --out-dir=target/bootstrap -L target/bootstrap --extern freight
./target/bootstrap/freight build
Compiling crate freight...Done
Compiling bin freight...Done
mkdir -p target/test
./target/debug/freight test
Compiling bin freight...Done
Compiling crate freight...Done
Compiling bin freight...Done
running 2 tests
test crate_type_from_str ... ok
test edition_from_str ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00sWith this we've actually built and run our tests! We setup most of the infrastructure needed now and in future posts we'll add rustdoc test support, tests folder support, and better output! Let's commit the code and push it up.
mgattozziConclusion
We've added tests finally! Not having tests was making me a bit nervous and so my goal was to keep freight small and focused until it could be added in. Now we can add more features with confidence as we go along by adding more tests. The more we automate these things, the more assured we can be that it won't just break in unexpected ways.
I'll be streaming more development on Freight this Friday July 15th, 2023 from 1pm to 4pm EDT/UTC-4 if you want to come along and get a preview of where Freight is at and future features. It's during the RustConf Unconf time so if you're not able to attend or just want to hack on some things yourself and have something to watch then come on by!
I'm also available for Hire or Contracting at this time so if you need a Rust expert with 8 years of experience on your team do drop a line.