Let’s Build a Cargo Compatible Build Tool - Part 4
This is the fourth post in an educational multipart series about how build tools like Cargo work under the hood, by building our own from the ground up. If you haven’t read the previous posts it’s recommended that you do so.
Previous posts:
First off I just wanted to thank everyone who came for the stream two weeks ago. I had an absolute blast doing it and I hope you got something out of it. If you weren’t able to attend you can watch it here! You’ll get a fun sneak peak of what’s to come in Freight.
In our last post we finally got Freight to build our Rust programs to make tests that we could actually run. Just like we would with Cargo. We now have some basic tests! This was a huge step forward to making Freight more usable outside of the context of building and running itself. Today’s post will mostly be some cleanup of the code base. Nothing ground breaking per se, but some chores that have needed to get done for some time and can’t really be ignored anymore. With that let’s get started.
Help Text
Ever since we introduced help text to be printed out on either failure or with the help command it has been living in src/main.rs looking like this:
const HELP: &str = "\
Alternative for Cargo\n\n\
Usage: freight [COMMAND] [OPTIONS]\n\n\
Commands:\n \
build Build a Freight or Cargo project\n \
test Test a Freight or Cargo project\n \
help Print out this message
";The reason was that with raw strings the text would have to be at the far left side with no indentation in order to be printed out properly. So instead we opted for normal strings in Rust so that we could have more aligned and formatted code, but with all of the formatting characters and escapes it’s not the best to look at and so we need a better solution. Luckily we can get the best of both worlds when it comes to raw strings and formatted data. We’re going to use the include_str! macro which lets us take an external file and store it into a const variable as an &’static str that we can use just as we were doing so before. The benefit is that now we can format the text file just like we want for the output with none of the formatting characters, and our Rust code should be a tad more readable.
This macro will use the given path relative to the current file. So in our case since we’re going to use this in src/main.rs it will be relative to src. Let’s first extract the help message into a new file src/help.txt:
Alternative for Cargo
Usage: freight [COMMAND] [OPTIONS]
Commands:
build Build a Freight or Cargo project
test Test a Freight or Cargo project
help Print out this messageAs you can see we’ve removed the \ and \n that were littering the string in src/main.rs. Lastly we just need to now use include_str! in it’s stead:
fn main() -> Result<(), Box<dyn Error>> {
const HELP: &str = include_str!("help.txt");
// Abridged code
}This is much nicer to look at and easier to add new commands to than before. Mostly though it’s just nicer to look at.
This is a small change that should be easy to test out. We know that we call the help command as part of our testing so as long as the output from that looks like we expect then we should be okay.
❯ just run
rm -rf target
mkdir -p target/bootstrap
# Build crate dependencies
rustc src/lib.rs --edition 2021 --crate-type=lib --crate-name=freight --out-dir=target/bootstrap
# Create the executable
rustc src/main.rs --edition 2021 --crate-type=bin --crate-name=freight --out-dir=target/bootstrap -L target/bootstrap --extern freight
./target/bootstrap/freight build
Compiling crate freight...Done
Compiling bin freight...Done
./target/debug/freight help
Alternative for Cargo
Usage: freight [COMMAND] [OPTIONS]
Commands:
build Build a Freight or Cargo project
test Test a Freight or Cargo project
help Print out this messageGreat everything looks like we expect so let’s commit that change and move on to the next task.
mgattozziTest Args
One of the nice things about Cargo is that it lets us actually pass args to the test binary itself. For instance if we call cargo test — —nocapture then the —nocapture flag gets passed down to the test binary as a flag and it will then print out everything to stdout like it normally would. I find this flag useful when I want to call println! to debug test cases and it would be nice if freight could support that as well.
We can quickly do this by expanding our arg parsing code and modifying the run_tests function to accept a Vec<String> that it can pass in as arguments when running a test. So let’s do that by first changing our arg parsing code in src/main.rs
fn main() -> Result<(), Box<dyn Error>> {
const HELP: &str = include_str!("help.txt");
let mut args = env::args().skip(1);
match args.next().as_ref().map(String::as_str) {
Some("build") => freight::build()?,
Some("test") => {
freight::build_tests()?;
// This loop block is the only new thing!
loop {
match args.next().as_ref().map(String::as_str) {
Some("--") | None => break,
_ => continue,
}
}
freight::run_tests(args.collect::<Vec<String>>())?
}
Some("help") => println!("{HELP}"),
_ => {
println!("Unsupported command");
println!("{HELP}");
process::exit(1);
}
}
Ok(())
}We first loop until we find a -- arg as this is what Cargo uses to note anything past it should be passed down as an arg. We stop if there are no args left or we find that symbol. Otherwise, we keep going until either of those conditions are true. We want to eat up any extra args passed in e.g. freight test foo bar – --nocapture should drop foo and bar and only pass in --nocapture to our tests.
From here we can call collect on the iterator and anything left will be our args for the test binaries or if there’s nothing left then we just get an empty Vec to pass in! Not to bad of a change for the argument parsing.
Next we need to open up src/lib.rs and actually modify run_tests to accept an argument to the function. The change looks like this:
pub fn run_tests(test_args: Vec<String>) -> Result<()> {
// Abbreviated function
}We now have args that we can reference when we use command. Our final change in the code is to make it actually pass in the args for the test binary:
pub fn run_tests(test_args: Vec<String>) -> Result<()> {
// Abbreviated function
if is_test {
Command::new(path).args(&test_args).spawn()?.wait()?;
}
// Abbreviated function
}It’s important to note that we use &test_args here and not test_args since this spawning is done inside a loop and we need these args passed to every test that we run. If we just passed in the Vec<String> then we wouldn’t have args for the next test binary going through the loop and rustc would rightfully complain at us for not following the borrowing rules.
One final thing is we need to actually test that this functionality does what we expect it too. So let’s open our justfile and modify our test command to look like this now:
test: build
mkdir -p target/test
# Test that we can pass args to the tests
./target/debug/freight test ignored-arg -- --list
# Actually run the tests
./target/debug/freight testWith this we test we’re ignoring the junk arg and then passing in everything else to the test binary and we should see this command pass with the tests listed in the output. In the future we should really error on the fact that it’s a bad argument being passed in, but for now this is fine.
Let’s give it a shot before committing everything.
❯ just test
rm -rf target
mkdir -p target/bootstrap
# Build crate dependencies
rustc src/lib.rs --edition 2021 --crate-type=lib --crate-name=freight --out-dir=target/bootstrap
# Create the executable
rustc src/main.rs --edition 2021 --crate-type=bin --crate-name=freight --out-dir=target/bootstrap -L target/bootstrap --extern freight
./target/bootstrap/freight build
Compiling crate freight...Done
Compiling bin freight...Done
mkdir -p target/test
# Test that we can pass args to the tests
./target/debug/freight test ignored-arg -- --list
Compiling bin freight...Done
Compiling crate freight...Done
Compiling bin freight...Done
crate_type_from_str: test
edition_from_str: test
2 tests, 0 benchmarks
0 tests, 0 benchmarks
# Actually run the tests
./target/debug/freight test
Compiling bin freight...Done
Compiling crate freight...Done
Compiling bin freight...Done
running 2 tests
test crate_type_from_str ... ok
test edition_from_str ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00sAs we can see in the output the tests are listed and the justfile didn’t error out so everything works as expected. Let’s commit those changes and move onto the final change for today.
mgattozziRustc Module
We have written a lot of code related to calling Rustc, it’s args, and even some tests for the types we can pass in as args. It's about time that we finally split it out of src/lib.rs into it's own module. With the power of copy and paste we can easily make the changes. First we grab all of the code from struct Rustc to the bottom of the src/lib.rs and yeet it into src/rustc.rs and that new file should look like this:
pub struct Rustc {
edition: Edition,
crate_type: CrateType,
crate_name: String,
out_dir: PathBuf,
lib_dir: PathBuf,
cfg: Vec<String>,
externs: Vec<String>,
test: bool,
}
impl Rustc {
pub fn builder() -> RustcBuilder {
RustcBuilder {
..Default::default()
}
}
pub fn run(self, path: &str) -> Result<()> {
Command::new("rustc")
.arg(path)
.arg("--edition")
.arg(self.edition.to_string())
.arg("--crate-type")
.arg(self.crate_type.to_string())
.arg("--crate-name")
.arg(self.crate_name)
.arg("--out-dir")
.arg(self.out_dir)
.arg("-L")
.arg(self.lib_dir)
.args(if self.test { vec!["--test"] } else { vec![] })
.args(
self.externs
.into_iter()
.map(|r#extern| ["--extern".into(), r#extern])
.flatten(),
)
.args(
self.cfg
.into_iter()
.map(|cfg| ["--cfg".into(), cfg])
.flatten(),
)
.spawn()?
.wait()?;
Ok(())
}
}
#[derive(Default)]
pub struct RustcBuilder {
edition: Option<Edition>,
crate_type: Option<CrateType>,
crate_name: Option<String>,
out_dir: Option<PathBuf>,
lib_dir: Option<PathBuf>,
cfg: Vec<String>,
externs: Vec<String>,
test: bool,
}
impl RustcBuilder {
pub fn edition(mut self, edition: Edition) -> Self {
self.edition = Some(edition);
self
}
pub fn out_dir(mut self, out_dir: impl Into<PathBuf>) -> Self {
self.out_dir = Some(out_dir.into());
self
}
pub fn lib_dir(mut self, lib_dir: impl Into<PathBuf>) -> Self {
self.lib_dir = Some(lib_dir.into());
self
}
pub fn crate_name(mut self, crate_name: impl Into<String>) -> Self {
self.crate_name = Some(crate_name.into());
self
}
pub fn crate_type(mut self, crate_type: CrateType) -> Self {
self.crate_type = Some(crate_type);
self
}
pub fn cfg(mut self, cfg: impl Into<String>) -> Self {
self.cfg.push(cfg.into());
self
}
pub fn externs(mut self, r#extern: impl Into<String>) -> Self {
self.externs.push(r#extern.into());
self
}
pub fn test(mut self, test: bool) -> Self {
self.test = test;
self
}
pub fn done(self) -> Rustc {
Rustc {
edition: self.edition.unwrap_or(Edition::E2015),
crate_type: self.crate_type.expect("Crate type given"),
crate_name: self.crate_name.expect("Crate name given"),
out_dir: self.out_dir.expect("Out dir given"),
lib_dir: self.lib_dir.expect("Lib dir given"),
cfg: self.cfg,
externs: self.externs,
test: self.test,
}
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Edition {
E2015,
E2018,
E2021,
}
impl Display for Edition {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
let edition = match self {
Self::E2015 => "2015",
Self::E2018 => "2018",
Self::E2021 => "2021",
};
write!(f, "{edition}")
}
}
impl FromStr for Edition {
type Err = BoxError;
fn from_str(input: &str) -> Result<Self> {
match input {
"2015" => Ok(Self::E2015),
"2018" => Ok(Self::E2018),
"2021" => Ok(Self::E2021),
edition => Err(format!("Edition {edition} is not supported").into()),
}
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum CrateType {
Bin,
Lib,
RLib,
DyLib,
CDyLib,
StaticLib,
ProcMacro,
}
impl Display for CrateType {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
let crate_type = match self {
Self::Bin => "bin",
Self::Lib => "lib",
Self::RLib => "rlib",
Self::DyLib => "dylib",
Self::CDyLib => "cdylib",
Self::StaticLib => "staticlib",
Self::ProcMacro => "proc-macro",
};
write!(f, "{crate_type}")
}
}
impl FromStr for CrateType {
type Err = BoxError;
fn from_str(input: &str) -> Result<Self> {
match input {
"bin" => Ok(Self::Bin),
"lib" => Ok(Self::Lib),
"rlib" => Ok(Self::RLib),
"dylib" => Ok(Self::DyLib),
"cdylib" => Ok(Self::CDyLib),
"staticlib" => Ok(Self::StaticLib),
"proc-macro" => Ok(Self::ProcMacro),
crate_type => Err(format!("Crate Type {crate_type} is not supported").into()),
}
}
}
#[test]
fn edition_from_str() -> Result<()> {
let e2015 = Edition::from_str("2015")?;
assert_eq!(e2015, Edition::E2015);
let e2018 = Edition::from_str("2018")?;
assert_eq!(e2018, Edition::E2018);
let e2021 = Edition::from_str("2021")?;
assert_eq!(e2021, Edition::E2021);
if !Edition::from_str("\"2015\"").is_err() {
panic!("bad string parsed correctly");
}
Ok(())
}
#[test]
fn crate_type_from_str() -> Result<()> {
let bin = CrateType::from_str("bin")?;
assert_eq!(bin, CrateType::Bin);
let lib = CrateType::from_str("lib")?;
assert_eq!(lib, CrateType::Lib);
let rlib = CrateType::from_str("rlib")?;
assert_eq!(rlib, CrateType::RLib);
let dylib = CrateType::from_str("dylib")?;
assert_eq!(dylib, CrateType::DyLib);
let cdylib = CrateType::from_str("cdylib")?;
assert_eq!(cdylib, CrateType::CDyLib);
let staticlib = CrateType::from_str("staticlib")?;
assert_eq!(staticlib, CrateType::StaticLib);
let proc_macro = CrateType::from_str("proc-macro")?;
assert_eq!(proc_macro, CrateType::ProcMacro);
if !CrateType::from_str("proc-marco").is_err() {
panic!("bad string parsed correctly");
}
Ok(())
}However, this will not compile until we add some imports. Let's add them to the top of the file:
use super::BoxError;
use super::Result;
use std::fmt;
use std::fmt::Display;
use std::path::PathBuf;
use std::process::Command;
use std::str::FromStr;Most of these are just imports from std, however, we also use our custom Result and BoxError aliases in our code here so we needed to import them from src/lib.rs via super. The final step here is to modify our imports and modules in src/lib.rs so that our code knows where the new module is and it can compile without warnings:
mod config;
mod logger;
pub mod rustc; // This is new
use crate::rustc::CrateType; // This is new
use crate::rustc::Edition; // This is new
use crate::rustc::Rustc; // This is new
use config::Manifest;
use logger::Logger;
use std::env;
use std::error::Error;
// Removed std::fmt and std::fmt::Display
use std::fs;
use std::path::Path;
use std::path::PathBuf;
use std::process::Command;
// Removed std::str::FromStrWith these changes everything should compile and run. Let's do a quick sanity check before committing:
❯ just test
rm -rf target
mkdir -p target/bootstrap
# Build crate dependencies
rustc src/lib.rs --edition 2021 --crate-type=lib --crate-name=freight --out-dir=target/bootstrap
# Create the executable
rustc src/main.rs --edition 2021 --crate-type=bin --crate-name=freight --out-dir=target/bootstrap -L target/bootstrap --extern freight
./target/bootstrap/freight build
Compiling crate freight...Done
Compiling bin freight...Done
mkdir -p target/test
# Test that we can pass args to the tests
./target/debug/freight test ignored-arg -- --list
Compiling bin freight...Done
Compiling crate freight...Done
Compiling bin freight...Done
rustc::crate_type_from_str: test
rustc::edition_from_str: test
2 tests, 0 benchmarks
0 tests, 0 benchmarks
# Actually run the tests
./target/debug/freight test
Compiling bin freight...Done
Compiling crate freight...Done
Compiling bin freight...Done
running 2 tests
test rustc::crate_type_from_str ... ok
test rustc::edition_from_str ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Great everything compiles and runs and so we can make that commit and wrap up for today.
mgattozziConclusion
Today wasn't crazy in terms of what we did. Mostly moving things around and adding an easy feature with lots of upsides for us. We've now made the code a bit nicer to work with and in the next article we'll finally be adding rustdoc test support (giving us an incentive to write some docs and tests), as well as changing up the logging output to be consistent with Cargo's mostly and fixing some undesired/confusing output when running tests.
As always thank you for reading and I hope you enjoyed today's update. I think the next post will be quite fun as well so do look out for that.
Also, if you need a Rust developer with over 8 years of experience do get in touch!