Use Rust in shell scripts

24 Aug

TL;DR

Write shell scripts in Rust using rust-script and rust-script-ext.

Lately I have been porting many of my bash scripts to Rust source files. It started with writing Rust scripts and compiling them using rustc to get an executable (eg rustc foo.rs && ./foo) for script tasks that were complicated enough to benefit from using Rust. For example, I have a script which reads a burn log directory and generates a bunch of plotly charts to analyse neural network training progression (see the gist). This task was complex enough that writing it in bash was not really an option, however it is simple enough that it fits within 215 lines of Rust code. For these types of tasks, what I really want is to be able to treat the Rust source code like a script.

`rust-script`

rust-script is exactly the tool to use. After installation, a script can be compiled and executed using rust-script foo.rs. Even better, by prepending the shebang line #!/usr/bin/env rust-script, the script can be compiled and executed with a simple ./foo.rs much like a bash script!

rust-script also supports incorporating dependencies, which is a game changer over using rustc. Take their README example:

$ cat script.rs
#!/usr/bin/env rust-script
//! Dependencies can be specified in the script file itself as follows:
//!
//! ```cargo
//! [dependencies]
//! rand = "0.8.0"
//! ```

use rand::prelude::*;

fn main() {
    let x: u64 = random();
    println!("A random number: {}", x);
}

$ ./script.rs
A random number: 9240261453149857564

With rust-script I started porting other scripts over to Rust. In doing so I found some emerging patterns that I decided to encapsulate in rust-script-ext. The crate exposes a bunch of crates and utility items that are common for scripting. The repository also provides a simple template to scaffold a script for use with rust-script:

$ curl -L https://github.com/kurtlawrence/rust-script-ext/raw/main/template.rs -o my-script.rs
$ chmod +x my-script.rs
$ cat ./my-script.rs
#!/usr/bin/env -S rust-script -c
//! You might need to chmod +x your script
//! ```cargo
//! [dependencies.rust-script-ext]
//! git = "https://github.com/kurtlawrence/rust-script-ext"
//! rev = "145ecc5015628f298be5eec5e86661c618326422"
//! ```

use rust_script_ext::prelude::*;

fn main() {
    // fastrand comes from rust_script_ext::prelude::*
    let n = std::iter::repeat_with(|| fastrand::u32(1..=100))
        .take(5)
        .collect::<Vec<_>>();

    println!("Here's 5 random numbers: {n:?}");
}
$ ./my-script.rs
Here's 5 random numbers: [28, 97, 9, 23, 58]

So what kinds of patterns are common?

Error handling

One of the main benefits of writing a script in Rust is the explicit error handling. Tasks like file reads and writes and parsing arguments can all benefit from informative errors and early exiting. Rust's error handling can be somewhat disjointed when building an application though. There are a few crates which work to remedy this issue, and miette was chosen to be exposed as the primary error infrastructure. It provides an easy mechanism to give errors context and provide more information on script errors.

// local.rs
use rust_script_ext::prelude::*;

fn main() -> Result<()> {
    std::fs::read("foo.txt")
        .into_diagnostic()
        .wrap_err("failed to read file `foo.txt`")?;
    Ok(())
}

$ ./local.rs
Error:   × failed to read file `foo.txt`
  ╰─▶ No such file or directory (os error 2)

File operations

Reading, writing, and appending to files are common tasks scripts do. rust-script-ext provides a wrapped File which provides a few utility functions for accessing files, automatically buffers writes, and provides contextual errors.

// local.rs
use rust_script_ext::prelude::*;

fn main() -> Result<()> {
    let bytes = std::iter::repeat_with(|| fastrand::u8(..))
                    .take(200).collect::<Vec<_>>();

    File::create("foo.data")?.write(bytes)?;

    // this will likely fail, and provides a decent reason why
    let _ = File::open("foo.data")?.read_to_string()?;

    Ok(())
}

$ ./local.rs
Error:   × failed to encode bytes from 'foo.data' as UTF8
  ╰─▶ invalid utf-8 sequence of 1 bytes from index 0

Argument parsing

rust-script-ext provides a minimal argument parser on top of std::env::Args. There are more fully featured crates which can provide a really good CLI experience, the goal is not to compete with these, but provide a minimal API which can still deliver decent error messages and some structural parsing. This keeps compilation time minimal.

// local.rs
use rust_script_ext::prelude::*;

fn main() -> Result<()> {
    // get the args
    let mut args = args();

    // require an argument
    let guess = args.req::<u8>("guess a number between 1..=3")?;

    // optional delay
    let delay = args.opt::<Duration>("thinking time")?;

    if let Some(delay) = delay {
        eprintln!("thinking...");
        std::thread::sleep(delay.into());
    }

    if guess == fastrand::u8(1..=3) {
        println!("🎊 You guessed correctly!");
    } else {
        println!("❌ Sorry, try again");
    }

    Ok(())
}

# Errors, since we did not specify a number
$ ./local.rs
Error: Error with argument <guess a number between 1..=3>

  × expecting an argument at position 1
   ╭────
   ╰────

# Works!
$ ./local.rs 2
🎊 You guessed correctly!

# Fails to pass the second arg
$ ./local.rs 2 5t
Error: Error with argument <thinking time>

  × failed to parse `5t` as humantime::wrapper::Duration
   ╭────
 1 │ 2 5t 
   ·   ──
   ╰────

$ ./local.rs 2 5s
thinking...
❌ Sorry, try again

Invoking commands

A shell script is usually created to batch up a bunch of other commands. Whilst Rust has a (pretty decent) mechanism for working with commands (std::process::Command), it is a bit unwieldy to work with for a typical script usage. rust-script-ext provides a few utilities to improve the ergonomics of working with commands and improving error messages. The first is the construction of commands. cmd! is a macro which can bunch the command and arguments together to make building a command a simple one-liner. Then comes the trait CommandExecute which gets implemented on Command, providing an execute and execute_str function which are tailored for scripting use. Importantly, they provide mechanisms to

Capture stdout so that it can be worked with further,
Capture stderr to provide a decent error message upon failure,
Print the incoming stdout/stderr much like what occurs in typical scripting

The default behaviour is to print both stdout and stderr, and it does so line-wise. This behaviour is similar to say tee which will capture stdout but also print it to terminal so a user can see progress of a command.

Command is still exposed if a user needs to build more complex IO handling.

// local.rs
fn main() -> Result<()> {
    // most simple call, cmd! builds 
    // the Command, and .run executes it inheriting
    // the stdio
    cmd!(ls).run()?;

    println!("=== under src/ ===");
    // Verbose makes stdout and stderr print
    cmd!(ls: src).execute(Verbose)?;
    Ok(())
}

$ ./local.rs
Cargo.lock  Cargo.toml    LICENSE  local.rs  README.md  src  target  template.rs
=== under src/ ===
args.rs
cmd.rs
fs.rs
lib.rs
snapshots

The error handling, when stderr is captured, provides some decent context.

// local.rs
fn main() -> Result<()> {
    cmd!(ls: non-existent-directory).execute(Verbose)?;
    Ok(())
}

$ ./local.rs
ls: cannot access 'non-existent-directory': No such file or directory
Error:   × failed to execute cmd: ls non-existent-directory
   ╭────
 1 │ ls: cannot access 'non-existent-directory': No such file or directory
   · ───────────────────────────────────┬──────────────────────────────────
   ·                                    ╰── stderr
   ╰────

The below snippet shows the differing levels of verbosity.

fn main() -> Result<()> {
    cmd!(ls).execute(Quiet)?; // no printing
    cmd!(ls).execute(Stderr)?; // prints stderr
    cmd!(ls).execute(Stdout)?; // prints stdout
    cmd!(ls).execute(Verbose)?; // prints both

    Ok(())
}

Conclusion

Write your scripts in Rust! It does have a few drawbacks. The compilation time can take a bit on the initial compile. rust-script does a good job of caching builds so that subsequent runs will use the built binary. It also requires a fair bit of knowledge of Rust, however I find the compilation errors much more helpful than (for example) bash. I have been porting many of my scripts over and I do prefer them written in Rust, I find it allows me to extend the script without the pain of realising you've made a monstrous bash script that really should have been written in a proper language initially. If you use rust-script-ext and want more features or find different patterns, please raise and issue or PR in the repository!

Kurt Lawrence

Use Rust in shell scripts

TL;DR

rust-script