Rust in Practice #5 Parallelism with rayon: Map-Reduce Without Data Races

Programming Language Rust CLI
4 min read

Part 4 pushed one core to its limit with the release build. But machines these days come with 8 or 16 cores. This part spreads the aggregation across them — and it is also the part where we finally witness the sentence basics Part 4 only promised: “the borrow rules block data races at compile time.”

First, the conditions under which parallelism pays #

Splitting work does not automatically make it faster. The criterion is where the bottleneck sits.

  • IO-bound (disk reads are the bottleneck): more cores, same single disk. Parallelism gains almost nothing.
  • CPU-bound (parsing and aggregation are the bottleneck): headroom up to the number of cores.

loglens does heavy per-line work — string splitting and number conversion — while a modern SSD’s sequential read is much faster than that. So the workload is CPU-bound, a candidate for parallelism. We also change one strategy. Part 3 chose streaming to save memory, but parallel processing needs data it can slice and distribute, so reading the whole file and handing chunks to threads is simpler and faster. The tool’s decision: streaming by default, whole-file reads only for the parallel command — strategy per command.

rayon: your iterators, parallel #

Install 
cargo add rayon

rayon’s core appeal: it is the parallel edition of the iterator chains you already write.

src/main.rs 
use rayon::prelude::*;

fn cmd_stats_parallel(path: &Path) -> anyhow::Result<()> {
    let text = std::fs::read_to_string(path)
        .with_context(|| format!("failed to read log file: {}", path.display()))?;

    let stats = text
        .par_lines()                       // parallel line iterator
        .fold(Stats::default, |mut acc, line| {
            acc.feed(line);                // per-thread partial aggregate
            acc
        })
        .reduce(Stats::default, Stats::merge); // merge the partials

    print_stats(&stats);
    Ok(())
}

Swap lines() for par_lines() and rayon slices the lines to match your cores, distributes them over a thread pool, and gathers the results. No thread spawning, no chunk-size arithmetic in the code. The structure is map-reduce exactly: fold builds one Stats per thread and aggregates that thread’s share of lines; reduce merges the partial aggregates into one. Only the merge function is new.

src/main.rs 
impl Stats {
    fn merge(mut self, other: Stats) -> Stats {
        self.total += other.total;
        self.parsed += other.parsed;
        self.failed += other.failed;
        self.bytes_sum += other.bytes_sum;
        for (status, count) in other.by_status {
            *self.by_status.entry(status).or_insert(0) += count;
        }
        self
    }
}

On a multi-million-line log you can expect speedups approaching the core count. On a few thousand lines, thread-distribution overhead can eat the gain and make things slower. Part 4’s stopwatch, run on your own data, is always the conclusion.

Why fold-reduce and not a shared Mutex #

The design that comes first to mind from other languages: one HashMap, protected by a lock, updated by everyone. Rust can do that too (wrap it in a Mutex). But that design takes the lock on every line, and threads queueing at the lock — lock contention — hands the parallel gains right back. Eight cores ending up slower than one is not rare. With fold-reduce there is no sharing at all during aggregation, and merging happens only once per chunk. “Divide and merge” over “share and lock” is the design instinct to take away from this part.

The data race the compiler catches #

The real highlight of this part is a mistake. What if, finding fold-reduce tedious, we update an outer HashMap directly from the closure?

src/main.rs 
let mut by_status: HashMap<u16, u64> = HashMap::new();
text.par_lines().for_each(|line| {
    if let Ok(entry) = parse_line(line) {
        *by_status.entry(entry.status).or_insert(0) += 1; // compile error
    }
});
Compile error 
error[E0596]: cannot borrow `by_status` as mutable, as it is a
              captured variable in a `Fn` closure

The moment a closure that many threads will run tries to borrow one value mutably, the borrow rule from basics Part 4 — one mutable reference only — fires exactly as written. In another language this compiles, and surfaces as an unreproducible bug where counts drift on an unlucky day. In Rust, a data race is not a debugging target but a compile error list. This is the moment the basics course’s investment in ownership and borrowing pays interest in parallel code.

Summary #

  • Judge parallelism by the bottleneck. CPU-bound parsing and aggregation qualify; IO-bound work gains nothing from more cores.
  • The parallel command switches to whole-file reads. Streaming (default) vs whole-file (parallel) — per-command strategy is the tool’s decision.
  • rayon starts at lines() → par_lines(). Per-thread fold plus merging reduce is the map-reduce standard form for parallel aggregation.
  • Sharing one map behind a Mutex locks on every line and refunds the gains through contention. Divide and merge is the norm.
  • Shared-mutable-state mistakes are caught by the borrow rules as compile errors. Data races becoming build failures instead of unreproducible bugs is Rust’s decisive advantage in parallel code.
  • Next part: testing. From the parser’s unit tests to integration tests that execute the whole tool via assert_cmd — a safety net around everything built so far.
X