Key ideas

Eight points cover how every part of proc behaves. The rest of the book is detail.

1. A pipeline is a source, some steps, and a consumer

import { read } from "@j50n/proc";

const errors = await read("app.log") // a source: the file's bytes
  .lines // a step: bytes to lines of text
  .filter((line) => line.includes("ERROR")) // another step
  .count(); // the consumer: pulls everything through

console.log(errors);

A source produces items: run() (a command’s output), read() (a file), or enumerate() (any iterable or async iterable). Each step (lines, map, filter, take, transform, …) returns a new Enumerable and does nothing yet; .run() is the exception, since it starts its command at once (point 2). The consumer (collect, forEach, count, reduce, first, writeTo, toStdout, or a for await loop) pulls items through one at a time and returns a promise. Await it.

2. A command starts when you call `run()`, and you must read its output

run() starts the child process at once; only the steps after it wait. The child’s stdout is a pipe with a small buffer (typically 64 KB). A child that writes more than that waits until someone reads, so if your code never consumes the output, the child never finishes and neither does your program. Always end a command’s pipeline with a consumer. If you don’t want the output, consume it anyway: await run("make").lines.forEach(() => {}).

stderr is not piped by default: it goes straight to your terminal.

3. Errors come out of the consumer’s `await`

import { ExitCodeError, run } from "@j50n/proc";

try {
  await run("sh", "-c", "echo one; echo two; exit 3")
    .lines
    .forEach((line) => console.log(line));
} catch (error) {
  if (error instanceof ExitCodeError) {
    console.log(`failed with exit code ${error.code}`);
  } else {
    throw error;
  }
}

one
two
failed with exit code 3

A command that exits with a non-zero code throws ExitCodeError once you have read all of its output, so the lines it wrote before failing still reach you. The same catch also gets an error from any command earlier in a pipeline (as an UpstreamError whose cause is the original), and anything your own callbacks throw. One try around the consumer covers the whole pipeline. See Errors.

4. Stopping early is fine

import { run } from "@j50n/proc";

// `yes` prints "y" forever; taking three stops it.
const ys = await run("yes").lines.take(3).collect();
console.log(ys);

[ "y", "y", "y" ]

When a consumer stops before the end (take, first, find, a break out of for await), proc closes the pipeline behind it, and the await returns at once, without an error. A command that is still writing dies at its next write; one that runs on quietly, like a server that printed “ready”, keeps running until it exits, or until main() stops it on the way out. Deno doesn’t exit while a child is running, so without main() the script ends only when that child does.

5. An `Enumerable` is used once

import { enumerate } from "@j50n/proc";

const doubled = enumerate([1, 2, 3]).map((n) => n * 2);

console.log(await doubled.collect());
console.log(await doubled.collect()); // already used up: nothing left

const saved = await enumerate([1, 2, 3]).map((n) => n * 2).collect();
console.log(await enumerate(saved).count(), await enumerate(saved).count());

[ 2, 4, 6 ]
[]
3 3

Consuming an Enumerable uses it up, and a second pass finds nothing, without an error. To go over data twice, collect it into an array first, or split the stream with tee().

6. Some members are properties

.lines, .chunkedLines, .first, .status, and .pid take no parentheses. .first and .status are promises: await p.status, not p.status(). Everything else is a method.

7. Parsers yield batches

import { read } from "@j50n/proc";
import { fromCsvToRows } from "@j50n/proc/transforms";

const rows = await read("people.csv")
  .transform(fromCsvToRows()) // yields batches: arrays of rows
  .flatten() // one row at a time
  .collect();

console.log(rows);

[ [ "name", "city" ], [ "Ada", "London" ], [ "Grace", "Arlington" ] ]

For speed, the parsers in @j50n/proc/transforms yield arrays of rows rather than one row at a time. Add .flatten() before a step that works on one row. The row writers (toCsv(), toTsv(), toRecord()) take either; toJson() takes one value per item.

8. Wrap a long-running program in `main()`

import { main, run } from "@j50n/proc";

await main(async () => {
  await run("./long-job.sh").lines.toStdout();
});

When Deno is told to stop (Ctrl-C, or a container’s SIGTERM), it exits at once without waiting for its children; in a container they are killed with it before they can clean up. When an error goes uncaught, Deno kills its children as it exits, on any host. main() runs your program, and however it ends (it returns, it throws, or a signal arrives), it makes sure every child has been asked to stop, and waits for them, up to 30 seconds, before exiting. See Shutting down cleanly.

Keyboard shortcuts

proc