Reading and writing data formats

@j50n/proc/transforms turns bytes into rows and rows back into bytes, a batch at a time, so a CSV export or a process’s output streams through your code without being read whole into memory.

import { read } from "@j50n/proc";
import { fromCsvToRows, toTsv } from "@j50n/proc/transforms";

await read("data-orders.csv")
  .transform(fromCsvToRows()) // bytes in, batches of rows out
  .flatten() // one row at a time
  .filter((row) => row[3] !== "1") // row is a string[]
  .transform(toTsv()) // rows in, bytes out
  .toStdout();

id	customer	item	qty
1	Ada	Widget, large	2
3	Linus	Bolt	10

Each function returns a transformer for .transform(). A parser (fromCsvToRows(), …) takes bytes from read(), a command’s output, or any async iterable of Uint8Array. A writer (toTsv(), …) yields bytes, ready for .writeTo(), .toStdout(), or the stdin of the next command with .run(). The transforms are a separate entry point, so import them from "@j50n/proc/transforms".

Which format

• CSV to exchange data with spreadsheets, databases, and other people. Its parser is lenient; CSV says exactly what it accepts.
• TSV for simple tabular text whose fields never hold a tab or line break: logs, command output, files you grep.
• JSON lines when each item is an object (or any JSON value) rather than a row of strings.
• Record to hand rows to another program when fields may hold tabs, quotes, or newlines. Any language splits it with two calls.

To convert, chain any parser to any writer: .transform(fromCsvToRows()) then .transform(toRecord()). The row writers take what the row parsers yield, batches and LazyRows included. Between CSV and TSV, csvToTsv() and tsvToCsv() convert bytes to bytes without making rows, several times faster; see CSV. JSON is the exception, since it holds values rather than rows: map rows to objects before toJson(), and objects to arrays of strings after fromJsonToRows(). JSON lines shows both.

Batches

Parsers yield batches (arrays of rows), not single rows. Add .flatten() before a step that works on one row (filter, map, take), as in Key ideas. The row writers take a row or a batch per item, so there is no need to flatten just to write. toJson() takes one value per item, so flatten before it; see JSON lines.

Row or LazyRow

A Row is a string[], every field decoded. A LazyRow from fromCsvToLazyRows() or fromTsvToLazyRows() holds the row’s bytes and decodes a field when you ask for it with getField(), or compares one without decoding it with fieldEquals(). Use plain rows unless you filter on or read a few fields of each row; LazyRow has the details.

Header rows

No format has a header. A header line is the first row, like any other, so in the example above it went through the filter and into the TSV. Skip it with .drop(1), or keep it to name the fields:

import { read } from "@j50n/proc";
import { fromCsvToRows } from "@j50n/proc/transforms";

// The header is the first row, like any other.
let header: string[] = [];
const orders = await read("data-orders.csv")
  .transform(fromCsvToRows())
  .flatten()
  .enum()
  .filter(([row, index]) => {
    if (index === 0) header = row;
    return index > 0;
  })
  .map(([row]) => Object.fromEntries(header.map((name, i) => [name, row[i]])))
  .collect();

console.log(orders);

// Or skip it.
const quantities = await read("data-orders.csv")
  .transform(fromCsvToRows())
  .flatten()
  .drop(1)
  .map((row) => Number(row[3]))
  .collect();

console.log(quantities);

[
  { id: "1", customer: "Ada", item: "Widget, large", qty: "2" },
  { id: "2", customer: "Grace", item: 'Gear "XL"', qty: "1" },
  { id: "3", customer: "Linus", item: "Bolt", qty: "10" }
]
[ 2, 1, 10 ]

enum() pairs each row with its index, counted from 0. Every field is a string; convert numbers yourself, and turn them back into strings before writing.

To write a header in front of rows, put it first: enumerate([header]).concat(rows), where rows is any Enumerable of rows.

The CSV and TSV parsers, and csvToTsv() and tsvToCsv(), run in WebAssembly bundled with the package; the rest is TypeScript. None of them needs permissions of its own; only read() and writeTo() do.

What writers refuse

A writer throws rather than write a field its format can’t hold, since the reader would split it into extra fields or rows:

import { enumerate } from "@j50n/proc";
import { toCsv, toTsv } from "@j50n/proc/transforms";

const rows = [["id", "note"], ["1", "line one\nline two"]];

try {
  await enumerate(rows).transform(toTsv()).toStdout();
} catch (error) {
  if (error instanceof Error) console.log(`[${error.message}]`);
}

// CSV can hold it: the field is quoted.
await enumerate(rows).transform(toCsv()).toStdout();

id	note
[Invalid character (LF) in TSV data at row 2, field 2]
id,note
1,"line one
line two"

The Error names the row and field, both counted from 1 across the whole stream. Items before the bad one have already been written, so a file you were writing holds the rows up to that point. toCsv() refuses no field.

A writer also refuses a row that would read back as no row, or as a different one, with an error such as Invalid row (no fields) in CSV data at row 3:

csvToTsv() refuses a CSV row of one empty field, as toTsv() does. Every row writer refuses a field holding a lone surrogate, which UTF-8 can’t hold (TextEncoder would write U+FFFD in its place). A first field starting with U+FEFF would be dropped by the reader as a byte order mark, so toCsv() and tsvToCsv() quote it and the others refuse it.

What parsers refuse

The CSV and TSV parsers take lines ending in LF or CRLF. Any other CR outside a quoted field, as in a file with old Mac CR-only line ends, throws an Error such as Invalid character (CR) in CSV data at row 1, field 2, rather than read the file as one long row. The CSV parser throws on a quote still open at the end of the input, Unclosed quote in CSV data at row 7, field 3, rather than make the rest of the file one field. Invalid UTF-8 throws a TypeError such as Invalid UTF-8 in CSV data at row 3001, field 2 (a file saved as Latin-1 or Windows-1252 is the usual cause) from the parser, or, for a LazyRow, when the field is decoded. Batches before the one holding the error have already gone down the pipeline.

A parser’s row numbers count rows, from 1, with the header row included. Blank lines are skipped and not counted, and a quoted CSV field can span lines, so a row number is the line number only in a file with neither.

The flatdata CLI converts between the same formats in a separate process, with the same checks.

How fast

From benchmarks/transforms-throughput.ts: 100,000 rows of 20 fields (UTF-8, about one field in ten quoted, some with newlines), median MB/s on one machine. Your numbers will differ; the ratios are what matter.

The LazyRow parsers are fast because they make no strings until asked, and converting between CSV and TSV never makes any.

Lots of rows

The figures above are for a program that handles each batch with plain code. A step after .flatten() (filter, map, forEach) is an await per row, about a microsecond each, which on small rows costs more than the parsing. With millions of rows, work a batch at a time, with the array’s own methods inside one step:

import { enumerate } from "@j50n/proc";
import { fromCsvToLazyRows } from "@j50n/proc/transforms";

// 100,000 orders; every tenth is from New Zealand.
const csv = Array.from(
  { length: 100_000 },
  (_, i) => `${i},customer${i},${i % 10 === 0 ? "NZ" : "US"},${i % 97}\n`,
).join("");
const orders = () =>
  enumerate([new TextEncoder().encode(csv)]).transform(fromCsvToLazyRows());

// Row by row: every step after .flatten() is an await per row.
const rowByRow = await orders()
  .flatten()
  .filter((row) => row.fieldEquals(2, "NZ"))
  .count();

// A batch at a time: plain array methods inside one step per batch.
const byBatch = await orders()
  .map((rows) => rows.filter((row) => row.fieldEquals(2, "NZ")))
  .flatten()
  .count();

console.log(rowByRow, byBatch);

10000 10000

Both count the same rows. On 100,000 rows of 20 fields, the filter ran at about 290 MB/s a batch at a time and 135 MB/s row by row; on rows of 25 bytes, row by row fell to about 25 MB/s. It is the same advice as .chunkedLines for lines of text.

Against other JavaScript

From benchmarks/compare.ts, on the same data. Here every reader keeps all 2,000,000 fields in memory, as a whole-string parser must, so the figures are lower than the table above. The others get their best case, the whole file decoded to one string; proc streams it in 64 KB chunks.

Converting CSV to TSV, proc’s csvToTsv() runs at about 540 MB/s here; reading with the hand-written parser and joining the rows back runs at 30. Reading TSV, proc and a plain split on the whole string are close (72 and 59): TSV has no quoting to work out, so making the strings is nearly all the work.