# Record serialization: is `records_to_columns` fast enough?

The question this page answers is whether the client's write path — build a
`Json` tree, then `Json::stringify` it — costs too much for large batches, and
whether it should be replaced by a hand-rolled streaming writer.

Short version: **it does not, at any batch size a Chroma server will accept.**
The measurements are below; the streaming alternative is about 8x faster in
isolation but saves tens of microseconds, while costing the column alignment
and null padding that make the body correct.

## What the write path does

`add` / `upsert` / `update` all call `records_to_columns(records)`, which walks
the batch once and builds four parallel `Array[Json]` columns (`ids`,
`embeddings`, `documents`, `metadatas`), then wraps them in one
`Json::object`. `ChromaClient::send` calls `.stringify()` on the result once,
producing the request body.

So a batch of *n* records costs: *n* small `Json` values per column, plus one
`stringify` pass over the whole tree.

## Measured

`bench_test.mbt` times `records_to_columns(...).stringify()` over a batch of
records with a 4-dimension embedding and a two-key metadata object each, and
compares it against a hand-written `StringBuilder` version that appends the
body directly. Both produce a comparable body; the streamed one is only
exercised by the benchmark.

Measured on a Linux amd64 container (CNB runner image, 8 vCPU, AMD EPYC 9754),
`native` target, 200 rounds per size, median of three runs, wall clock per
batch:

| records | body size | JSON tree | streamed |
| --- | --- | --- | --- |
| 1,000 | 49 KB | 2 us | not measured |
| 5,000 | 262 KB | 11 us | 1 us |
| 20,000 | 1.09 MB | 58 us | 8 us |

The streamed column is only filled in for the batch sizes `bench_batch_streamed`
runs; 1,000 is not one of them, so it reads "not measured" rather than a number
invented for the table. The 20,000 row is the one the ratio below uses: it is
the size where the difference is largest, so it is also the most generous
number the streaming alternative can claim.

Reproduce with:

```bash
# The benchmark is skipped unless CHROMA_BENCH=1 is set, so ordinary runs do
# not pay for it.
CHROMA_BENCH=1 moon test --target native -p SundayT1998/chroma-client
```

Two tests print numbers: `bench: serializing a large record batch` (the tree
path, all three sizes) and `bench: json tree versus a streamed body` (both
paths at 5,000 and 20,000 records, which is where the ratio comes from). The
numbers move by 10-20% between runs on a shared runner; the order of magnitude
and the single ratio above are stable, the microseconds are not.

## What that means

At 20,000 records the tree path spends **58 microseconds** preparing a
**1.09 MB** body — roughly 20 GB/s, which is memory-bandwidth territory and
nowhere near the cost of the request that follows it. A local HTTP round-trip
to Chroma is on the order of milliseconds; the serialization is two orders of
magnitude below it.

A batch that large is already unusual: Chroma's own max batch size default is
well below it, and a caller sending 20,000 vectors is paying far more for the
embedding computation that produced them and for the server's index update
than for the JSON.

The streamed writer is ~8x faster (8 us vs 58 us), so it would save 50
microseconds per 20,000-record write — under 0.01% of the round-trip, in
exchange for:

- hand-writing the JSON escaping and number formatting the `Json` package
  already does correctly;
- keeping the column/null-padding rules (`null` for a missing document or
  metadata, `null` for an `update` field left alone) in two places rather than
  one;
- a second body-building path that has to be kept in step with the first.

That trade is not worth making. The write path stays as it is.

## When to revisit

The measurement above is per-batch CPU, which is the part this client
controls. Two things would change the conclusion:

- **A much larger batch size.** If Chroma raised its limit into the
  hundreds of thousands of records per request, 58 us would become ~1 ms and
  the streamed path ~0.1 ms — still small next to the round-trip, but no
  longer free. Re-run the benchmark before assuming either way.
- **A real profile showing serialization in the hot path.** If an application
  profile ever puts `stringify` at the top of the stack, that is the evidence
  to act on, not this table.

Neither is true today, so no change is made.