# Independent row-group fixtures

All rows are synthetic. The generated data, generator, and development footer
editor are licensed Apache-2.0. Existing upstream fixtures elsewhere are not
modified or relicensed. These files exercise the MoonBit reader and the external
consumer; the generator contains no statistics pruning implementation.

The original seven research fixtures were replayed byte-for-byte: `base`,
`no_statistics`, `zero_rows`, `unsupported_float`, `unsupported_string`,
`unsupported_uint32`, and `poison_skipped`. Their SHA256 values match the
independently generated research manifest. `poison_skipped.parquet` is explicitly
invalid: group 0, column 0 has its first page-header byte changed from `15` to
`00`. Its footer is untouched, group 2 independently reads correctly, and a full
PyArrow read fails.

## Reproduction

Pinned tools: CPython 3.13.5, PyArrow 18.0.0, NumPy 2.2.4, Apache Thrift 0.22.0.
Install these development dependencies in an environment dedicated to fixtures:

```powershell
python -m pip install pyarrow==18.0.0 numpy==2.2.4 thrift==0.22.0
python -c "import subprocess,sys; subprocess.run([sys.executable,'scripts/generate_row_group_fixtures.py'],check=True,timeout=60)"
python -c "import subprocess,sys; subprocess.run([sys.executable,'scripts/generate_row_group_fixtures.py','--check'],check=True,timeout=60)"
```

`--check` regenerates all 32 files and the manifest in a temporary directory and
compares every generated byte and the complete generated file set. `--output`
can generate to a separate directory. The manifest has deterministic provenance
instead of a variable timestamp: commands and their SHA256, generator source
SHA256, package versions, runtime source SHA256, per-file SHA256, writer options,
schema, exact original rows, and actual raw footer fields.

This Windows environment's initial Thrift native-extension build exceeded the
60-second limit. Validation used the official 0.22.0 source distribution's
unmodified pure Python `src` package in a temporary `thrift` directory, with its
`thrift.egg-info` and `PYTHONPATH` set to that directory. No existing dependencies
were upgraded. The pinned source archive is `thrift-0.22.0.tar.gz`, SHA256
`42e8276afbd5f54fe1d364858b6877bc5e5a4a5ed69f6a005b94ca4918fe1466`.
The same pure Python `TCompactProtocol` runs when the package is installed
normally; the optional fastbinary extension is not used by this editor.

Protocol fields are taken from
[Apache parquet-format at 53b8bd8576827080d4382fa457003f45b41cda6e](https://github.com/apache/parquet-format/blob/53b8bd8576827080d4382fa457003f45b41cda6e/src/main/thrift/parquet.thrift).
`scripts/compact_fixture.py` uses Apache Thrift's official `TCompactProtocol` to
read and write a schema-independent field tree. For every complete fixture,
decoding and re-encoding the footer must reproduce its bytes. Statistics variants
also assert that schema, page offsets, and all fields outside Statistics and
column_orders are unchanged. Structural negative cases intentionally alter the
stated schema/count fields, while still asserting unchanged codec, page sizes,
and page offsets. Every footer variant keeps its base's complete page prefix
byte-identical; the overflow case deliberately uses the poisoned prefix.

## Data and expected results

Base schema is nullable, unannotated `i32: INT32` and `i64: INT64`, 16 rows in four
groups of four. Each row aligns across these columns:

| Group | i32 | i64 |
| --- | --- | --- |
| 0 | -20, -19, null, -17 | -9007199254741003, -9007199254741002, null, -9007199254741001 |
| 1 | 0, 1, null, 3 | -1, 0, null, 2 |
| 2 | 100, 101, 102, 103 | 9007199254740993, 9007199254740995, 9007199254740997, 9007199254740999 |
| 3 | null, null, null, null | null, null, null, null |

All footer variants below retain that schema and those page bytes. Retention is
the conservative consumer expectation for `i32 >= 100`, followed by residual
filtering. The four matching rows are group 2 in every valid base variant.

| File | Footer change / expected metadata behavior | Retained groups |
| --- | --- | --- |
| base.parquet | Modern and legacy bounds, null counts, TYPE_ORDER | 2 |
| no_statistics.parquet | Statistics absent, including unknown null counts | 0, 1, 2, 3 |
| missing_null_count.parquet | Remove every null_count; all-null status becomes unknown | 2, 3 |
| legacy_only.parquet | Remove modern fields 5/6 and exact flags; preserve legacy 1/2 without substitution | 0, 1, 2 |
| min_only.parquet | Keep modern min_value, remove max_value field 5; retain legacy without substitution | 0, 1, 2 |
| max_only.parquet | Keep modern max_value, remove min_value field 6; retain legacy without substitution | 2 |
| missing_column_orders.parquet | Modern ordering undefined; preserve raw bounds | 0, 1, 2 |
| unknown_column_order.parquet | ColumnOrder union has unknown struct field 42 | 0, 1, 2 |
| inexact_wide_bounds.parquet | Every nonempty modern bound widened by 1; both exact flags false | 2 |
| zero_rows.parquet | Same schema, zero rows, one empty Arrow row group | none |
| poison_skipped.parquet | Intentionally corrupt group 0 page; metadata and selected group 2 succeed | 2 |

`inexact_wide_bounds` also has an oracle for `i32 >= 104`: retain group 2 because
the conservative upper bound is 104, then residual filtering produces zero rows.
False exact flags do not invalidate conservative bounds. Groups whose known
null_count equals num_values may be excluded even when their bounds/order are
unavailable. A missing null_count is always unknown, never zero.

The following statistics are intentionally invalid. Metadata parsing retains raw
binary fields so the typed signed view can validate lengths and ordering. These
are expected errors, not missing statistics and not a reason to silently scan.

| File | Exact mutation | Expected error stage |
| --- | --- | --- |
| bad_i32_bound_length.parquet | Group 0 i32 min_value is 3 bytes rather than 4 | signed_integer_statistics: InvalidData |
| bad_i64_bound_length.parquet | Group 0 i64 min_value is 7 bytes rather than 8 | signed_integer_statistics: InvalidData |
| all_null_bad_i32_bound_length.parquet | Group 3 i32 max_value is 3 bytes, despite null_count=4 | signed_integer_statistics: InvalidData |
| reversed_bounds.parquet | Group 0 i32 min_value=-16, max_value=-17 | signed_integer_statistics: InvalidData |
| negative_null_count.parquet | Group 0 i32 null_count=-1 | read_metadata: InvalidData |
| excess_null_count.parquet | Group 0 i32 null_count=5, num_values=4 | read_metadata: InvalidData |
| negative_distinct_count.parquet | Group 0 i32 distinct_count=-1 | read_metadata: InvalidData |

The structural negative cases below keep actual page offsets, codecs, and sizes.
They are verified through the official Thrift tree without invoking PyArrow on
intentionally inconsistent metadata.

| File | Exact mutation | Expected behavior |
| --- | --- | --- |
| schema_path_mismatch.parquet | Group 0 column 0 path is renamed_i32 rather than i32 | read_metadata: InvalidData |
| schema_type_mismatch.parquet | Group 0 column 0 is INT64 while leaf schema remains INT32 | read_metadata: InvalidData |
| column_order_count_mismatch.parquet | One ColumnOrder entry for two leaf columns | read_metadata: InvalidData |
| selected_row_count_overflow.parquet | Start with poison_skipped pages; group 0 num_rows and both chunk num_values become 2147483648; file num_rows becomes 2147483660 | Metadata and empty selection succeed; selection `[0]` reports Selected row count exceeds columnar batch Int capacity before the poison page is decoded |

The overflow file's claimed row counts are internally consistent at the footer
level. Actual page contents remain intentionally poisoned and do not contain the
claimed billions of rows. This isolates selection's Int capacity check from
structural count consistency and page decoding; it is not a valid full dataset.

The generator does not call PyArrow on intentionally invalid statistics. During
development, accessing the 3-byte INT32 statistics caused PyArrow 18 to terminate
the native process with Windows exit status 3221226505. Official Thrift round-trip
and unchanged layout/page assertions still run for all seven statistics files. Whether full
decoding consults the typed statistics API is outside these fixture assertions.

Unsupported type fixtures each have a nullable `value` column, 12 rows in three
groups, with the last group all null. Full PyArrow decoding is verified, while
the signed unannotated INT32/INT64 statistics view must return None:

| File | Arrow / physical schema | Distinguishing values |
| --- | --- | --- |
| unsupported_float.parquet | float32 / FLOAT | -2.5 through 103.5 |
| unsupported_string.parquet | string / BYTE_ARRAY + STRING | a, b, c, x, y, z |
| unsupported_uint32.parquet | uint32 / INT32 + unsigned INTEGER | 2147483648 through 4294967295 |
| unsupported_uint64.parquet | uint64 / INT64 + unsigned INTEGER | 9223372036854775808 through 18446744073709551615 |
| unsupported_decimal.parquet | decimal128(9,2) / INT32 + DECIMAL | -2.00 through 103.00 |
| unsupported_date.parquet | date32 / INT32 + DATE | 1969-12-30 through 2026-10-22 |

PyArrow 18 cannot expose decoded statistics for INT32 DECIMAL. Its full read is
verified, and actual raw decimal bounds and null counts are checked against the
source decimals scaled exactly by 100. This limitation is explicit in the
manifest; no decoded statistics are invented.

`bad_footer_magic`, `bad_footer_length`, `truncated_file`, and `truncated_footer`
are intentionally corrupt complete-file negative cases. They respectively
replace final magic, extend footer length before the header, remove six trailing
bytes, or remove the final Compact STOP while updating trailer length. PyArrow
metadata reads reject all four; MoonBit read_metadata must report InvalidData.

In `manifest.json`, `footer_fields` records actual binary fields, their presence,
legacy/modern separation, exact flags, logical annotations, ColumnOrder union
members, and counts. Missing fields are JSON null. INT64/UINT64 values and raw
counts are decimal strings; raw bounds are hex. These representations preserve
values beyond 2^53 without JSON-number rounding.

## Isolated decoder-work measurement

From the repository root, with the validated MoonBit toolchain root:

```powershell
python scripts/verify_decode_work.py --toolchain <MOON_HOME> --target js
```

The runner copies current sources and fixtures under OS TEMP, reuses installed
`.mooncakes` with `--frozen`, and changes only the private columnar chunk decoder
entry in the copy to trace actual page offsets. Its new whitebox test calls the
real public APIs and checks values and offset order as well as counts:
full=8, selected `[2]`=2, empty=0, ordered `[2,0]`=4, invalid `[0,4]`=0.
The invalid-index test uses the poisoned file to prove validation precedes page
decoding. Original decoder statements and return paths remain byte-identical.

Evidence is saved under `validation/decode-work-TARGET/`: source and fixture SHA,
the exact single-location patch, the generated whitebox test, command and hash,
toolchain versions, stdout/stderr, measured counts, and source-integrity checks.
The runner supports js, wasm, wasm-gc, and native. Every subprocess has a hard
60-second timeout. These measurements claim actual decoder entries only, without
unmeasured runtime, memory, or I/O benefits.