# The Ion data model (`moonrockz/ion/ion`)

The core package of `moonrockz/ion`: the thirteen Ion types and the values
that hold them, with no dependency on either encoding. The text and binary
codecs, Ion Hash, JSON conversion, and Ion Schema all build on it.

- `Ion` is the node type: an `IonValueKind` plus an ordered list of
  annotations. `IonValue` remains an alias for the same type. Build values
  with `Ion(value)` and read them with `kind()`. Explicit helpers such as
  `Ion::typed_null`, `Ion::sexp` and `Ion::from_fields` select Ion-specific forms.
- `Decimal` and integers are exact (`BigInt` coefficients), so precision and
  negative zero survive every round trip.
- `Timestamp` keeps the components it was given; its precision follows from
  which ones are present.
- `SymbolToken` has optional text and an optional symbol ID, and
  `SymbolTable` maps IDs to text as a stream's symbol tables declare them.
- `IonVisitor` and `IonFold` traverse a value depth-first.
- Every failure is an `IonError`: `Syntax`, `DataModel`, `Unsupported`, or
  `FuelExhausted`.
- Nesting is unrestricted by default. Readers and schema operations accept
  optional `Fuel::Limited(Budget(...))` for defensive coding. Fuel bounds
  open nested operations, with siblings reusing the enclosing budget.
  Exhaustion carries the budget, operation and location.
- Readers, equality, folds, visitors, writers, events, Ion Hash and JSON
  conversion keep a bounded machine stack at any nesting depth.

## Building values

Values are immutable. Equivalence follows the Ion data model: annotations
count, struct fields form an unordered multiset, and all NaNs are equivalent.

`Ion(value)` uses the open `ToIon` trait. Integers retain their full precision,
floating-point values become Ion floats, and strings become Ion strings.
Arrays become lists, string-keyed maps become structs in iteration order,
bytes become blobs, and unit becomes untyped null. Decimals, timestamps and
symbol tokens keep their Ion types. Constructing from an existing Ion value
preserves its annotations. Core `Json` values follow the JSON-to-Ion number
mapping documented in the JSON package.

The short helpers `decimal`, `timestamp` and `symbol` construct `Decimal`,
`Timestamp` and `SymbolToken` payloads. Wrap a payload with `Ion(...)` to
construct an Ion value. The `field` helper constructs a `Field` from a name
and an Ion value.

```mbt check
///|
test "construct Ion values from MoonBit values" {
  let counts : @ion.IonValue = @ion.Ion(Map([("counts", [1, 2, 3])]))
  inspect(@text.write_ion(counts), content="{counts: [1, 2, 3]}")
  let json = @ion.Ion(Json({ "ready": true }))
  inspect(@text.write_ion(json), content="{ready: true}")
  let decimal = @ion.decimal(123N, -2)
  inspect(@text.write_ion(@ion.Ion(decimal)), content="1.23")
  let timestamp = @ion.timestamp(year=2024)
  inspect(@text.write_ion(@ion.Ion(timestamp)), content="2024T")
  let field = @ion.field("status", @ion.Ion(@ion.symbol("ready")))
  inspect(
    @text.write_ion(@ion.Ion::from_fields([field])),
    content="{status: ready}",
  )
}
```

Implement `ToIon` for application types. Callers use the same constructor:

```mbt check
///|
struct ExampleCount {
  count : Int32
}

///|
impl @ion.ToIon for ExampleCount with fn to_ion(self) -> @ion.Ion {
  @ion.Ion(Map([("count", self.count)]))
}

///|
test "construct Ion from an application type" {
  inspect(
    @text.write_ion(@ion.Ion(ExampleCount::{ count: 3, })),
    content="{count: 3}",
  )
}
```

The old `IonValue::new`, `Field::new`, `Decimal::new`, `Timestamp::new` and
`SymbolToken::new` calls remain available with deprecation warnings. Use
`Ion(kind)`, `field(...)`, `decimal(...)`, `timestamp(...)` and `symbol(...)`.
The short helpers alias the named constructors, which also remain available.

```mbt check
///|
test "build a value and compare it" {
  let book = @ion.IonValue::from_fields([
    @ion.field("title", @ion.IonValue::string("Ion")),
    @ion.field(
      "tags",
      @ion.IonValue::list([
        @ion.IonValue::symbol("data"),
        @ion.IonValue::symbol("format"),
      ]),
    ),
  ]).with_annotation(@ion.symbol("book"))
  inspect(book.ion_type().to_text(), content="struct")
  debug_inspect(
    book.annotations().map(token => token.text()),
    content=(
      #|[Some("book")]
    ),
  )
  // Field order does not matter to equivalence.
  let reordered = @ion.IonValue::from_fields([
    @ion.field(
      "tags",
      @ion.IonValue::list([
        @ion.IonValue::symbol("data"),
        @ion.IonValue::symbol("format"),
      ]),
    ),
    @ion.field("title", @ion.IonValue::string("Ion")),
  ]).with_annotation(@ion.symbol("book"))
  assert_true(book.equals(reordered))
  // Annotations do.
  assert_false(book.equals(reordered.with_annotations([])))
}
```

Match on `kind()` to read a value:

```mbt check
///|
test "read a value by its kind" {
  let value = @text.read_ion("{ name: \"Ada\", born: 1815 }")
  match value.kind() {
    Struct(fields) =>
      debug_inspect(
        fields.map(field => field.name().text().unwrap_or("?")),
        content=(
          #|["name", "born"]
        ),
      )
    _ => fail("expected a struct")
  }
}
```

## Exact numbers

A decimal keeps its coefficient and exponent as written, so `1.50` and `1.5`
are different Ion values, and so are `0.0` and `-0.0`. Arithmetic goes
through `moonbitlang/x/decimal`, and returns `None` for what that type cannot
represent.

```mbt check
///|
test "decimals keep their precision" {
  let price = @ion.Decimal::parse("1.50")
  inspect(price.coefficient(), content="150")
  inspect(price.exponent(), content="-2")
  assert_false(
    @ion.IonValue::decimal(price).equals(
      @ion.IonValue::decimal(@ion.Decimal::parse("1.5")),
    ),
  )
  inspect(@ion.Decimal::parse("-0.0").is_negative_zero(), content="true")
  debug_inspect(
    price.add(@ion.Decimal::parse("0.25")).map(sum => sum.to_ion_string()),
    content=(
      #|Some("1.75")
    ),
  )
}
```

## Timestamps

A timestamp has the precision its components give it, and an optional offset.
`validate` checks that it describes a real moment, which the readers apply to
every timestamp they read.

```mbt check
///|
test "timestamps" {
  let launch = @ion.Timestamp::parse("1969-07-20T20:17:40Z")
  debug_inspect(launch.precision(), content="Second")
  inspect(launch.to_ion_string(), content="1969-07-20T20:17:40Z")
  // The offset is kept, in minutes; `-00:00` is an unknown offset.
  let eastern = @ion.Timestamp::parse("2024-01-01T01:30+02:00")
  debug_inspect(eastern.offset(), content="Some(Known(120))")
  debug_inspect(
    @ion.Timestamp::parse("2024-01-01T01:30-00:00").offset(),
    content="Some(Unknown)",
  )
  // 2001 was not a leap year.
  let invalid = @ion.timestamp(year=2001, month=Some(2), day=Some(29))
  let message = try invalid.validate() catch {
    error => error.message()
  } noraise {
    _ => "valid"
  }
  inspect(message, content="timestamp day is out of range for its month")
}
```

`with_offset` moves a timestamp to another offset, keeping the instant it
names, and `as_utc` moves it to `Z`. `compare_instant` compares the instants
two timestamps name, whatever their precision and offset; `equals` compares
Ion values, which differ when the precision or the offset does.

```mbt check
///|
test "offsets and instants" {
  let eastern = @ion.Timestamp::parse("2024-01-01T01:30+02:00")
  inspect(eastern.as_utc().to_ion_string(), content="2023-12-31T23:30Z")
  inspect(
    eastern.with_offset(@ion.TimestampOffset::Known(-300)).to_ion_string(),
    content="2023-12-31T18:30-05:00",
  )
  let midnight = @ion.Timestamp::parse("2024-01-01")
  let also_midnight = @ion.Timestamp::parse("2024-01-01T01:00:00.000+01:00")
  inspect(midnight.compare_instant(also_midnight), content="0")
  assert_false(
    @ion.IonValue::timestamp(midnight).equals(
      @ion.IonValue::timestamp(also_midnight),
    ),
  )
}
```

## Symbols and symbol tables

A symbol has text, a symbol ID, or both. The Ion 1.0 system symbols take IDs
1 to 9; a local symbol table declares more from 10. The readers keep a
`SymbolTable` per stream and resolve IDs against it.

```mbt check
///|
test "a symbol table resolves symbol IDs" {
  let table = @ion.SymbolTable::system()
  debug_inspect(table.text(4), content="Some(\"name\")")
  let declaration = @text.read_ion("{ symbols: [\"red\", \"green\"] }").with_annotation(
    @ion.symbol("$ion_symbol_table"),
  )
  assert_true(table.apply_local_symbol_table(declaration))
  inspect(table.max_id(), content="11")
  debug_inspect(
    table.lookup(11).text(),
    content=(
      #|Some("green")
    ),
  )
  // An ID beyond the table is an error.
  let beyond = try table.lookup(12) catch {
    error => error.message()
  } noraise {
    token => token.text().unwrap_or("?")
  }
  inspect(
    beyond,
    content="symbol ID $12 is beyond the symbol table, whose largest ID is 11",
  )
}
```

A local symbol table can import shared symbol tables by name and version.
The readers resolve them against a `Catalog`, which a program loads from its
`$ion_shared_symbol_table::{...}` structs:

```mbt check
///|
test "imports resolve against a catalog" {
  let catalog = @ion.Catalog::load(
    @text.read_ion_datagram(
      "$ion_shared_symbol_table::{ name: \"colors\", version: 1, symbols: [\"red\", \"green\"] }",
    ),
  )
  let values = @text.read_ion_datagram(
    "$ion_symbol_table::{ imports: [{ name: \"colors\", version: 1 }], symbols: [\"local\"] } $11 $12",
    catalog~,
  )
  inspect(
    @text.write_all(values),
    content=(
      #|green
      #|local
    ),
  )
}
```

Unknown symbols imported from a shared table retain an `import_location`
with its name and one-based slot. `SymbolToken::equivalent` and
`IonValue::equals` use that location, ignoring the stream-local ID and table
version. Unknown local symbols remain equivalent to `$0`. The token's derived
`Eq` and `Hash` compare its complete representation, including reader metadata.
Shared-table `imports` fields are informational metadata; loading a catalog
does not resolve them or add slots for them.

```mbt check
///|
test "unknown imports keep their identity through text" {
  let symbol = @ion.SymbolToken::from_import(20, "example", 1, 2)
  let shifted = @ion.SymbolToken::from_import(30, "example", 2, 2)
  assert_true(symbol.equivalent(shifted))
  assert_false(symbol.equivalent(@ion.SymbolToken::from_sid(0)))
  let value = @ion.IonValue::symbol_token(symbol)
  assert_true(@text.read_ion(@text.write_ion(value)).equals(value))
}
```

Text and binary writers preserve the import name, version, and slot while
assigning output-local IDs. Unknown local IDs are reserved with null slots.
The incremental writers reset their context when an unknown imported symbol
needs a different table. A single writer context containing unknown symbols
from conflicting versions of one shared table raises `IonError::Unsupported`.
This applies to one text value or one binary datagram; write separate contexts
with the incremental writer when the versions differ between values.

## Traversal

`IonValue::accept` walks a value depth-first with an `IonVisitor`, whose
methods all default to doing nothing. `IonValue::fold` threads an accumulator
instead, and each callback can continue, skip a value's children, or stop.
None of these recurse: they work at any depth.

```mbt check
///|
struct FieldNames {
  names : Array[String]
}

///|
impl @ion.IonVisitor for FieldNames with fn visit_field_name(self, token) {
  self.names.push(token.text().unwrap_or("?"))
}

///|
test "visit every field name" {
  let value = @text.read_ion("{ a: 1, b: { c: 2 }, d: [ { e: 3 } ] }")
  let visitor : FieldNames = { names: [], }
  value.accept(visitor)
  debug_inspect(
    visitor.names,
    content=(
      #|["a", "b", "c", "d", "e"]
    ),
  )
}

///|
test "sum the integers outside annotated values" {
  let value = @text.read_ion("[1, 2, skip::[100, 200], (3 4)]")
  let sum = value.fold(0, {
    ..@ion.IonFold::default(),
    visit_value: (total, value) => {
      if value.has_annotations() {
        return SkipChildren(total)
      }
      match value.kind() {
        Int(n) => Continue(total + n.to_int())
        _ => Continue(total)
      }
    },
  })
  inspect(sum, content="10")
}
```

`IonValue::walk` gives the same depth-first order as events: `Enter` and
`Exit` around each list, S-expression and struct, and `Scalar` for every
other value, each with its struct field name. `IonValue::walker` gives the
walk as a pull-based `ValueWalker`, which can also skip a container's
children. `IonValue::fold_up` folds bottom-up: each container gets its
children's results. None of them recurse, so they work at any depth.

```mbt check
///|
test "walk a value as events" {
  let value = @text.read_ion("{ a: [1, 2], b: null.list }")
  let events = value
    .walk()
    .map(event => {
      match event {
        Enter(v, ..) => "enter " + v.ion_type().to_text()
        Exit(v, ..) => "exit " + v.ion_type().to_text()
        Scalar(v, ..) => "scalar " + v.ion_type().to_text()
      }
    })
    .collect()
  debug_inspect(
    events,
    content=(
      #|[
      #|  "enter struct",
      #|  "enter list",
      #|  "scalar int",
      #|  "scalar int",
      #|  "exit list",
      #|  "scalar list",
      #|  "exit struct",
      #|]
    ),
  )
}

///|
test "skip the children of annotated containers" {
  let walker = @text.read_ion("[1, skip::[2, 3], 4]").walker()
  let ints : Array[String] = []
  while walker.next() is Some(event) {
    match event {
      Enter(value, ..) => if value.has_annotations() { walker.skip_children() }
      Scalar(value, ..) =>
        if value.kind() is Int(n) {
          ints.push(n.to_string())
        }
      Exit(_, ..) => ()
    }
  }
  debug_inspect(
    ints,
    content=(
      #|["1", "4"]
    ),
  )
}

///|
test "measure the depth with fold_up" {
  let value = @text.read_ion("[1, [2, [3]], {a: (4)}]")
  let depth = value.fold_up(scalar=_ => 0, container=(_, children) => {
    let mut deepest = 0
    for child in children {
      if child.1 > deepest {
        deepest = child.1
      }
    }
    deepest + 1
  })
  inspect(depth, content="3")
}
```