# The Ion data model (`moonrockz/ion/ion`)

The core package of `moonrockz/ion`: the thirteen Ion types and the values
that hold them, with no dependency on either encoding. The text and binary
codecs, Ion Hash, JSON conversion, and Ion Schema all build on it.

- `Ion` is the node type: an `IonValueKind` plus an ordered list of
  annotations. `IonValue` remains an alias for the same type. Build values
  with `Ion(value)` and read them with `kind()`. Explicit helpers such as
  `Ion::typed_null`, `Ion::sexp` and `Ion::from_fields` select Ion-specific forms.
- `Decimal` and integers are exact (`BigInt` coefficients), so precision and
  negative zero survive every round trip.
- `Timestamp` keeps the components it was given; its precision follows from
  which ones are present.
- `SymbolToken` has optional text and an optional symbol ID, and
  `SymbolTable` maps IDs to text as a stream's symbol tables declare them.
- `IonVisitor` and `IonFold` traverse a value depth-first.
- `FromIon` decodes MoonBit values from Ion, the inverse of `ToIon`.
- Every read, write or validation failure is an `IonError`: `Syntax`,
  `DataModel`, `Unsupported`, or `FuelExhausted`. A decode failure is a
  `FromIonError` that names the path to the value that failed.
- Nesting is unrestricted by default. Readers and schema operations accept
  optional `Fuel::Limited(Budget(...))` for defensive coding. Fuel bounds
  open nested operations, with siblings reusing the enclosing budget.
  Exhaustion carries the budget, operation and location.
- Readers, equality, folds, visitors, writers, events, Ion Hash and JSON
  conversion keep a bounded machine stack at any nesting depth.

## Building values

Values are immutable. Equivalence follows the Ion data model: annotations
count, struct fields form an unordered multiset, and all NaNs are equivalent.

`Ion(value)` uses the open `ToIon` trait. Integers retain their full precision,
floating-point values become Ion floats, and strings become Ion strings.
Arrays become lists, string-keyed maps become structs in iteration order,
bytes become blobs, and unit becomes untyped null. Decimals, timestamps and
symbol tokens keep their Ion types. Constructing from an existing Ion value
preserves its annotations. Core `Json` values follow the JSON-to-Ion number
mapping documented in the JSON package.

The short helpers `decimal`, `timestamp` and `symbol` construct `Decimal`,
`Timestamp` and `SymbolToken` payloads. Wrap a payload with `Ion(...)` to
construct an Ion value. The `field` helper constructs a `Field` from a name
and an Ion value.

```mbt check
///|
test "construct Ion values from MoonBit values" {
  let counts : @ion.IonValue = @ion.Ion(Map([("counts", [1, 2, 3])]))
  inspect(@text.write_ion(counts), content="{counts: [1, 2, 3]}")
  let json = @ion.Ion(Json({ "ready": true }))
  inspect(@text.write_ion(json), content="{ready: true}")
  let decimal = @ion.decimal(123N, -2)
  inspect(@text.write_ion(@ion.Ion(decimal)), content="1.23")
  let timestamp = @ion.timestamp(year=2024)
  inspect(@text.write_ion(@ion.Ion(timestamp)), content="2024T")
  let field = @ion.field("status", @ion.Ion(@ion.symbol("ready")))
  inspect(
    @text.write_ion(@ion.Ion::from_fields([field])),
    content="{status: ready}",
  )
}
```

Implement `ToIon` for application types. Callers use the same constructor:

```mbt check
///|
struct ExampleCount {
  count : Int32
}

///|
impl @ion.ToIon for ExampleCount with fn to_ion(self) -> @ion.Ion {
  @ion.Ion(Map([("count", self.count)]))
}

///|
test "construct Ion from an application type" {
  inspect(
    @text.write_ion(@ion.Ion(ExampleCount::{ count: 3, })),
    content="{count: 3}",
  )
}
```

The old `IonValue::new`, `Field::new`, `Decimal::new`, `Timestamp::new` and
`SymbolToken::new` calls remain available with deprecation warnings. Use
`Ion(kind)`, `field(...)`, `decimal(...)`, `timestamp(...)` and `symbol(...)`.
The short helpers alias the named constructors, which also remain available.

```mbt check
///|
test "build a value and compare it" {
  let book = @ion.IonValue::from_fields([
    @ion.field("title", @ion.IonValue::string("Ion")),
    @ion.field(
      "tags",
      @ion.IonValue::list([
        @ion.IonValue::symbol("data"),
        @ion.IonValue::symbol("format"),
      ]),
    ),
  ]).with_annotation(@ion.symbol("book"))
  inspect(book.ion_type().to_text(), content="struct")
  debug_inspect(
    book.annotations().map(token => token.text()),
    content=(
      #|[Some("book")]
    ),
  )
  // Field order does not matter to equivalence.
  let reordered = @ion.IonValue::from_fields([
    @ion.field(
      "tags",
      @ion.IonValue::list([
        @ion.IonValue::symbol("data"),
        @ion.IonValue::symbol("format"),
      ]),
    ),
    @ion.field("title", @ion.IonValue::string("Ion")),
  ]).with_annotation(@ion.symbol("book"))
  assert_true(book.equals(reordered))
  // Annotations do.
  assert_false(book.equals(reordered.with_annotations([])))
}
```

Match on `kind()` to read a value:

```mbt check
///|
test "read a value by its kind" {
  let value = @text.read_ion("{ name: \"Ada\", born: 1815 }")
  match value.kind() {
    Struct(fields) =>
      debug_inspect(
        fields.map(field => field.name().text().unwrap_or("?")),
        content=(
          #|["name", "born"]
        ),
      )
    _ => fail("expected a struct")
  }
}
```

## Decoding values

`FromIon` is the inverse of `ToIon`. Call it with the type you want:
`let x : T = @ion.FromIon::from_ion(value)`. Each type reads exactly the Ion
type that its `ToIon` writes, and nothing else:

| MoonBit type | Ion type |
|---|---|
| `Unit` | untyped `null` |
| `Bool` | `bool` |
| `Int`, `Int64`, `UInt`, `UInt64`, `Byte`, `BigInt` | `int`, in the type's range |
| `Float`, `Double` | `float` (`Float` rounds to nearest) |
| `Decimal`, `Timestamp` | `decimal`, `timestamp` |
| `String`, `Char` | `string` (`Char`: exactly one character) |
| `SymbolToken` | `symbol` |
| `Bytes` | `blob` |
| `Array[T]`, `(A, B)`, `(A, B, C)` | `list` (tuples: exactly that length) |
| `Map[String, T]` | `struct` with unique field names that have text |
| `T?` | any null is `None`; any other value is `Some` |
| `Ion` | any value, unchanged |
| `Json` | any value, by the JSON package's down-conversion |

So a symbol is not a `String`, a decimal is not a `Double`, an int is not a
`Decimal`, a clob is not `Bytes` and a typed null such as `null.int` is not an
`Int`. Annotations on the decoded value are ignored. `None` writes the untyped
`null`, so `Some(x)` reads back as `None` when `x` itself writes a null.

```mbt check
///|
test "decode MoonBit values from Ion" {
  let value = @text.read_ion("{name: \"Ada\", born: 1815, tags: [\"math\"]}")
  let fields : Map[String, @ion.Ion] = @ion.FromIon::from_ion(value)
  let born : Int32 = @ion.FromIon::from_ion(fields.get("born").unwrap())
  assert_eq(born, 1815)
  let pair : (Int32, String?) = @ion.FromIon::from_ion(
    @text.read_ion("[1, null]"),
  )
  assert_eq(pair, (1, None))
}
```

A failure raises `FromIonError`. Every case carries the `path` from the
decoded root to the value that failed, as field names and list indexes:
`TypeMismatch` keeps the expected Ion type and the actual kind (a typed null
is `Null(Int)`), `OutOfRange` states the range, `WrongLength` the length,
`DuplicateField` the name, `MissingSymbolText` the token, and `JsonDecode`
the message of core's JSON decoder. `message()` renders the path in `$`
notation:

```mbt check
///|
test "a decode error names the path to the value" {
  let value = @text.read_ion("{events: [1, null.int]}")
  let error = try
    (@ion.FromIon::from_ion(value) : Map[String, Array[Int32]])
  catch {
    error => error
  } noraise {
    _ => fail("expected a decode error")
  }
  assert_true(error is TypeMismatch(expected=Int, actual=Null(Int), ..))
  inspect(error.message(), content="$.events[1]: expected int, found null.int")
}
```

Implement `FromIon` for application types. Use `within` to add a field name
to the path of an error from a field. Here a missing field reads as `null`,
so it fails as a type mismatch at that field:

```mbt check
///|
impl @ion.FromIon for ExampleCount with fn from_ion(value) {
  let fields : Map[String, @ion.Ion] = @ion.FromIon::from_ion(value)
  let count = fields.get("count").unwrap_or(@ion.Ion::untyped_null())
  {
    count: @ion.FromIon::from_ion(count) catch {
      error => raise error.within(Name("count"))
    },
  }
}

///|
test "decode an application type" {
  let count : ExampleCount = @ion.FromIon::from_ion(
    @ion.Ion(ExampleCount::{ count: 3, }),
  )
  assert_eq(count.count, 3)
  let empty = @text.read_ion("{}")
  let missing = try (@ion.FromIon::from_ion(empty) : ExampleCount) catch {
    error => error.message()
  } noraise {
    _ => "decoded"
  }
  inspect(missing, content="$.count: expected int, found null")
}
```

A type that derives core `ToJson` and `FromJson` gets both codecs through its
JSON form with `to_ion_via_json` and `from_ion_via_json`. The trip follows the
JSON rules, so annotations are dropped and symbols become strings. A
`FromJson` failure becomes `JsonDecode` with its path in Ion terms:

```mbt check
///|
struct ExampleEvent {
  id : Int32
  reason : String
} derive(ToJson, FromJson, Eq, Debug)

///|
impl @ion.ToIon for ExampleEvent with fn to_ion(self) {
  @ion.to_ion_via_json(self)
}

///|
impl @ion.FromIon for ExampleEvent with fn from_ion(value) {
  @ion.from_ion_via_json(value)
}

///|
test "bridge a derived JSON type to Ion" {
  let event = ExampleEvent::{ id: 7, reason: "timeout", }
  inspect(
    @text.write_ion(@ion.Ion(event)),
    content="{id: 7, reason: \"timeout\"}",
  )
  let back : ExampleEvent = @ion.FromIon::from_ion(@ion.Ion(event))
  assert_eq(back, event)
  let wrong = @text.read_ion("[{id: \"seven\", reason: r}]")
  let error = try (@ion.FromIon::from_ion(wrong) : Array[ExampleEvent]) catch {
    error => error
  } noraise {
    _ => fail("expected a decode error")
  }
  assert_eq(error.path(), [Index(0), Name("id")])
}
```

## Exact numbers

A decimal keeps its coefficient and exponent as written, so `1.50` and `1.5`
are different Ion values, and so are `0.0` and `-0.0`. Arithmetic goes
through `moonbitlang/x/decimal`, and returns `None` for what that type cannot
represent.

```mbt check
///|
test "decimals keep their precision" {
  let price = @ion.Decimal::parse("1.50")
  inspect(price.coefficient(), content="150")
  inspect(price.exponent(), content="-2")
  assert_false(
    @ion.IonValue::decimal(price).equals(
      @ion.IonValue::decimal(@ion.Decimal::parse("1.5")),
    ),
  )
  inspect(@ion.Decimal::parse("-0.0").is_negative_zero(), content="true")
  debug_inspect(
    price.add(@ion.Decimal::parse("0.25")).map(sum => sum.to_ion_string()),
    content=(
      #|Some("1.75")
    ),
  )
}
```

## Timestamps

A timestamp has the precision its components give it, and an optional offset.
`validate` checks that it describes a real moment, which the readers apply to
every timestamp they read.

```mbt check
///|
test "timestamps" {
  let launch = @ion.Timestamp::parse("1969-07-20T20:17:40Z")
  debug_inspect(launch.precision(), content="Second")
  inspect(launch.to_ion_string(), content="1969-07-20T20:17:40Z")
  // The offset is kept, in minutes; `-00:00` is an unknown offset.
  let eastern = @ion.Timestamp::parse("2024-01-01T01:30+02:00")
  debug_inspect(eastern.offset(), content="Some(Known(120))")
  debug_inspect(
    @ion.Timestamp::parse("2024-01-01T01:30-00:00").offset(),
    content="Some(Unknown)",
  )
  // 2001 was not a leap year.
  let invalid = @ion.timestamp(year=2001, month=Some(2), day=Some(29))
  let message = try invalid.validate() catch {
    error => error.message()
  } noraise {
    _ => "valid"
  }
  inspect(message, content="timestamp day is out of range for its month")
}
```

`with_offset` moves a timestamp to another offset, keeping the instant it
names, and `as_utc` moves it to `Z`. `compare_instant` compares the instants
two timestamps name, whatever their precision and offset; `equals` compares
Ion values, which differ when the precision or the offset does.

```mbt check
///|
test "offsets and instants" {
  let eastern = @ion.Timestamp::parse("2024-01-01T01:30+02:00")
  inspect(eastern.as_utc().to_ion_string(), content="2023-12-31T23:30Z")
  inspect(
    eastern.with_offset(@ion.TimestampOffset::Known(-300)).to_ion_string(),
    content="2023-12-31T18:30-05:00",
  )
  let midnight = @ion.Timestamp::parse("2024-01-01")
  let also_midnight = @ion.Timestamp::parse("2024-01-01T01:00:00.000+01:00")
  inspect(midnight.compare_instant(also_midnight), content="0")
  assert_false(
    @ion.IonValue::timestamp(midnight).equals(
      @ion.IonValue::timestamp(also_midnight),
    ),
  )
}
```

## Symbols and symbol tables

A symbol has text, a symbol ID, or both. The Ion 1.0 system symbols take IDs
1 to 9; a local symbol table declares more from 10. The readers keep a
`SymbolTable` per stream and resolve IDs against it.

```mbt check
///|
test "a symbol table resolves symbol IDs" {
  let table = @ion.SymbolTable::system()
  debug_inspect(table.text(4), content="Some(\"name\")")
  let declaration = @text.read_ion("{ symbols: [\"red\", \"green\"] }").with_annotation(
    @ion.symbol("$ion_symbol_table"),
  )
  assert_true(table.apply_local_symbol_table(declaration))
  inspect(table.max_id(), content="11")
  debug_inspect(
    table.lookup(11).text(),
    content=(
      #|Some("green")
    ),
  )
  // An ID beyond the table is an error.
  let beyond = try table.lookup(12) catch {
    error => error.message()
  } noraise {
    token => token.text().unwrap_or("?")
  }
  inspect(
    beyond,
    content="symbol ID $12 is beyond the symbol table, whose largest ID is 11",
  )
}
```

A local symbol table can import shared symbol tables by name and version.
The readers resolve them against a `Catalog`, which a program loads from its
`$ion_shared_symbol_table::{...}` structs:

```mbt check
///|
test "imports resolve against a catalog" {
  let catalog = @ion.Catalog::load(
    @text.read_ion_datagram(
      "$ion_shared_symbol_table::{ name: \"colors\", version: 1, symbols: [\"red\", \"green\"] }",
    ),
  )
  let values = @text.read_ion_datagram(
    "$ion_symbol_table::{ imports: [{ name: \"colors\", version: 1 }], symbols: [\"local\"] } $11 $12",
    catalog~,
  )
  inspect(
    @text.write_all(values),
    content=(
      #|green
      #|local
    ),
  )
}
```

Unknown symbols imported from a shared table retain an `import_location`
with its name and one-based slot. `SymbolToken::equivalent` and
`IonValue::equals` use that location, ignoring the stream-local ID and table
version. Unknown local symbols remain equivalent to `$0`. The token's derived
`Eq` and `Hash` compare its complete representation, including reader metadata.
Shared-table `imports` fields are informational metadata; loading a catalog
does not resolve them or add slots for them.

```mbt check
///|
test "unknown imports keep their identity through text" {
  let symbol = @ion.SymbolToken::from_import(20, "example", 1, 2)
  let shifted = @ion.SymbolToken::from_import(30, "example", 2, 2)
  assert_true(symbol.equivalent(shifted))
  assert_false(symbol.equivalent(@ion.SymbolToken::from_sid(0)))
  let value = @ion.IonValue::symbol_token(symbol)
  assert_true(@text.read_ion(@text.write_ion(value)).equals(value))
}
```

Text and binary writers preserve the import name, version, and slot while
assigning output-local IDs. Unknown local IDs are reserved with null slots.
The incremental writers reset their context when an unknown imported symbol
needs a different table. A single writer context containing unknown symbols
from conflicting versions of one shared table raises `IonError::Unsupported`.
This applies to one text value or one binary datagram; write separate contexts
with the incremental writer when the versions differ between values.

## Traversal

`IonValue::accept` walks a value depth-first with an `IonVisitor`, whose
methods all default to doing nothing. `IonValue::fold` threads an accumulator
instead, and each callback can continue, skip a value's children, or stop.
None of these recurse: they work at any depth.

```mbt check
///|
struct FieldNames {
  names : Array[String]
}

///|
impl @ion.IonVisitor for FieldNames with fn visit_field_name(self, token) {
  self.names.push(token.text().unwrap_or("?"))
}

///|
test "visit every field name" {
  let value = @text.read_ion("{ a: 1, b: { c: 2 }, d: [ { e: 3 } ] }")
  let visitor : FieldNames = { names: [], }
  value.accept(visitor)
  debug_inspect(
    visitor.names,
    content=(
      #|["a", "b", "c", "d", "e"]
    ),
  )
}

///|
test "sum the integers outside annotated values" {
  let value = @text.read_ion("[1, 2, skip::[100, 200], (3 4)]")
  let sum = value.fold(0, {
    ..@ion.IonFold::default(),
    visit_value: (total, value) => {
      if value.has_annotations() {
        return SkipChildren(total)
      }
      match value.kind() {
        Int(n) => Continue(total + n.to_int())
        _ => Continue(total)
      }
    },
  })
  inspect(sum, content="10")
}
```

`IonValue::walk` gives the same depth-first order as events: `Enter` and
`Exit` around each list, S-expression and struct, and `Scalar` for every
other value, each with its struct field name. `IonValue::walker` gives the
walk as a pull-based `ValueWalker`, which can also skip a container's
children. `IonValue::fold_up` folds bottom-up: each container gets its
children's results. None of them recurse, so they work at any depth.

```mbt check
///|
test "walk a value as events" {
  let value = @text.read_ion("{ a: [1, 2], b: null.list }")
  let events = value
    .walk()
    .map(event => {
      match event {
        Enter(v, ..) => "enter " + v.ion_type().to_text()
        Exit(v, ..) => "exit " + v.ion_type().to_text()
        Scalar(v, ..) => "scalar " + v.ion_type().to_text()
      }
    })
    .collect()
  debug_inspect(
    events,
    content=(
      #|[
      #|  "enter struct",
      #|  "enter list",
      #|  "scalar int",
      #|  "scalar int",
      #|  "exit list",
      #|  "scalar list",
      #|  "exit struct",
      #|]
    ),
  )
}

///|
test "skip the children of annotated containers" {
  let walker = @text.read_ion("[1, skip::[2, 3], 4]").walker()
  let ints : Array[String] = []
  while walker.next() is Some(event) {
    match event {
      Enter(value, ..) => if value.has_annotations() { walker.skip_children() }
      Scalar(value, ..) =>
        if value.kind() is Int(n) {
          ints.push(n.to_string())
        }
      Exit(_, ..) => ()
    }
  }
  debug_inspect(
    ints,
    content=(
      #|["1", "4"]
    ),
  )
}

///|
test "measure the depth with fold_up" {
  let value = @text.read_ion("[1, [2, [3]], {a: (4)}]")
  let depth = value.fold_up(scalar=_ => 0, container=(_, children) => {
    let mut deepest = 0
    for child in children {
      if child.1 > deepest {
        deepest = child.1
      }
    }
    deepest + 1
  })
  inspect(depth, content="3")
}
```