# Pulsar client capability and test matrix

Performance coverage is tracked separately in [Performance testing](performance.md):
local codec/routing/batch microbenchmarks and an explicit Broker throughput and
send-receipt latency harness. Functional/live compatibility results below are
not performance baselines; no performance regression threshold is enforced.

Compared with the public APIs of [pulsar-rs](https://github.com/streamnative/pulsar-rs)
and [pulsar-client-go](https://github.com/apache/pulsar-client-go). This is a
functional comparison, not an API-by-API claim of parity.

The Status column describes the 0.1.1 candidate source in this checkout.
The Verification column summarizes historical runs recorded through 2026-09-28;
it is not a passing result for every later commit. The original matrix was
recorded on 2026-09-25, with the native dependency update below on 2026-09-28.
Entries without a linked run and source revision are historical coverage notes,
not independently reproducible acceptance evidence. For a release, record the
candidate commit, commands, environment and results using the [release process](releasing.md).

Native dependency update (2026-09-28, local validation): the enhanced TLS and
ECIES adapters no longer require OpenSSL development headers; they still load
OpenSSL 3 at feature use. RSA PEM parsing selects its algorithm before probing
ECIES. The ABI table is checked against official headers with a failing negative
control, and loader failures have isolated sanitizer checks. The root public
API is unchanged; the `native_ecies` helper package adds `availability_error`.
Java sync-flush decoding now uses a bounded pure MoonBit DEFLATE boundary
scanner and the released flate decoder; the product has no system zlib
dependency. Differential checks cover 1000 sync-flush and 1000 complete streams,
18000 mutations, truncations and size mismatches. Linux isolated build/runtime
environments verified absence of development headers and optional libraries,
then TLS/ECIES with only OpenSSL runtime libraries restored. Pulsar
4.2.4/3.3.9 runs recorded in that update passed mTLS, certificate rejection, three EC curves, RSA and
Java/Go compressed chunk exchange. See the source repository's
[boundary proof](https://github.com/pangbit/moonpulsar/blob/main/docs/archive/native-deps-2026-09/ZLIB_BOUNDARY_PROOF.md) and [completion report](https://github.com/pangbit/moonpulsar/blob/main/docs/archive/native-deps-2026-09/NATIVE_DEPENDENCY_COMPLETION.md)
for acceptance details and evidence boundaries. Remote CI and registry
publication were separate, unperformed actions at the time of that report.

| Capability | Status | Verification |
| --- | --- | --- |
| TCP handshake, lookup, pooling, keepalive, reconnection | Implemented | Mock broker, connection refusal, handshake timeout, malformed handshake and response, explicit broker CONNECT rejection, challenge limit, pending-request failure and release; real-broker restart and token-rotation recovery scenarios |
| Client configuration | Partial: connection/operation timeouts, keepalive, listener and lookup properties, reconnect backoff/attempts, per-broker pool sizing and idle eviction; `max_memory_bytes` charges pending producer payloads, keys, properties, encoded replay frames, incomplete chunks, queued consumer/reader messages (payload and dynamic metadata), and each inbound broker frame from its declared length through dispatch. TLS/socket internal buffers and runtime heap overhead remain outside the budget | Mock timeout, LOOKUP wire, invalid retry policy, exhausted producer/consumer reconnect, pool bound/replacement, active-handler-aware idle eviction, shared byte-budget close and encoded-frame rejection; receive queue tests cover delivery/reuse, metadata, overflow, composite forwarding/peek/stale filtering and close; chunk tests cover completion/clear/over-limit release; inbound frame tests cover pre-read admission, dispatch release, malformed-frame release, and consumer/reader error propagation without reconnect. Two-connection and expanded shared-budget live suites on 4.2.4 and isolated 3.3.9 cover producer budgeting. `examples/memory_budget` passed bounded sequential send/receive, chunk and Reader roundtrips in isolated 4.2.4 and 3.3.9 containers; the same live program verified consumer and Reader inbound-frame over-limit errors and recovery with a fresh, sufficiently budgeted client on both versions. `chunk_live_wbtest.mbt` verified consumer and Reader incomplete-chunk budget failure and release; `receive_queue_live_wbtest.mbt` verified retained receive-queue budget failure, release and fresh-client replay for both receiver types on both versions |
| TLS service URL, CA, client certificates and protocol/cipher policy | Server-authenticated TLS and native OpenSSL 3 client-certificate authentication implemented. File identity and async identity supplier are reloaded on connection; TLS 1.2/1.3 minimum/maximum, TLS 1.2 cipher list and TLS 1.3 ciphersuites supported. macOS and Linux live mTLS verified; Windows remains unverified | TLS mock validates private CA, rejects untrusted certificate and missing key, selects TLS 1.3 cipher, rejects invalid cipher, and refreshes identity on reconnect. Isolated authenticated 4.2.4 and 3.3.9 brokers passed client-certificate send/receive on macOS and Linux; Linux runs rejected absent client cert and wrong CA and reused producer/consumer after each broker restart. Official Go RSA and Java ECIES clients exchanged encrypted messages with MoonBit over mTLS on both versions |
| Binary-protocol token and basic auth | Implemented | Token: live broker; basic: unit test; Admin REST supports bearer token |
| Auth challenge refresh | Implemented for data refresh | Handshake and post-handshake mock tests; an isolated 4.2.4 and 3.3.9 Broker with a disposable authentication provider issued a real binary AUTH_CHALLENGE during CONNECT. MoonBit supplied credentials twice, completed authentication and rejected a wrong second response. Challenge-specific algorithms remain open |
| Async auth provider and rotating token file | Implemented | Local file rotation and broker challenge mock tests; authenticated Pulsar 4.2.4 rejected an invalid rotated file token, then accepted a second valid token on a new connection from the same client and completed send/receive. On isolated 4.2.4 and 3.3.9 brokers, `examples/token_reconnect` rotated the token file while a producer and consumer were active; after switching the broker signing key and restarting, the old token was rejected and both existing handlers resumed send/receive with the new token. A stock token-authenticated Broker does not issue AUTH_CHALLENGE; the isolated challenge-provider Broker test covers the wire response on both versions |
| Producer send, async receipt, batching, compression, delayed delivery | Implemented, including CRC32-C on outgoing SEND frames and spec-compliant batch payloads. Zlib decompression accepts both complete streams and Java's sync-flushed stream when the declared uncompressed size is valid | Mock broker and live broker scenarios; official Go client decodes MoonBit-produced Protobuf Native, INT64 and two-message STRING batch payloads on 4.2.4. Local Java-style Zlib sync-flush, wrong-size and truncation tests pass. Pulsar Snappy payloads use raw blocks matching the pinned Go client; the separate public framed-stream helpers retain their existing format |
| Producer pending queue and send timeout | Per-producer pending-message limit and client-wide retained-message and encoded-frame byte limit shared with receive queues and incomplete chunks. Admission can block for the message; encoded-frame admission fails immediately if both retained copies cannot fit, avoiding a wait while holding the original reservation. Buffered batch messages time out from admission and are removed before flush. Full heap-memory accounting remains open | Mock cross-producer full/block/release, timeout recovery, close wakeup, metadata and frame rejection, pre-flush and in-flight timeout, late and early receipts, batch removal, send failure and invalid options; expanded shared-budget, batch expiry followed by successful flush and bounded sequential sends passed on 4.2.4 and isolated 3.3.9 |
| Producer access modes, metadata, reconnect epoch, and last confirmed sequence | Implemented | Mock wire, broker-history sequence, wait-ready, and reconnect tests; live exclusive conflict and wait-for-exclusive takeover |
| Producer chunking and compressed chunking | Explicit or broker-limit-driven automatic chunking implemented; chunk `SEND` frames set `is_chunk=true`, and oversize frames fail before write | Wire-size, `is_chunk` flag and failure-path mocks; 6 MiB auto-chunk roundtrip and `examples/memory_budget` chunk roundtrip on isolated Pulsar 4.2.4 and 3.3.9; official Go and MoonBit exchanged 256 KiB random payloads as forced chunks in both directions with LZ4, Zlib, Zstd and raw Snappy compression on both versions. Official Java and MoonBit exchanged the same payload with LZ4, Zlib and Zstd in both directions on both versions |
| Partition routing and producer expansion | Key hash, round-robin, single-partition, and custom callback routing implemented for initially partitioned topics, including one-partition topics | Mock routing and invalid-index tests, one-to-two and live two-to-three partition expansion; custom routing on 4.2.4 and 3.3.9 |
| Single, partitioned, and explicit multi-topic consumption | Implemented; initially partitioned consumers discover added partitions | Mock one-to-two expansion and bounded slow-consumer forwarding; live two-to-three expansion. Malformed compressed payloads wake consumer and Reader callers with `InvalidFrame` instead of being silently dropped |
| KeyShared auto-split and sticky hash ranges | Implemented | Range validation and SUBSCRIBE wire mock; sticky policy live broker |
| Pattern subscriptions | Topic additions and removals implemented; a broker `CLOSE_CONSUMER` on a pattern source defers recreation to the next namespace discovery | Mock addition, removal, stale-queue filtering, broker-initiated close without premature re-subscription, and re-addition; isolated 4.2.4 and 3.3.9 live runs deleted active non-partitioned and two-partition topics, observed source removal, recreated each topic and received again |
| Consumer options, ack/nack, ack timeout, seek, unsubscribe | ACK can optionally await a broker response; opt-in batch-index ACK and default whole-batch ACK implemented. Transactional ACK supports both batch modes, registering the subscription and confirming each wire ACK. Optional ACK grouping buffers by count/time, coalesces cumulative positions, flushes on close/seek and bypasses grouping for confirmed ACKs. Zero-sized on-demand receive queue is limited to non-partitioned single-topic consumers, with explicit errors for composite subscriptions; opt-in auto-scaled receive queue implemented for single-topic consumers; the latter is propagated to composite subscriptions. Composite consumers still prefetch into their merge queue. ID-list ACK combines ordinary IDs in one frame and preserves batch-index handling; per-topic last-message IDs are available | Mock grouped frames, timer/close flush, confirmed ordering, batch bitmaps, cumulative coalescing, reconnect drop/retry and invalid options; grouped ACK persistence on 4.2.4 and 3.3.9. Mock batch ACK bitmaps (including 65 indexes), single-message batch for consumer/reader, broker ACK-set filtering, rejected ACK retry and transactional distinct-index bitmap retry; whole-batch and per-index ACK with consumer reopen on both versions (`acknowledgmentAtBatchIndexLevelEnabled=true`); transactional whole-batch and per-index ACK rollback redelivery and commit persistence on 4.2.4 and isolated 3.3.9; official Go two-message INT64 batch decoded by MoonBit on 4.2.4; mock auto-scale FLOW growth/bound/reset; zero-queue and auto-scale receive, ack timeout and ID seek on both versions; new mock checks reject partitioned and composite zero queues, verify one-frame ID-list ACK with rejection/retry, and query last IDs for each composite source with malformed-response recovery; 4.2.4 and isolated 3.3.9 live suites verified composite last IDs and confirmed ID-list ACK; dedicated isolated broker runs on both versions verified Consumer timestamp seek to first and second publish times, including replay |
| Exponential negative-ack backoff | Implemented for `Message::nack` | Timing and close-cancellation mock tests; live broker redelivery |
| Chunk reassembly and ACK | Implemented with bounded memory and configurable expiry/pending count; incomplete chunks charge the shared client byte budget, and budget exhaustion closes the affected consumer/reader queue with `ClientMemoryFull`; `ChunkAssemblyPolicy::new(auto_ack_incomplete=true)` ACKs received IDs of evicted incomplete chunks, default false | Mock wire, capacity eviction, periodic expiry, optional ACK and default no-ACK tests for consumer/reader; shared-budget failure and release on complete/clear/discard for both receiver types. Complete-chunk consumer/reader replay passed live. `scripts/test-chunk-live.sh` injects incomplete broker messages and verifies consumer and Reader buffer expiry, broker cursor advancement and subsequent receipt, then verifies `ClientMemoryFull` and released budget after two chunks for both receiver types on isolated 4.2.4 and 3.3.9 |
| Reader, seek, replay, and added-partition discovery | Implemented for initially partitioned topics; custom name/properties/subscription, last-message ID per partition, inclusive start and seek | Mock inclusive/exclusive, empty-latest and nonempty inclusive-latest cursor reset paths, one-to-two expansion, SUBSCRIBE metadata, invalid name, malformed SUBSCRIBE and last-ID replies with handler cleanup, bounded slow-reader forwarding, chunk expiry, and disconnect wakeup; live explicit/latest inclusive on 4.2.4 and 3.3.9, two-to-three expansion; 4.2.4 and isolated 3.3.9 live single-topic last-ID maps; dedicated isolated broker runs on both versions verified Reader timestamp seek forward and backward across two publish times |
| Transaction coordinator lookup and transaction lifecycle | Implemented; participant registration checks broker response type and errors, and transactional batch ACK confirms each sent ACK | Mock lifecycle, disabled and out-of-range coordinator IDs, rejected/malformed coordinator attach and failed owned connection, begin/end and participant replies, transactional batch-index registration and rejected-ACK retry; live coordinator/abort and producer/subscription participant registration on 4.2.4 and transaction-enabled isolated 3.3.9; transactional whole-batch and per-index commit/rollback on both versions |
| Admin topic and subscription operations | Selected endpoints implemented, including partitioned-topic list/stats, topic properties, and subscription create/skip/expire | Mock HTTP path, body, JSON-shape, and invalid-input tests; live create/expand/query/stats/delete, topic property updates, and subscription cursor/backlog behavior |
| Schema declaration, versions, and typed payload conversion | STRING, JSON, BYTES, integer/float primitives, Avro datum and typed mapping, Protobuf Native, and legacy Protobuf codecs; broker schema/version lookup and message version propagation implemented. INT16/32/64 default to Java-compatible big-endian encoding and accept `little_endian=true` for Go-client encoding. Async `Client::decode_avro` fetches and caches writer schemas by message topic/version, then resolves field aliases/defaults, unions, collections, enum fallbacks and allowed numeric promotions; incompatible schemas return `SchemaIncompatible` | Local wire-byte, range, malformed-payload, endian, Avro schema/datum validation, descriptor validation, GET_SCHEMA error and version tests; STRING, JSON, INT64, Protobuf Native and schema-version broker roundtrips on 4.2.4 and 3.3.9. Pinned official Go client and MoonBit exchanged Avro records, Protobuf Native, legacy Protobuf, INT8/16/32/64, FLOAT, DOUBLE, and BYTES values in both directions on 4.2.4 and 3.3.9. Mock Avro evolution covers normal, incompatible, failed lookup/retry and cached reuse; official Go v1 writer to MoonBit v2 reader passed on 4.2.4 and isolated 3.3.9. The pinned Go reader cannot fill a v2 default field from v1 bytes (`interop/go/avro_evolution_test.go` expects its EOF), so that reverse evolution case is a reference-client limitation; Official Java v1 writer to MoonBit v2 reader and MoonBit v1 writer to Java v2 reader passed on isolated 4.2.4 and 3.3.9; both directions verified field alias, default value and int-to-long promotion |
| Dead-letter policy | Implemented for consumer delivery and negative ack; producer created when the consumer is created | Mock publish-before-ACK, send-failure NACK, direct consumer NACK, composite non-blocking receive, and live broker explicit-NACK routing (including a broker that reports redelivery count zero) |
| Retry-letter topic policy | Implemented through explicit `create_retry_consumer` or `retry_topic` on single, multi-topic and pattern constructors | Mock retry then DLQ publish-before-ACK, send-failure NACK, and invalid options; live Shared delayed retry/DLQ and all three ordinary constructors on 4.2.4 and 3.3.9 |
| TableView (raw and typed payload values) | Implemented with schema declaration, optional message decryption and decode errors for typed views | Mock typed snapshot, malformed update/snapshot and incompatible schema; raw initial replay/live updates/empty topic/query failure; typed INT32 and encrypted STRING snapshots on 4.2.4 and 3.3.9, encrypted no-key failure, raw update/tombstone and added partition live |
| In-place conversion of a non-partitioned topic to a partitioned topic | Broker does not support this on the tested Pulsar 4.2.4 instance | An isolated Admin REST probe returned HTTP 409 (`This topic already exists`); existing clients can discover added partitions of an already partitioned topic |
| OAuth2, Athenz, TLS client-certificate authentication | OAuth2 client-credentials, external Athenz role-token supplier/file providers, RSA service-NToken ZTS exchange, and certificate identity for an HTTPS ZTS exchange implemented; native TLS client-certificate authentication implemented | OAuth2 form encoding, fresh token retrieval, rejected/malformed token endpoint, and broker challenge covered by local HTTP and mock broker; local OAuth2 token endpoint to authenticated Pulsar 4.2.4 send/receive passed. Athenz service-NToken mock covers verifiable RSA signature, request path/header, early refresh, cached reuse, 503 retry, valid-token fallback, malformed/expired role token, 401 no-retry, missing key and broker challenge; role-token file rotation and invalid token covered by mock. Certificate exchange mock covers HTTPS requirement, HTTP response framing, 503 retry and cache; a local ZTS fixture requiring a client certificate passed on macOS and Linux, including failed client-auth and wrong-CA checks. An isolated Athenz-authenticated broker and a signed-role-token ZTS fixture passed on 4.2.4 and 3.3.9: wrong-domain role tokens and wrongly signed service NTokens received HTTP 401, while both a role-token file and RSA service-NToken ZTS exchange sent and received over the binary protocol. No external Athenz service was run. Broker client-certificate authentication passed on isolated 4.2.4 and 3.3.9 from macOS and Linux; Windows remains unverified |
| Producer and consumer interceptors, events, tracing, metrics | Producer and consumer hooks implemented. `ClientOptions::new(on_event=...)` reports structured send, receive/try-receive success and failure, ACK/nack and producer/consumer reconnect events; `ClientMetrics` counts outcomes including receive and negative-ACK failures. Optional on_span callbacks emit paired send, receive, ACK and reconnect spans with correlation IDs, results and durations; ClientMetrics exports fixed-label Prometheus counters and millisecond histograms | Producer mocks cover order, routing, failures, async completion and replay; consumer mocks cover message and ID ACK, nack, rejected confirmed ACK, blocking/nonblocking receive-after-close failure and composite delivery; reconnect event and metrics classification tests; 4.2.4/3.3.9 callback and counter roundtrips; mock span pairing, asynchronous receipt, closed-operation failures, broker-rejected ACK, cancellation and reconnect, plus histogram buckets/sum and label bound; 4.2.4 and isolated 3.3.9 live span exporters exercised |
| Message encryption | RSA-OAEP-SHA1 key wrapping and AES-256-GCM payload encryption/decryption implemented for producer, consumer, reader and TableView; explicit file reload or synchronous named-key callbacks support rotation. Encrypted retry/DLQ messages are re-encrypted when a recipient key is supplied. Consumer/Reader decryption failure policies `Fail` (default), `Consume` (ciphertext; non-batched only) and `Discard` are implemented. P-256, P-384 and P-521 ECIES key wrapping is implemented with OpenSSL 3; other key-wrapping algorithms remain open. The pinned Go client exposes the 16-byte GCM tag for MoonBit's encrypted empty payload because its consumer does not replace the payload buffer when `UncompressedSize=0`; Go-to-MoonBit empty payload works | Local fresh-key, tamper, missing/wrong-key, multi-recipient, file and callback rotation, malformed-PEM and empty-payload tests; mock policy tests cover default failure, ciphertext delivery and ACK, encrypted-batch rejection, discard ACK and replacement FLOW, and Reader skip. Official Go client exchanged encrypted single, Zlib-compressed and batched messages with MoonBit on 4.2.4 and isolated 3.3.9. Live empty-payload Go-to-MoonBit consumer and Reader checks passed on both versions; `receive-empty` reproduces the Go-side limit on both. Live Reader, typed TableView, retry, DLQ, no-key failure, Consume ciphertext ACK and Discard skip/ACK paths passed on both versions. Official Java client exchanged P-256, P-384 and P-521 ECIES messages with MoonBit in both directions on isolated 4.2.4 and 3.3.9 brokers; local EC rotation, tampering and mixed RSA/EC recipients passed on macOS and Linux. The pinned Go client only accepts RSA encryption keys. Bidirectional RSA/Go and P-521 ECIES/Java exchanges over mutually authenticated TLS passed on isolated 4.2.4 and 3.3.9 brokers |
| Full Admin REST surface | Outside the data-client parity target | Only listed topic/subscription endpoints are implemented |
| Broker version compatibility | Historical live runs covered 4.2.4 and 3.3.9; rerun for the release candidate | 4.2.4 test container and isolated 3.3.9 container with transaction coordinator enabled exercised `examples/live_capabilities` with Admin REST; separate isolated 4.2.4 and 3.3.9 containers exercised `examples/memory_budget` in an independent namespace |

`moon test --target native` runs the in-process mock suite. The
`examples/live_capabilities` program exercises an authenticated Pulsar broker;
set `PULSAR_ADMIN_URL` to include Admin REST checks. The live program is not
part of `moon test` and requires a broker. `moon coverage` measures mock-suite
line execution and therefore does not include the live program's execution.
Historical `moon coverage analyze` results showed uncovered connection and
Reader branches. No current-commit coverage percentage is asserted here;
regenerate coverage for the candidate being assessed. Live examples are not
included in the mock coverage report. Malformed frame lengths and
missing handshake metadata have executable regression tests.

The CI broker matrix runs the isolated chunk/receive-queue memory and timestamp
seek scripts against 4.2.4 and 3.3.9. The six optional live tests return early
without their `PULSAR_LIVE_*` inputs; a passing default `moon test` run is not
evidence that those broker scenarios ran. CI supplies those inputs through the
scripts. Other live examples and Go/Java interoperability remain separate checks.