# Moonfish incremental market-data plane upgrade ## Decision Moonfish will use embedded SQLite as its operational market-data store and MoonBook as its immutable evidence store. Moonfish owns the data schemas, normalization, update planning, reconciliation, factors, and policy-ready queries. MoonClaw schedules and executes declared jobs. Moondesk only operates and displays the pack. This is not a database-server subsystem and does not introduce Python or a localhost pilot. SQLite is linked into the native MoonBit binaries. The database is a rebuildable domain index; raw provider responses, source manifests, policy bindings, reviews, and released artifacts remain MoonBook evidence. Policy analysis uses a separate evidence-coverage contract. Market bars alone never imply policy readiness. `ashare/policy_evidence` requires one explicit record for security master, adjusted history, intraday/accessibility, liquidity, fundamentals, audit/governance/enforcement, pledging/corporate actions, catalysts and contradictions, industry strength, market regime and breadth, and valuation/risk-reward. Every observed record must carry source URI, effective time, retrieval time, license reference, and SHA-256 digest. Unknown, stale, conflicted, unlicensed, duplicate, or incomplete records block policy analysis. ## Why incremental storage The current provider commands can acquire a complete Sina market snapshot and per-symbol daily history, but repeated history downloads waste provider quota and prevent efficient policy runs. After a one-time bootstrap, every completed session should append one row per traded security. Fundamentals, disclosures, security-master changes, and corporate actions update only affected symbols. Cross-sectional ranking still scans the current local universe because a rank is relative to every eligible security. That local scan is cheap and does not require a full provider-history scan. ## Sina load reduction Moonfish uses separate paths for separate responsibilities: 1. `Market_Center.getHQNodeData` is the structured JSON universe-discovery and reconciliation endpoint. It is capped at 100 rows per request. 2. `hq.sinajs.cn/list=` is the aggregate daily/intraday quote path. Testing on 2026-07-24 confirmed 300 symbols in one request; a 500-symbol request timed out. Production therefore caps batches at 300 and falls back to 100 on timeout or response truncation. 3. `CN_MarketDataService.getKLineData` is used only for bootstrap, missing sessions, candidate history expansion, and corporate-action repair. It is not called for every symbol on every daily run. For roughly 5,529 A-shares, 300-symbol quote batches require about 19 requests instead of 56 market-center pages. A daily count check detects universe-size changes. Full structured universe reconciliation runs when the count changes and at a periodic safety interval. The provider-wide request budget is stored transactionally. Separate MoonClaw jobs and command processes cannot each consume the full Sina quota. ## Authoritative and operational data MoonBook retains: - immutable raw provider payloads; - retrieval/effective time, license identity, and content digest; - session manifests and completeness reports; - accepted corrections and rejected records; - policy versions, run inputs/results, reviews, and release decisions. SQLite retains: - security-master projection; - raw daily bars and revisions; - adjustment factors; - normalized fundamental facts and disclosures; - provider cursors and request-budget state; - atomic session commits; - rolling factor projections and score snapshots; - evidence references back to MoonBook records. The SQLite database can be deleted and rebuilt from accepted MoonBook evidence. No policy conclusion may depend on an untraceable database row. ## Initial schema ### `schema_migration` Tracks applied data-plane schema versions. ### `security_master` Primary key `(provider, symbol)`. Stores name, board, listing status, first and last observed sessions, and source digest. ### `daily_bar` Logical key `(provider, symbol, session, revision)`. Stores unadjusted OHLC, previous close, volume, amount, retrieval time, and source digest. An identical digest is an idempotent no-op. A changed source row becomes a new revision. ### `provider_cursor` Primary key `(provider, dataset)`. Stores the last complete session, cursor, retrieval time, and source digest. ### `provider_rate_limit` Primary key `provider`. Stores quota, current window, requests used, cooldown, and lease owner. Request acquisition uses `BEGIN IMMEDIATE`. ### `session_commit` Primary key `(provider, dataset, session)`. Stores expected/stored counts, source digest, status, and commit time. A session is queryable by policy code only after status becomes `committed`. Later migrations add adjustment factors, fundamental facts, disclosure events, rolling factors, and score snapshots. ## Atomic daily ingestion 1. Acquire the provider-wide request lease. 2. Fetch aggregate quote batches into MoonBook staging. 3. Validate batch membership, response count, symbol uniqueness, quote date, OHLC consistency, amount/volume ranges, and source digest. 4. Start an immediate SQLite transaction. 5. Upsert security-master observations. 6. Insert new daily-bar revisions idempotently. 7. Verify the stored distinct-symbol count. 8. Write the committed session record and provider cursor. 9. Commit SQLite. 10. Persist the accepted MoonBook session manifest. 11. Update rolling factors for changed symbols. Any validation or count failure rolls back the database transaction and leaves the session unavailable to policy runs. ## Update modes ### Bootstrap - discover the complete security master; - acquire the required trailing daily history once; - record missing histories and retry from durable cursors; - create the first rolling-factor projection. ### Daily close - count-check the universe; - fetch aggregate quotes in batches of at most 300; - append one completed daily bar per traded security; - update rolling technical/liquidity state; - run local cross-sectional normalization. ### Intraday - query only the active watchlist and focus candidates; - evaluate opening and accessibility rules; - never poll the full universe tick by tick. ### Event updates - ingest only new disclosures and financial reports after the provider cursor; - recompute fundamental/catalyst factors only for affected symbols; - rebuild adjusted projections only when an adjustment factor changes. ### Reconciliation - periodically compare security-master counts and session coverage; - fetch only missing symbols/sessions; - append provider corrections as revisions; - never overwrite accepted raw evidence. ## Readiness stages The data plane reports: - `screen_ready`: complete session market data is committed; - `research_ready`: required fundamental, disclosure, and catalyst evidence is linked; - `entry_ready`: adjustment, accessibility, opening, risk, and regime evidence is current; - `reviewed_for_release`: named independent and policy-owner review is complete. Sina market data can make a session `screen_ready`. It cannot set later stages without their required evidence. ## Delivery phases ### Implementation status on 2026-07-27 - Phase 1 is complete: native SQLite schema/migration, atomic daily commit, idempotency, revisions, cached security master, cursor, and transactional provider quota are implemented and tested. - Phase 2 is operational for saved full-snapshot bootstrap and live aggregate updates. The native command skips committed sessions, uses 300-symbol batches, falls back to 100, and refuses incomplete or pre-close sessions. Aggregate and Market Center calls now have bounded timeouts and retries. History bootstrap accepts a structured shortlist, resumes per-symbol, and can request a 127-row window to clear the 120-completed-bar policy gate with seven rows of margin. - Phase 2 still needs MoonBook raw-batch/session-manifest persistence, scheduled universe reconciliation, and durable partial-download resume. - Phase 3 now includes a deterministic price-only acquisition shortlist using 61 completed sessions and the versioned technical-alert price formula. Candidate count is an explicit operator budget. On 2026-07-27, a frozen SSE/SZSE main-board queue selected 45 symbols from uniformly current histories. The shortlist always reports `follows_policy: false`, `investment_score: false`, and `recommendation_eligible: false`. - Phase 4 provider groundwork is in progress. Native Sina fundamentals, valuation, QFQ/HFQ factors, actions, disclosures, accessibility, breadth, industry, and stock-industry mapping providers preserve raw payloads, timestamps, source URIs, digests, and explicit unknowns. Tushare is not an active provider. The 45-symbol queue has complete per-symbol fundamentals, actions, and disclosure bundle files plus one shared accessibility batch. Native CNInfo support now adds immutable-cohort, rate-limited, resumable disclosure-index and reported-facts backfill. It stores raw pages, source digests, effective dates, policy-category evidence references, and per-symbol cursors in SQLite. Tagged PDF acquisition is bounded and optional. CNInfo title tags and provider field codes remain evidence routing only; structured policy normalization, document-text interpretation, public-web terms assessment, and independent source reconciliation remain explicit blockers. - `moonfish_policy_filter` now binds both the JSON policy contract and Markdown narrative digests and evaluates a fixed 33-rule catalog across 14 policy groups. Every rule is emitted as `pass`, `fail`, `unknown`, or `not_applicable`; an unknown blocks qualification. Alerts are evidence-only and carry `policy_failure: false`. - The 2026-07-27 audit evaluates all 45 names. All 45 fail the complete policy: permissioned Sina evidence does not satisfy the policy's complete licensed point-in-time bundle, and the policy-owner operating parameters are not set. Nine also fail the observed RMB 5 billion float-cap gate; 36 pass that gate. Every name has 29 unknown composite rules and one inapplicable trapped-position rule. Twelve names have recent secondary announcement-title alerts, but those alerts do not create a policy failure. Zero names qualify, and no investment score or recommendation is emitted. - Phases 3, 5, and 6 remain open. Therefore the data plane can become `screen_ready`; the overall product is not yet policy-complete or investment-ready. ### Bounded live evidence pilot on 2026-07-24 The first non-Sina pilot is persisted under the MoonBook raw evidence tree. It is an ingestion check, not a recommendation: - CNInfo resolved `002440` to organization `9900013207`, returned 30 disclosure records for the bounded search page, one latest record for each of main indicators, income statement, cash-flow statement, and balance sheet, and five stock-structure records. The bounded lift-ban and equity-pledge responses contained no records. No absence claim is inferred from an empty bounded response. - SZSE returned a captured response for `002440`, but listing, ST, delisting, suspension, and enforcement status remain explicit unknowns. This is not exchange clearance. - The refreshed Sina breadth snapshot covers 5,530 symbols through 2026-07-24: 555 advanced, 4,939 declined, and 32 were unchanged, for 10.10% directional breadth. Shanghai Composite, Shenzhen Component, and ChiNext were all below their 60-session averages. This is recorded as a proxy risk-off alert. It does not set policy capacity because Sina's `sh000985` history is stale at 2016 and therefore cannot establish the required CSI All Share trend. - Sina's published Shenwan Level-1 nodes map all 28 unresolved symbols to an industry. Industry-history returns, constituent breadth, and a current source-attested session remain incomplete, so industry scoring stays blocked. - Sina's current aggregate and Market Center data provide an exact 2026-07-24 amount for all 28 unresolved symbols, each above RMB 100 million for that one session. The probed daily-history and decoded archive formats contain OHLCV but no historical amount. The 20-session median therefore remains unavailable; `close * volume` is never substituted. - Sina live validation for `002440` retrieved QFQ/HFQ factors, dividends, share-structure observations, restricted-share unlocks, four financial datasets, valuation inputs, and bounded announcements/news. Structured pledging, official accessibility clearance, complete catalyst verification, and historical point-in-time valuation remain unavailable or unknown. All pilot artifacts set `investment_ready` or `recommendation_eligible` to false. The pilot does not follow the complete trading policy because required policy evidence and named independent review remain missing. ### Closed-session refresh on 2026-07-27 - The permissioned Sina Market Center snapshot contains 5,532 unique A-share identities and is preserved under the MoonBook raw tree. The SQLite daily session atomically commits 5,522 traded symbols with exact amount fields. - Ten exact non-trading rows have zero open, high, low, volume, and amount. They remain in the security universe but do not create synthetic daily bars or advance their symbol-history cursors. Partially malformed market rows still fail the complete session. - Daily session ingestion now advances each traded symbol's history cursor in the same transaction as its bars, session commit, and provider cursor. The idempotent replay inserted zero revisions and retained all 5,522 rows. - `moonfish_sina_regime breadth-snapshot` derives breadth from the preserved permissioned snapshot instead of downloading the complete market twice. It observed 5,194 advances, 286 declines, and 44 unchanged priced rows, giving 94.78% directional breadth. - Shanghai Composite, Shenzhen Component, and ChiNext histories are complete through 2026-07-27 but all three remain below their 60-session averages. Sina's CSI All Share history remains stale at 2016-06-13, so the exact policy regime and new-position capacity remain blocked. - The local price-only projection has 5,496 evaluable symbols and a uniformly current top 100. Applying only the 120-session, observed-name, and SSE/SZSE main-board acquisition constraints produces 45 downstream names. This is an evidence-acquisition queue, not a policy pass, investment score, or recommendation. ### Full 45-name policy audit on 2026-07-27 - Selective enrichment completed fundamentals, actions, and disclosures for all 45 frozen queue symbols, plus one complete shared accessibility batch. The importer reported zero acquisition failures. - The full policy catalog ran for every symbol, producing 1,485 rule outcomes. This means the complete rule inventory was attempted; it does not mean the required evidence was complete. - All 45 have overall status `fail`. The two universal failures are `evidence.fresh-complete-licensed-point-in-time` and `governance.owner-parameters-complete`. - `eligibility.minimum-float-market-cap` passes for 36 and fails for nine: `sh603221`, `sh605028`, `sz000608`, `sz000779`, `sz001317`, `sz002303`, `sz002879`, `sz002900`, and `sz003001`. - Exact 20-session median transaction amount remains unknown for all 45. The current session has exact Sina amount, but historical OHLCV rows do not provide the required amount field; `close * volume` is not substituted. - Current CSI All Share evidence remains stale at 2016-06-13, so the exact regime and capacity rule is unknown. Strategy mode, fixed policy score, industry strength, intraday controls, quarterly scenarios, invalidation, named challenge, operating lifecycle, and execution-cost controls also remain unknown. - The full raw report is `full-policy-tri-state-2026-07-27.json`; the compact per-symbol companion is `full-policy-summary-2026-07-27.json` under the 45-name enrichment evidence root. Both report `investment_score_status: not-evaluated`. ### Phase 1: SQLite foundation - add a MoonBit SQLite dependency behind `ashare/market_store`; - create migrations, daily bars, session commits, cursors, and rate limits; - prove atomicity, idempotency, corrections, rollback, and quota serialization. Status: complete. ### Phase 2: Incremental Sina ingestion - normalize saved Sina session snapshots into SQLite; - add aggregate multi-symbol quote batches with 300/100 fallback; - persist raw batches and accepted manifests into MoonBook; - prove second-run idempotency and missing-batch resume. Status: in progress. SQLite ingestion, aggregate batching, fallback, idempotency, committed-session skipping, exact non-trading-row handling, and transactional per-symbol cursor advancement are complete. Saved full snapshots are archived in MoonBook; aggregate raw-batch archival, scheduled reconciliation, and partial-download resume remain. ### Phase 3: Local factor projection - calculate rolling 5/10/20/60-session values from stored bars; - add adjustment-factor projections; - construct complete `FactorSecurityInput` values without model-authored numbers. Status: in progress. Price-only acquisition ranking is implemented. It reduces expensive provider requests but is not a policy eligibility gate. On 2026-07-27, the full universe was reconciled through the closed session, 5,496 symbols were evaluable for the local price projection, and the bounded main-board acquisition queue contains 45 names with uniformly current history. ### Phase 4: Fundamental and disclosure evidence - ingest official CNInfo/SSE/SZSE records incrementally; - add audit, governance, enforcement, pledging, corporate-action, financial, catalyst, and contradictory-evidence projections. Status: in progress. Native Sina evidence adapters, an evidence artifact ledger, and resumable cursors exist. Expensive per-symbol imports wait for complete price history and then run only for an explicitly bounded shortlist. Existing per-symbol bundles and the shared accessibility batch are reused on retry. The 45-symbol 2026-07-27 queue is fully acquired at the artifact-file level and has a complete tri-state rule-catalog audit, but most policy projections and named review are not yet connected to the daily workflow. ### Phase 5: Workflow and review - replace the installed fixture input with committed-session construction; - run deterministic eligibility, scoring, regime, and quarterly-upside gates; - generate dossiers and named independent reviews; - expose staged readiness and evidence gaps in Moondesk. ### Phase 6: Acceptance - backtest the versioned policy; - run governed paper operations; - set owner loss, concentration, drawdown, cooldown, cost, and strategy limits; - permit investment-ready output only after acceptance thresholds pass. ## Non-goals - no custom database engine; - no market data in MoonClaw core; - no finance branching in Moondesk core; - no full-universe tick store; - no silent provider fallback; - no database row without evidence lineage; - no forced candidate when policy evidence is incomplete.