Commit before publish
Presentation facts and immutable payloads commit before external side effects. A crash does not erase the intended transition.
Realtime Protocol Audit:
The Clock Has Two Owners
Verdict: The recovery protocol is thoughtfully built, but the game protocol is not sound under its full failure model.
Safe today only if: IVS publishes before each deadline, transient timer callbacks do not fail, and device clocks are close to server time.
The core split is good: Ably carries immutable payloads and IVS carries the presentation cursor. The break is one layer earlier. A round becomes consequential before the system knows that the audience received its clock.
3High
5Medium
1Low
NOArbitrary-failure safe
2. Four planes, three cursors, two clocks
The implementation is not one realtime stream. It is four coordinated planes with different durability and ordering guarantees.
01 / CONTROLPlanetScaleshow, game, round, revision
commit
02 / PAYLOADAbly + event logsealed, immutable, replayable
preload
03 / CLOCKIVS metadatapresentThroughEventId + elapsedMs
apply
04 / CLIENTExpo reducerpayloads gated by IVS watermark
| State | Authority | Cursor / clock | Repair |
|---|---|---|---|
| Show lifecycle | PlanetScale | controlRevision, stateRevision | Fresh snapshot |
| Presentation payload | PlanetScale event log | Session-scoped eventId | Exact range replay |
| Visible presentation | IVS timed metadata | presentThroughEventId | Repeated watermark |
| Answer lock | Local SQLite WAL | Per-round settlement watermark | Host-local reopen |
Protocol law A payload may be known before it is visible. It becomes visible only when IVS advances the presentation cursor.
3. What is already strong
This is not a rewrite recommendation. Several difficult invariants are implemented correctly and should be preserved.
Ably before IVS
The server awaits Ably acknowledgement before IVS can expose an event ID, preventing a normal cue from outrunning its payload.
Cursor-gated rendering
Expo buffers synchronized events and applies a contiguous range atomically only through the IVS watermark.
Exact gap recovery
Missing presentation IDs are fetched as a complete bounded range and validated for identity, order, and integrity.
Idempotent answer locks
A round/user unique lock and stable submission key make HTTP retries safe after lost acknowledgements.
Settlement drain
Active permits and the local settlement watermark prevent in-process batches from being tallied halfway through a commit.
4. Findings
Ordered by player and show impact. Each finding is a concrete execution trace, not a theoretical consistency preference.
A question can settle without ever reaching the audience
The server starts openedAt and closesAt, commits question_live, returns success, and schedules settlement before the background IVS publish succeeds. Retries are suppressed once the answer window expires.
ImpactAn IVS or Ably outage can turn every eligible player into a no-answer elimination. Q1 can produce zero survivors and a false rollover without any player seeing a question.
live-gameplay-ledger.ts:243-248 gameplay.ts:951-1016 question-open-visibility.ts:44-51
An unacknowledged optimistic answer can become a false winner
Expo stores the selected answer before HTTP succeeds. Network and 5xx exhaustion retain it. Results treat that local selection as the player's outcome when no server lock exists, and terminal projection promotes a still-active local player to winner.
ImpactOn a final question, a correct tap that never reaches the server can be shown as a win if result and terminal personalization also fail. The server correctly eliminated the player and created no payout.
GameCoordinator.tsx:2158-2249 game-reducer.ts:814-830 game-reducer.ts:1020-1062
A failed personalization burst can disable a giveaway for its full window
After bounded retries fail, request keys remain claimed while pending work is cleared. Later IVS heartbeats cannot request the same personalization again.
ImpactAn eligible player can remain on "Checking eligibility" through the complete 15-second entry window and never receive an enabled entry action.
Immediate ordering has no durable cursor
Immediate IDs come from a process-local map and Date.now(). The snapshot boundary filters only events already buffered during reconcile(); an older event delivered afterward is accepted without another boundary check.
ImpactA stale
game.status can overwrite a newer snapshot. After restart or clock skew, that stale high ID can also make Expo reject valid replacement-process events as out of order.ably-realtime.ts:17-19 ably-realtime.ts:333-342 reconciliation.ts:13-48
A same-revision snapshot can discard a proven answer lock
Answer locks do not advance the show revision. If older personalization has the same revision, snapshot merge logic preserves it and ignores a newly returned non-null lock.
ImpactAfter an acknowledgement loss and reconnect, the app can forget the server's answer lock and temporarily derive the wrong personal outcome.
The reveal-to-results transition is a volatile one-shot timer
If the callback's database query or transaction fails, the catch path logs only. It does not retry and does not leave a durable transition intent.
ImpactThe stage remains at answer reveal. The next Advance hides it, allowing the run to skip
answer.results and its counts.Deadline admission measures after validation, not at ingress
The first authoritative wall-clock check occurs after authentication, profile, audience, legal, token, and game access reads.
ImpactA request that reaches the server before cutoff can be rejected because unrelated auth or database work consumed the remaining milliseconds.
Client deadlines still trust mutable device wall time
Expo maintains a monotonic question anchor, but question closure, countdown ticks, and absolute giveaway closure still compare against Date.now() without a server-time offset.
ImpactA forward clock correction can close answer or giveaway UI early. A backward correction leaves actions visible after server cutoff and produces avoidable rejection.
GameCoordinator.tsx:1820-1861 CountdownTimer.tsx:61-94 GameCoordinator.tsx:1973-1980
Round deadlines lose fractional precision in PlanetScale
Local SQLite and presentation events retain milliseconds. Durable rounds.openedAt and rounds.closesAt use MySQL timestamps without fsp: 3.
ImpactRecovery and visibility checks can disagree with local admission by up to roughly one second at the boundary.
5. The failure trace that matters most
The system can be internally consistent and still produce the wrong game result.
- T+000msAdvance is pressed
The local ledger records
openedAtandclosesAt. - T+080msAuthority commits
PlanetScale says
question_live. The immutable event is retained. HTTP returns201. - T+120msIVS publish fails
The question payload exists in Ably, but no IVS watermark reveals it. A background retry is scheduled.
- T+10sThe answer window expires
Visibility checks now suppress every question publish retry. No player has seen the card.
- T+24.25sSettlement succeeds
The protocol correctly drains zero answers and deterministically eliminates the roster.
Safety violation A consequential round was settled without durable evidence that its presentation clock was ever accepted by IVS.
6. Repair order
The first two changes establish the missing safety boundary. The rest make recovery converge monotonically.
Make delivery an explicit round state
Add a durable publishing to visible transition. Settlement must require durable IVS publish evidence. If first delivery never succeeds, automatically void instead of eliminating players.
Make timer transitions durable
Represent reveal-to-results and deadline work as persisted intents with idempotent workers. Timers should wake work, not be the only record that work exists.
Merge facts monotonically
A non-null answer lock must dominate an older absence. Apply the snapshot boundary to every later immediate event, not just the buffer present at reconciliation time.
Use one time domain per deadline
Capture trusted request ingress time. Use IVS-relative elapsed time or an explicit server clock offset on clients. Store all authoritative timestamps at millisecond precision.
Add adversarial protocol tests
Exercise publish outage through deadline, two owners, response loss before reconnect, post-boundary delayed events, timer callback failure, and device clock skew.
7. Method and scope
This was a static, end-to-end protocol audit of maintained server, Expo, and admin paths at commit 9c3b609. The review traced authority, publication, IVS timing, snapshot reconciliation, answer admission, settlement, process restart, and operator feedback.
IncludedPlanetScale stateLocal SQLite WALAbly channelsIVS timed metadataExpo reconciliationAdmin Advance path
Not claimedNetwork chaos executionProduction traffic replayFormal model checkingPhysical device validationAbuse/legal analysis
The event cursor is the strongest part of this design. The next step is to give delivery and server ownership equally explicit authority.