Missing Not at Random

Last week built the drop/quarantine/refuse decision tree — a taxonomy for classifying malformed input before any arithmetic. This week the campaign turned inward on itself, and the unifying discovery is statistical. Three separate systems — a data validator, a trading agent’s attention, an exception handler — were each silently censoring observations in a way that depends on the content of what was being observed. In survival analysis this has a name: missing not at random. The missingness mechanism correlates with the data itself, so every downstream statistic is biased in a direction no amount of averaging can fix.

Three instances, stated once:

  1. The validator that ate “latest.” A filter meant to exclude one historical test artifact rejected any daily result whose reasoning contained the bare substring "test" — silently invalidating 15 of 150 legitimate results (10%), including the day it was discovered on. “Sharpe ratio test”, “multiple-testing bias”, the innocent word “latest”: all censored by a mechanism that triggers because of what the data says.
  2. The attention asymmetry. For three consecutive sessions the trading agent cited the weekly trade cap as a complete answer to every deployment question — while the cash-band mandate, nominally forcing deployment at ≥30% cash, was never surfaced. Keyword trends confirm it: “trade cap” mention rising to 80% of sessions, “cash buffer” 0% and falling. The mandate wasn’t absent from the prompt; it was absent from the attention. Salience is a censoring mechanism too.
  3. The silence that sounded like success. The last silent-swallow site in the repository laundered any unexpected exception into the same NaN as “forward window not yet observable.” Failure and absence became indistinguishable — and the caller’s exclusion, correct for both, manufactured evidence of absence.

The MNAR framework matters because the remedy differs from MCAR (just average) and MAR (condition and average). Under MNAR you must fix the mechanism — declare it, instrument it, or remove it. You cannot average your way out of a filter that eats “latest.”

Monday & Tuesday: Deployment, Budget, and a Controlled Arm

PR #88 routed the behavioral-analysis ledgers through the quarantine ingress — strict parse hook, per-element loud drops, degradation warnings embedded in the report artifact itself. The evening’s live session executed the week’s first trade: the FEZ stop-loss, mechanically, at −5.05% drawdown, the override gate (RSI < 30 and below the lower Bollinger band) correctly rejecting on a half-satisfied condition. A clean stop is a data point too.

Monday also shipped the H4 experiment’s Phase 1: the mid-band deployment prompt variant went live in shadow mode — no execution, no budget consumption, no contamination of the live decision history. A controlled arm for exactly the censoring hypothesis the week would later confirm.

Tuesday, Rule-5 fired: 47.26% cash is far above the 30% band, and the mandate resolved into two buys (REET, ^FCHI) with low correlation to existing exposure, returning cash to the band at 30.17% and exhausting the three-trade weekly budget. PR #89 hardened the churn-analysis ledger ingress, closing the analysis-codomain class. The first shadow record landed: both arms agreed on deploy, diverging only on what. n=1, logged, not judged. Suite: 1595 passing.

Wednesday: The Omission That Liquidated the Position

PR #90 closed the campaign’s most expensive translation bug. A sell action with a missing pct key passed validation; the executor coerced pct=0 into pct=None; Portfolio.sell(pct=None) documents its default as 100% — a full position exit. One omitted key, one laundered translation, blast radius exceeding the malformation by the full portfolio dimension: position deleted, exit cooldown armed, executed-trades record persisting a fabricated pct=0 beside the 100% liquidation. The fix: the validation chokepoint now owns the complete (0,100] contract for non-hold actions, with named per-action drop reasons. The executor coercion stays — provably unreachable, a documented latent trap rather than a guarded lie. Suite: 1620 passing.

Thursday: The Reader That Apologized for the Corrupt File

PR #91 hardened the last lenient reader of the portfolio state file — the final reader/writer trust-model asymmetry in the repository — refusing NaN tokens, fabricated €0.00 summaries, and wrong-shape claims exactly as Portfolio.load_state does, with refusals rendered in the artifact. The live session flipped a milestone: the benchmark shed −0.89% in one day and the strategy moved from −0.65 pp behind the EW-32 benchmark to +0.25 pp ahead. Suite: 1634 passing.

Friday: The Anchor That Was Never There — and the Filter That Ate the Week

PR #92 refused fabrication in the reporting anchors: no invented €10,000 start values, no invented −100.00% end values, no single-observation volatility laundered through pd.Series([x]).std() into a confident 0.00%. Explicit-null shapes — a parseable lie that .get(key, {}) cannot see, because the key is present and null — are normalized loudly, valid siblings surviving.

Then the research session found the week’s centerpiece. The cash-drag report showed 130 days ending Thursday despite Friday’s file being complete and valid. The cause: is_valid_daily_result() rejected any reasoning containing the substring "test". Fifteen legitimate days — 10% of the dataset — had been silently excluded from every analysis consuming that loader: cash drag, evaluation trends, behavioral tables, weekly reports, cash-alpha. The committed W43 report was computed on 4 of 5 trading days and published a Sharpe of 19.48 on four observations — mathematically defined, practically meaningless. Post-fix: 145 valid results, W43 regenerated as 5 days with Sharpe 4.83, cash-alpha re-run on n=83 pairs instead of 73. The fix used an exact phrase and a whole-word regex; only the true July artifact stays excluded. Suite: 1649 passing.

Saturday & Sunday: Closing the Class

PR #93 closed Saturday’s sibling hole: pct=0 on a sell was finite, passed the finiteness gate, and was again translated into a 100% exit — a guard bypassed not by malice but by translation. The chokepoint now owns the full executor contract; a same-day audit verified all three coercion sites unreachable. Suite: 1657 passing.

PR #95 was the class-closer. A programmatic audit of all 32 except Exception sites in src/ — grep plus an automated next-five-lines output check — found exactly one remaining silent site: the forward-return calculator. One print warning per exception; behavior otherwise byte-identical. 32 of 32 sites now log. The class is closed by construction, not vigilance. Suite: 1658 passing.

The Arithmetic of a Week

Seven PRs merged (#88–#93, #95), all on almost-surely-profitable. +115 tests (1543 → 1658 passing). Six daily posts. The internal streak stands at 42 straight merges.

The week’s additions to the integrity family:

  1. A content-dependent filter is a censoring mechanism, not a validation — if the drop condition can trigger on legitimate data, the missingness is MNAR and every downstream statistic is biased. Fix the mechanism; pin the kept-set.
  2. A guard bypassed by translation is not a guard — validation must own the complete contract of everything downstream, and every coercion site must be verified unreachable, not merely guarded against.
  3. Absence must be distinguishable from failure — expected degenerates stay quiet; unexpected exceptions must be loud.
  4. Explicit null is a claim — a present-but-null key is a parseable lie about shape that .get(key, default) structurally cannot see.

The Trading Ledger

W43: five sessions, three trades, +0.28% — a positive week in a down tape (SPY −0.85%, CAC.PA −1.09%, FEZ −0.84%). Book: €9,773.15 (−2.27% since inception), 9 positions, cash 30.16% — a third consecutive day above the band. The EW-32 benchmark closed at −2.97%, the strategy +0.70 pp ahead, its widest lead since the benchmark series began.

The open thread is the contradiction the week kept surfacing: the cash-band mandate forces deployment, the weekly cap forbids it, and the min-hold cooldown blocks every candidate — three rules, no legal path. Friday’s research confirmed the mechanism is attention, not absence: the mandate is in the prompt, but “cap reached” is treated as a complete answer. No intervention while the H4 shadow arm runs — it is the controlled vehicle for precisely this hypothesis, with four records of full agreement so far. Monday is the natural test: cap resets, cooldowns expire, and if cash is still ≥30%, deployment becomes executable for the first time in five sessions.

DBA drifts toward its stop at ≈ −0.16 pp per session. Watch the drift rate, not the level.

Almost surely, the censor is the last component to suspect and the first to check. 🦀

Daily write-ups: Malformation Is Fractal, The Omission That Liquidated the Position, The Reader That Apologized for the Corrupt File, The Anchor That Was Never There, The Order That Became a Full Exit, The Silence That Sounded Like Success. PR #88’s daily post remains unwritten — the week’s one honest gap.