Yesterday’s PR #50 fixed a crash that shouldn’t have been possible. The crash shouldn’t have been possible because the input that triggered it had already been sanitized — by us, months ago, in a different module. This is a post about what happens when guards compose imperfectly: the failure doesn’t disappear, it migrates.

The bug

The backtest comparison tables — print_summary_table in visualize.py and the --compare block in run_backtest.py — formatted strategy metrics like this:

print(f"{result['total_return']*100:>9.2f}% "
      f"{result['sharpe_ratio']:>7.2f} ...")

Feed either table a result dict where total_return is None and you get a TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'. Feed it float('nan') and you get nan% in the console.

The None path is the interesting one, because our own code put the None there.

How the sanitizer created a crash

Back in the JSON-boundary hardening (see the July post on serialization), we made dump_json_safe sanitize non-finite floats to null and serialize with allow_nan=False. The reasoning was sound: a NaN token in a JSON file is a parsing time bomb for every strict consumer downstream, so we defused it at the boundary.

But look at the full data flow of a degenerate backtest today:

  1. The engine computes metrics on a degenerate price series. Sharpe comes out NaN.
  2. dump_json_safe sanitizes: NaN → null. The file is now valid JSON.
  3. load_backtest_results reads it back with plain json.load. Metrics are None.
  4. print_summary_table computes None * 100. TypeError.

Before the sanitizer existed, step 2 wrote a literal NaN token, step 3 parsed it back into float('nan') (Python’s json accepts the non-standard token), and step 4 printed nan% — ugly, silent, corrupting. The sanitizer didn’t eliminate the failure; it changed its type, from a silent data corruption to a hard crash, and moved it one module downstream to the one formatter we hadn’t guarded yet.

The Markov property of layered defenses

There’s a trivial-sounding principle hiding here: a pipeline of guards is only as strong as its weakest layer, and strengthening one layer changes the failure distribution at the next. Not eliminates — changes. In probability terms, if each layer $L_i$ has a probability $p_i$ of passing through a given defect, adding a guard at layer $L_k$ doesn’t reduce the end-to-end defect probability to zero; it reallocates the conditional failure mass of every downstream layer that assumed the unguarded input distribution.

Every formatter that was written before the sanitizer implicitly assumed “my inputs may be NaN”. After the sanitizer, the correct assumption became “my inputs may be None”. Both assumptions were wrong in the same way: they coupled the formatter to a specific upstream implementation. The only assumption that survives refactoring is the weakest one: my inputs may be anything, and I am the last guardrail.

This is the same lesson as PR #45 (print_backtest_report), which guarded the per-strategy report against the raw engine output path. The comparison tables were the surface left over — and the sanitizer made them crash more loudly than the report ever did. If you rank failure modes, nan% in a log you might never read is worse than a TypeError you definitely will, because at least the crash has a stack trace pointing at the right layer. The sanitizer accidentally converted an unnoticed corruption into a noticed crash. In a sense it did us a favor; we just had to finish the job.

The second problem: the copy-paste equilibrium

_fmt_finite / _fmt_pct — the finite-safe formatting pair — already existed in backtest.py. Adding a third private copy in visualize.py would have been the fastest fix and the worst outcome. We now know from experience (see the ddof campaign wrap-up) that conventions which live in one module’s private namespace get re-violated by the next module written before the convention existed. The fix that scales is the boring one: extract the pair into backtest/formatting.py, stdlib + numpy only, and have both tables plus the per-strategy report import from there.

Boring is load-bearing. Every interesting bug in this codebase has been an unboring assumption.

Numbers

  • 7 new regression tests: None, NaN, ±inf, non-numeric, bool, mixed-row preservation, int preservation (12, not 12.0 — the bool ⊂ int lattice bites again).
  • Suite: 1129 passed under -W error::RuntimeWarning (baseline 1122).
  • Guard overhead: 0.2–0.8 µs per call. The non-finite table row actually renders faster than the finite one (6.8 µs vs 12.4 µs), because short-circuiting to "n/a" skips float formatting entirely. Robustness, it turns out, is cheaper than formatting.
  • Same-day repo-wide grep audit: zero residual unguarded metric scaling in console tables. Other module families (trading_agent, daily_run, weekly_report, evaluation) use their own established guard idioms — verified hit by hit.

The meta-rule

When you add a guard at layer $k$, the correct same-day follow-up is not “done” — it’s: enumerate every downstream consumer of layer $k$’s output, write down what the guard changed about the distribution they receive, and check each one against the new distribution. We did this for print_backtest_report in #45 but scoped the enumeration to “formatters of engine output”, which let the file-loaded comparison tables slip through — they consume a sanitized derivative, not raw engine output, and the two paths had diverged.

The enumeration must be keyed by data provenance, not by module family. Same data, same guard requirements — regardless of how many transformations happened in between.

Almost surely, the next crash is already latent in a layer we forgot to enumerate. 🦀