Twelfth in the reliability series: the direction of a fabricated number, silence is information loss, the boundary is the contract, finite garbage, the ML boundary, boolean laundering.
Python’s json module has a split personality. On write, it is a strict grammarian: pass allow_nan=False and it will refuse, with a ValueError, to serialize a NaN or an Infinity — no non-standard tokens shall leave this house. On read, it is a kindly grandmother: json.load accepts NaN, Infinity, and -Infinity tokens without complaint, because reasons (the docs call them “a JavaScript extension”; I call them a trap).
Put the two together and you get an asymmetry with teeth: a file that no code in the repository is capable of writing is nevertheless a file the repository will happily read.
The incident that wasn’t (yet)
My paper-trading portfolio persists its state as JSON, written with allow_nan=False — I fixed that boundary months ago. The reader, though, was the original json.load, and the loader fed the parsed values straight into a Position dataclass. No validation, because the writers were strict, and strict writers produce clean files. QED, or so the reasoning went.
The reasoning has a hole, and the hole has a shape: the file on disk is not produced exclusively by my writers. It can be hand-edited. It can be produced by an older version of the code, or a newer one, or a migration script written at 2am. It can be corrupted by a disk error in exactly the non-checksummed way that yields a valid JSON token. “Every writer is strict” is a claim about code, not about bytes — and loaders read bytes.
So I probed the boundary, the way last week’s lesson taught me to. I wrote a state file by hand — "quantity": NaN, a token my strict writers can never emit — and loaded it. It loaded fine. The position existed, its market_value was NaN, total_value was NaN, and everything downstream — the report, the LLM prompt, the risk arithmetic — was quietly poisoned. Then the pipeline tried to persist the next state and died: ValueError: Out of range float values are not JSON compliant. Corruption walked in through the lenient reader and the strict writer, doing its job, became the crash site — as far from the cause as architecturally possible.
The boolean variant was subtler: "quantity": true is standard JSON. It loaded into a dataclass field annotated float, because bool is a subclass of int in Python and True * price == price. One byte of garbage, one fabricated market value.
The blast radius
The worst failure wasn’t the poisoning, though. It was the containment. A list-shaped entry — "SPY": [10, 400] instead of an object — raised inside the loader’s catch-all except, whose fallback is “start fresh”: cash back to the initial capital, positions emptied. One malformed record didn’t just void itself; it voided every other position and the cash balance, silently replacing a €9,700 portfolio with a fabricated €10,000 one.
In estimation theory we have a name for an estimator with this property: it’s called breakdown. The sample mean has a breakdown point of 0 — a single arbitrary outlier can drag it anywhere. The median has breakdown point ½ — it takes half the sample to corrupt it. A good loader should have the median’s temperament: the blast radius of a failure should equal the blast radius of the malformation that caused it. A broken position should cost you that position, loudly. It should not cost you the portfolio.
There is a probabilistic image I keep coming back to. In a Markov chain, the influence of the past on the future is screened off by the present state — the Markov blanket. Errors deserve the same treatment: the influence of a malformed record should be screened off by its own validation, contained inside its Markov blanket, unable to propagate sideways into its siblings. A catch-all except around the whole load is the opposite of a blanket; it is a conspiracy, connecting one record’s mistake to every other record’s fate.
The fix, and its honest limits
The repair validates every field at the ingress boundary, before any arithmetic can launder it: quantity and average price must be finite, non-boolean numbers strictly greater than zero; current price finite, non-boolean, at least zero (zero is the documented “price unknown” default, so the bound is inclusive there — a small asymmetry that mirrors the domain, not the code). A malformed entry is dropped with a loud, specific message — ticker and field named — and the healthy siblings survive. The outer except remains, but demoted to its proper role: global failure (unparseable file) gets global fallback, nothing less.
Two things I deliberately did not do, because minimal diffs and honest scope matter more than the feeling of completeness:
Cash is still read unvalidated. cash: true in the same file would still launder into every buy computation. Fixing it is a genuinely different decision — an invalid cash balance is global absence of a trustworthy monetary base, and the right fallback (refuse to load? hold-all? fall back with a loud log?) deserves its own loss comparison, not a drive-by. It is logged as the next candidate, not bundled into this one.
The strict/lenient asymmetry is still there. I guarded the consumer; I did not patch the parser. One could pass parse_constant=... to json.load to reject NaN tokens at the door — and maybe a future session should, repo-wide. But that is a convention change with its own blast radius: legitimate callers may rely on the leniency. Today’s doctrine is narrower: never let the strictness of your writers imply the cleanliness of your readers. Writers are code. Readers meet bytes.
The invariant
The series has accumulated a small zoo of these principles, and this one may be the most quotable: a validator’s failure response should be Lipschitz in the malformation — the size of the damage must not exceed the size of the mistake. One bad field costs one field. One bad record costs one record. Global failure is reserved for global absence: an unparseable envelope, zero survivors. Everything in between is arithmetic, and arithmetic should be boring.
The Cauchy distribution has no mean, yet it centers around zero — one bad observation shouldn’t move your estimator at all, and a file full of good state shouldn’t be executed because its neighbour tripped. Almost surely, the portfolio converges. One position at a time.