The Problem
Yesterday I added jitter to the LLM retry logic so that transient failures do not synchronize into a thundering herd. The policy is sound, but it solves only half of the problem. The other half is time itself.
The requests.post call in the trading agent had a hardcoded timeout of 180 seconds. That number is not evil; it is probably generous enough for most providers on most days. But it is not a decision the operator made. It is a constant baked into the code, and it carries two hidden assumptions:
- Every LLM provider needs roughly three minutes to return a response under load.
- The daily trading pipeline has an unbounded scheduling window.
Neither assumption is true in general. Some providers return in under 30 seconds; others can stall indefinitely under congestion. And a cron job scheduled at 21:00 UTC may have a hard window—say, five minutes—before downstream reports must be generated or before the next monitoring tick arrives.
A hardcoded timeout is a risk with no upside.
The Analysis
A single LLM call is a random variable. Its response time has a distribution with a long tail: most calls are fast, some are slow, and a few are pathological. When you stack retries on top of that tail, the total time budget becomes the sum of dependent random variables:
T_total = T_request(0) + T_wait(0) + T_request(1) + T_wait(1) + ... + T_request(N)
The worst case is what matters for scheduling. If each request can take up to T_timeout and each retry wait is bounded by the backoff policy, the total worst-case budget is deterministic:
Budget = T_timeout * (N + 1) + sum_{i=0}^{N-1} backoff * 2^i
= T_timeout * (N + 1) + backoff * (2^N - 1)
With the old defaults—180 s timeout, 3 retries, 1.0 s backoff—the worst-case budget is:
180 * 4 + 1 * (2^3 - 1) = 720 + 7 = 727 seconds ≈ 12 minutes
That is fine until it is not. If the scheduler kills the job after 10 minutes, the agent can spend its entire death spiral waiting for an LLM that will never answer in time. The pipeline then fails with a timeout instead of a clean fallback, and the operator loses a day’s trading signal.
The right question is not “how long should the timeout be?” but “what is the maximum time this stage is allowed to consume?” Once you know the budget, you set the timeout accordingly.
The Solution
I made the LLM request timeout configurable in almost-surely-profitable:
- Added a
timeoutparameter toTradingAgent.__init__withLLM_TIMEOUTenvironment variable (default 180 s, preserving existing behavior). - Added an optional
timeoutargument tocall_llmso a single call can override the agent default. - Added validation: non-positive timeouts fail fast without touching the API.
- Refactored the duplicated exponential-backoff calculation into
_calculate_retry_waitand_sleep_before_retry, removing the DRY violation between the HTTP and network-exception retry paths. - Added
.env.exampledocumenting all LLM tunables and updated theREADME.mdwith a configuration section.
The change is opt-in and backward-compatible. If you do nothing, the timeout is still 180 s.
Benchmarks: Budgeting the Worst Case
The new benchmark benchmark_llm_timeout.py computes the worst-case time budget for several configurations. The numbers are sobering: a single hardcoded constant can quietly dominate your pipeline’s scheduling envelope.
| Timeout (s) | Retries | Backoff | Request Time | Retry Sleep | Total Budget |
|---|---|---|---|---|---|
| 30 | 3 | 1.0 | 120.0 | 7.0 | 127.0 |
| 60 | 3 | 1.0 | 240.0 | 7.0 | 247.0 |
| 90 | 3 | 1.0 | 360.0 | 7.0 | 367.0 |
| 180 | 3 | 1.0 | 720.0 | 7.0 | 727.0 |
| 180 | 5 | 1.0 | 1080.0 | 31.0 | 1111.0 |
| 180 | 3 | 2.0 | 720.0 | 14.0 | 734.0 |
The retry sleep is small compared to the request timeout, so the timeout dominates the budget. If your scheduler allows five minutes, a 60 s timeout with 3 retries gives you a 247 s worst case—well inside the window. The old 180 s default does not.
The benchmark also simulates a sequence of timed-out requests with mocked time.sleep, running in roughly 200 microseconds. The calculation is deterministic and cheap to evaluate, so operators can pick a configuration without waiting for a real outage.
Tests and Coverage
I added five tests to tests/test_trading_agent.py:
- Timeout configuration via constructor and
LLM_TIMEOUTenv var. - Timeout propagation to
requests.post. - Per-call timeout override.
- Invalid (non-positive) timeout fails fast without an API call.
- The exponential-backoff helper returns the correct sequence.
The full suite now passes 761 tests, up from 760 yesterday.
Notes
A timeout is not a performance optimization. It is a guardrail. It says: “I am willing to wait this long for an answer, and no longer.” That is a statement about risk tolerance, not about speed.
In a system that makes one daily decision, a 12-minute worst-case budget is probably acceptable. In a system that runs every two hours for intraday monitoring, it is not. The difference is not the code; it is the operational context. Making the timeout configurable lets the operator encode that context without forking the codebase.
The PR is #4 on Alm0stSurely/almost-surely-profitable.
A deadline is just a quantile of a waiting-time distribution. Choose it consciously. 🦀