3,125 Backtests Before Lunch: Parameter Sweeps with Parquet Caching

A single backtest tells you whether a strategy worked with one set of parameters. That's almost worthless on its own. The question that actually matters is: is the edge robust, or did you just get lucky on bb_period=20?

Answering it means a sweep — running the strategy across a grid of parameter combinations and looking at the shape of the results. A strategy whose Sharpe collapses the moment you nudge a parameter is overfit. A strategy that holds up across a broad plateau is real. You can only tell the two apart by running hundreds or thousands of backtests and looking at them together.

The problem has always been cost. Five parameters across five values each is 5^5 = 3,125 backtests. If each one re-queries your historical data from the database, you've turned a research question into an overnight batch job — and research that takes overnight is research you do once a quarter instead of once an hour.

We attacked the cost directly. The sweep harness is trivial; the parquet cache is what makes it fast. Together they turn a sensitivity study into something you run before lunch.

The sweep itself is boring (on purpose)

There's no magic in the grid expansion. It's a Cartesian product over your parameter ranges:

def _cmd_sweep(args):
    grid = json.loads(args.grid)
    keys = list(grid)
    combos = list(itertools.product(*[grid[k] for k in keys]))
    for combo in combos:
        params = dict(zip(keys, combo))
        strat = build_strategy(args.kind, params)
        res = run_backtest(strat, args.start, args.end, config=cfg,
                           use_cache=not args.no_cache)
        # collect res.metrics keyed by params

From the command line that's:

python -m development.fund.backtest.cli sweep --kind pairs_bollinger \
    --start 2024-06-01 --end 2025-06-01 \
    --grid '{"bb_period":[15,20,30],"bb_stdev":[1.5,2.0,2.5]}'

Nine combinations there; expand each list to five values and you're at 3,125. Each runs the exact same engine a single backtest uses — same fills, same funding accrual, same point-in-time discipline — so the sweep results are directly comparable to (and as trustworthy as) any one run.

The boring part is the point. A sweep is just "the same honest backtest, many times." All the engineering effort went somewhere more useful: making "many times" cheap.

Where the time actually goes — and the cache that reclaims it

Run a naive sweep and profile it, and you'll find the strategy logic is a rounding error. The wall-clock is dominated by one thing: pulling bars and funding out of the database, over and over, for the same window.

Every combo in {"bb_period":[15,20,30],"bb_stdev":[1.5,2.0,2.5]} runs over the identical date range on the identical instruments. The parameters change how the strategy interprets the data — they don't change the data. So fetching it 3,125 times is 3,124 wasted round-trips.

The parquet cache eliminates them. Resolved canonical frames are keyed and written to local parquet on first fetch, and every subsequent run over the same window reads from disk:

"""Local parquet cache for resolved canonical frames — the dev-loop speedup.

The live signal runner always bypasses the cache (freshness); only backtests use it."""

def path_for(physical_table, interval_seconds, instruments, start, end,
             cache_dir=None, as_of=None):
    d = cache_dir or _DEFAULT_DIR
    return os.path.join(
        d, f"{_key(physical_table, interval_seconds, instruments, start, end, as_of)}.parquet")

The cache key is the content of the request — table, interval, instruments, window, and any knowledge-time pin. So the first combo pays the database cost once; combos 2 through 3,125 hit warm parquet and skip the network entirely. A sweep that would have been thousands of identical queries becomes one query and thousands of fast local reads.

Two details show this was built by people who've run real sweeps:

• Knowledge-time isolation. The as_of pin is part of the key: "a knowledge-time pin makes a distinct cache entry so a reproducible (pinned) run never reads a parquet built under a different knowledge state." You can't accidentally contaminate a point-in-time-pinned sweep with cache from a live run.

• The cache never breaks a run. Writes are best-effort. A locked-down container with a read-only ~/.cache raises PermissionError, and the engine swallows it and proceeds — you lose the speedup, not the result. The cache is an accelerator, never a dependency.

And the live signal runner bypasses the cache entirely — freshness wins over speed when real orders are at stake. The cache speeds up research; it never touches production decisions.

Reading a sweep: shape over peak

Once you have 3,125 results, the temptation is to grab the top-Sharpe combo and ship it. Resist it — that's how you overfit. The whole reason to sweep is to read the surface. Each result carries the full metric set the engine computes per run — total and annualized return, Sharpe, Sortino, max drawdown, win rate, profit factor — so you can rank and filter on whatever matters to your mandate.

What you're looking for isn't the single highest peak. It's a plateau: a broad, contiguous region of the grid where the strategy stays good. A plateau means the edge tolerates parameter error — it'll survive the inevitable drift between your backtest window and live. A lone spike surrounded by mediocrity means you found noise, and noise doesn't pay.

That read is only possible when running the whole grid is cheap enough to do casually. When a sweep costs you an afternoon, you run the minimum and grab the peak. When it costs you a coffee, you run the full grid, see the surface, and ship the plateau.

Velocity compounds

There's a reason we obsess over this. Research throughput compounds. A quant who can test ten hypotheses a day finds robust edges an order of magnitude faster than one who can test one. The bottleneck is almost never ideas — it's the latency between having an idea and seeing whether it holds.

By making the data cost of a sweep approximately one fetch instead of thousands, the platform collapses that latency. Sweeps stop being a scheduled batch ritual and become part of the thinking — you change a hypothesis, re-sweep, and look at the new surface before the old one's left your head.

That's what "3,125 backtests before lunch" actually buys: not a faster computer, but a faster you.

---

Define a grid, fire a sweep, and read the surface — the data's fetched once and reused across every combo. Run a sweep →