My Backtest Was Lying.
So I Built One That Can't.

The ₹9 crore spreadsheet

At one point a side strategy of mine backtested at 76.9% CAGR. Compounded forward, the spreadsheet promised ₹9 crore. When a backtest says you're about to be rich, it isn't a forecast. It's a bug report.

The first bug was the universe. I'd been testing on today's Nifty 500 projected backwards, a list that only contains companies that survived. I guessed that was worth 12.7pp a year. Wrong again: measured properly it's +3.8pp a year, handed out free to anyone, including someone picking at random.

Beat a random portfolio, not zero

So the benchmark changed. Every idea now has to beat five random rankers drawing from the same pool, on the same dates, through the same production code, paired date by date. Five monkeys with dartboards, except the monkeys pay the same costs and taxes I do.

The monkeys went four for four:

• The ranker from my August write-upIts "top picks" were a 19-way tie at a capped score, broken by a feature with no signal.coin flip
• Its fixA continuous score instead of a checklist.+0.44pp · t = +1.6
• The next twoMeasured on a universe rebuilt as of every date.−0.02pp · −0.08pp

All four retired before they touched real money. That is the control doing its job.

Six ways the data lied

NSE's raw bhavcopy is a daily file of what printed, not a research dataset. Each of these was caught by a check built for exactly that job, then fixed for good. Some favourites:

Then I audited the "honest" version

By mid-September the cleaned-up backtest said 29.6% CAGR. Before publishing it, I did what the system is built for and audited it, as developer, market analyst and researcher at once. It found 14 defects and took the number to 19.1%.

The scariest one wasn't in the data at all. A date join plus a pandas 3 change (pct_change stopped padding gaps by default) meant production's signal correlated only 0.92 with the one I'd validated. On the same panel they scored 23.5% vs 30.1%. The backtest had been grading a strategy I wasn't running.

Here's the part I'm proud of: the edge barely flinched. Through all 14 fixes, the per-pick edge over random went from +4.83pp to +4.39pp, still at t = +11.3. The bugs had inflated the compounding, not the signal.

A two-day change worth +4.9pp. Allegedly.

Different start dates share most of their trades, so luck survives averaging and t-stats. Two of my September "wins" turned out to live inside this noise floor. Now parameters are chosen on a path-free per-pick measure, and CAGR is quoted as a mean over 15 economically equivalent configs × 45 start dates = 675 paths. By that estimator the old book was 21.8% (its single config had landed in an unlucky pocket). Re-shaping the book on the per-pick measure took it to 24.6%, ahead of the old book in 15 of 15 equivalent configs.

+4.77pp per pick over random. t = +13.0.

Positive in both halves of the sample, and simulated the boring, honest way: point-in-time universe, production code path, next-day-open fills, gap-downs filled at the open, 0.5% round trip, and STCG/LTCG plus cess netted per financial year.

Pre-tax it's 30.9%. Even the 10th-percentile path beats the index by 5 points, and the unluckiest of 675 paths still made money (+5.4% a year). The price is the ride: median max drawdown −30%, worst −54%. Blending in an ETF basket keeps the return (~23.4%) and pulls the worst drawdown in to −29%.

The system that caught all of it

A backtest that lies is normal. One that gets caught, every time, before money moves, is the actual product. Random controls, noise twins, a path-noise null and as-of reads exposed every lie above, and they now gate every new idea. Here's what else held up:

Most ideas die. That's the job.

250+ specifications tested; roughly 1 in 70 survived. That isn't a bad hit rate, it's the published one: Hou, Xue and Zhang found 64% of 447 published anomalies insignificant. Every result, pass or fail, is logged with its number, so no idea gets paid for twice.