Hi all, my team and I just finished a research paper and I’d really like people with quant, trading, or stats experience to review it and tell us what we got wrong.
WHAT WE TESTED
A rule-based strategy on BTC/USDT 4-hour candles combining MACD momentum, a KDJ entry gate with an adaptive exit, dual-ATR risk sizing (14 and 200), and Smart Money Concepts Order Block detection. We compared it against weekly dollar-cost averaging and lump-sum buy-and-hold from the same $10,000 start.
HOW WE TRIED NOT TO FOOL OURSELVES
- Rules were set by eye on 2018-2022 charts, then frozen. No optimizer or parameter sweep.
- Reported window is 2022-2026 (8,767 candles), plus a separate forward test for Jan-Jul 2026.
- We found and fixed a lookahead bug in two Order Block criteria (it didn’t change the baseline trades).
- Every number is traced to the function that produced it (Appendix A), and Appendix D lets you recompute the headline figures by hand from the trade log.
RESULTS (INCLUDING THE UNFLATTERING ONES)
- 27 trades, 70.37% win rate, +30.31% total return, one-sided binomial p = 0.0261. Bootstrap 95% CI on total return: [+5.99%, +54.99%].
- With a realistic 0.20% round-trip cost, win rate drops to 62.96% and p rises to 0.1239. The significance doesn’t survive, and it hinges on just two marginal trades.
- Weekly DCA clearly won on final value ($22,313 vs $13,418).
- The strategy had a higher Sharpe (1.11 vs 0.56) and a shallower drawdown (-6.5% vs -27.9%), but it was in the market only 2.63% of the time, so it’s not like-for-like.
- Putting 10-30% of weekly contributions into the strategy improved both Sharpe (0.556 to 0.637) and max drawdown (-27.85% to -23.18%). This is descriptive only, no p-value.
- Forward test: -3.45% over 4 trades.
- Three trades make up 51.7% of total return, and the edge is concentrated in bull regimes.
WHAT I’D LOVE FEEDBACK ON
- Is the train/test separation (visual rule-building, then a frozen test window) defensible, or too weak?
- Is a binomial test on win rate plus a percentile bootstrap on 27 trades reasonable, or is there a better approach?
- Any flaws in how I handled costs, the Sharpe comparison, or the blended-DCA framing?
- Any lookahead or leakage risks I’ve missed?
FULL PAPER (Google Doc, comments open):
https://docs.google.com/document/d/1uITFSqdj0iqwA91pXcfSRGGTkWFoQulLYYyRZUcynQw/edit?usp=sharing
Not financial advice. This is a backtest, and the only forward evidence so far is negative. Harsh feedback is welcome. Thanks for reading!