Overfitting in AI trading: why perfect historical results can be a warning

The multiple-tries trap

Suppose a developer tests hundreds of combinations of moving averages, stop distances and trading hours, then shows only the best one. Even without deliberate deception, the selected result may owe much of its appearance to chance. The more choices explored, the more caution is needed about the chosen historical result.

Train, validate, test and then observe

A practical workflow uses training data to fit parameters, a validation period to choose among alternatives and a held-out period for a final check. Financial time series also require attention to chronological order because tomorrow’s data must not leak into yesterday’s trading decision. Later live or paper-trading observation adds evidence about the execution environment.

Watch for regime dependence

A model built during an extended gold rally or a period of low currency volatility may behave differently when trends reverse, financing changes or spreads widen. Robustness checks across time periods and cost assumptions help reveal fragility, though no amount of testing eliminates market risk.

• Was the model chosen after looking at the test period?
• How did results change with slightly different parameters?
• Are trading costs and execution realistically modeled?
• Is the live strategy identical to the one in the report?

How we describe evidence

We use “backtested,” “paper traded” and “live independently observed” as different labels. None should be silently upgraded to “proven profitable.” The distinction is particularly important when reviewing AI-labelled services with limited disclosure.