Curve Fitting

The enemy of a backtest is curve fitting. In this lesson, we explain what this is and how you can avoid it.

Let’s start by defining what curve fitting really is:

Curve fitting is when a strategy or edge is not fit to market behavior, but market noise, leading to failure in live trading. Curve fitting is overoptimization that is unlikely to fit into future unknown data.

We have in a previous lesson explained how optimization helps you in trading, but you need to be careful so as not to fall into the curve fitting trap.

The counterintuitive rule about backtesting

We believe backtesting is a great tool, but always keep in mind that there is a counterintuitive rule in backtesting: The more you test and the more ideas you bring to the table, the chances of finding something that works increase.

Be careful! Finding something of real value often diminishes the more you backtest.

Why is that?

There are tens of thousands of traders using a lot of computer power to find trading edges, but they still lose money. Backtesting often has diminishing utility and goes toward zero.

This is why we recommend testing and trading something that is logical. You must have a good idea to backtest. Then, when you have backtested it, you should try to debunk it, and not add more variables or filters. By doing the latter (adding variables), you increase the chances of curve fitting!

What can be done to avoid curve fitting?

Let’s look at some ways to avoid or reduce curve fitting from your backtests:

1. Less is more in trading

In trading, you should always strive to make your edge as simple as possible. Overcomplicated edges with numerous conditions and rules tend to be curve fit.

• 2 Reasons Why Less Is More In Trading
• Simple Vs Complex Trading Strategies
• Street Smarts Beat Book Smarts In Trading?

It’s no problem finding something that looks great on paper as long you add filters and conditions. But such a strategy is unlikely to hold up well in the future.

2. Parameter stability

Parameter stability is a good indication of whether a strategy is curve fitted or not. In general, we want as many parameter combinations to produce desirable results as possible. This is why you want to “optimize” a backtest. Preferably, an optimization should show good results for a wide range of parameters.

3. In-sample and out-of-sample testing

To avoid curve fitting, out-of-sample and in-sample testing is a crucial part of every trader’s methodology, in one way or another, as described in a previous lesson.

By dividing our data into a training set and a validation set, we can test our idea on the training set, and later verify it on the validation set.

4. Incubation

To put it outright, incubation is out of sample testing, but with one major difference:

With typical out-of-sample testing, all data is historical. With incubation, we use future data that is not available yet! This is a process that should take at least several months.

5. Monte Carlo simulation

Monte Carlo simulation in trading is a method that can be helpful in some cases. It works by reshuffling the order of the trades in a backtest and has the potential to expose weaknesses that otherwise had been hidden in the backtest.

Monte Carlo could be a lesson on its own, but we believe you can test yourself in your trading software.

Curve fitting – conclusion

Most backtests you see on the internet are a result of curve fitting. The first thing you should always ask yourself after a successful backtest is to debunk it. Why should this work? Is this simple enough to work? Is there any logic behind this idea?

It’s much better to get rid of a strategy before you commit money than after!