Common Mistakes In Backtesting

There are many traps in backtesting. We end the article with a list of common mistakes most of us do, no matter how experienced we are.

Whenever you have a ready trading strategy, or perhaps even better before you make a strategy, you should ask yourself the following questions:

Curve fitting

Is the data curve fitted? Is it too many parameters?

We have a simple trading rule:

Simple trading strategies with few variables or parameters trump complex strategies. Complex strategies are more likely to go wrong. Instead, try to create many strategies that complement each other. Less is more in trading!

Once you have a minimum of information to form a judgment, it doesn’t improve the outcome by adding more information. Nassim Taleb once wrote that the easiest way to bankrupt a fool is to give him lots of information.

Curtis Faith wrote the following in The Way Of The Turtle:

It takes a lot of time and study before one realizes just how simple trading is, but it takes many years of failure before most traders come to grips with how hard it can be to keep things simple and not lose sight of the basics (page 115)…..Keep it simple. Simple time tested methods that are well executed will beat fancy complicated methods every time (page 131)……In a similar manner, simple rules make systems more robust because those rules work in a greater variety of circumstances (page 212)…….People have a tendency to believe that complicated ideas are better than simple ones….Some of us thought that trading successfully couldn’t possibly be that simple; that there must be something else to it (page 224).

We have a bit more about curve fitting in the next lesson.

Is the strategy looking forward?

For example, you might have done a coding error and entered at the open, when the close is correct. Thus, the results might be overstated. This error might sound obvious, but more subtle mistakes can be difficult to understand.

Survivorship bias

Is the strategy likely to have ignored survivorship bias?

Bad data?

Are you using cheap free data or are you subscribing to quality data? Always ask yourself if the data is trustworthy. We recommend subscribing to a quality service, like Norgate, for example.

Lacking logic?

The markets are often counterintuitive and it’s very hard to make predictions. Jim Simons says that they stopped looking for logic in their strategies. If the strategy has many trades and the results are statistically significant, they trade it (if it adds diversification).

However, we somewhat disagree with this. You probably have more faith in the strategy if you believe you understand the logic behind the strategy.

A few winners make all the difference?

A look at the winners and losers might be useful. 

For example, remove the top 1% of winners.

Sometimes a couple of bigger winners can make up the majority of your strategy’s historical performance.

This is not necessarily bad, but it also depends on the type of strategy. Mean reversion strategies tend to have many winners and occasionally big losers, while trend following tends to have many small losers and a few big winners. 

No journal(s)

You need both a trading and backtesting journal. We are of the following opinion: The more data you record, the better you’ll trade. You need to record all trades, but equally important is keeping track of all your backtests.