Out-of-sample and incubation

In the previous lesson, we looked at the importance of good data and backtesting.

It’s easy to get fooled by backtesting: you curve fit, you accidentally find some strategy that only works during certain cycles, or you backtest on bad data.

This lesson is about how you can build a safety valve to avoid this: you test your data on real future data.

Out-of-sample test

Out-of-sample testing is simple:

Let’s say you have data from 2005 until 2021 and want to backtest an idea. Then you can split your data into two parts, for example from 2005 until 2017, and then use 2018 to 2021 as out-of-sample test.

Basically, you make your strategy on the data until 2017 and base your conclusion on that time period.

When you have made a strategy on the data up to 2017 you test the strategy on the “unknown” data from 2018 to 2021. If the strategy performs well on the “unknown” data, you might have a very good strategy.

Then you can jump to the incubation period:

Incubation:

We recommend you have a demo account. For example, Interactive Brokers automatically makes a demo account available to you on-demand (if you have a real account).

In this demo account, we recommend you paper trade on real data all your potential strategies for several months before you commit your hard-earned capital.

This has two advantages:

First, you get to see how the strategy performs in real-time. Doing a backtest compresses time, and you don’t get any “feel” for how it is to trade the actual strategy.

Secondly, the incubation period acts as a second safety valve (the out-of-sample backtest being the first). In our own trading, we filter out a lot of bad strategies this way and this has saved us a lot of money over the years.

 

This is all there is to making safety valves. It takes little time, except for the incubation period, but it is well worth it!