Pros And Cons Of Backtesting
Backtesting is, unfortunately, not a bulletproof way to riches. It has many advantages, but it also comes with disadvantages.
In this lesson, we look at both the pros and cons of backtesting. Let’s start with the advantages:
Advantages of backtesting
There are 5 main advantages of backtesting:
Advantage 1: you can confirm or falsify a trading idea
Backtesting works because you can easily check if something has worked in the past or not.
- Is, for example, the Turnaround Tuesday in stocks true or just a myth? Just define the rules and start the backtest. You’ll find out in five minutes.
- Do you believe you see an interesting pattern in the chart? Then quantify it with strict buy and sell rules and test it. We tend to see patterns when in reality there are none.
Did the strategy work in the past? If something has not worked in the past, you can easily falsify your hypothesis and go on to test another idea. If it has not worked in the past, why should it work in the future?
Because most ideas don’t work, you should not spend much time testing a hypothesis. Many traders waste months, even years, in both programming software and tweaking their strategies only to find out it was a waste of time. You don’t need “perfect” strategies to make money in the markets. You need many strategies that complement each other (more about this later).
One of the main reasons for your success (or no success) depends on your ability to test and generate trading ideas. One of our main trading lessons is that we spend about 80% of our time testing ideas back and forth between ourselves.
Advantage 2: Backtesting lets you automate
If you have successfully backtested a strategy, you can easily go “live” with the strategy. In Tradestation, you just check a box and you are good to go.
In Amibroker, you need to add code to automate and let Amibroker keep track of your positions and strategies. We have even made a course that lets you automate your Amibroker strategies to Interactive Brokers:
Obviously, this saves you a lot of time. There is no need to check quotes, prices, or follow the markets. The computer does all the work!
Advantage 3: You can exploit the law of large numbers
Your computer can easily trade and supervise hundreds of strategies. This lets you exploit the law of large numbers and you can diversify into time frames, asset classes, directions, and types of strategies. When you have done a successful backtest you might go live with the strategy.
The main reason for the success of the Medallion Fund is twofold: they use enormous amounts of data to generate hundreds of uncorrelated strategies. Because of the low internal correlation among the strategies they can use leverage to boost returns.
We’ll get back to correlation later in the course.
However, leverage is dangerous and certainly not something we recommend. Only use leverage if you have many years of experience.
Advantage 4: Backtesting reduces emotions
Ample evidence points out that individual investors underperform the averages, and women are better investors than men. The main reason for this is behavioral mistakes:
Investors tend to sell in a panic and buy after a big rise. Most of the time you need to do the opposite. A backtest can’t capture such mistakes and that’s why you need to stick to the trading plan.
When you have a clear plan with written rules and the computer does all your buying and selling, you reduce the emotional pull to money. To trade well you need detachment to money.
Overruling your systems and strategies is unlikely to end well. You have not backtested overruling, so how do you know if it works? That’s why you backtest a trading strategy!
Advantage 5: Backtesting saves time
You can backtest about 15 hypotheses in Silvers Miners (ticker code SIL) in about 1.5 hours. You can generate and test hundreds of strategies in just a single day. This way, you can falsify or confirm the ideas quickly.
Trading is mainly about trial and error. And luckily, backtesting a trading strategy is a great tool for that and at the same time, it saves you a lot of time.
Disadvantages of backtesting
There are many reasons why backtesting doesn’t work, like curve fitting, market cycles, chance, luck, and randomness. But the good thing is that you can avoid or at least minimize many of the disadvantages of backtesting.
We are proponents of quantifying trading strategies, but you need to understand how to avoid the many pitfalls in backtesting.
Disadvantage 1: The backtest might be liable to favorable conditions
The backtest is perhaps performed in a certain period and the markets may have been favorable to that trading strategy in that period. Most traders neglect this aspect. Testing over a longer time frame might minimize this.
Take for example so-called trend following strategies. They perform quite badly in certain periods covering many years, and much better in others.
If you are testing a moving average breakout, this might yield mediocre results over a five-year period. This is a typical strategy that needs to be tested over a period of at least 10 years.
Obviously, you don’t want to change a strategy in response to one year just because something didn’t work. That’s when you’re almost guaranteed that it would have worked the next year had you kept it as it was.
The ever-changing market cycles make trading a difficult task to follow. You have to accept drawdowns to make money and every strategy has drawdowns.
Disadvantage 2: Backtests work because you know the after the fact
You can only trade the strategy after the fact.
For example, you are using entry at the close. Problem is, you only know if it’s a trade after the close. In order to trade on the close, you might have to guess/estimate there wil be a trade at the close.
So when backtesting, it’s crucial you take this factor into consideration. One way to do this is to trade at the open the day after the signal or to trade just seconds before the exchange closes.
Disadvantage 3: Backtesting involves elements of curve fitting
Another reason is the curve fitting (see separate lesson later) aspect. This certainly applies if you’re having a lot of parameters or variables.
It’s easy to come up with a system that has performed remarkably well. You just need to put in a lot of parameters. That will explain the past, but most likely not the future.
The more simple the system, the more likely it’s to stand the test of time.
Curtis Faith explains in The Way of The Turtle some trend-following strategies that are incredibly simple. That’s probably why they work! Over several decades they have worked well in currencies and commodities (not on stocks).
However, over a period of 1-5 years, they sometimes experience quite huge drawdowns. Still, these systems are so simple that they are less prone to be curve fitted.
Disadvantage 4: Survivorship bias
Another problem is survivorship bias.
Put simply, this relates to the use of stocks/tickers that have “survived” the testing period.
For example in 2008/2009 a lot of stocks went bust (Lehman being an example). This means that companies that have gone bust are excluded from the analysis at later dates.
In day trading this might be less of an issue, but not when testing over a much longer time frame. If you download quotes for REITs back to 2005, this will exclude several stocks that went bust during the financial crisis.
We’ll cover this aspect later in the course.
Disadvantage 5: Chance, luck, and randomness should not be ignored
If you’re testing a lot of strategies, some will show good returns simply by chance.
Unless there might be some logical reason for a strategy, you are guaranteed to find many good strategies the more you test.
Hence, we recommend it must be some reasoning behind the parameters.
Disadvantage 6: Backtesting involves garbage in, garbage out
The quotes you buy or download are often incorrect. If using the high and the low quotes in the backtest, you can be sure there are errors compared to live trading. There are a lot of wrong quotes in high and low!
In real trading, this will have a huge impact, probably most of all the factors mentioned in this article.
Disadvantage 7: Transaction costs are unknown
Slippage and commissions are easy to underestimate. Commissions are easy to calculate, but slippage is not.
This is a huge unknown, especially if you base your strategies on chasing the stock. If you wait to get hit, this is of course less of an issue.
Disadvantage 8: Markets change – no backtest can accommodate change
The market changes all the time.
Obviously, the future is unpredictable and you can bet there will be totally random and dramatic changes in the marketplace. No one expected terrorists to hijack planes and send them into a skyscraper.
Such totally unpredictable disasters will happen sooner or later. Correlations among different asset classes also increase during such happenings. You can never backtest such things.
Disadvantage 9: All traders do behavioral mistakes
You need to understand the most obvious trading biases. The psychological aspect is just as important as the strategy. Can you handle drawdowns and continue trading? Can you actually follow the strategy?
Based on our personal experience, this is something you have to consider thoroughly before you implement a strategy. It’s a lot easier said than done to follow a strategy 100%. A backtest can never accommodate all the mistakes you do along the way.
About disadvantages of backtesting: Always prepare for the worst
How can you prepare for the worst?
As a rule of thumb, it might be wise to expect a maximum of 50% of the profits from backtesting. If a strategy made 100 000, assume you’ll make 50 000 in live trading.
Hope for the best, but prepare for the worst.
Never be too optimistic when seeing a very nice equity curve, the downfall will be bigger. There is only real trading that matters.
Summary of advantages and disadvantages of backtesting
Despite having more disadvantages than advantages in this lesson, we still believe backtesting is the far superior way of making money in the markets. If you have no idea if your strategy has made money in the past, how can you be sure it’s profitable?
Keep in the back of your mind that markets are pretty counterintuitive. What you expect to happen rarely happens.