Can Curve Fitting Actually Be Good?
In the last lesson, you learned the pitfalls of curve fitting. In this lesson, we take the contrarian view.
Curve fitting is always mentioned negatively in trading. By all means, that is how it should be. If you are backtesting, you need to keep the risk of curve fitting in the back of your head all the time.
However, this lesson argues why curve fitting is good, as long as you know what you are doing.
Let’s give you an example of what many argue is curve-fitting. That said, it is still a procedure we did ourselves for a couple of decades (with success).
When curve fitting is good – an example
Let’s assume you have a stock trading strategy. This is what might be a viable “curve fitting” to improve the strategy (albeit many disagree):
- Define a trading strategy with specific trading rules.
- Backtest the strategy on a wide range of listed stocks.
- Pick the stocks that have worked well over the last N years, for example, 5 years. Test those stocks going forward.
- Is there a pattern? Are there groups of stocks that have performed badly?
- Exclude sectors/groups performing poorly.
- Perhaps exclude all stocks that performed poorly.
- Rinse and repeat quarterly, semi-annually, or annually.
This might not be a recipe for success for all traders. But in our opinion, it makes sense. Why?
All backtesting is curve fitting
Trading is all about finding tradeable patterns, and we might argue that if you reject 10 patterns but trade one, you are curve-fitting.
Jim Simons (the man behind the world’s most successful hedge fund) has multiple times said that they don’t ask why a strategy might work, they just trade it as long as it’s statistically significant.
We might argue that the Medallion Fund is curve fitting. After all, they test a vast number of hypotheses, and scrap those that don’t work, and keep those that do work.
If you backtest thousands of strategies you’ll be sure to find something by chance, simply because you backtest a lot. This is kind of curve fitting.
Can curve fitting be a trading edge?
Can we argue that curve fitting is the edge?
Plenty of traders skip a trading strategy because of:
a) doesn’t work well; or
b) doesn’t work on all stocks or markets.
But there is no reason for oil stocks to trade like consumer staples, for example. Thus, rejecting a strategy might be a mistake.
The a) above might be due to very volatile small-caps, for example. We don’t think it’s curve fitting to exclude those stocks.
What could be the deciding factors for including or excluding a stock? We list the two most obvious factors:
- Sectors or industries. Some sectors don’t work. Oil stocks and commodities are notoriously bad, in our opinion. But it might change over time. For example, REITS were good in the early 2000, but it all changed in 2007 when the financial crisis started coming in.
- Volatility: volatile stocks are not good. Of course, it depends on the type of strategy. We have always regarded the most boring stocks as the best one to trade.
To summarize, don’t expect a strategy to work on all stocks or all markets.
Markets differ.
But make sure you know what you are doing! This is a part of the backtesting process that is highly dependent on experience.