Hedge Fund Due Diligence

Backtesting – Data Mining

There is good and there is bad Data Mining. It is all a question of the attitude of the person conducting the data mining exercise. This is why when I am presented with backtests' results, I am more interested in knowing who did it, why he/she did it and how. In the due diligence section, I mentioned that a reputed researcher working for one of the largest global organization had presented me with honest backtests’ results, but backtests that did not reflect the way the portfolio was actually managed – in this case with constraints imposed by the risk department which he did not agree with.

While I personally think that backtests can be very valuable and can be representative of the performance of a live strategy, I know that backtests can be deceiving.

1. Different Alphas depending on the portfolios!

I once met with local representatives of one of the largest global investment management organization. They had in the past developed Canadian active strategy which ended-up being last percentile in a leading performance survey which lead clients to exit the strategy and employees involved being terminated.

They had developed a new Canadian active equity strategy for our local market. They had actually developed three such strategies: one for a normal long only strategy benchmarked against our TSX index, a 130-30 strategy and a long-short market neutral strategy. What shocked me is that each of the three strategies had different sets of alphas for each stock! When I asked why they had three different sets of prediction for all companies, they said that they get better performance in their backtest this way. None of the people presenting to me were aware that what they were doing was a classic example of bad data mining! I tried to explain them that their approach was wrong… It would be interesting to hear how they explain to their clients why they like company x in a portfolio, yet don’t like it in another portfolio.

I shared this with another leading global manager whom I respect for their knowledge of these matters and he quickly agreed that this was a clear example of bad data mining. The sad thing is that these people are not only fooling themselves, they are also fooling investors and consultants. People doing due diligence should be familiar with the negative consequences of bad data mining.

2. Great story teller flipping his story!

I once came across a great guy – on a personal basis. Paul (not his true name) had a lot of motivation to succeed, but he did not have the background to do advanced financial research.

Investors need to realize that if we split the universe of stocks in two halves according to any system, one half will do better than the other half. This may look like a remarkable tautology and it is. But it can have bad consequences.

Any research project starts from an idea. Companies with a high exposure to a factor x (could be any ratio, valuation model or signal) should outperform because of reason y. Paul would test his theory. It did not work. He would reverse his rationale/story, reverse his backtest and voila: companies with a low exposure to factor x should outperform because of reason z and here is the proof (backtest with positive results)...

 I would look at Paul and say “Paul, do you really believe in what you just said?” Paul would smile and go back searching for better investment strategies with better rationale. Eventually Paul left to work in an area which better exploited his skills.

If you want to find stock picking stories, it is very easy. If you want to find a well thought investment strategy, you need a forward looking and honest mind. Investors and consultant need to be aware that there are many Paul out there.

For more information on proper backtesting and good data mining methodology, please contact us.

Dominic Clermont, ASA, MBA, CFA

Back to Due Diligence Page