Please read the disclaimer at the bottom of this page

Thursday, November 8, 2007

Some more analysis (2)

In my previous post I compared the performance of the highest ranking systems during their first 3 months from inception to their performance during month 4-6. The conclusion of that analysis was that for all but one of the 10 highest ranking systems in the first period, performance during the 2nd period was substantially worse. Still, not taking into account transaction costs and slippage, the average Sharpe ratio of these 10 systems during the second period was slightly better than that of the S&P500 index.

Obviously the choice of 2 periods of 3 months each is arbitrary, and one could argue that 3 months is too short to judge the quality of a system. If that is true, we would expect to see more promising results if we would allow us a longer time period to evaluate a system before subscribing. So, let's look at the performance of 10 end-of-day stock systems with the highest Sharpe ratio's for their first 6 months:

As we can see, "Good NEWS Predictor" is again leading for the first period. However, this time performance during the 2nd period is quite disappointing. "Momentum #3" and "Momentum Breakout" show how bad it can get... We have to be a little cautious though, because I expect that at some point the vendor of Momentum #3 terminated trading the system without closing positions (while the equity curve continuous uncontrolled). In that case subscribers would have halted trading before digesting the full -0.68 Sharpe ratio. Momentum Breakout is quite a tragic case, as subscribers during the 2nd period found there account losing money during a runaway bull market (as indicated by the large negative excess Sharpe ratio).

Also of interest is Trend Plays #1. While it had a very decent Sharpe ratio (and equity curve) for the first period, in fact it underperformed the S&P500 index during that time on a risk-adjusted basis (as indicated by the -0.21 excess Sharpe ratio).

I didn't find these numbers particularly encouraging. Even with half a year of history it is quite a gamble what you'll get as a subscriber during the next 6 months for these systems that all had these attractive equity curves. In a way, it's interesting to look at some current end-of-day systems that will show up in this table half a year from now (i.e. they currently have about half of a year history).

Consider Small Cap Fundamental Value with a Sharpe ratio of 4.3 over 29 weeks:

It would show up right in between Momentun #3 and Good NEWS Predictor. Any guesses about its performance over the next 6 months??? I simply don't know. Perhaps it will do great, perhaps not. So far history suggests it's difficult to judge based on the Sharpe ratio.

Wave Rider (Sharpe ratio 2.2 over 27 weeks) is another system that will be included in the table 6 months from now:

We will continue with longer histories (9 and 12 months) in a few days.


ProbTrader said...

Interesting analysis. My take is that 6 months is a too short period to evaluate a system performance.
I am assuming the Sharpe ratio data uses month-to-month returns variability (monthly SD). In that case risk evaluation is based on only 6 data points?

Science Trader said...

No, I use daily returns. It's ~125 data points for half a year and the Sharpe ratio is annualized--multiplied by SQRT(250).

Anonymous said...

Most of the systems on C2 are too young to even consider. Your analysis is interesting, but much more data is needed to make a good call. I would be more interested if your data collection had a minimum of 5 years, which is literally imposible with most C2 systems.

Science Trader said...

The oldest system on C2 started about 4 years ago. You'll probably have to wait a decade before you'd have a reasonable sample of systems with at least 5 years of history.

Anonymous said...

Then wait is what I would have to do. Unproven systems are a dime a dozen. Many pretenders out there on C2.

Jules Ellis said...

Hi ST,
Your analysis confirms the conjecture of the regression effect that we considered earlier. Nice to see that it applies to stock systems as well, and also your analysis (comaring the same systems over different time periodes) is more convincing than mine was. So we may conclude that the statistics of young systems are more vulnerable to good or bad luck. The question is what "young" is. Preferably the analysis should be based on time periods that are independent, and I don't think this is "days". Also, as I argued, it is to be expected that the effect will still exist when you take longer segments of history, although the effect will probably be weaker. However, the truly interesting question is if this holds for *all* systems or only for *most* systems.
Regards, Jules

Science Trader said...

Those are all interesting thoughts. I'm planning to post longer timeframes soon. I ran some regressions as well (including more systems than just the top-10 shown in the table), and the prior Sharpe ratio's predictive power turns out to be insignificant (also, the scatterplot shows a very nice cloud with no apparent structure).