Please read the disclaimer at the bottom of this page

Saturday, November 10, 2007

Some more analysis (3)

We're continuing the previous analysis, but instead of considering 3 and 6 months of history, we're now looking at 9 months:



The first thing we can see is that as the time periods get longer, the Sharpe ratios get smaller. Only one end-of-day stock system was ever launched on C2 with more than 18 months of history, and a Sharpe ratio > 2 for the first 9 months. I won't discuss all the details of the table, as by now I assume readers are familiar with interpreting these results (otherwise: see the previous 2 posts). However, what should be mentioned is the fact that 80% of the top-10 systems for the first 9 months underperformed the S&P 500 during the next 9 months, and in most cases by a substantial amount.

Extending the timeframe further, we get:



Before coming to a conclusion, I'd like to collect some feedback from readers. So what is your interpretation of all this?

7 comments:

Anonymous said...

Another way to test a system is to compare risk/return performance against a well-defined benchmark, but the systems advertised on collective2 don't seem to have any.

Why did you limit your analysis to stocks systems?

Science Trader said...

I limited myself to end-of-day stock systems for two reasons. First, because I have a day-time job and my experiences with "auto trading" are not great. Second, I expect that the equity curves for these systems on C2 are close to reality, as opposed to various scalping and daytrading systems (with a few hundred to a thousand trades a year), where slippage and transaction costs might have a severe impact. But I'll see if I can replicate the analysis including all systems.

Regarding the benchmark: I agree that most systems just aim for absolute returns rather than trying to outperform a particular benchmark. So, I'm assuming the typical subscriber will consider to (a) not invest his money at all; (b) invest his money in a broad based market index; or (c) pick a (set of) C2 stock system(s).

The Sharpe ratio compares (c) to (a), the excess Sharpe ratio compares (c) to (b).

(As a side note: For many systems the correlation with the S&P500 is small however, and the common "alpha" statistic is not meaningful in that case. This is the reason I report the excess Sharpe ratio.)

Anthony said...

Looking for the best systems on C2 based on early performance is like looking at 4,512 coin-flippers, and picking the ones that flipped the most "heads" in their first 20 coin flips. Although they look really good, they don't really have an "edge". On the other hand, a few really great system might have an unlucky streak in their early trading, so they don't look great initially, but over a period of years (3+) will produce great returns. An early equity curve doesn't really tell you much at all. All the mathematical and statistical analysis in the world won't help you make money if all you are doing is chasing after the latest most successful coin-flippers. What's more informative is a longer term back-tested or real history, including bull and bear markets. Its also important to know that the system is based on some market fundamentals rather purely behavioral aspects of the market that are subject to change at any moment. If the system vendor won't share such supporting information, then buyer beware.

Anonymous said...

-ST
I understand why Bottoms up was left due to it not being an entirely EOD system in its complete history, and that I used a couple of options a few times to minimal profit.

But I only traded intraday for 3 months and it was only 1 or 2 trades during the lunch hour to make it easy to follow without having to use autotrading.

I happened to be forthcoming about my trading style from day one. Is it possible that some of the EOD systems you analyzed weren't EOD systems in their entirety? (since not all vendors are completely forthcoming about their system)

If I would have known about being penalized for those 3 months I would have ended BottomsUp and restarted for statistical's sake. Problem was, I didn't want to lose my record and have to start over.

I understand your frustration so far with the results so far with the systems you have traded. If I were you, I would wait about 6 months for the bear market to fully come out, and watch to see how the systems perform during this turbulant, volitle time.

oh... wanted to ask a question- have you tried the new "generation" autotrader with optionsxpress? since it doesn't rely on your connection, maybe you might have better luck with it? - just a thought- (i am going to try it out once my account gets set up over there at optionsxpress)

-David

Science Trader said...

David,
Whereas many of my analyses in the past months focused on the merits of a particular system, that is not what the current analyses are about. The purpose of this analysis is not so much to single out one particular system as being really "bad" or really "good". So, for the purpose of this analysis it's not so important if one system less or more is included. What matters is the general picture: past results seem to have little predictive power for the future for the majority of systems with great looking statistics.

What I sincerely hope is that as time passes by and/or market conditions change, I can show new tables that refute this hypothesis. Perhaps that happens when we can observe systems for 5 years or more, or when the share of high quality vendors on C2 increases. Perhaps adding other statistics than just the Sharpe ratio to the analysis could also lead to higher predictability.

The main barrier for me to not use OptionsXpress is that their commissions on stocks are quite large when trading small size (which typically is the case when you spread capital over multiple systems).

Anonymous said...

Thanks ST,

Good points.

Yes options xpress is pricey. It would be nice for IB to set up the advanced autotrading future (being server based).

Take care and wish you the best and hopefully after more time, some systems will prove to be a valuable tool for you.

-David

liveforexcalls.com said...

You are absolutly right in this. Many sites are giving Forex Signals and forex alert. trade signals and forex pips are good way to earn money for people. http://www.liveforexcalls.com is one of best forex signal provider.