Trading System Performance – Skill or Luck

So, you have finally devised your trading system which seems to have produced good returns in the back-testing you have conducted and is ready for trading with real money. Pat yourself on the back since regardless of what happens next, you have taken a step towards being a better trader by using a rule based strategy than implementing a discretionary strategy where every rule has a exception and every failure is easily rationalized.

By accepting that we have our limitations which can prevent us from succeeding in markets, we are able to make our trading more process driven and one where mistakes aren’t rationalized but seen through the eyes of it being either a part of the system or the failure of the system. Either way, the results showcase how good or bad your system is (assuming you take every trade and do not try to over-ride your own system).

While a back-test is a good way to analyze as to whether the system has any merit or not, it is not the end of testing but actually the beginning. It is said that if you were to give typewriters to monkeys, its just a matter of time before one of them hits all the right keys and gets you a Shakespeare’s plays in order.

Or as the character Dr.Alfred Lanning in the film I, Robot says “Random segments of code, that have grouped together to form unexpected protocols”. If one keeps testing ideas, especially using optimization feature, one is bound to hit the right system. Its just a matter of time.

The question hence is, is the system performance due to Skill or pure Luck. A system that is based on some skill or market  inefficiency. A system that is based on luck is one where in the streak of profits is just purely driven by luck and its just a matter of time before the said system starts misbehaving.

So, how does one determine whether the profits that are shown in the back-test is due to it being inherently good or just a matter of luck.

There are a few ways to analyze that and I shall lay out a few of them out here. To start with, let me choose a system which will be used for the testing.

The system that I shall use for the test below is a EMA Crossover. To get the two parameters, I ran through a Walk Forward Optimization which finally gave me the 2 variables I was after.

Now that the main requirement is fulfilled, lets move onto what are the ways by which you can detect whether the performance of your system is more based on luck than skill.

The first thing to do is to test your system on other time frames / markets. No single system performs exceptionally well in all time frames / markets and hence if you are looking for a 100% score, you are bound to be disappointed. What we are looking instead is performance that is well within limits.

Lets take the EMA Cross system that we have now created. The testing was done on Daily time frame. Lets compare this to Hourly as well as Weekly and see how different the numbers will be. To keeps things simple and as well as have a standard format, we shall use Expectancy as the parameter to judge. Time frame of the Test is from 1st Jan 2010 to 1st Jan 2015.

Other test settings are

• We trade on Close Value
• No Commissions or slippage is included.
• We do not compound our position size
• We take both Long & Short trades (Always in the trade)

We shall first test the system on the Daily time frame. While the optimization was done on CNX Nifty (Spot), we shall see how the system fares on other Indices

The system is profitable in most indices save for CNX Energy where it saw a loss. This is good since it indicates that the system is not curve fitted to Nifty.

We shall now move to testing the system on hourly data.

Once again the system trumps (though if you were to observe closely, expectancy is in dangerous area for stocks such as ICICI Bank & Reliance while it out right loses money on ITC.

While we cannot decide based on above data as to whether the system is good or not, what it does show is that there is some amount of skill and its not entirely due to luck.

We next move back to Nifty F1 on Daily time frame and test the result using the Bootstrap method as well as Monte Carlo Testing.

The p-value using Bootstrap for Nifty F1 (Daily) comes to 0.55 which is way above the null value requirement of 0.05 to reject the thesis that the results seen above are based on random luck. In fact, the 0.55 number indicates that 55% of the result probability is just due to random luck and no more. This is pretty damming evidence of the fact that while our system seems to perform great, it actually is nothing more than a lucky streak.

Lets move forward and test our hypothesis using Monte Carlo Analysis. Once again, we are looking at the whether our result is based upon blind luck or is there something worth while.

Before we get into Equity Curves, lets compare and contrast a few other details.

In our regular back-test, we get a max draw-down of 19.70%. In contrast, the 95 percentile says that the probability is high that we may see a draw-down as high as 60%.

In our regular back-test we see a maximum losing steak of 7 trades while in the Monte Carlo approach, the figure comes to 20

And finally, lets look at  the various ways in which our equity curve could have been plotted. We ran a total of 10000 stimulations and as expected, chart is messy. But what is does show is the high probability (test of the lower line) of the system losing more than 50% of capital and hence running out of capital.

But should we be surprised with the answer that this system is not suitable for trading since far too many trades hinge on luck than actual skill?

Too many traders believe that one can come up with great system by just optimizing data. If only life was that simple, we would all be chilling out in our favorite place while the system kept earning for us without a sweat 🙂