In-sample vs out-of-sample testing represents one of the basic methods by which it is possible to find out if a trading strategy works the same on the data set it was built on or if it can also be successful using new data. A trading strategy might give excellent backtesting results in the development process, but this does not necessarily guarantee that it will act the same way in the real-world market data.
There is a difference in the purpose of each type of test. The in-sample data is used for the development process, whereas the out-of-sample data is not used in any way until the strategy is finished.
In this blog you will find out the main differences between these two and what specific purpose they play.
What Is In-Sample Testing?
In-sample testing takes place with the use of the selected data used in the development of a trading strategy. It is used at the point when traders make decisions on establishing the rules, testing their ideas and adjusting the parameters according to the behavior of the market.
For instance, assume that a trader has a ten-year history of price data. Traders may use the first seven years to formulate a strategy using moving averages, volatility, and price momentum. The seven years will be considered the in-sample period.
Some of the issues that could be tested include:
- Do the entry rules give the desired signals?
- Are the exit conditions reasonable?
- How often does the strategy trade?
- How does the strategy perform when there are different market moves?
- Are the parameters selected appropriate?
The main issue to consider is that the trader already has the data available to him. Thus, even a successful test does not necessarily mean that the strategy will work somewhere else.
What Is Out-of-Sample Testing?
The out-of-sample testing technique involves testing a finalized strategy with historical data that have been purposely left out in its development process.
Taking the above example, the trader could hold off the last three years of the ten-year set of historical data. The strategy is developed from the first seven years and then tested with the remaining three years.
It is important that the trader does not modify the strategy because of the disappointing results from the out-of-sample testing.
This will enable the out-of-sample testing process to answer another question:
Can the strategy perform itself reasonably on data that did not contribute to its development?
In-Sample vs Out-of-Sample Testing
The in-sample data helps in the formation of the strategy, whereas the out-of-sample data is independent from the former to be used for testing the formulated trading rules. Both periods taken together provide more insight into the consistency of results obtained.
| Factor | In-Sample Testing | Out-of-Sample Testing |
|---|---|---|
| Main purpose | Develop the strategy | Evaluate the finished strategy |
| Data usage | Available during development | Kept separate |
| Rule changes | Can be made during development | Should not be based on results |
| Parameter optimization | Performed here | Avoided |
| Main question | Does the strategy work on development data? | Does it work on unseen data? |
| Risk of overfitting | Higher | Helps reveal overfitting |
| Position in Workflow | Before final testing | After strategy development |
How In-Sample and Out-of-Sample Testing Work Together
In order for both test stages to work properly, they need to be defined in terms of their functions in the strategy development process. The first stage involves strategy development based on in-sample data, the second one involves applying the already completed rules on the independent out-of-sample period.
Below is the step-by-step guide for you:
1. Divide the Historical Data
Start by separating the available historical dataset into development and testing periods.
For example:
10 years of data → 7 years in-sample + 3 years out-of-sample
The exact split can vary depending on the strategy, market, timeframe, and amount of available data.
2. Develop the Strategy on In-Sample Data
Only use the in-sample period in formulating your trading rules. These may include criteria for entering positions, exiting trades, sizing trades, and other rules.
The aim here is to formulate a coherent strategy and not necessarily generate maximum profits from history.
3. Finalize the Rules
When the development stage is over, the strategy must be frozen.
It means that it is not reasonable to make changes to the parameters just because one needs a more favorable outcome of the test run.
4. Apply the Strategy to Out-of-Sample Data
Test the finalised strategy against the holdout data set.
No new rules. This is done in order to see how the strategy reacts to history that was not seen during the development process.
5. Compare Both Periods
Lastly, contrast the outcomes of the two time periods.
Consider performance measures like:
- Net return
- Maximum drawdown
- Number of trades
- Win rate
- Average trade
- Profit factor
The comparison should be aimed at whether the general behavior of the strategy remains sensible and not at whether both periods generate the exact return.
How to Read In-Sample and Out-of-Sample Results
Comparison of the two periods will make clear how the strategy behaves out-of-sample relative to how it was developed. Rather than looking just at return behavior, consider whether key performance metrics remain relatively constant between the two samples.
Strong In-Sample and Strong Out-of-Sample Results
This is usually a good sign. The model worked quite well during the development phase and still performed relatively well when used on new data.
While this does not ensure future profitability, it is a better indicator than just an in-sample result.
Strong In-Sample but Weak Out-of-Sample Results
This is a red flag.
The model has probably been tailored too much to the development dataset. The disparity may also arise due to changes in the market environment or certain assumptions which may be wrong after the original time frame.
Weak Results in Both Periods
There will hardly be any sense in assuming that the testing phase will save the strategy from failure.
It might be a question of the wrong trading concept, its rules, or the wrong testing environment.
Similar but Lower Out-of-Sample Performance
Out-of-sample performance does not have to be exactly the same as in-sample performance.
Take for example an investment technique which returns a 25% profit in-sample and 18% out-of-sample, which might be more consistent than an approach returning 80% in-sample and 2% out-of-sample.
The size and stability of the performance discrepancy is important here.
Common Mistakes in In-Sample vs Out-Of-Sample Testing
Avoiding common testing errors also makes the comparison between both periods more meaningful.
- Repeatedly optimizing the same data
- Looking at the testing period too early
- Moving the data boundary after seeing results
- Judging performance only by profit
- Using an out-of-sample period that is too short
Conclusion
In-sample vs out-of-sample testing creates a distinction between strategy design and testing on historical data that was not seen before. When testing in sample, traders have an opportunity to develop their strategies, but out-of-sample testing will demonstrate how these developed strategies work using data that did not contribute to the strategy's development.
An evident discrepancy between the two periods does not necessarily mean a problem; however, a significant deterioration definitely requires further consideration. The idea is not to achieve the same results but to check whether the strategy works in a similar way using new data.
FAQs
1. What is the main purpose of in-sample testing?
The in-sample test is employed in the creation and modification of a trading strategy. Traders employ the in-sample test period in order to test out the rules and parameters prior to conducting an out-of-sample test on different data.
2. Why is out-of-sample testing performed?
This is where the strategy’s performance is tested using data that was not used in creating the strategy.
3. What is a common in-sample and out-of-sample split?
There is no one-size-fits-all ratio. One such case can be allocating 70 percent of the historical data set for training and using 30 percent for out-of-sample testing, given that both sets have adequate market information.
4. Can I change my strategy after seeing out-of-sample results?
It is possible to do so, but the data will no longer be an out-of-sample test after incorporating such outcomes into the strategy. The modified strategy must be tested on another data set.
5. Should in-sample and out-of-sample results be identical?
No. Some deviation is natural due to changing market conditions. What really matters is that the strategy retains relatively consistent performance, not that it deteriorates drastically beyond the development phase.