MultiOpt vs MultiWalk Walkforward Strategy Results
Results in MultiWalk may vary from results obtained in MultiOpt, or even non-database versions of MultiWalk Beta (pre-March 2023 ). One reason is due to implementing a database environment in MultiWalk instead of using text-based CSV files. Metrics from the walkforward process, such as Net Profit, R^2, etc., are now stored in their binary representation rather than as text strings. This made them subject to what is known as “floating point machine error”. This essentially means that it is impossible to represent real numbers in binary, so they are estimated and can, depending on machine state and slight changes in floating point values, slightly change. I therefore needed to implement “floating-point error mitigation” techniques so that the walkforward selection of iterations remained consistent from project to project, whether the same walkforward project was run one month later or five years later.
However, what this means is that the walkforward results obtained from MultiOpt may not always match (and can sometimes be dramatically different) with MultiWalk. Previous non-database beta versions of MultiWalk were not subject to this issue.
Recommendations
1. Use MultiOpt to generate parameters. If you have a strategy from MultiOpt that you are trading live and results are different in MultiWalk, then continue to use MultiOpt to re-optimize the strategy.
2. Use MultiWalk to generated parameters. re-run the strategy through MultiWalk and choose a different in/out period or fitness function. This new strategy would be subject to all the same Strategy Factory rules as when developing any new strategy. So be sure to follow the process and use an incubation period, either simulated or real-time, before trading the new strategy.
3. Enable the “MultiOpt legacy option” in MultiWalk and compare results to MultiOpt. You will need to enable the Deverloper Screen Tab for this option. This option attempts to emulate the way that MultiOpt handled dollar (floating point) numbers. It may or may not replicate MultiOpt results but, if it does, then you can use MultiWalk to generate re-opt parameters when combined with this option.
The Nitty Gritty Explanation
Numbers, such as Net Profit, R^2, etc. are termed “floating point” numbers in programs because they have fractional components. These numbers cannot be precisely represented in binary computer systems and can be prone to what is known as “floating point error estimation”. Now that MultiWalk stores data in a database environment that stores numbers as floating point representations rather than CSV text files (that stored them as text), decimal precision is strictly enforced. This may slightly change the outcome of your walkforward strategies when optimized iterations have the same in-sample net profit, but were slightly different due to floating point error estimation.
Consider the following example. Let’s say that you had two optimized iterations that varied a stop loss and had the SAME in-sample net profit results:
IN-SAMPLE Optimized Parameters Iteration 1 (stop loss = $1000): 6283.90
IN-SAMPLE Optimized Parameters Iteration 2 (stop loss = $1500): 6283.90
BUT, due to binary floating point imprecise representation, this becomes:
IN-SAMPLE Optimized Parameters Iteration 1 (stop loss = $1000): 6283.89999999999781721272
IN-SAMPLE Optimized Parameters Iteration 2 (stop loss = $1500): 6283.89999999999872670742
Normally, since they are both the “same”, the first iteration will be chosen as the “best” net profit since the second iteration did not increase in value. However, look closely at the numbers due to machine floating point imprecision. The second iteration was 0.0000000000009094947 greater than the first, and would therefore be chosen as “best” during the walkforward process in MultiOpt. MultiWalk, however, would choose the first iteration since it employs floating point error mitigation techniques and recognizes that the second iteration did not actually change in value.
Using this example, consider what happens with the out-of-sample period for each of the above in-sample periods. Each in-sample period was the same. However, their out-of-sample trading results were not. They had the same in-sample net profit, but then diverged in their trading results during the out-of-sample period:
IN-SAMPLE Optimized Parameters Iteration 1 (stop loss = $1000): 6283.90
IN-SAMPLE Optimized Parameters Iteration 2 (stop loss = $1500): 6283.90
OUT-SAMPLE Optimized Parameters Iteration 1 (stop loss = $1000): 5219.39
OUT-SAMPLE Optimized Parameters Iteration 2 (stop loss = $1500): 3890.10
Even though both iterations had the same in-sample net profit, choosing one over the other would make a big difference in the out-sample trades and equity curve results! This is what is occuring between MultiOpt and MultiWalk. MultiOpt may choose the second iteration while MultiWalk may choose the first iteration. But since the out-of-sample results were different between the two iterations, the walkforward strategy traded differently.
Now that MultiWalk stores data in its binary form in the database, this issue needs to be attended to so that strategies with “tie-breaker” in-sample iterations will always choose the same iteration regardless of floating point imprecision.
Therefore, previous walkforward strategies developed using MultiOpt or MultiWalk Beta (pre-database) may choose different iteration parameters and have slightly different outcomes due to this phenomenon.
MultiWalk will now strictly enforce floating point comparisons so that they are decimal precision appropriate. All strategies developed after the database release (April 2023) will be immune to this effect and perform consistently when confronted with “tie breaker” iterations.