I'm a bit confused how the memory management in MC work. After some digging (mainly testing different scenarios), it seems to me that MC first loads the necessary data into the memory, and then for each strategy instance (logical core) works through this data and stores the calculations (and data?) to the RAM again. After seeing this, I would expect the total RAM requirement to look something like “X + N*Y” where X is the initially loaded tick data, and Y>X is the RAM used for each of N instances, but this doesn’t seem to be the case as the requirement doesn't scale linearly with the number of cores. Often Y is larger than X though, even when I run an empty strategy that doesn't generate any orders and doesn't keep any data, which seems a bit "wasteful" as it shouldn't have to hold that much data for the instance. How come so much memory is needed for each instance, even if no trades are generated?
Are there any Multicharts specific actions the users can take, in order to speed up optimiziation and minimize RAM requirements? I'm not referring to tweaks outside of MC, but rather changes to the code/settings or any tricks to turn off further functionality in MC that might not be required in the optimization.
A few ideas are:
- Minimize the number of bars the strategy can reference.
- Set the 'NumberOfThreadsOnOptimization' registry key to equal the number of real cores instead of logical cores (will decrease the memory requirement without adding to computational time, assuming CPU utilization is 100%).
I was hoping to learn about some attributes (or similar) the strategy can use to avoid unnecessary computations, but that is probably wishful thinking.
Memory management and speeding up the portfolio backtester
- Svetlana MultiCharts
- Posts: 645
- Joined: Oct 19 2017
- Has thanked: 3 times
- Been thanked: 163 times
Hello, SysInv,
Memory usage is different during backtesting and optimization.
Backtesting
The memory is allocated for the entire data series + for the number of generated orders.
Memory usage cannot be significantly reduced during backtesting, but the following methods still can be used:
- Reduce the amount of data;
- Reduce the number of generated orders.
Optimization
The memory is allocated for the entire data series (as for backtesting) + for displaying the Optimization Report. The amount of memory allocated for the Optimization Report depends on the number of lines in the Report. Consequently, one can reduce the memory usage during optimization by reducing the number of lines in the Report. This can be done the following way:
- For Genetic Optimization: reduce the value of “Population Size” parameter in the Genetic Algorithm Properties window -> Algorithm-specific Properties tab. Population Size is selected automatically depending on the number of СPU cores (the more cores you have – the larger Population Size can be selected), but it is possible to configure it manually as described above.
- For Exhaustive Optimization: reduce the value in “Show … results” field in the Exhaustive Search Properties window -> Optimization Criteria tab.
Memory usage is different during backtesting and optimization.
Backtesting
The memory is allocated for the entire data series + for the number of generated orders.
Memory usage cannot be significantly reduced during backtesting, but the following methods still can be used:
- Reduce the amount of data;
- Reduce the number of generated orders.
Optimization
The memory is allocated for the entire data series (as for backtesting) + for displaying the Optimization Report. The amount of memory allocated for the Optimization Report depends on the number of lines in the Report. Consequently, one can reduce the memory usage during optimization by reducing the number of lines in the Report. This can be done the following way:
- For Genetic Optimization: reduce the value of “Population Size” parameter in the Genetic Algorithm Properties window -> Algorithm-specific Properties tab. Population Size is selected automatically depending on the number of СPU cores (the more cores you have – the larger Population Size can be selected), but it is possible to configure it manually as described above.
- For Exhaustive Optimization: reduce the value in “Show … results” field in the Exhaustive Search Properties window -> Optimization Criteria tab.