MultiCharts: Trading Software for Automated Trading and Backtesting

If you vote for the feature request to use Rx.net and reactive programming in multicharts to increase overall performance, I think it will at least minimize many of the performance problems that have been described time and again.


https://www.multicharts.com/pm/public/m ... es/MC-2961

Disclaimer : Not familiar with Rx at all, but I like the concept. That said :

I think it will at least minimize many of the performance problems

I'm afraid it will make it worse (but please keep in mind the disclaimer). I mean, Rx seems to take out heavy lifting tasks from us, the programmer, hence brings all to a higher level of coding. This will not help the means MC Devs apply modal forms and the threading environment which, well ... I would have done all 180 degrees differently. Or should I say all s*cks ?

Side note : I sure wanted to vote for the Rx idea, but it seems I need a special login for that ? Must go to MC Support to ask for it ?

People may complain about the responsiveness (or sluggishness or being slow) of MultiCharts, but I don't think it is that really. What it is - in my view - is the wrong approach in multi threading in general, with so many "wrong" design decisions that it is too much to list with some sense. I could better hint at some culture difference and how things seem to be approached upside down. I sense this from day one with MC and I sense this today each minute - it being so annoying all over.
The net effect of this all is that I am waiting for minutes before I can apply a next input (like starting a next backtest) while a fraction of a second would suffice. This is not "some times", this is about always, everywhere.
Chinese people work there ways different from ours (like from left to right and from bottom to top, in the end even starting translated manuals at the end instead of at "our" beginning), but this (MC) feels not much different. It is in all the explanations, it is in all the setup(s) of the software ... it is in the technical approaches we can't see really.

I could also attest that way back - at least for the .Net version an approach was chosen which was very much "object oriented" but today this approach does not work out, but can't be changed *because* of the object oriented approach.

Trying to be on topic somewhat, all what's "data series" is unnecessary slow. And "unnecessary" really is that. I really don't need backfilling, when backfilling just has been applied with the Data being actively loaded - now shut off. I saved it, it resides on disk, so MC, what do you think you are doing but pestering me with backfilling *again* after restarting MC ? Is the data residing in the wrong format on disk or something ?
I suppose MC does not even see that 1 month worth of tick data - used at the tick level - is totally unworkable. ONE-MONTH. 2 weeks barely is, and with one day all goes unnoticed, though the anomalies still there under the hood (1000 times too long on 0.01 second is doable).

I very much like the functional concepts of MC. But if you can't start laying out where the laying out should start (which is at the inside) then MultiCharts will not be for those for whom it was made. The lack of starting at the inside, IMO causes that MC Devs themselves also work from outside in. Now nice feature X cause all to stall inside.
Ah wrappers. Yes. Rx would be a (virtual) wrapper - am I right ? but garbage-in garbage-out. So it would not work out.

So my 2c is that MC should get their act together, which seems to start with people who left the building ?
Support is great, but they too (like us) can only work with the tools they have. Still, the silence (often at questions) is speaking and the machine seems to get rusty. I think this is a real waste (for this fine product !) ...

First of all, yes, as far as I know, you have to register/activate yourself there once.

You must not forget that the software has been around for a very long time, you don't just change a running software, this can lead to many side effects. Object-oriented programming has the advantage of modularity and, with the right software architecture, also the possibility of interchangeability and expandability, which should also be possible if you follow clean code rules, for example. These include dependency injection and inversion of control.

Rx.net will not be a panacea, but if used correctly it should speed up one thing or another and, as mentioned, it offers multithreading capabilities (asynchronous programming) as standard and is very well developed and tested due to the open source principle.

It is not suitable for every scenario, but can be helpful and speed up many scenarios because the data streams are available when they are needed.

Incidentally, Rx.net is common practice in Java and has proven itself there.

Multicharts should primarily function reliably, because it is about real money trading. All other things are secondary.

Because multicharts now uses outdated technology (such as Windows Forms instead of WPF, but also generally no newer frameworks), you will always come across obstacles. Perhaps it simply needs a completely new development.

What always bothers me about MS is that I have no access to the extended functions of multicharts - e.g. backtest engine, the database, etc. It is a partially visible black box, so you always have to build a lot of things around it. This is certainly good and sufficient for simple indicator developers, but not for a multicharts.net version in my view, after all, as a developer you also want to be able to realize your own potential. Multicharts.net could be expanded to include so many functions if you let the community in the spirit of open source, but if the only access to the programming is via an indicator or signal file, that's not exactly great. I could list a thousand things that should be done differently or better, but in the end I still like that MS.net offers the possibility of programming at all. I have also looked at other trading software, but none of them offer the possibility of programming in .net(C#).

I am just hoping for a small improvement with Rx.net. And yes, you can call this a wrapper, but it offers many possibilities (also a different type of programming) - this is the way in which programming languages ​​develop and enable you to concentrate on the essentials, on the goal "what" you want to achieve and no longer on the "how". Very few people will (want to) write machine code these days. Higher levels of abstraction should be our wish, at some point you will just tell an AI what you want, the how, the AI ​​will take care of that. It will take a while, but at some point you will only program in human language.

Really, the last thing I would do is debunk the software or the helpful people supporting it (say those in the Chats).
It seems that we must approach all as a fait accompli, but this seems no good solution/approach in the long run.

Understanding what happens under the hood is something to graduate upon, and working in between those "prerequisite lines" is a kind of killing (the challenge is too great at times).

My first baby steps ever back were about bits and bytes and ORs and NANds and the like; back then I never understood why that would be the necessary base of it all. However, later, when I worked at that higher level - like everybody normally did - I praised those lessons more and more, because without that insight nothing could run "optimal".
The fun is : with MC (and less than a handful of others), you can't do without this "knowledge" or else you can't achieve what the environment was made for. So yes, I said that in my previous post with different wording, and this is about the ability to apply e.g. HFT which (IMO) really requires all the bits and bytes insight to let that run well.

All 'n all I fully agree with the impossibility to amend a few parts / modules of MC so all runs smoothly again, but I am also afraid that no Rx is going to help with that. Mind you, assuming that I am correct in the cause and solution of things being sluggish.

One thing to add (and winking to my first post in this thread) : when a code runs in an IMHO most lean fashion for a couple of days (Tick chart), then eventually it will die of too much overhead on the for me unknown (the forever dealing with history data). It is hard for me to prove just that, but for MC it is also impossible to solve unless they would trade themselves via the means I use myself (I regard this impossible). We could ask for a "FiFo of history data" because what to do with the growing history in the first place, but it is exactly this which won't be honored because of too many questions of "why" and the too many hurdles of the "you should not do that in the first place". This all in the realm of readily made features for the job, which nobody seems to understand really well. Not any more. And this is about those who left the building, mind you, as far as I can judge it.

It is clear that I am a newcomer here, but I hope to be able to judge the merits of a piece of software and its goals. I still think that if something lacks these days in the realm of the subject, it should be tackled in the kernel and not by a wrapper-like means. Why ? well, because it will be too slow. Too many CPU cycles required. If you trade with timed bars (like 1 second and up) then nothing will be too slow (there is just no means to notice it unless you'd compare 20ms with 25ms on the 1000ms).

I hope to bring across that I am not complaining. But really challenging it is.

I agree that it is difficult to change some parts or modules of Multicharts to make everything run smoothly. The deep integration and complexity of the software make comprehensive changes a real challenge. While I understand the view that Rx.NET may not be able to help with all optimizations, I still think there are some scenarios where it could provide valuable services. Rx.NET provides powerful tools for handling asynchronous data streams and events. These capabilities can be particularly beneficial in Multicharts.net, as they can greatly simplify and optimize the handling and processing of real-time data streams. Rx.NET allows developers to use declarative and reactive programming patterns to implement complex data flows and event processing. This results in cleaner, more maintainable code and can help improve performance by providing more efficient methods for event processing. Error handling is also made much easier by Rx.NET's extensive operators, resulting in more robust and reliable code.

I also think that a FIFO strategy for handling historical data is a good idea. Historical data can cause significant overhead and lead to performance problems, especially if it is constantly growing. Efficient handling and storage of this data is crucial to maintain performance. A FIFO strategy could help here by limiting the amount of data and thus reducing overhead. However, it must be ensured that the old data is no longer needed for historical analysis.

Regarding solving problems with wrapper solutions versus core changes, I partially agree. Core changes are often necessary to achieve deep and sustainable improvements, especially for fundamental performance problems. At the same time, however, I also think that wrapper solutions like Rx.NET can be very effective in certain contexts, especially in handling asynchronous processes and improving code organization and maintainability. They should therefore not be rejected out of hand. A balanced approach that considers both core changes and wrapper technologies would be most effective in my opinion.

On the project management page I have also included some links where the advantages and disadvantages of Rx.net for various scenarios are presented very well and better than I could describe. In the end Multicharts has to decide whether it fits into their concept. However, software maintenance does not bring any financial advantage in the short term and only pays off in the long term. With a new Future you are usually more likely to reach new paying customers, but Multicharts does offer the opportunity to contribute ideas, so I think it makes sense to continue to introduce this idea, as I hardly see any disadvantages. While Fifo has more of an impact on the memory, Rx.net tends to reduce the load on the CPU, so both can be useful.

Thinking this over (requires more research from my side as well), but before responding more ...
In the last paragraph you mentioned MS twice, what you could have meant. Or was it MC ? this would fit slightly better.

Thanks !

yes, that's right, I made a mistake MS = Multicharts, I'll correct it

Here is another example of why using Rx.net can be helpful and how I currently use it:

Because Rx implements the Publisher Subscriber pattern, you have the ability to write completely decoupled classes that are universal. Since you usually always work with price/time data, you can create numerous filters (classes) that can be easily strung together. You can then easily use the dependency injection principle to map the filters, or in other words, you just have to connect your filters anywhere and in a central location. This may be a very simple code overhead, but it is helpful and clear for more extensive strategies.

Please note : I have asked for a Login to that Voting area (what, a month ago by now ?), but never heard back a thing. It would be a fifth time or so that I'd have to repeat my question to MC, while they "timely respond" as they say. I am fed up with this.
-----------------

I sorted this out to some extent (and capabilities) and first off, it would not be about coding overhead. It would also not be redundant, because whatever it is you'd code in that area, would be the only place (I think). It may become more fuzzy to understand your own coding logic, but that's part of this game. My point would be (and was already without too many words) :

In the background this "aggregator" (my term) would continuously run in the background to collect and prepare the data you would need. This is a for me well known way of working, like preparing the data for Excel sheets users may like to get emailed to them (or download). Thus, Excel is generated in no-time, while collecting the data for could take 5 minutes. That data now is always there, and the run to prepare it may be continuously running on another PC or server. Not even on the same PC in another task, but elsewhere, thus taking the resources from elsewhere.

And this latter part would not work here, unless it could be set up like that. But do you see this happening ?

My thinking on this matter could be more specific, as I'd always think in terms of being called per tick, and all the hoopla required to let that work sufficiently lean. This most certainly means no background processes tearing down the same CPU and other shared resources, no matter how many cores that has because they are not under my control (they would be if I had written MC).
Let me add this too far out thinking : I would already arrange for a redundant internet (data) connection, so that would not disturb on the low latency required for the trading.

If this is not really the/your idea about Rx in this realm then my perceptions are off.

--------------

The idea itself is something I adhere and thus apply already in an other life (without the formal Subscriber thing). But it really requires a distributed "resource model" which we - I think - can't leave to MC without a huge overhaul first. This starts with the current "model" which in my eyes s*cks a bit, coincidentally because a base was not set up properly. This lies in the fact that some core processes are being dealt with one processor core only, which today is a huge bottleneck (bothers me all day long). Maybe I pointed it out already, but this boils down to some expression of Henry at some (longer ago) stage : "Because there needs to be a central control organ(ization), 'that' all needs to be dealt with by one processor core" (my wording of what he said) ... This sounds nice and gets rid of the difficult question(s), but in the base this testifies of a not-so-good threading model. And oh, which in MC is not thread safe as well (https://www.multicharts.com/trading-sof ... Calculated - see the red text in there).
And thus it would come down to a too much horse behind the wagon principle and possibly not-reliable data on top of it.

As always, I try not to turn this into a complaint about MC. But the subject of "decency" surfaces automatically with a subject like this.
I will also admit that there is a LOT to learn first, and I should refrain of inappropriate comments without knowing a year or two more first.

It's best to write to support in live chat, as this is always the quickest and easiest way to solve the problem!


Personally, given my current usage behavior, I don't notice too many disadvantages in multicharts. It could definitely be better in many places. But as I said, it is now quite "old" software, which means it is robust, but does not always correspond to the latest standards (can correspond to them - without generating numerous side effects).
Please also keep in mind that even the .net framework used has not been further developed by Microsoft since version 4.8 and there are regular advances in "software architecture/programming" that cannot all be applied.

Rx.net simply offers more options that I would often prefer.

I generally think that normal multicharts are great for quickly developing an indicator. For more complex scenarios, however, I don't see enough attention to today's programming paradigms. Many things are now better: DI containers, AOP (e.g. for the output instead of having to write output everywhere), WPF instead of windows forms, Rx.net for live data streams. Increased use of GPUs instead of everything via CPU. etc. ASP.net instead of .net Framework etc.

But they will hardly experience this without a necessary new development and it seems to be seen as sufficient enough for the majority of customers, so why should you throw everything overboard?

It's true that you should focus more on using more CPU cores. This is one of the few opportunities to enable significant performance increases in the future. (Other aspects such as the MHz height are now hardly scalable).
Alternatives here are “cloud computing”, possibly something like Microsoft Orleans.

But this is all going too far; in the end, this should only be about this one framework and not a fundamental discussion.