Page 1 of 1
MC 7.0 Question - improvements to precision in data handling
Posted: Jan 20 2011
by Nick
Hi there,
For some while we have been unable to do accurate work with 'MD' 'cumulative delta' 'volume@bid vs volume@ask' type indicators.
These are pretty important to a large number of traders as you can see from numerous forums. Several competitors obviously do too:) Software that now includes these studies include IRT (and MD), Neoticker, Ensign, SC, NT (requires user study for historic data) and TS has it in beta test (so I am informed, I have not seen the beta myself). These are just the ones that I am familiar with! Delta is probably the biggest application but there are other things that can not be done because of issues MC has with 'precision' in data handling.
For MC to support this functionality it needs a couple of changes:-
Storing historical data to a higher resolution than 1 second. I would go the whole way and stamp in micro seconds.
You need a way to load historical data (bid ask and last) into a chart from the database in the same sequence as it was received. You also need to be able to construct higher resolution bars from this sequenced tick data. A user simply needs to be able to select 'delta' or cumulative delta as an indicator.
Currently there also some issues with live data. To find out if a tick is at best bid or best ask one uses intrabar persist and insidebid / insideask. These peek at the current value asynchronously so if you are processing a tick that was received a few milliseconds ago and these change you get the wrong value.
I know I wrote about this a long time ago but with competitors increasingly supporting this I feel it is pretty important that this finds it's way into V7.0 rather than V7.5. Will we see it there?
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 20 2011
by Dave Masalov
Dear Nick,
None of the competitors that you have mentioned have mili- or microseconds. We consider adding miliseconds an important feature and we will certainly implement it, but not in MC 7.0.
The data is stored in the database in the same sequence as it was received. If you have multi-data series chart, then the sequence of ticks may not be the same on the chart, the same issue can be observed in other trading software. We will address this issue in the future.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 20 2011
by Nick
The important thing is to have correct sequencing of bid ask last data so you can correctly build delta (volume @ bid vs volume @ ask) all of them do this. Arguably the best way to do it is with higher resolution time stamping. Either would do.
Will MC offer delta type indicators in version 7.0? (which requires correct sequencing of bid ask and last data).
Edit: Just to be clear all of those competitors order MD type indicators at this moment in time. I really like MC I have a great deal of time invested in it, but I am increasingly finding that traders I work with require delta and so we just can not use MC for those projects.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 21 2011
by Dave Masalov
Dear Nick,
Bid and ask data is stored in the correct sequence. When you plot bid and ask data on the same chart, this sequence gets corrupted. But we see the same behavior in other software, such as NT for example.
As for delta type indicators for MC, they are already available. Please contact FulcrumTrader on Big Mike Trading Forum:
http://www.bigmiketrading.com/platforms ... ts-38.html (see the last post). He has tested MD for MC and they seem to work fine.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 22 2011
by CrazyNasdaq
Hi,
I've read this thread carefully because Delta is very important for me too.
The very problem to be very accurate with the synchronization of bid/ask and trade is the time resolution and sub second sequence.
Milliseconds would be very interesting, but surely the market is just going toward microsecond, so if you have millisecond in Multicharts and the market moves to microseconds, you are in the same postion than now.
In my opinion the best way to project this is not using logical time based, but using sequence/counter based.
What I mean is........if you collect each trade of the market and if you give it a specific unique ID and in the same time that you record that ID you record the Bid or the ask side with the same ID, you can move under microsecond not only millisecond. Time will become secondary in synchronization and you can put toghether each trade with that unique ID with the same Bid or ASK ID, so you can build even ex-post the correct sequence of trade and bid/ask even if they are not recorded in real time.
This way the trade with ID # 3568 goes with ASK #3568, trade with ID#7864 goes with BID#7864 and it doesn't matter what time it is.
I think this would be a better way and even if I'm not a programmer, I think it's not so hard to code.
That's my thought
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 22 2011
by Nick
Dear Nick,
Bid and ask data is stored in the correct sequence. When you plot bid and ask data on the same chart, this sequence gets corrupted. But we see the same behavior in other software, such as NT for example.
As for delta type indicators for MC, they are already available. Please contact FulcrumTrader on Big Mike Trading Forum:
http://www.bigmiketrading.com/platforms ... ts-38.html (see the last post). He has tested MD for MC and they seem to work fine.
The trouble is they do not work with historic data. You need your have your charts running 24/7 and if you need to hit reload or have a glitch or simply miss some ticks you are finished. It is just not at all practical if you can not load historical data especially with
cumulative work.
The charting packages I mention all work with historical data MC 6.x does not (hence my question about 7). To be fair NT relies on a
third party suite from a poster called Gomi. This suite stores bid ask data with millisecond precision and sequenced across all 3 data streams. I did point that out NT could not 'do it on its own' in my original post. Incidentally Neoticker has had tick precise technology for I dunno over 10 years I guess.
Indeed MC stores bid/ask/last data in the correct sequence to the database but as there is no sequence number
across the bid/ask/last records and the granularity of the database is only 1 second.
There is no way to recall the bid/ask/last/data in the correct sequence.
I know Christopher (FulcrumTrader) is speaking with TSSuport about fully supporting cumulative delta, but currently he recommends people to use IRT, or NT (with Gomi's addons and qcollector). SC is absolutely fine too and more and more people at BigMikes are starting to take an interest or changing to it.
Currently MC is just not suitable for delta work and I am sure Christopher has informed you guys exactly why. His last post on the thread you linked is encouraging though but delta absolutely needs to be loaded from history.
Can you tell us a bit more about these tools. Are they based on EasyLanguage reading InsideBid, InsideAsk functions? There are currently issues with that approach in MC 6.x. as those functions return the
current value of bid & ask and not the value at the time of the tick. It is very possible to get race conditions and so inaccurate results. Do I have to sign up to MC 7.0 to test these tools? I'd really like to help test these specific tools.
Let me enphasise I
really like Multicharts
![Smile :)](./images/smilies/icon_smile.gif)
:) I have been with you since V2.0 was first launched (V1 was just too buggy for me! to see where you have got to today is remarkable). The only reason I am making a big deal of this is that I want to see version 7.0 do great. Actually it's not the only reason, I'd rather not have to run two charting packages each day! If you look at BigMike it really is the hot topic at the moment and has been for some while now.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 22 2011
by CrazyNasdaq
It seems that the problem has been solved in MC 7.
Take a look at Bug tracker link
http://www.multicharts.com/pm/viewissue ... e_no=MC-25
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 22 2011
by Nick
Hi CrazyN, It looks encouraging however I have a nasty suspicion that they are relying on an EasyLanguage indicator. Could perhaps Dave or Andrew confirm or deny?
If that's the case then sadly it does not give accurate results. I should know I wrote the indicator linked here
viewtopic.php?f=5&t=6640&p=37671#p37671 (I post as 'BlowFish' at TradersLaboratory). It's based on code I did way back when god was a boy. Since then I have tried another technique that does not use InsideBid/InsideAsk functions but that has issues too sadly.
Of course this is all academic for any 'real world' applications without a way of
calculating delta on historic data.
Still I remain cautiously optimistic but wonder how exactly it has been fixed and if and when it will support historic data?
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 22 2011
by CrazyNasdaq
Hi Nick,
nice to meet you here and happy to know that you are Blowfish at Traderslaboratory.
We talked toghether about the same topic on CVD in Traderslaboratory and there I gave the same answer about #ID as I post here above.
I hope as you about the result on a reliable CVD in MC.
my best Regards to you
CrazyNasdaq
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 22 2011
by Nick
Yes I remember! I too am hopeful and excited but at the same time slightly nervous that TSSuport might not be fully aware of all the issues. Even if things are spot on you absolutely need history to trade live day in day out with cumulative delta.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 22 2011
by janus
I hope you all know that in IB's case they often incorrectly record the historical volume. A long time ago I submitted a ticket to IB and they came back explaining it away by saying they use different snapshot intervals for real-time and historical data. I'm not convinced this is the complete explanation since the real-time volume is often but not always about double that of the historical volume. It appears the discrepancy increases as the market activity increases, and the two are almost always equal during very quiet periods. I also figured out that the real-time volume is the more accurate representation of the true volume at the exchange. I've confirmed this using various other sources of the data. What makes it more worrying is the patterns sometimes change. By this I mean the volume peaks on the volume chart sometimes move to another time after a reload. In any case, this is an IB problem but it appears they refuse to fix it. It has been a problem that was reported by others years ago. Some say IB refuses to fix it because they are very conservative and fear upsetting the reliability of the service if they make a change. Of course this is rubbish. Besides the fix is very simple. Just derive the historical volume from the real-time one. Why compute it twice? So, in IB's case at least providing higher precision data storage in MC would be of limited value, although I would still be in agreement of the move since it would be beneficial with other data source providers who do provide accurate data.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 22 2011
by CrazyNasdaq
YES, IB is not a reliable datafeed. Months ago I've directly asked them which was their protocol to record data regard bid, ask and trade and they confirm me that their data is not so accurate.
This was their answer (see the image below).
BTW I don't use IB anymore.
DTN IQ feed is the best at the moment, Rithmic is not so bad too and Zenfire is not bad, but not so accurtae regard synchronization of Bid & ask with trade
![Image](http://img687.imageshack.us/img687/3926/ibdatafeed.gif)
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 22 2011
by Nick
Yes most feeds are unsuitable you need a full feed complete with all bid ask data. It needs to be clean and properly sequenced. The requirements for cumulative delta are a bit more onerous than simply looking at delta shifts over shorter periods. If you are doing the latter you may get away with less.
IQfeed is pretty much the gold standard for retail. I am currently finding Rithmic pretty darn good too. Zen used to be before CME changed how they reported (creating a lot more ticks) my hunch is they may well be again as they are based on Rithmic. I did not have as many issues with Zen as some have but at the time I was concentrating on short term patterns. CQG are supposed to be good too though I have not tried myself. That's pretty much it until you start getting into the top dollar league.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 22 2011
by janus
Yes CrazyNasdaq, that was the exact same response I got from IB support. If they continue to stick their heads in the sand they will gradually lose market share as others provide better quality data. Meanwhile, I'm stuck with them as I trade a couple of exchanges that the others do not cover as yet. As soon as one of them does I will switch.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 22 2011
by Nick
Very few of the broker feeds are good enough for delta work. At least IB come clean and say they aggregate. Personally I find there data 'adequate' for price and volume based work. Obviously you have less ticks but all the volume is reported.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 22 2011
by janus
Yes, IB is adequate for most traders, but clearly other vendors are not providing more accurate data feeds, such as true tick data just for the fun of it. IB has to come clean one day (perhaps still many years from now) and provide as good if not better data quality, otherwise as trading tools become more sophisticated even for the non-expert trader, IB will be left in the dust. This is the case already for most high frequency day traders and scalpers.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 26 2011
by Dave Masalov
Dear Nick,
Firstly, I would like to point out that in all posts I refer to Fulcrum Trader's indicators that we have tested on our end. As far as we are concerned, they have nothing to do with Gomi's add-ons. These seems to be a different type of indicators. The indicators that Fulcrum Tarder have sent us represent a histogram, while when I have searched for Gomi's indicators I have found that they look like bricks.
Regarding Fulcrum Trader's MD indicators, they can be of two types: BidAsk and UpDownTick based. UpDownTick MD indicators work the same way in real-time and on historical data (they use one data series). BidAsk MD indicators work the same way in real-time and in history with time-based charts (minutes, hours, days etc.), but can show different values in real-time and historical calculations with tick-based charts due to the fact that two data series are used (Bid and Ask). In this case Ask and Bid ticks are not plotted in the same sequence as they were received. I would like to emphasize again that ticks are saved in the database in the same sequence as they were received and you can recall tick data in the correct sequence if you use one data series on your chart. The issue occurs only with several data series on the same chart synchronized with each other as without millisecond or even microsecond precision there is no way to know in which order they should be placed – they can all come in one second. The solution would be to use microsecond granularity (milliseconds are not precise enough).
Also, I would like you to clarify how NT can use milliseconds for indicators calculation even with the help of some add-ons that store data. We are not very familiar with how NT works with regard to indicators calculation, but, normally, the main driver off all processes affecting charting is the chart itself and, if this chart have data with one second precision, how can it use milliseconds for indicators calculation?
As for Neoticker, I have made an experiment showing that Tick Precise technology does not mean that ticks are displayed on history in the same order as they were received in real-time. Please see the attached screenshot. It is a simple case: 2 data series – bid and ask, higher picture is real-time and lower – the same chart after reload. I am not trying to prove anything, but as far as I am concerned multiple data series can be mixed correctly only if milliseconds or microsecond granularity is used.
Can you confirm that Neoticker, NT and SC are capable of doing what you describe?
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 26 2011
by Nick
Hi Dave,
Gomi has a whole suite of indicators for NT. At there core is the GomRecorder and a database that holds bid ask & last ticks with millisecond time stamps. This is completely separate to Ninjas own database! What you probably saw was the volume ladder chart (an add on to the basic indicator suite) in the basic indicator suite there is a delta indicator, cumulative delta, upticks and down ticks indicator and other stuff too. Delta (non cumulative) is usually plotted as a histogram. Cumulative Delta is generally displayed as a regular bar(or candle) with an OHLC. As a side note Gomi's stuff relies on real time data to build the database but it can use Qcollector to populate the database from IQfeed's history. The general consensus is it is accurate. V2 is in the elite section here
http://www.bigmiketrading.com/elite-cir ... r-2-a.html not sure if V1 is still available in the regular section.
SC is looking good I am getting to know the package and have not personally done in depth tests yet (hopefully I won't need to as you guys are working on delta!) I don't know whether they achieve it with millisecond time stamping or with a sequence number across the 3 data streams. Quite a few people have started using it over at BigMikes with encouraging results. It certainly appears to work as it should.
Neoticker I have not used for a long time but if my memory serves me correctly it maintains millisecond precision in it's database. I am not sure why the ticks on your chart have a different time stamp after refresh maybe something to do with live data being time stamped by the PC clock and the refreshed data by the providers ticker plant? The key thing is the sequence of course. I do know a small prop shop that use Neoticker to do full order book analysis but as I say it's years since I used it personally.
I posted some code I wrote a while ago in the Multicharts thread at BigMikes (first posted on TradersLaboratory) it plots delta or cumulative delta. All of the TS/Multicharts code I have seen is either based on it or uses the same technique. I hope it might be helpful.
One thing I think may need attention (as I have mentioned before) is that InsideAsk InsideBid are not guaranteed to return the best bid and ask
from when the tick occurred.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 27 2011
by Dave Masalov
Dear Nick,
InsideAsk and InsideBid are not used in MultiCharts MD indicators as they return price and not volume. Their values are taken from the status line and come directly from data provider.
Regarding your code plotting delta or cumulative delta, could you please give us the link so we can take a look on it?
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 27 2011
by Nick
Hi Dave,
you need to know the price of the best (inside) bid and ask so you can determine if the last tick was at one of those levels. You then accumulate the volume of the last tick accordingly.
Code: Select all
//
// Plots Cumulative Delta
//
// Start at the begining of day and plot as a 'bar'
// Has large trade and small trade filter
//
// By BlowFish / NickA
inputs:
UpColor(darkgreen),
DownColor(red),
MaxBlock(9999),
MinBlock(0),
ResetDeltaEachBar(0);
variables:
MyVol(0),
Block(0),
color(yellow),
firstrunthrough(true),
intrabarpersist MyCurrentBar(0),
intrabarpersist VolTmp(0),
intrabarpersist Delta (0),
intrabarpersist DeltaH (0),
intrabarpersist DeltaL (0),
intrabarpersist DeltaO (0);
if firstrunthrough then begin // We need to do this in case indicator starts mid bar
Voltmp = Iff(BarType < 2, Ticks, Volume);
firstrunthrough = False;
end;
if LastBarOnChart then begin
if CurrentBar > MyCurrentBar then begin
VolTmp = 0;
MyCurrentBar = CurrentBar;
if ResetDeltaEachbar = 1 then Delta =0;
DeltaO = Delta;
DeltaH = Delta;
DeltaL = Delta;
end;
MyVol = Iff(BarType < 2, Ticks, Volume);
Block = Myvol - VolTmp;
if (Block >= MinBlock) and (Block <= MaxBlock) then
if Close <= InsideBid then
Delta = Delta - MyVol + VolTmp
else if Close >= InsideAsk then
Delta = Delta + MyVol - VolTmp ;
VolTmp = MyVol ;
DeltaH = maxlist(DeltaH, Delta);
DeltaL = minlist(DeltaL, Delta);
if Delta <= Delta[1] then color = DownColor else color = UpColor;
plot1(DeltaO, "DO",color);
Plot2(DeltaH, "DH",color);
Plot3(DeltaL, "DL",color);
plot4(Delta, "DC",color);
end;
They key lines that accumulate the delta are
Code: Select all
if Close <= InsideBid then
Delta = Delta - MyVol + VolTmp
else if Close >= InsideAsk then
Delta = Delta + MyVol - VolTmp ;
VolTmp = MyVol ;
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 27 2011
by Dave Masalov
Dear Nick,
Thank you for sharing the code. As we can see it can be used only with real-time data as it uses InsideAsk and InsideBid. Could you tell me why do you think these keywords may not return the correct values? Do you have any example that we can investigate on our end?
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 27 2011
by RobotMan
lol
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 28 2011
by Nick
Hi Dave,
The matter is mentioned briefly here
viewtopic.php?f=5&t=6640 in the indicator section. This describes the problem towards the end of the thread (it's a short thread). It links to a thread at traders laboratory (probably not worth reading at 50+ pages) it also links to an earlier thread here at TSsuport by Robotman that talks about intrabarpersist and insidebid/ask.
I have explored another way of doing things using 3 data streams 1 for bid 1 for ask and 1 for last. That reveals other issues. I can post some code to illustrate that but thought it might be less confusing to tackle one thing at a time.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 31 2011
by Dave Masalov
Dear Nick,
Indeed, InsideAsk and InsideBid keywords are not suitable for these indicators as they return the values from Status Line which are not synchronized with actual ticks. We suggest you to refer to two data series (bid and ask) in your indicators. Thus, you will get correct and consistant results in real-time. Do you agree with that?
We do recognize that there is an issue with MD historical calculation, which comes from technical limitation, and we will adress this issue in the nearest future.
Again, as I have already written, ticks are saved in the database in the same sequence as they were received and you can recall tick data in the correct sequence if you use one data series on your chart. The issue occurs only with several data series on the same chart synchronized with each other.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Jan 31 2011
by Nick
Hi Dave, indeed. I have to say I have noticed some errors with the 3 data stream approach too. The problem is I don't believe that when a tick occurs that every study that relies on that tick is processed from beginning to end before the next tick is dealt with. Let me ask this....
Lets say a tick for data 1 arrives and then very close behind (milliseconds) a tick for data 2 arrives.
If my indicator is processing the data 1 tick and reads "close of data 2" will it get the new value of data 2 or the old (correct) value? I am worried about sequencing errors in processing. Are the ticks held in some sort of queue to be processed? My testing suggests that it is possible to get sequencing errors though it is not possible to say for sure. I certainly get bid/ask crossed (should never happen) and far too may trades that are between best bid & best ask (should be fairly rare).
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 01 2011
by Dave Masalov
Lets say a tick for data 1 arrives and then very close behind (milliseconds) a tick for data 2 arrives.
If my indicator is processing the data 1 tick and reads "close of data 2" will it get the new value of data 2 or the old (correct) value?
Dear Nick,
In this case your indicator will get old (correct) data 2 value for calculation.
My testing suggests that it is possible to get sequencing errors though it is not possible to say for sure. I certainly get bid/ask crossed (should never happen) and far too may trades that are between best bid & best ask (should be fairly rare).
Best bid & best ask come from the Status Line and not from ticks. Thus, they are not reliable for this purpose. Please see my last post in this thread for explanation:
viewtopic.php?f=1&t=6307
Also, I would like to ask you to contact us directly if you have any questions regarding your test results, so we can analyze the situation and give you appropriate answer. This will help to avoid confusion in the future.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 01 2011
by Nick
Hi dave, in my last post all of the questions and observations I was mentioned where when getting the best bid and ask from data2 and data3 NOT from the insidebid and insideask functions. Could the issues I am seeingit be to do with the issue that you mentioned in the intrabar persist thread recently?
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 01 2011
by Dave Masalov
Dear Nick,
Could you explain what exactly do you mean by bid/ask crossed? Please illustrate it with a screenshot if possible. Do you see any issues when using the code attached to my last post in the following thread:
viewtopic.php?f=5&t=6640 ?
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 02 2011
by Nick
Hi dave, yeah I do sometimes I see the bid higher than the ask (crossed). Whilst that can be legitimate (very infrequent though) on ECN's it is not on centralised exchanges like Globex. I am on MC 6.xx so have not had a chance to test on 7.0.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 02 2011
by Nick
You can add something like this to check to see if it occurs
Code: Select all
if (Ask <= Bid) then
print ("crossed, BB:", Bid, " | BA: ",Ask); // Market is crossed
As I say should never happen on a centralised exchange as it will be immediately matched and result in a trade.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 02 2011
by Dave Masalov
Dear Nick,
Please tell us what data provider do you use.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 02 2011
by Nick
Hi Dave, I have been using Rithmic & Zenfire recently (basically the same infrastructure). In the past I have tested with IQFeed. It did occur to me that it might be a data issue but I did not see the same with Zen/NT. My subscription to IQFeed has lapsed and I don't currently have NT set up anyway. Obviously the more side by side testing one can do the better.
If you add that code snipet and it ever triggers it would suggest further investigation is warranted. Last I used it was on the DAX, something that jumps around is possibly better to see it on than say the ES which is quite orderly. Of course that begs the question is it bad prints from Eurex!
I am not in the position to do rigorous testing at the moment (and seems kind of pointless on v 6.xx) but do I plan to make the time and resource available some time after MC beta 7.0 launches.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 03 2011
by Dave Masalov
Dear Nick,
Thank you for the explanation. Also, please confirm that you are talking about real-time calculation (not on historical data).
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 03 2011
by SP
Last I used it was on the DAX, something that jumps around is possibly better to see it on than say the ES which is quite orderly. Of course that begs the question is it bad prints from Eurex!
Nick,
these are most of the time Eurex OTC trades (FESX > 1000 lots, FGBL>2000, FDAX>250). The OTC trade doesnt need to match recent bid/ask values, some datafeeds like Bloomberg have
an extra labeling for these trades.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 04 2011
by Nick
Dear Nick,
Thank you for the explanation. Also, please confirm that you are talking about real-time calculation (not on historical data).
Yes, real time. Nothing gets processed on historic. Incidentally I am not sure but I dont think things will work simply by adding. bid = close of data2; ask = close of data3. You do not want to accumulate volume unless you get a
data1 update. NI have not had a chance to check things but you need to do nothing except update bid & ask if it is a data2 or 3 tick causing the indicator to run.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 04 2011
by Nick
Last I used it was on the DAX, something that jumps around is possibly better to see it on than say the ES which is quite orderly. Of course that begs the question is it bad prints from Eurex!
Nick,
these are most of the time Eurex OTC trades (FESX > 1000 lots, FGBL>2000, FDAX>250). The OTC trade doesnt need to match recent bid/ask values, some datafeeds like Bloomberg have
an extra labeling for these trades.
Hi SP yes OTC trades can be a problem though you don't tend to see them often (round expiration they seem more prevalent presumably as people role over positions into the new front month). By setting the large block filter in my code to the minimum large block size for the instrument you can ensure that is not what is causing the problem. (you may filter a legit trade but its pretty unlikely) Thanks though very valid point.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 08 2011
by Andrew Kirillov
Nick,
Dave is on vacation so I will try to continue investigation of this important issue. We conducted a few tests and found that overlapping happens on Zen-Fire/Rithmic plug-in only. It is a single problem that doesn’t characterize MultiCharts. It is explainable, because all other data feeds provide ask/bid as a pair and overlaps are impossible.
Zen-Fire/Rithmic provides asks and bids separately and as a result we can have updated ask, but old bid, which we use as a current bid.
We've changed the logic in our plug-in and now if we see that ask/bid are overlapping, we will not use the old value, but will wait for the updated one. As a result, the overlaps will not happen anymore. Please download and run updater.exe. Let us know how it works.
http://dl.dropbox.com/u/5815637/ZF_AskBid.zip
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 08 2011
by Nick
Hi Andrew,
I'll give it a go. I am not sure that this is the best way to go. To be honest I do not fully understand how Rithmic/Zen report things let me explain....
lets say that
15 are bid @ 1153.25 | 1153.50 has 25 Asked
OK lets say someone places a market order to buy 25 (which will hit the ask)
how is this reported I am assuming/guessing that
1) we get a tick saying 25 bought @ 1153.50
2) we get a tick saying the ask at 1153.50 is zero or perhaps instead we get
2) a tick saying the best ask is now NNNN @ 1153.75 perhaps we get both
or maybe we get both ask changes? The other thing is that I am not sure that the quote changes will not be before the actual trade
it is possible we will get a quote change first so best ask @ 1153.50 is zero followed by the trade 25 bought @ 1153.50
Long story short I/we need to know how things are reported. All I can say is that bid and ask should never be crossed and if we see them crossed something pretty fundamental is going wrong. Put another way putting a "sticking plaster" (throwing data away until things are not crossed) is a dangerous solution I think.
I wonder if we can find out exactly how Rithmic/Zen report stuff? A trade will nearly always cause a quote change and a trade. I did have some code that just read ticks straight from the Zen API but abandoned the project (and the code).
Hope Dave is having a good holiday!
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 09 2011
by Andrew Kirillov
Hi Nick,
I understand your concern. Obviously we don’t throw data away. We just wait for milliseconds to get an updated value for bid if ask changed or vice versa. Again the issue is they are not updated synchronously and as a result the overlaps are possible when market changes quickly.
Please try the updated data feed and you will see the difference.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 09 2011
by Nick
Hi Andrew,
I'm still on version MC 6 will this be a problem? I was planning to upgrade everything before doing further testing. Seems to make more sense to me.
By delaying are you saying that after a trade tick you will delay until you get a corresponding bid ask update (because the quote update occurs after the trade tick)? In my previous example of a trade at the ask I am thinking you should always get new ask data before a bid update even if someone has bid the price up pretty much at the same time.
I am still worried something is wrong somewhere for example I have never seen a crossed bid/ask on R|Trader or on a Zenfire DOM in NT.
I wish I still had the piece of code I had written direct to the Zen API that just captured raw ticks. It might help me understand what is going on.
Cheers.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 09 2011
by Andrew Kirillov
Nick,
MC6 will work. Just try the dll. I don’t want to spend more time discussing it. See Zen-fire/Rithmic API and you will see the ask/bids fields are sent separately, not as single data field. It is easy to understand that they will be updated at different times and that overlaps are possible.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 09 2011
by Nick
With respect I am only discussing it because I believe there may be a more fundamental problem. I have obviously just not stated it clearly enough.
I understand that the bid and ask will be updated separately but the sequence should never alter and they should never ever cross. Not at any time between the exchange and the client. The fact they do indicates a serious issue (I am not saying that this is a problem with MC just that there is a serious issue).
Now might be a good time to mention the second problem I have observed. I was holding it back so as not to confuse things.
Way too many trades seem to be reported
between the best bid and the best ask. This is not impossible like the previous example however it should be a very rare occurrence. You will see a lot.
Anyway I will try the code and let you know how I get on. It is quite time consuming and to be sure I often need to compare tick by tick!! I am a bit stretched at the moment (off to Venice for a week for Valentines and then I only have a couple of weeks before a month in the sun). Anyway, I will try and do all I can to help you get to the bottom of it, I just haven't been able to give it the time I would like.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 09 2011
by Nick
Hi there... I am getting a message in quote manager log....
unable to connect to trading system. Please check connection settings.
V6 build 3276
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 09 2011
by Andrew Kirillov
Nick,
I do understand your concern, but the issue is Zen-fire/Rithmic specific. Let me give you an example.
Let’s say we have
Best Bid = 10 and Best Ask = 11.
The new Best Bid comes and it is equal to 12. MultiCharts design requires that we send both ask and bid at the same time. So we must use some Best Ask for our new Best Bid. Since we didn’t have any updates for Best Ask we take the previous one which overlaps with the new Best Bid.
The new version will wait for a few milliseconds till the new Best Ask will come from Z/R and we will send it both. Obvously the updated Best Ask will be 13 or higher.
It will assure there are no overlaps.
Is this clear?
I assume the updater erased your registry. Please go to QM->Data Sources->Zen-fire to see if the server/login/password is there.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 09 2011
by Nick
Hi Andrew,
The settings seem to be intact. I was wondering whether it was because I was on an older version? I wouldn't mind updating to V7.0 alpha/beta to test this. Seems to make sense to all be on the same page. Cheers.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 09 2011
by Andrew Kirillov
Nick,
There is no MC 7 beta yet. We are working on it. The plug-in is not compatible with MC DT. So if it doesn't work you can visit our helpdesk for further investigation or wait for the beta. I would prefer if you take advantage of the helpdesk!
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 09 2011
by geizer
subscribing to the thread
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 09 2011
by Nick
Nick,
I do understand your concern, but the issue is Zen-fire/Rithmic specific. Let me give you an example.
Let’s say we have
Best Bid = 10 and Best Ask = 11.
The new Best Bid comes and it is equal to 12. MultiCharts design requires that we send both ask and bid at the same time. So we must use some Best Ask for our new Best Bid. Since we didn’t have any updates for Best Ask we take the previous one which overlaps with the new Best Bid.
The new version will wait for a few milliseconds till the new Best Ask will come from Z/R and we will send it both. Obvously the updated Best Ask will be 13 or higher.
It will assure there are no overlaps.
Is this clear?.
I understand. However you should
never get a new best bid at 12 without having a higher (13) best ask
first. Zen should send BA 13 followed by BB 12. If it sends theme the other way round there is a serious problem. I guest if they get sent together but in no specified order as two seperate events there could be trouble.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 09 2011
by Andrew Kirillov
"However you should never get a new best bid at 12 without having a higher (13) best ask first. "
Is this statement based on some facts or your opinion? I'm telling you what we can see. Did you come to helpdesk?
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 09 2011
by Nick
Hi Andrew thanks for the help getting the new zen dll's going.
It's simply based on the fact that it will result in a crossed market! It depends somewhat on the matching algorithm of the exchange but if contracts are bid and asked at the same price they will never show on the book as they will immediately be matched as a trade.
Lets say a market is 3 contracts bid @ 11 --- 5 contracts asked @ 12. If I place a limit order to buy 10 contracts at 12
3 things happen
A) a trade for 5 occurs @ 12 as my limit buy is immediately matched to the inventory already at 12
B) a new quote will be received for whatever the ask is now (because I bought all 12's e.g 2 @ 13)
C) a new quote will be received for the new best bid which is 5 @ 12 (the remainder of my order to buy 10 @ 12).
All these things happen simultaneously however if Zen reports them as separate events they must be reported the same way each time. if you know the sequence you should be able to reconstruct things. That is unless they have picked an order that does not make any sense like reporting C anywhere but last. I can see if this is reported as 3 events and stuff gets delayed or lost there will be problems.
P.S. Running some preliminary tests now.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 09 2011
by TIKITRADER
subscribing to the thread
ditto
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 10 2011
by Nick
Still seems to be issues
This image shows a couple of 'disturbances in the force' within a few ticks. Remember if there is no red/green dot you need to look to the left to see what the prevailing bid/ask is at. Put another way you only get a dot when they change.
EDIT: I should say this is a preliminary test and so not very rigorous. I'm trying to rectify that now but it requires some work on my part:) As I am progressing it does appear that there may well be bits of bad data coming from Zenfire!
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 10 2011
by Andrew Kirillov
Nick,
Please send me a screenshot of QuoteManager->Data Sources-> Zen-fire data feed.
It should show 1.0.1284 version.
We have no issues with the latest dll on our end. Please let me know.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 10 2011
by Nick
Image attached with version number.
It looks like the problem could actually be the way the data is presented by Zenfire. If you recall my example a couple of posts ago it would appear Zenfire are are actually reporting things as A C B. (Which is crazy imho but such is life
![Smile :)](./images/smilies/icon_smile.gif)
). I think it can be worked around by maintaining a tally of resting orderbook inventory in the indicator. Only just worked out what's going on and still not sure if a) that is the only issue and b) what is the best way of tackling it.
This bit is
important I think :-
I can see a separate problem with EasyLanguage for this sort of work. With the 3 datastreams (bid, ask and last), we need to know which one is causing the indicator to run. This is because the actions you need to perform are different for bid ask and last updates. A keyword that you could read to determine which data is causing the indicator to update would solve this. Unless there is already a way of telling which data stream is causing the 'tick'??
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 10 2011
by Andrew Kirillov
Nick,
It is a good point. Did you use an indicator to plot the data for print screen you posted? If so please give me the indicator code. I encourage you to add ask and bid data streams only and see overlaps. It will allow us to make an isolated test.
Thank you
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 12 2011
by Nick
Hi guys,
Just to be clear I don't think that the delay is needed or any other adjustments to the zenfire dll. (Unless we discover any other 'wrinkles').
Knowing the sequencing is A C B (from my example a few posts back) users can construct their code accordingly. We will need to store the number of contracts available at BB & BA and if a trade takes all of them flag that old BB/BA price level as 'unreliable' until we get an update (which will be the tick after next).
So the only remaining thing that is needed (as far as I am aware) is a way of telling which data stream has caused the indicator to run.
Might not be able to look in for a day or two. Cheers!
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 14 2011
by Dave Masalov
Nick,
Please tell us if you have used an indicator to plot the data for the screenshot you have posted. If yes, please send it to us so we can test it on our end.
We need to know if you have used an indicator to localize the issue. Please let us ask questions and please answer them to make the process faster and easier.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 18 2011
by Nick
Sorry for the delay in replying. Sometimes when I am away the laptop stays in its bag and phone stays off
![Smile :)](./images/smilies/icon_smile.gif)
Not often however it did this time.
Here is the code in case anyone else wants to try things. It is pretty fluid code I edit depending what I want to look at. The real test is to highlight interesting areas and then look at things tick by tick. Turn on the first plot statement to get tick by tick logging. It must be used on 1t charts.
As I have said a couple of time now I think that the crossed condition is simply a side effect of how Zen reports a limit order at BB or BA that fully depletes the inventory at that level. In other words
not a Multicharts issue!!! I hope I have described that adequately in previous posts. Let me know if I have not.
There is another condition I want to fully understand and that is trades between BB & BA. I just have not had the time to go through logs and try and work out what is happening. If i was a betting man I would bet that is also due to how things are reported rather than a problem with MC.
Code: Select all
{
Bid Ask Last - Core Functionality test. Zenfire Edition
Test data stream sequencing iregularities and race conditions.
Place last bid & ask on a single tick chart as data 1 2 & 3
set plot1 & plot2 to high a low bar to mark the place on the chart
where a race condition ocurred. Code is changing regularly.
By: NickA / BlowFish
}
once begin
ClearPrintLog;
end;
Var:
one(0),two(0),three(0),
intrabarpersist bestBid(0),bv(0),
intrabarpersist bestAsk(0),av(0),
intrabarpersist myLast(0),lv(0),
intrabarpersist n(0);
if LastBarOnChart then begin
// print (n, " ", currentbar data1, " ",currentbar data2, " ",currentbar data3, " ", " ",c data1, " ",c data2, " ",c data3);
if currentbar data1 <> one then begin // is this a data 1 tick.
myLast = c data1;
one = currentbar data1;
lv = ticks data1;
// still to check print log tick by tick to check when this condition occurs
//if (myLast < bestAsk) and (myLast > bestBid) then print ("last between BB BA");
//if (myLast > bestAsk) and (myLast < bestBid) then print ("last outside BB BA");
end;
if currentbar data2 <> two then begin
bestBid = c data2;
two = currentbar data2;
bv = ticks data2;
end;
if currentbar data3 <> three then begin
bestAsk = c data3;
three = currentbar data3;
bv = ticks data3;
end;
{ if a limit buy occurs at BA that is greater than the conracts available at BA it is reported as
Trade at BA - New BA - New BB
This invaliates this test- need to maintain volume at BB & BA if any trade depletes it those levels
are undefined until a new BB BA tick is recieved}
if (bestAsk <= bestBid) then begin
print ("crossed, BB:", bestBid, " | BA: ",bestAsk); // is market crossed?
plot1(c-3);
plot2(c+3);
end;
for value3 = 1 to 1 begin //increase loops to load indicator and see if race occurs
for value2 = 1 to 1 begin
value1 = SquareRoot(n) * SquareRoot(n);
end;
end;
if myLast <> c data1 then print ("race condition data1 |" ,myLast, " ", c data1);
if bestBid <> c data2 then print ("race condition data2 |", bestBid, " ", c data2);
if bestAsk <> c data3 then print ("race condition data3 |",bestAsk, " ", c data3);
end;
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Feb 18 2011
by Nick
Hi there it occurs to me one last thing would be very useful for this and all sorts of other things too.
There is currently a check box for 'update on every tick'. For this sort of indicator to work well on historical data a 'update historical bars on every tick' option would be good.
I guess you guys have considered this already, but thought I should mention it.
Incidentally I should apologise if I gave wrong impression early in the thread. I have used various packages but Multicharts is the one I come back to for day to day use, it is my day to day workhorse. My motivation here is simply to highlight an area that I feel is important, and I am very grateful to TSS that it is getting attention. It certainly was not my intention to promote other products (which have there own issues of course). Apart from that being pretty bad manners, I hope that is evident from my posts that I am firmly behind MC. I just wanted to be clear in case anyone got the wrong impression.
Re: MC 7.0 Question - improvements to precision in data hand
Posted: Mar 01 2011
by Dave Masalov
Nick,
This indicator does not show any overlapping on our end. We have performed multiple tests with Zen-Fire demo account. Are you using demo or live account for your tests? Please send us the output file showing 'market crossed' prints.