New Subject: Studies not getting executed on every tick.
Posted: Nov 17 2009
Old Subject: Understanding Ticks command & controlling study execution
Start of New Special Note:
This thread
http://forum.tssupport.com/viewtopic.php?t=6871
on the very last post, has a document you can download which at the very end has a good short explaination of the ticks reserve word as well as good explaination of the intrabarpersist command.
End of New Special Note:
There are two scripts in this thread that report on what appear to be 3 bugs in MC (Andrew confirmed they are bugs and said the will likely be fixed in MC 6.0). Specifically:
1/ Every so often a tick does not get processed by a study when "execute on every tick" is checked.
2/ BarStatus = 0 does not occur at all every so often.
3/ More often BarStatus = 0 occurs 2 times at the start of a bar.
I don't know if it is symbol specific so you might want to run the reports.
Two warnings about running the study:
Warning1: This code is designed for 1 minute bars or second bars such as 10 second bars. It is not designed for tick charts. It is a time based report. If you want to run against tick charts use the version attached to the post which is three posts below this one.
Warning2: As the help for Ticks states "Build Volume On" must be set to "Tick Count". This is found in the "format Symbol", "Settings" drop down box. I forgot to do this when I first applied it to the minute bars and the “Ticks” gaps were huge (often as high as 3000). When you set it properly the gaps in the “Ticks” are more reasonable (0, 2, 3). A gap of 1 is normal and is not reported.
I learned a lot from this problem such as thinking out where the tick data is likely to be centrally stored and accessed by each chart on a tick by tick basis without duplicating the data. In other words the studies are likely called with pointers back to the central data in the buffer. I realized this as part of dealing with this missing ticks problem and building the tables below in my mind to think out what was going on. As the table below demonstraites, it made me realize that this central buffer pool of ticks has to have a master Tick Number and I am guessing it has a master tick Arrival time too (both of which would be handy if we could get at them to rewrite our studies more efficiently).
New features of the attached study which creates the missing ticks report:
1/. Bar type, bar size and symbol are part of the report name so you can run it on seconds bars and minutes bars at the same time and have separate sets of reports being created.
2/ The “?” which shows a gap in the “ticks” reserve word is replaced with the size of the gap. Each tick should be an increment of 1 (a gap of 1) and when this occurs nothing is reported in that column. If it is less than 1 or greater than 1 it is reported. Note that the Stats report only counts the number of bad gaps that occurred (it does not total the size of the gaps).
3/ “<<” is placed beside large gaps and you can control the size for this. You also have the option to have it play a buzz sound if you want when this happens. I used it during testing.
4/ The stats report now reports every X number of bars. Doing it by minutes was too hard to verify.
5/ If you want you can have a sound come out when a stats record is written. I needed it for testing.
6/ A raw report is optional (no formating). I needed it to test the study.
The zip file now has the .pla file of the code. No more cut and paste.
Start of New Special Note:
This thread
http://forum.tssupport.com/viewtopic.php?t=6871
on the very last post, has a document you can download which at the very end has a good short explaination of the ticks reserve word as well as good explaination of the intrabarpersist command.
End of New Special Note:
There are two scripts in this thread that report on what appear to be 3 bugs in MC (Andrew confirmed they are bugs and said the will likely be fixed in MC 6.0). Specifically:
1/ Every so often a tick does not get processed by a study when "execute on every tick" is checked.
2/ BarStatus = 0 does not occur at all every so often.
3/ More often BarStatus = 0 occurs 2 times at the start of a bar.
I don't know if it is symbol specific so you might want to run the reports.
Two warnings about running the study:
Warning1: This code is designed for 1 minute bars or second bars such as 10 second bars. It is not designed for tick charts. It is a time based report. If you want to run against tick charts use the version attached to the post which is three posts below this one.
Warning2: As the help for Ticks states "Build Volume On" must be set to "Tick Count". This is found in the "format Symbol", "Settings" drop down box. I forgot to do this when I first applied it to the minute bars and the “Ticks” gaps were huge (often as high as 3000). When you set it properly the gaps in the “Ticks” are more reasonable (0, 2, 3). A gap of 1 is normal and is not reported.
I learned a lot from this problem such as thinking out where the tick data is likely to be centrally stored and accessed by each chart on a tick by tick basis without duplicating the data. In other words the studies are likely called with pointers back to the central data in the buffer. I realized this as part of dealing with this missing ticks problem and building the tables below in my mind to think out what was going on. As the table below demonstraites, it made me realize that this central buffer pool of ticks has to have a master Tick Number and I am guessing it has a master tick Arrival time too (both of which would be handy if we could get at them to rewrite our studies more efficiently).
Code: Select all
Below is a diagram of the potential reality of tick arrival and processing (or at least I am concerned about it because I have on occasion written loops in error that stop MC dead with an immediate freeze- I immediately realize it, fix it and the freeze is gone). In the table below one study can keep up with tick arrival speed and the other study can not (other factors outside of bugs in a study or inefficient studies could be the speed of the computer, number of studies, MC/TS itself, operating system catchup, whatever).
Studying the table creates these questions:
“How does MC currently handle it when a big study can not keep up?”
“How should MC report the problem to the trader so they can adjust their studies to fix the problem?”
“What tools can the trader be given to allow them to bypass a low priority study if it falls behind?”
"For slow ticking symbols can Studies have an option to execute every X.X seconds in addition to every tick (and know how it was executed).
For TSS “How can we market this to make traders realize MC handles this reality better than TS/NT? - In other words, helps the traders find studies that need to be made more efficient.”
Symbol ES tick receive buffer Info:
Note: Tick# is not the ticks command value (for demo only)
Tick# ReceiveTime Price, Volume
1 10:00:01 1000.00 ?
2 10:00:02 1000.25 ?
3 10:00:03 1000.50 ?
4 10:00:04 1000.75 ?
5 10:00:05 1001.00 ?
6 10:00:06 1001.25 ?
(Here is a study that can keep up to the ticks as fast as they arrive)
The study uses a pointer to the buffer table info.
At 10:00:01 Call Study2(Using Tick1 table entry) done at 10:00:01
At 10:00:02 Call Study2(Using Tick2 table entry) done at 10:00:02
At 10:00:03 Call Study2(Using Tick3 table entry) done at 10:00:03
At 10:00:04 Call Study2(Using Tick4 table entry) done at 10:00:04
At 10:00:05 Call Study2(Using Tick5 table entry) done at 10:00:05
At 10:00:06 Call Study2(Using Tick6 table entry) done at 10:00:06
(Here is a study that can not keep up in that it takes 2 times as long to execute as the speed of tick arrival)
The study uses a pointer to the buffer table info.
At 10:00:01 Call Study1(Using Tick1 table entry) done at 10:00:03
At 10:00:03 Call Study1(Using Tick3 table entry) done at 10:00:05
At 10:00:05 Call Study1(Using Tick5 table entry) done at 10:00:07
Code: Select all
TSS may want to verify my conclusions here.
Based upon the attached study results against the ES symbol, I think that when you start MC or TS it starts grabbing ticks (trades) and assigning a master number internally to each new tick (for each symbol probably). A small sample of these is shown on the left side of the little table below. Now at the start of each bar MC internally gets the master tick number and stores it for use by the ticks command. If you use the ticks command on each new tick it could only show the missing ticks by using this calculation shown here in brackets which is associated with what I diagram below in the table. (Master# 12 minus Master# 8 + a constant 1 Gives tick 5 within the current bar). In other words it stores the master number 8 at the start of the bar in the example below and works with that to compute the ticks command output. Again it properly computes the occasional gap which Andrew said they would fix in MC 6.0.
Main..................Study
Tick..................Starts
Receive...............at
Buffer................Base Tick# 8
Tick..................Ticks Command
Number................Value=1
1
2
3
4
5
6
7.........Constant
8..........minus 7......1
9..........minus 7......2
10.........minus 7......3
11.........................(Gap of 2 here as the study shows)
12.........minus 7......5
If we can get at the master tick number in the buffer from all our studies we can make good use of that to audit our studies. Also if we can get at the master arrival time that would be even more useful.
Note that MC does not count study executions to calculate the ticks command output or it would not be able to detect missing ticks. It has to use the master tick number from the buffer. Users can count study executions. No need for a reserve word command to do that.
New features of the attached study which creates the missing ticks report:
1/. Bar type, bar size and symbol are part of the report name so you can run it on seconds bars and minutes bars at the same time and have separate sets of reports being created.
2/ The “?” which shows a gap in the “ticks” reserve word is replaced with the size of the gap. Each tick should be an increment of 1 (a gap of 1) and when this occurs nothing is reported in that column. If it is less than 1 or greater than 1 it is reported. Note that the Stats report only counts the number of bad gaps that occurred (it does not total the size of the gaps).
3/ “<<” is placed beside large gaps and you can control the size for this. You also have the option to have it play a buzz sound if you want when this happens. I used it during testing.
4/ The stats report now reports every X number of bars. Doing it by minutes was too hard to verify.
5/ If you want you can have a sound come out when a stats record is written. I needed it for testing.
6/ A raw report is optional (no formating). I needed it to test the study.
The zip file now has the .pla file of the code. No more cut and paste.
)