[ts-gen] Keeping TWS/shim up for an entire week?
boadie at gmail.com
Thu Sep 3 15:50:46 EDT 2009
On 03/09/2009, at 2:24 PM, Ken Feng wrote:
>> So I have switched to a model where I keep TWS up all the time by
>> resetting the log off time hourly, and I keep the shim up all the
- How are you resetting the log off time hourly?
>> Russ or Bill who hinted a few weeks ago that ticks may be more
It was Bill and he said it was easy to get free realtime ticks from
other sources, I thought
he meant other than IB.
That was interesting for me and I found this ruby gem, which works
for realtime on many, many markets:
I worry about using it in mass of course because I am sure that yahoo
has to have a
mechanism in place to avoid "the tragedy of the commons" which is
sure to happen
when something so useful is free.
This warning from Yahoo is also worth reading:
On 03/09/2009, at 2:24 PM, Ken Feng wrote:
> This is an update to a thread I started a few weeks ago.
> After several weeks of trying, I gave up on accessing bars through
> ushmds2a. The problem was, I couldn't get TWS to reconnect to
> ushmds2a consistently after the daily reset. Sometimes, re-selecting
> the bars would force TWS to reconnect, while other times, it wouldn't
> work for at least an hour before reconnecting. I tried restarting
> both TWS and shim, but it made no difference.
> Initially, I tried to avoid select ticks because I didn't want all the
> extra data and I didn't need the resolution. However, during the
> experiment, I noticed that cashfarm seems to be more reliable than
> ushmds2a; outages are not as frequent and when they do occur,
> reconnects tend to be resolved more quickly.
> So I have switched to a model where I keep TWS up all the time by
> resetting the log off time hourly, and I keep the shim up all the
> time. I query for ticks instead of bars, and wrote a little Ruby to
> convert the ticks to bars. The only thing to watch out for is, during
> the times where the server is reseting, -1.0 for the open, high, low,
> or close may be sent by IB - not difficult to filter out, but be aware
> if you use cashfarm.
> I am now carefully monitoring this new setup to see if it is robust
> and stable. So far this week, it has failed once, when IB had some
> routing problems, as mentioned on the server refresh page:
> I will have to write a script to detect these outages so I can restart
> TWS and shim during these periods.
> But overall, I think selecting ticks through cashfarm appears to be
> more reliable than selecting bars through ushmds2a. I think it was
> Russ or Bill who hinted a few weeks ago that ticks may be more
> - Ken
> On 8/18/09, Ken Feng <kfmfe04 at gmail.com> wrote:
>> Hi Bill & Russ,
>> Regarding my daily 3:55UTC problem,
>>> 1. For example, I know for a fact that around 3:55UTC every single
>>> day, I get an odd behavior in the system where the TWS stops
>>> fulfilling the bar requests, even though TWS itself still receives
>>> ticks and continues to display them in the GUI. When I kill the
>>> and restart it manually, the response has been sporadic - sometimes,
>>> it works on the first try. Sometimes on the second or third try.
>>> guess is, the TWS has fallen into a state that I have not yet
>>> anticipated, for whatever reasons (maybe daily maintenance by IB?),
>>> and I need to to either detect the state at any time or patch around
>>> the 3:55UTC gap.
>> following Bill's suggestion that I cancel the bar requests first, I
>> was able to continue receiving ticks without manual intervention.
>> gap, however is kind of large:
>> AUDJPY|2009-08-18 03:54:55|77.960|77.960|77.960|77.960
>> AUDJPY|2009-08-18 03:55:00|77.960|77.960|77.960|77.960
>> AUDJPY|2009-08-18 04:05:25|78.040|78.050|78.040|78.040
>> AUDJPY|2009-08-18 04:05:30|78.040|78.040|78.040|78.040
>> I dug into ShimText and found the following in summarized form (table
>> lines up in a monospaced font:
>> secs utime utc comment
>> 46505 1250567700 03:55:00 Last Good Tick at 03:55:00
>> 46509 MDS data farm connection is broken:ushmds2a
>> 46512 ----- TRIGGER ----- MDS data farm connection is OK:ushmds2a
>> 46512 Ruby: begin sending 15 cancel bars to shim
>> 46527 Ruby: begin sending 15 select bars to shim
>> 46861 Market data farm connection is
>> 46939 Market data farm connection is OK:cashfarm
>> 47111 ----- TRIGGER ----- MDS data farm connection is OK:ushmds2a
>> 47111 Ruby: begin sending 15 cancel bars to shim
>> 47113 1250568305 04:05:05 EURCHF, EURJPY, USDCAD, EURUSD, USDJPY,
>> EURCAD, GBPCHF, NZDUSD, USDCHF, GBPUSD,
>> 47126 Ruby: begin sending 15 select bars to shim
>> 47130 1250568325 04:05:25 AUDJPY, AUDUSD, and others come in
>> I am TRIGGERing 15 cancel bars and 15 select bars to be sent to the
>> shim on "MDS data farm connection is OK:ushmds2a", and even though
>> this works, the behavior appears to be sub-optimal.
>> 1. Observe 47113 - the fact that I am already receiving ticks here
>> implies that 46512 and 46527 are sufficient, while 47111 and 47126
>> appear to be redundant.
>> 2. If I use no TRIGGERs at all, I will fail to receive any ticks at
>> all after 46505.
>> So given 1. and 2., we now have some state-based ugliness. It almost
>> seems that instead of TRIGGERing on "MDS data farm connection is
>> OK:ushmds2a", I must TRIGGER if and only if "Market data farm
>> connection is broken:cashfarm" and "Market data farm connection is
>> OK:cashfarm" do not immediately proceed it.
>> I think this kind of state-based code should work to prevent 47111
>> 47126 redundancies, but it is extremely brittle and prone to
>> and difficult to debug when the server changes. I can go for
>> "tight-coding" where the state-based code is expected to work only
>> this particular case, or I can go for "general-coding" where it will
>> apply to similar events throughout the day. Without any
>> specifications from IB, I can only guess which type of coding may be
>> more "robust". At this point, I am towards going with the
>> "tight-coding", perhaps even limiting the TRIGGER to a certain range
>> of UTC time.
>> If anyone has any better, simple solutions, please do let me know!
>> BTW, Russ' suggestion of modifying .shimrc values like ShimText
>> shim startup also works very well. I now make my Ruby script modify
>> the .shimrc to output to yyyymmdd.log before starting up the shim.
>> Thank you.
>> - Ken
> ts-general mailing list
> ts-general at trading-shim.org
More information about the ts-general