[ts-gen] Keeping TWS/shim up for an entire week?

Richard Pruss boadie at gmail.com
Thu Sep 3 15:50:46 EDT 2009


On 03/09/2009, at 2:24 PM, Ken Feng wrote:
>> So I have switched to a model where I keep TWS up all the time by
>> resetting the log off time hourly, and I keep the shim up all the
>> time.
- How are you resetting the log off time hourly?

>> Russ or Bill who hinted a few weeks ago that ticks may be more
>> reliable.

It was Bill and he said it was easy to get free realtime ticks from  
other sources, I thought
he meant other than IB.
That was interesting for me and I found this ruby gem, which works  
quiet nicely,
for realtime on many, many markets:
http://www.transparentech.com/opensource/yahoofinance
I worry about using it in mass of course because I am sure that yahoo  
has to have a
mechanism in place to avoid "the tragedy of the commons"  which is  
sure to happen
when something so useful is free.
This warning from Yahoo is also worth reading:
http://help.yahoo.com/l/us/yahoo/finance/my_portfolios/fitarealtimedata.html

- Ric

On 03/09/2009, at 2:24 PM, Ken Feng wrote:

> Hi,
>
> This is an update to a thread I started a few weeks ago.
>
> After several weeks of trying, I gave up on accessing bars through
> ushmds2a.  The problem was, I couldn't get TWS to reconnect to
> ushmds2a consistently after the daily reset.  Sometimes, re-selecting
> the bars would force TWS to reconnect, while other times, it wouldn't
> work for at least an hour before reconnecting.  I tried restarting
> both TWS and shim, but it made no difference.
>
> Initially, I tried to avoid select ticks because I didn't want all the
> extra data and I didn't need the resolution.  However, during the
> experiment, I noticed that cashfarm seems to be more reliable than
> ushmds2a; outages are not as frequent and when they do occur,
> reconnects tend to be resolved more quickly.
>
> So I have switched to a model where I keep TWS up all the time by
> resetting the log off time hourly, and I keep the shim up all the
> time.  I query for ticks instead of bars, and wrote a little Ruby to
> convert the ticks to bars.  The only thing to watch out for is, during
> the times where the server is reseting, -1.0 for the open, high, low,
> or close may be sent by IB - not difficult to filter out, but be aware
> if you use cashfarm.
>
> I am now carefully monitoring this new setup to see if it is robust
> and stable.  So far this week, it has failed once, when IB had some
> routing problems, as mentioned on the server refresh page:
>
> http://www.interactivebrokers.com/en/software/systemStatus.php?ib_entity=llc
>
> I will have to write a script to detect these outages so I can restart
> TWS and shim during these periods.
>
> But overall, I think selecting ticks through cashfarm appears to be
> more reliable than selecting bars through ushmds2a.  I think it was
> Russ or Bill who hinted a few weeks ago that ticks may be more
> reliable.
>
> - Ken
>
> On 8/18/09, Ken Feng <kfmfe04 at gmail.com> wrote:
>> Hi Bill & Russ,
>>
>> Regarding my daily 3:55UTC problem,
>>
>>> 1.  For example, I know for a fact that around 3:55UTC every single
>>> day, I get an odd behavior in the system where the TWS stops
>>> fulfilling the bar requests, even though TWS itself still receives
>>> ticks and continues to display them in the GUI.  When I kill the  
>>> shim
>>> and restart it manually, the response has been sporadic - sometimes,
>>> it works on the first try.  Sometimes on the second or third try.   
>>> My
>>> guess is, the TWS has fallen into a state that I have not yet
>>> anticipated, for whatever reasons (maybe daily maintenance by IB?),
>>> and I need to to either detect the state at any time or patch around
>>> the 3:55UTC gap.
>>
>> following Bill's suggestion that I cancel the bar requests first, I
>> was able to continue receiving ticks without manual intervention.   
>> The
>> gap, however is kind of large:
>>
>> AUDJPY|2009-08-18 03:54:55|77.960|77.960|77.960|77.960
>> AUDJPY|2009-08-18 03:55:00|77.960|77.960|77.960|77.960
>> AUDJPY|2009-08-18 04:05:25|78.040|78.050|78.040|78.040
>> AUDJPY|2009-08-18 04:05:30|78.040|78.040|78.040|78.040
>>
>> I dug into ShimText and found the following in summarized form (table
>> lines up in a monospaced font:
>>
>> ----------------------------------------------------------------------
>> secs  utime      utc      comment
>> ----------------------------------------------------------------------
>> 46505 1250567700 03:55:00 Last Good Tick at 03:55:00
>> 46509                     MDS data farm connection is broken:ushmds2a
>> 46512 ----- TRIGGER ----- MDS data farm connection is OK:ushmds2a
>> 46512                     Ruby: begin sending 15 cancel bars to shim
>> 46527                     Ruby: begin sending 15 select bars to shim
>> 46861                     Market data farm connection is  
>> broken:cashfarm
>> 46939                     Market data farm connection is OK:cashfarm
>> 47111 ----- TRIGGER ----- MDS data farm connection is OK:ushmds2a
>> 47111                     Ruby: begin sending 15 cancel bars to shim
>> 47113 1250568305 04:05:05 EURCHF, EURJPY, USDCAD, EURUSD, USDJPY,  
>> EURGBP,
>>                          EURCAD, GBPCHF, NZDUSD, USDCHF, GBPUSD,  
>> GBPJPY
>> 47126                     Ruby: begin sending 15 select bars to shim
>> 47130 1250568325 04:05:25 AUDJPY, AUDUSD, and others come in
>> ----------------------------------------------------------------------
>>
>> I am TRIGGERing 15 cancel bars and 15 select bars to be sent to the
>> shim on "MDS data farm connection is OK:ushmds2a", and even though
>> this works, the behavior appears to be sub-optimal.
>>
>> 1. Observe 47113 - the fact that I am already receiving ticks here
>> implies that 46512 and 46527 are sufficient, while 47111 and 47126
>> appear to be redundant.
>> 2. If I use no TRIGGERs at all, I will fail to receive any ticks at
>> all after 46505.
>>
>> So given 1. and 2., we now have some state-based ugliness.  It almost
>> seems that instead of TRIGGERing on "MDS data farm connection is
>> OK:ushmds2a", I must TRIGGER if and only if "Market data farm
>> connection is broken:cashfarm" and "Market data farm connection is
>> OK:cashfarm" do not immediately proceed it.
>>
>> I think this kind of state-based code should work to prevent 47111  
>> and
>> 47126 redundancies, but it is extremely brittle and prone to  
>> breakage,
>> and difficult to debug when the server changes.  I can go for
>> "tight-coding" where the state-based code is expected to work only  
>> for
>> this particular case, or I can go for "general-coding" where it will
>> apply to similar events throughout the day.  Without any
>> specifications from IB, I can only guess which type of coding may be
>> more "robust".  At this point, I am towards going with the
>> "tight-coding", perhaps even limiting the TRIGGER to a certain range
>> of UTC time.
>>
>> If anyone has any better, simple solutions, please do let me know!
>>
>> BTW, Russ' suggestion of modifying .shimrc values like ShimText  
>> before
>> shim startup also works very well.  I now make my Ruby script modify
>> the .shimrc to output to yyyymmdd.log before starting up the shim.
>>
>> Thank you.
>>
>> - Ken
>>
> _______________________________________________
> ts-general mailing list
> ts-general at trading-shim.org
> http://www.trading-shim.org/mailman/listinfo/ts-general



More information about the ts-general mailing list