[ts-gen] Keeping TWS/shim up for an entire week?

Bill Pippin pippin at owlriver.net
Mon Aug 17 13:02:13 EDT 2009


First, what seems to be your core question:

> Has anyone managed to keep TWS/shim up automagically for an entire
> week?  If so, could you please share with us some of the things you
> have done to ensure robustness of your system?

I haven't tried to run the IB tws past the recommended service period
of 24 hours.  In fact, I see doing so as directly contrary to your
stated goal of robust operation.  About ignoring IB's recommended
practice of restarting the tws once per day:

> Has anyone succeeded in keeping TWS/shim up for an entire week?

Again, I haven't tried.  That being said, there's no reason I know of
why the shim couldn't keep up with the IB tws if it was working properly
over the course of a week, that is: 1.  you were to use one of the
various work-arounds to keep the IB tws running and 2.  it was not just
running, but running *correctly*.  So, feel free to explore this area
with the paper account and let us know here on the list what you find
out.  In particular, please let us know about any bugs in the shim, if
any, that you uncover.

By the way, you should know that one IB employee with detailed
knowledge of IB tws internals has given [on the Yahoo list] what I
read as friendly but very direct warnings about the unsafe nature of
the IB tws over multi-day timeframes.

> ... my initial attempt is to query for forex 5 second bars.
> It seems to be working most of the time, but recently, the system
> falls into a strange state where the TWS remains up, but the
> Ruby/shim needs to be restarted.

You should be able to completely disambiguate the performance of your 
downstream ruby script from that of the shim, either by writing restart
logic for the shim in your downstream program, or, in this case, by
discarding the downstream program completely and instead feeding simple
commands to the shim through the stdin, say from a shim script.
If you're uncertain about whether your problem is with the shim, or
else your downstream script, I'll leave you to settle that question

As for the shim, you can use ping to make sure it's alive, select next
to make sure the IB tws is alive (both of these steps are mentioned
in the manual), and then drop and reselect your symbol of interest to
see if the upstream is providing data.  You should know that a recent
post to the Yahoo list suggested that the kind of data quality problems
you are seeing are sometimes a sign of communication problems with the
exchange, and that the occurrence of such was a key reason why the
writer did not try to run his system unattended.

> I believe the problem is, when TWS loses its connect and reconnects,
> my initial query for 5 second bars is lost.  I will code around this
> by requerying for bars ...

Make sure to drop the old subscription first, and then, yes absolutely,
if you lose data flow for any reason, resubscribing makes perfect sense,
and, as you imply, waiting until network connectivity is restored is
probably your best bet:

> But there seems to be two kinds of 3 4 2 recovery message sequences:

> 1.  Connectivity between IB and TWS has been lost.
> HMDS data farm connection is OK:ushmds2a

> 2.   Connectivity between IB and TWS has been lost.
> Connectivity between IB and TWS has been restored - data maintained.
> HMDS data farm connection is OK:ushmds2a

> The second one almost seems to iimply that I don't need to re-request
> for 5 second bars, but it doesn't happen very often.  I will try to
> re-request when I see the following message:

> HMDS data farm connection is OK:ushmds2a

Makes sense ...

> in hopes that re-requesting something that is already requested
> doesn't hurt anything.

It might well use up multiple market lines when you just need one,
so again, unsubscribe first.

>                    ...  In trying to keep TWS/shim up, it seems
> to me that I have to code for all sorts of states - I guess this
> is a characteristic of real-time systems.

Indeed, and some of the behavior you'll see, such as the evidently
bad data you mention later, is to my way of thinking a good reason to
restart your IB tws at one day intervals as recommended by them, in
which case you'll be restarting the shim as well.  Note that there is
no particular reason you can't keep your downstream running, just that
you almost certainly should restart the IB tws, and that in that case
you will need to restart the shim also.

> One strange side-effect of this reconnecting seems to be - sometimes,
> querying continues and I get highs and lows of 0.0, but okay values
> for open and close.  Very odd.

Does the data improve if you restart the IB tws?  In any case, this is
not a problem with the shim.  By the way, if you use the file and save
options, you can compare the binary log to the text log, and you'll
see that the zeros are in the input, and that the shim is just
faithfully reproducing them to the text log.



More information about the ts-general mailing list