[ts-gen] Unable to retrieve historical data for german stocks

Bill Pippin pippin at owlriver.net
Tue Oct 7 15:37:50 EDT 2008

With respect to which release of the IB tws to use, the shim should
work well against any of the working releases.  Note that IB kills
off older releases with some enthusiasm, so that this doesn't cover
that much ground.

On occasion new IB tws downloads increase the version number for
one of the message types, to alert developers to the presence of
new features in later versions of the api.  The actual syntax for
the rest of the message stays unchanged, since that is determined
by the client version sent by the shim during the connect handshake.

For this case where the IB tws is advertising new message features,
then in the absence of updates to the shim, where it is behind in what
message version number to expect, it prints a warning, so that you
know to download a newer version of the shim:

Message  1 version: expected  4, received  5 for    Price
Problem: 427 api msg version mismatch -- shim is outdated

In this case I forced the above by setting the version back from
the current 5 to 4 for the market data price message, the first
of the tws api message types, and then recompiling the shim, after
which the exs/tick script sprayed the above all over the screen.

It prints these warnings to the stderr for *every* message of the
given type it sees, so once observed, you may see a lot of these
messages, but they are otherwise harmless; they don't prevent
acceptance of the related messages, which show up in the log just
as you would expect.

They do mean the shim's parser is guessing about message headers,
but this is typically not a problem in the short run.  Over time
it would lead, given the occurrence of other parse errors, to greater
confusion.  That's why the warning occurs, and why it's a good idea
to get an update when you see it.

The 427 warning message can also occur by mistake after some
other parse error, if the message token stream becomes misaligned.
The shim's approach to parse errors for IB tws messages is to 
discard one token and try again, so that the parser is self-aligning,
but there may be a fair number of discarded messages before
realignment is achieved.  So, only initial warnings are definitely
trustworthy, and once other message parse errors have occurred, 427
warnings in the immediate sequel are most likely misleading.

As for message version conflicts in the other direction, the shim
ignores differences where it is ahead by no more than 1, so using
the shim with an old release of the IB tws should just work (IB
doesn't seem to let them fall behind by more than that).

The only other possible problem is that in the presence of multiple
related parse errors the shim may have a harder time recovering from those
errors, since there is no explicit message boundary in the IB tws
message protocol.  That's one reason why the shim requires that message
version numbers be at most one off from, if not equal to, the expected
value, and why it is so meticulous about checking token values against
tables; false message matches would otherwise be much more likely.

So, given that Paolo was observing syntax errors, and trying to track
them down, Sam was giving good advice:

> ... test with a Release 887.2 TWS ...

though users are free to ignore it, as explained above.  Be aware that
IB's servers will probably refuse to talk to your old favorite tws long
before the shim has problems with it.

By the way, in this case, since the command and message streams have
distinct parsers, scanners, and token queues, there would not have been
any interaction between the problem with command notation, and the
ability of the shim to parse the resulting message stream.  There 
might not have *been* any messages --- no command, then no request,
then no messages --- but the shim's message parser would have been

Outside of 427 warnings when new features are released, parse errors
for the IB tws message stream are very rare for ordinary use.  One
of our users recently observed one because he connected to an FA
account, and the shim was not able --- by the then-current design ---
to parse the unsolicited FA account list message that showed up
immediately after the handshake.

We don't use the shim with FA accounts, and I was unaware up til
then of the need to parse this message, since it occurs in the
absence of an explicit request.  I went ahead and updated the shim's
message parse tables in response to his report, which eliminated the
problem.  (It didn't provide the other features needed for FA support,
but that's a whole different subject.)

Errors can also occur during the database load phase at startup,
although here once you get past the banner you should be fine.  Such
errors are due either to bugs in the shim, or bad data in the
database, and for the former case should be caught before the shim
is released, although here I'll note that the 080930 release was an
unfortunate exception.

Command syntax errors are by their nature much easier to generate.
After all, the shim downstream input is free form text, possibly
from people, as opposed to the upstream feed from the IB tws program.



More information about the ts-general mailing list