[ts-gen] cross session orders, client ids

pippin at owlriver.net pippin at owlriver.net
Thu Dec 18 14:16:44 EST 2008


I'd like to explain what I think the current problem most likely
is, and not so incidently a more precise explanation of what I
meant by "overly sensitive behavior by the shim".  I'd also like
to suggest another configuration variable you should feel free to
vary, although it's admittedly a long shot, and finally, I'll give
you some idea of what to expect from the modified shim, that is
the one with a variant banner that I pushed earlier today.

1.  When scanning for account data, the shim does not yet have
the information it needs to accept commands from the user, and
it's already read in the database, so its only input at this
point is the upstream tws.  Since initialization time must be
predictably bounded, just blocking on input is not acceptable;
instead, the shim times out if it can't get the account number.

The IB tws unfortunately does not respond to the account data
query right away, instead sending various other status messages
about matters such as the availability of the market data and
history farms, and possibly your portfolio value.  Then, when
it sends the account data, it sends over one hundred account
data quads, all of which should have the account number as the
last attribute, here just a string.

The shim takes the first one of these account data quads it
reads, and validates its account code string as an account
code object, first looking up the string in the AccountCode
table, and either finding an account code object already in
the AccountCode table, or else writing the newly observed
AccountCode record to the database, reading the record back
in, and then looking it up again, and this time necessarily
succeeding given a successful database write/read.  There
is a strong presumption here of success with that IO, since
errors with mysql queries should result in other error messages,
large chunks of text sprayed across the screen according to
the mysql_errno() procedure, and it would seem that you have
not seem such errors. 

Any failure along the way to this lookup gives a 524 exit,
for failure to obtain account info, where the resulting message
reads as follows:

     Problem: 524 could not obtain IB account info at startup

So, your problem, given a 524 exit, could be due to time out
waiting on the IB tws; unexpected text from the IB tws, and
delay of the account data query answer following; failure by
the shim to read an account data quad; or failure by the shim
to translate the account code string to a validated account
code object.

The rapid exit you see indicates that the problem is with
some failure beyond the timed read, the first step in the
chain above.

I've confirmed that the timeout variable works properly; if I
cripple the code so that the upstream reader (falsely) claims
not to have received the text for the account quads, the
shim blocks for the expected time period.

I wouldn't be surprised if the problem stems from either
unexpected data from the IB tws that arrives in front of
the account data, or some kind of problem with the database
lookup.  It's unlikely that the IB tws is just refusing to
answer the account data query; we would expect you to observe
a timeout delay in this case, although a bunch of portfolio
value messages could in effect "poison the well", in which
case please see [2] below.

The filter translations of the socket stream text should help
us narrow the problem down.

2.  There is a configuration variable NumWords, default value 16.
Please feel free to increase it, so that the shim will try to
read in more text from the IB tws before scanning that text for
an account quad.  Don't waste too much time on this --- it may
well not be the problem --- but feel free to try it.  Also, please
understand that if it's sufficiently large, the shim will just
hang, as both it and the tws wait for messages from the other.

3.  The modified shim I've pushed at the ftp site purposely
provides a somewhat messy banner:

    src$ clear; (cd ..; make && exs/exit --data file save)

              The trading shim has connected to the database server
              and loaded 51649 products;
    Text:     0
    Read:    10   2  10

    Text:     8
    Read:     8   1  10
    connected to the IB tws as
              client id  1;
    Text:     5
    Read:   654  21  10
    Time: 1 Error Error
    Time: 1 Error Error
    Time: 1 KeyValue KeyValue
    and queried for the account code, which
              seems to be  DUxxxx.  Program initialization has been

The Text/Read trace lines give the number of words queued from the
stream before and after the read loop, and, following as part of
the Read trace, the minimum number of words required, and the timeout.

The Time traces give the type and object labels, for the common case
both the same, here the history farm status messages [Error, since
they are after all tws error messages, even if just info in this case]
followed by an account data quad [KeyValue] message.  The completion
of the banner indicates that the database lookup succeeded.

I'd like to see your version of this banner, if you would be so kind
as to post it.

Also, feel free to check your database, in particular the journal table

    sql$ ../bin/mysql
    Reading table information for completion of table and column names
    You can turn off this feature to get a quicker startup with -A

    Welcome to the MySQL monitor.  Commands end with ; or \g.
    Your MySQL connection id is 1923
    Server version: 5.0.45 Source distribution

    Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

    mysql> select * from AccountCode;
    | uid | code   | time                |
    |   1 | DUxxxx | 2008-12-17 12:06:35 |
    1 row in set (0.00 sec)

    mysql> Bye

Looking at our ftp site, the tarball isn't there yet, but hopefully
it'll show up within the hour.

[Update]  The tarball has arrived, sort of.  Due to a glitch I'm
not going to try to explain here, you should make sure to pull an
explicitly named shim-081218.tgz tarball, and ignore the link to
shim-latest.tgz; that is, once at the download page, choose one
of the underlined links, and not the xx'd-under one in the diagram

    Download information
    The current version may be downloaded at:


    or through the courtesy link for the shim-latest.tgz

    Earlier versions may be found in the FTP attic.



More information about the ts-general mailing list