[ts-general] Problem 410 - History Data

R P Herrold herrold at owlriver.com
Thu May 9 11:23:17 EDT 2013

On Wed, 8 May 2013, David wrote:

> It appears to be a limit imposed by the Shim, but I belive I'm
> following the guidelines of IB's history requirements.

There are a complex set of rules on number of requests in 
flight in a sliding time window, as well as over all.  The 
shim logic in this regard was always tempermental, and 
manually tuned as to what it sought to constrain a client from 
doing, to avoid running into IB upstream limits

> I get when I'm doing history requests in risk mode, but I pace the
> commands, only 59 commands in 610 seconds max, which more than covers
> the 60 in 600 seconds, or 1 per 10 seconds average.

I think that is optimistic, and that simple 'metering requests 
out' at a fixed rate, will eventually hit a stall, and 
over-run the limits.  This is an 'open loop' approach -- 
firing a request without the feedback of knowing that a 
previous one has been completed

> Has anybody found a way around it if they have experienced it?

Yes.  This is mentioned in some posts from me, but not 
stressed perhaps

The more correct solution is to 'close the feedback loop' 
outside in the client driving the shim -- that is, start a 
separate program thread to load a queue on one end, and to 
drain it from the other.  Initially load it up to perhaps 75 
pct of capacity.  Thereafter, watch the returns, and only send 
another request from the head of the queue, when a previously 
sent request is completed.  Add monitoring logic to identify 
'stalled' conditions and emit debugging notices

MQ's and such are used upstream for a reason.  The TWS from IB 
is conversing with such, with the elements dialog described in 
the state machine and dictionaries of the two Java files that 
the shim is designed to talk with: EReader.java and 
EClientSocket.java, which are literally define and state the 
shim's complete marching orders as to what it can say, and 
what it expects to hear from the TWS

The shim's rate limit code is based on observation of IB 
behaviours that they can and do change over time at their side 
of the transaction, and without a mechanism to give or receive 
notice of which, beyond the message you quote.  For example, 
one can 'buy' additional market data lines, etc.  There are 
reasons that this is a understandable set of choices for IB to 
have made, that are out of scope here

> -I tried disabling the insert of history data into 
> HistoryBar by modifying post.c, (by adding 'return data;' 
> after 'data.clear();' in one::Poster::insert) but that did 
> not make a difference.

ick -- Solving the problem in the code will not be the right 
place.  Solving it in how you feed it (eyes closed 'open 
loop' vs eyes open 'closed feed back loop') will be more 
likely to succeed, and is durable

-- Russ herrold

More information about the ts-general mailing list