[ts-general] Problem 410 - History Data

David tshim at allone.org
Thu May 9 14:51:30 EDT 2013


Russ-

Thanks very much for your reply! See below.

At 11:23 AM 5/9/2013, you wrote:
>On Wed, 8 May 2013, David wrote:
>
> > It appears to be a limit imposed by the Shim, but I belive I'm
> > following the guidelines of IB's history requirements.
>
>There are a complex set of rules on number of requests in
>flight in a sliding time window, as well as over all.  The
>shim logic in this regard was always tempermental, and
>manually tuned as to what it sought to constrain a client from
>doing, to avoid running into IB upstream limits
>
> > I get when I'm doing history requests in risk mode, but I pace the
> > commands, only 59 commands in 610 seconds max, which more than covers
> > the 60 in 600 seconds, or 1 per 10 seconds average.
>
>I think that is optimistic, and that simple 'metering requests
>out' at a fixed rate, will eventually hit a stall, and
>over-run the limits.  This is an 'open loop' approach --
>firing a request without the feedback of knowing that a
>previous one has been completed
>
> > Has anybody found a way around it if they have experienced it?
>
>Yes.  This is mentioned in some posts from me, but not
>stressed perhaps
>
>The more correct solution is to 'close the feedback loop'
>outside in the client driving the shim -- that is, start a
>separate program thread to load a queue on one end, and to
>drain it from the other.  Initially load it up to perhaps 75
>pct of capacity.  Thereafter, watch the returns, and only send
>another request from the head of the queue, when a previously
>sent request is completed.  Add monitoring logic to identify
>'stalled' conditions and emit debugging notices

I could see happening in some cases, and it does make sense to code 
for that situation. Very good point, and I will add that in the future.

However, in my particular case, when I look at ShimText, I could see 
the history request from Shim to IB go out, and the data from IB come 
back typically within 1-2 seconds, so I didn't hit a stalled reply:
1) In the previous history request data reply from IB, IB came back 
with the data at time 81681.
2) The next request sent by Shim for more history came at time 81693, 
12 seconds later. So there were no outstanding requests.

When I track down "cmd dropped, would exceed capacity limits"
1) sets.c shows it to be 410:
new(perm) T(w,p, self, 410, "cmd dropped, would exceed capacity 
limits"   , *x),

2) data.c: drop_cmd_capacity_limits
drop_cmd_capacity_limits(dynamic_cast<T_c>(*X__I[Index(nat(410))])),

3) else.c:
Sig(capacity) { drop_cmd(e, t, fail.drop_cmd_capacity_limits); }
which is called by:
Sig(drop)                     { if (v.tick) capacity(e, *v.tick, e.fail); }

A simple test of replacing Sig(drop) with NOOP code shows that the 
Sig(drop) is called when the shim decides it needs to drop the command.

But I'm currently lost from there. What code could call Sig(drop)?

Is there a way to change the capacity limits of the shim to see if by 
chance the shim is getting to restrictive? In other words, where is 
the code and variables that goes into determining the "capacity 
limit"? What does "capacity limit" mean by the shim?

I'm sending one history request every 12 seconds, and I'm getting 
each reply (history data) from IB within about seconds, so I'm not 
sure why the shim would think I've hit the "capacity limit".

Any help with this would be warmly welcomed.


>MQ's and such are used upstream for a reason.  The TWS from IB
>is conversing with such, with the elements dialog described in
>the state machine and dictionaries of the two Java files that
>the shim is designed to talk with: EReader.java and
>EClientSocket.java, which are literally define and state the
>shim's complete marching orders as to what it can say, and
>what it expects to hear from the TWS
>
>The shim's rate limit code is based on observation of IB
>behaviours that they can and do change over time at their side
>of the transaction, and without a mechanism to give or receive
>notice of which, beyond the message you quote.  For example,
>one can 'buy' additional market data lines, etc.

Yes, and unrelated to my post, but maybe worth mentioning here, if 
you try to request more than 60 active real time bars you need to 
change "RealBar" from 60 to 100 (or however many you may want active) 
in constant.h

>  There are
>reasons that this is a understandable set of choices for IB to
>have made, that are out of scope here
>
> > -I tried disabling the insert of history data into
> > HistoryBar by modifying post.c, (by adding 'return data;'
> > after 'data.clear();' in one::Poster::insert) but that did
> > not make a difference.
>
>ick -- Solving the problem in the code will not be the right
>place.  Solving it in how you feed it (eyes closed 'open
>loop' vs eyes open 'closed feed back loop') will be more
>likely to succeed, and is durable
>
>-- Russ herrold
>_______________________________________________
>ts-general mailing list
>ts-general at lists.trading-shim.org
>http://www.trading-shim.org/mailman/listinfo/ts-general

Have a GREAT day!



More information about the ts-general mailing list