[ts-gen] Quality of market data via api [Was: Keeping TWS/shim up ...]

Bill Pippin pippin at owlriver.net
Tue Aug 18 12:42:00 EDT 2009


Ken,

You describe heroic efforts to squeeze every last possible bar from
the IB upstream:

> I think this kind of state-based code should work ...

For selected values of the verb "work".

> ... is extremely brittle and prone to breakage ... difficult to
> debug when the server changes.

Indeed.

You've just begun your adventure of exploring the quality of soft
or nearly realtime market data from IB.  It's nothing like the
cleaned-up time-delayed stuff that is freely available elsewhere.
Aiming for perfection here is not useful, and you should try instead
for simple and robust collection mechanisms.  And yes, you'll almost
certainly be seeing erroneous data, as well.

You will need to either learn to live with such glitches and outages,
or else get a supplementary data provider.  As for a related and more
specific point, your request for "better, simple solutions" for
realtime bar outages, I'm not sure if the following is what you're
looking for, but again, there is no perfect answer here.

The rest of this message suggests that a robust downstream client
should compute local history bars from the market data stream, if
only as a backup to, say, the so-called realtime bars.  Now on to
the details of my argument:

With respect to variations in market price data quality as provided
by IB, please keep in mind that IB is not in the business of providing
market data, as indicated by the limits they set on how much you can
get via your account.  There have been *many* discussions of alternate
data vendors on the Yahoo list, and you might like to see the archives
there.  Otherwise, once given that you're limited to the data you can
get via your IB account and the IB tws api, then consider the following
hypotheses:

    * the only truly [soft] realtime data requests are the original
      market data [select tick], and market depth [select book]
      requests;

    * all other ways to collect market data via the api --- history,
      "realtime" bars, and the scanner [not yet supported by the
      shim] --- involve more processing by IB's upstream servers;

    * the nearly realtime requests --- market data and market
      depth --- provide the greatest volume of price data, and the
      other data requests condense or project market data in some
      way, so that they allow IB to trade off network bandwidth for
      server processing;

    * the notion of the so-called realtime bars as being "realtime"
      when they capture data over five seconds is a misnomer, and they
      actually serve primarily as an alternative to history requests,
      one that saves trouble for IB and by which they encourage users
      to make fewer history requests;

    * all price data other than that derived from the market data and
      market depth requests is in some sense inferior; less timely,
      more subject to "correction", vulnerable to history farm and other
      processing outages, likely of lower service priority; and

    * given the tremendous load market depth places on IB's servers,
      and their known practice of using synthetic values for market
      depth with the paper account, even market depth is probably
      less reliable than market data.

Conclusion: whatever variation on data requests you might choose or
prefer, if you're concerned about reliability, you should also be
collecting market data via the "select tick" cmd, and synthesizing
a backup value for your preferred request/message form if at all
feasible. 

Granted, even the price ticks from the market data request are
condensed over a 300 millisecond interval, in part so that the IB
price feed to you can keep up with heavy volumes during chaotic
market conditions; and again, if you don't like this, again,
consider getting an alternative/supplementary market data provider.

> I was able to continue receiving ticks ...  The gap, however is
> kind of large:
 
> AUDJPY|2009-08-18 03:54:55|77.960|77.960|77.960|77.960
> AUDJPY|2009-08-18 03:55:00|77.960|77.960|77.960|77.960
> AUDJPY|2009-08-18 04:05:25|78.040|78.050|78.040|78.040
> AUDJPY|2009-08-18 04:05:30|78.040|78.040|78.040|78.040

True.  So, in any case, don't depend entirely on real time bars;
you're buying yourself trouble.  There have been many comments to
this effect on the Yahoo list.

Thanks,

Bill



More information about the ts-general mailing list