[ts-gen] Re: The Shim as a data recorder?
R P Herrold
herrold at owlriver.com
Tue Sep 26 17:16:32 EDT 2006
On Mon, 25 Sep 2006, (an anonymous user) wrote:
> I'm currently involved in a (very much amateur) automated
> trading project, and am looking for an ideal way to build an
> ultra-fine-grained database of historical data--basically, I
> want every detail of every order (on the exchanges I
> monitor) stored in a database the moment TWS reports them.
Some issues off the top: TWS, and so near as we can find, NO
source short of an extremely expensive data feed with SIAC,
can provide in real time the 'unified' 'broad tape' of "very
detail of every order" [we assume here you mean 'trade' rather
than 'order' as orders are held confidential until they result
in a trade, in some cases]
TWS updates its 'Market Data' information at a rate not more
frequently than 300 mSec intervals, when there is detail to
report. That is, there is a peak data rate load of about 3
updates a second; also updates may contain both a trade price
and volume information, or just a trade price, or just volume
of a trade, or just Bid/Ask spread data, or when no trades are
occurring the bid, and the ask prices, so one needs to keep
track of the last observed values for each, and push out stale
information of various types.
A trap for the unwary -- one cannot simply start a 'market
data' collector before the day starts, and stop it after the
day ends, and 'catch' all trades; The CBOT make their
complete trade detail data available for free download as a
matter of historical record, and we are told that people
trying to 'reconcile' IB market data to it cannot get it to
Also, it is not at all clear in light of 'off floor' trades
reporting (often delayed up to 15 minutes), error correction,
and such, we doubt that anyone can see _all_ trades, ever,
anywhere, with sufficient detail to fully catch 'everything'.
Perhaps the Exchanges can, perhaps the SEC can, but I do not
think there is a realtime feed available outside of those
places to get 'everything'.
> I don't have a ton of programming experience (mostly
> scripting languages) so I was looking at the third-party
> Python and Ruby APIs, but then I noticed that the
> description of the shim seems to hint that it might be
> capable of doing precisely what I'm interested in.
> However, after reading most of the documentation (geared
> mainly towards experts, it seems) I'm still not entirely
> sure of its focus and capabilities.
The trading-shim is designed so that a non-programmer person
1. do a database insert with the desired Underlyings,
specified with sufficient particularity as to remove all doubt
as to what is being sought, send a 'signal' to the shim, and
a. an ongoing 'market data' (viz.,
tickstream) dataset, visible through the syslog,
b. recurring 'history farm' data (viz.,
OHLC, Vol, and VWAP), which is 'close' to the present, and with
c. full historical data to the limits of the
data farms maintained by IB.
This works for Stocks, Futures, and Indices just fine; support
for Currencies, and Options is not yet fully complete. It is
possible to ask for 'too much' history data 'too fast', and to
trigger IB's protective mechanisms to prevent denial of
service attacks, or undue load on the history farms.
We have disabled order management in the released versions
to date as not in our initial release path. We did a proof of
concept some time back on this and there is some remnant code
lingering, but it is probably not in a working state at the
> So, just to clarify, is it possible to instruct the shim to
> monitor and record (in real time--this is important) all
> orders of select securities in a MySQL database?
With the prior qualification as to 'all' and redefinition as
The latency introduced by the shim once it is presented the
data from the TWS, is under a second, and probably much less.
We have no way to know what latencies exist between IB and the
Exchanges, and assume it varies but is reasonable short -- on
the order of under a second or two of a report of trade being
filed, based on study of the way that market data is
distributed in the US securities markets; lags or dropouts of
up to 5 seconds seem to occur getting data from the Sydney
exchange to the US.
> Would this require much mucking around in C++, or could I
> simply use the command prompt?
No C++ mucking about other than compiling a binary suitable
for your system, exists; with a proper environment (ie, all
headers present, and so forth), it is as simple as
untarring it, and typing:
There are also scripts to set up a basic database of
Underlyings, and Exchanges, and so forth in the ./bin/
directory of the tarball, again assuming you have a MySQL
database up and running, and ACL issues solved.
Personally, I have a set of scripts which take the tarball,
and patch it as to the database dataset and credential details
to use (I run an environment separate from another here at the
office, and we wanted freedom to each use our own database
instance without 'stepping on' the other.)
> Also, am I to take this statement: "The command interface
> also serves as an alternative to the existing tws gui
> interfaces, to permit you to use downstream programs,
> whether gui or not, to drive the tws through its api." to
> mean that I can use pipes to pass text commands to the shim?
yes -- indeed, I do just that
> I apologize for the simplistic questions, but given my
> overall skill level it would probably take me several hours
> (minimum) to install and thoroughly examine/test the shim.
smile -- We anticipate a release with much more robust history
parseing momentarily; the market data portion is fine and I
have used it for perhaps the last year.
As noted in the release announcement archive, I am running the
shim against tws version 857; there is some datastream change
later which we have not needed to solve yet.
The mailing lists are available at:
with the customary archiving features at:
for the general list.
-- Russ Herrold
More information about the ts-general