[ts-gen] Adding new symbols [Was: Syntax error due to unknown symbol]

Bill Pippin pippin at owlriver.net
Wed Aug 19 15:29:16 EDT 2009


About your question with respect to contract expiration, and the
implied issue of adding the new symbols for derivatives of new time
periods: although it's a challenge for us to collect up-to-date
symbols data for everyone from a source that allows for legal
redistribution, and it is a challenge as well to extend the shim
so that it can dynamically learn symbols on the fly, while still
providing type checking and journalling, still it's easy for
users to add a few symbols on their own.

I'll address your question about using invalid IB conid tuples
later, near the end of this post.  By the way, the reason I'm
explaining to you first how to add the symbols yourself rather
than my doing it for you is:

    1. IB tws account license restrictions prevent us from
       redistributing data obtained via the api, including 
       that obtained from contract info queries;

    2. our previous approach, of collecting symbols data from
       the network, is under redevelopment, and the symbols
       site we previously collected from does not at this time
       play nicely with our old scripts; and

    3. my existing approach of manual collection from the net
       [for FUT:ECBOT:YM: ] does not scale, as is typical for
       any essentially manual data input process.

Let me demonstrate how users can collect their own symbols data,
for your example of FUT:DTB:GBL:EUR: .

In brief, you'll need to collect contract info, add it to a symbols
load file, [for futures] update mod/LocalFut.sql to confirm the
expiry dates, unload to save your existing journal as needed,
repopulate the database using sql/create.sql, and update your
sql/syms.txt file using sql/bin/get_id.sql if you so choose.

A: collect contract info.

The following is a contract info query for FUT:DTB:GBL:EUR: 

    #!./shim -f
    select info FUT:DTB:GBL:EUR: all; wait 4;

    sql$ grep GBL ../log/ShimText > gbl.txt
    sql$ cd ../sym
    sym$ grep -l "'GBL'" A0Fut0*

B: add it to a symbols load file.

Within the sym directory, there are GBL symbols in both of the
futures load files.  Looking at the second with a text editor,
I see that in the past I was adding new symbols for GBL at the
end of the A0Fut01.sql file, so I might as well put the new ones
their as well.  Location is not critical since sql/more.sql sorts
the symbols in the process of moving them from the initial load
tables to their final destinations.

I'll use a text editor --- in this case, vi, with :set nowrap --- to
add data by hand from the gbl.txt file to the end of A0Fut01.sql,
following the entries I'd made earlier.  The existing entries look
in part more or less as follows:

(51728729, 'FUT', 'GBL', 'EUR', 'DE' , '20090306', 'FGBL MAR 09' ...
(53716236, 'FUT', 'GBL', 'EUR', 'DE' , '20090608', 'FGBL JUN 09' ...

Duplicating the lines, and editing the 1st: IB contract id, and 6th:
expiry, and, for good measure, 7th: local symbols name fields to
vary in the natural way with respect to the existing entries is
straight forward.  The new data comes from the contract information
saved earlier to the file gbl.txt.  Since A0Fut01.sql is a sql
insert statement, I also have to ensure that, at the far right
margin, only the very last line ends with a semilcolon, and that the
prior lines end with a comma. 

C: update mod/LocalFut.sql

> I use sql/mod/LocalFut.sql, set the appropriate expiry date ...

As you suggest above, here again some text editing of a sql script
is needed.  Again, the previous entries similar to the ones to be
added are at the end, so again, dup lines and edit, again with the
ibc and expiry values.  Since these are distinct update statements,
each one has its own semicolon.

D: repopulate the database

>           ... and source it.

Rather than modify the database, I'd like to suggest that you
clear and repopulate it.  Unload your journal first, since all
tables will be emptied, and then run the sql/create.sql script

    sql$ ../bin/mysql < create.sql

The above statement takes about 30 seconds to execute, and provides
no feedback beyond the eventual command prompt, so I typically source
the file while within the mysql interpreter, that is, I would run the
above command leaving off the redirection, and from the sql command
prompt, source the create.sql script.  Since this is my most common
reason to run mysql, up arrow is all I need here.  Also, the current
working directory matters for this process; the create.sql script
should be run from within the sql directory of the directory where
you unwrapped your tarball, so that mysql can see its subscripts.

The connection parameters in your ../bin/mysql script will determine
whether the testing or trading database is repopulated; the other
will be left alone.

Once all this is done, the current front month for FUT:DTB:GBL is
recognized; the following script, given the appropriate setting of
the .shimrc Locality to DE, would serve just fine to collect two
seconds of GBL tick data, except that my account lacks the proper

    #!./shim -f

    select tick  FUT:DTB:GBL:EUR:20091208 1;	wait 2;
    cancel tick  FUT:DTB:GBL:EUR:20091208;	wait 1;

>From the log, I see "354|Requested market data is not subscribed ... "
The shim read the contract value expressions just fine, however.


The above completes the demonstration, and now on to your question
about the ibc attribute, that is the IB contract identifier, what
they call the conid :

> ... ibc stands for IB Contract id or something like this, right? 


> ... I can send orders with probably wrong contract id, but updated
> expiry date.  
>                          ... This way I can keep a fixed ibc:xxxx
> identifier for the front month. This procedure worked fine so far,
> however I do not have any idea .. any side effects ... related to
> a missing resolution of expressions like FUT:DTB:GBL:EUR:20090908. 

You perhaps can for now, but need not --- see the above --- and, by
doing so, rule out compatability with newer versions of the IB tws
api protocol. 

I'm not sure whether your approach of using an invalid ibc mapping
works at this time, however I can assure you that with newer versions
of the IB tws api protocol there will be problems, e.g., symbol not
found messages, since newer levels of the protocol use the IB conid
as part of the contract specification.

If you use the contract expression value form in your commands, you
only need to update the expiry value, which can be stored in a single
variable near the top of your script, and filled in elsewhere via
string interpolation.  This is, however, up to you; one nice thing
about an interpreter interface is that downstream developers are free
to code as they wish.  By the same token, if it breaks, you get to
keep both pieces.

Good luck, and feel free to ask more if you have problems adding
new symbols.



More information about the ts-general mailing list