Current time: 18:28   
Descriptive files from the source distribution
COPYINGINSTALLNEWSREADMEROADMAP ]
NEWS
copyright (c) 2005--2009 Trading-shim.com, LLC  Columbus, OH
GPL version 3 or later, see COPYING for details

date: Fri Nov 26 2010 (db: 1.89; ts: 1.00; data-100614)
date: Wed Nov 24 2010 (db: 1.89; ts: 0.99; data-100614)
date: Fri Nov 19 2010 (db: 1.88; ts: 0.96; data-100614)
date: Thu Nov 18 2010 (db: 1.88; ts: 0.96; data-100614)
date: Thu Nov 11 2010 (db: 1.87; ts: 0.96; data-100614)
date: Thu Nov  4 2010 (db: 1.86; ts: 0.96; data-100614)
date: Wed Nov  3 2010 (db: 1.86; ts: 0.96; data-100614)
date: Fri Sep  3 2010 (db: 1.86; ts: 0.96; data-100614)
date: Thu Sep  2 2010 (db: 1.86; ts: 0.96; data-100614)
date: Tue Aug 31 2010 (db: 1.86; ts: 0.96; data-100614)
date: Thu Aug 26 2010 (db: 1.86; ts: 0.96; data-100614)
date: Thu Aug 12 2010 (db: 1.86; ts: 0.96; data-100614)
date: Fri Jul 29 2010 (db: 1.85; ts: 0.95; data-100614)
date: Fri Jul 16 2010 (db: 1.85; ts: 0.95; data-100614)
date: Fri Jul  9 2010 (db: 1.85; ts: 0.95; data-100614)
date: Wed Jul  7 2010 (db: 1.85; ts: 0.95; data-100614)
date: Wed Jun 30 2010 (db: 1.84; ts: 0.95; data-100614)
date: Wed Jun 23 2010 (db: 1.84; ts: 0.95; data-100614)
date: Mon Jun 14 2010 (db: 1.84; ts: 0.95; data-100614)
date: Thu Jun 10 2010 (db: 1.84; ts: 0.95; data-100614)
date: Wed Jun  9 2010 (db: 1.83; ts: 0.95; data-100331)
date: Fri May 28 2010 (db: 1.83; ts: 0.95; data-100331)
date: Mon May 10 2010 (db: 1.82; ts: 0.95; data-100331)
date: Fri May  7 2010 (db: 1.81; ts: 0.95; data-100331)
date: Fri Apr 30 2010 (db: 1.81; ts: 0.95; data-100331)
date: Thu Apr 29 2010 (db: 1.81; ts: 0.95; data-100331)
date: Tue Apr 20 2010 (db: 1.81; ts: 0.95; data-100331)
date: Mon Apr 19 2010 (db: 1.81; ts: 0.95; data-100331)
date: Fri Apr 16 2010 (db: 1.81; ts: 0.95; data-100331)
date: Thu Apr 15 2010 (db: 1.81; ts: 0.95; data-100331)
date: Mon Apr  5 2010 (db: 1.81; ts: 0.95; data-100331)
date: Thu Apr  1 2010 (db: 1.81; ts: 0.95; data-100331)
date: Tue Mar 30 2010 (db: 1.80; ts: 0.95; data-100226)
date: Fri Mar 19 2010 (db: 1.80; ts: 0.95; data-100226)
date: Thu Mar 18 2010 (db: 1.80; ts: 0.95; data-100226)
date: Wed Mar 17 2010 (db: 1.80; ts: 0.95; data-100226)
date: Fri Mar 12 2010 (db: 1.80; ts: 0.95; data-100226)
date: Mon Mar  1 2010 (db: 1.79; ts: 0.95; data-100226)
date: Fri Feb 26 2010 (db: 1.79; ts: 0.95; data-100226)
date: Fri Feb 19 2010 (db: 1.79; ts: 0.95; data-090415)
date: Mon Feb 15 2010 (db: 1.79; ts: 0.95; data-090415)
date: Thu Feb  4 2010 (db: 1.79; ts: 0.95; data-090415)
date: Thu Jan 14 2010 (db: 1.79; ts: 0.95; data-090415)
date: Fri Dec  4 2009 (db: 1.78; ts: 0.95; data-090415)
date: Mon Nov 23 2009 (db: 1.78; ts: 0.95; data-090415)
date: Fri Nov 20 2009 (db: 1.78; ts: 0.95; data-090415)
date: Thu Nov 19 2009 (db: 1.78; ts: 0.95; data-090415)
date: Wed Nov 18 2009 (db: 1.78; ts: 0.95; data-090415)
date: Fri Nov 13 2009 (db: 1.78; ts: 0.95; data-090415)
date: Thu Nov  5 2009 (db: 1.78; ts: 0.95; data-090415)
date: Mon Nov  2 2009 (db: 1.77; ts: 0.95; data-090415)
date: Fri Oct 30 2009 (db: 1.77; ts: 0.95; data-090415)
date: Fri Oct 16 2009 (db: 1.77; ts: 0.95; data-090415)
date: Tue Oct  6 2009 (db: 1.77; ts: 0.95; data-090415)
date: Mon Oct  5 2009 (db: 1.77; ts: 0.95; data-090415)
date: Thu Oct  1 2009 (db: 1.77; ts: 0.94; data-090415)
date: Wed Sep 30 2009 (db: 1.77; ts: 0.94; data-090415)
date: Mon Sep 21 2009 (db: 1.76; ts: 0.94; data-090415)
date: Fri Sep 18 2009 (db: 1.76; ts: 0.94; data-090415)
date: Tue Sep 15 2009 (db: 1.76; ts: 0.94; data-090415)
date: Fri Sep 11 2009 (db: 1.76; ts: 0.94; data-090415)
date: Fri Sep  4 2009 (db: 1.76; ts: 0.94; data-090415)
date: Thu Sep  3 2009 (db: 1.76; ts: 0.94; data-090415)
date: Tue Sep  1 2009 (db: 1.76; ts: 0.94; data-090415)
date: Wed Aug 26 2009 (db: 1.76; ts: 0.94; data-090415)
date: Tue Aug 25 2009 (db: 1.76; ts: 0.94; data-090415)
date: Mon Aug 24 2009 (db: 1.76; ts: 0.94; data-090415)
date: Fri Aug 21 2009 (db: 1.75; ts: 0.94; data-090415)
date: Tue Aug 11 2009 (db: 1.75; ts: 0.94; data-090415)
date: Fri Aug  7 2009 (db: 1.75; ts: 0.94; data-090415)
date: Tue Aug  4 2009 (db: 1.75; ts: 0.94; data-090415)
date: Mon Aug  3 2009 (db: 1.75; ts: 0.94; data-090415)
date: Fri Jul 31 2009 (db: 1.75; ts: 0.94; data-090415)
date: Thu Jul 23 2009 (db: 1.75; ts: 0.94; data-090415)
date: Wed Jul 22 2009 (db: 1.74; ts: 0.94; data-090415)
date: Fri Jul 17 2009 (db: 1.74; ts: 0.94; data-090415)
date: Tue Jul 14 2009 (db: 1.74; ts: 0.94; data-090415)
date: Mon Jul 13 2009 (db: 1.74; ts: 0.94; data-090415)
date: Fri Jul 10 2009 (db: 1.74; ts: 0.94; data-090415)
date: Mon Jul  6 2009 (db: 1.74; ts: 0.94; data-090415)
date: Thu Jul  2 2009 (db: 1.74; ts: 0.94; data-090415)
date: Wed Jul  1 2009 (db: 1.74; ts: 0.94; data-090415)
date: Fri Jun 26 2009 (db: 1.74; ts: 0.94; data-090415)
date: Tue Jun 23 2009 (db: 1.74; ts: 0.94; data-090415)
date: Fri Jun 19 2009 (db: 1.73; ts: 0.94; data-090415)
date: Tue Jun  9 2009 (db: 1.73; ts: 0.94; data-090415)
date: Fri May 29 2009 (db: 1.72; ts: 0.94; data-090415)
date: Thu May 28 2009 (db: 1.72; ts: 0.93; data-090415)
date: Fri May 22 2009 (db: 1.72; ts: 0.93; data-090415)
date: Tue Apr 28 2009 (db: 1.71; ts: 0.93; data-090415)
date: Fri Apr  3 2009 (db: 1.71; ts: 0.92; data-090320)
date: Fri Mar 27 2009 (db: 1.70; ts: 0.91; data-090320)
date: Fri Mar 20 2009 (db: 1.70; ts: 0.91; data-090320)
date: Fri Mar 13 2009 (db: 1.70; ts: 0.91; data-090210)
date: Tue Mar 10 2009 (db: 1.70; ts: 0.91; data-090210)
date: Fri Feb 27 2009 (db: 1.69; ts: 0.91; data-090210)
date: Fri Feb  6 2009 (db: 1.69; ts: 0.91)
date: Wed Jan 21 2009 (db: 1.69; ts: 0.91)
date: Tue Jan 20 2009
date: Mon Jan 12 2009
date: Fri Jan  9 2009
date: Fri Jan  2 2009
date: Wed Dec 24 2008
date: Mon Dec 22 2008
date: Fri Dec 19 2008
date: Wed Dec 17 2008
date: Tue Dec 16 2008
date: Mon Dec 15 2008

      * Announcing: version 1.0 of the shim!  Open bugs have
        been fixed, and additional features such as the following
        may be considered for, and may become part of, a post 1.0
        effort:

            + add man page, add autotools "make config" packaging,
              and add man page text to help command;

            + add contracts dynamically, as contract data query
              answers occur;

            + complete port of database to postgresql;

            + add newer, extended order attributes to cmd language;

            + add support for the market scanner;

            + add user process() per-message callback for C++
              developers;

            + add grammar rule driven configuration to the ruby
              sample client; and

            + port to Windows

        The order --- and for that matter, the existence --- of the
        above is not a commitment to deliver, or deliver in the
        order listed.  Readers on the list may want to consider how
        they can contribute to any of the above goals that they find
        of interest.  The shim is free software!

      * Fix bug in ErrorMsg ApiA lookup action, where type tag
        for tick id was conflated with type tag for api error
        index.  This prevents the syntax errors that were
        occurring for most api error messages, of the form:
            Syntax error:
            Grammar rule: Compound ErrorMsg
            Cursor state: match; text is:

        The trace text above was empty since the input itself
        parsed fine, and it was the trailing action that failed.

      * Relax quantity term type, to allow negative values in
        size tick messages, since the option open interest value
        for subtick type 27 can be negative; and generalize the
        TickData parent class data member to be integral, rather
        than natural.

      * Fix timestamp bug, by splitting all base and offset
        calculations for the two clocks until the final
        comparison in DualClock::check_time().  Monotonicity
        is maintained, at the cost of allowing the shim's
        clock to run ahead of the wall clock time in the
        worst case, where either the time stamp counter
        speeds up significantly over time, or the wall clock
        time is set back significantly. 

      * Update shim version to 0.99, reflecting, on the one
        hand: the simplifications to the database; use of
        exchange sets in contract lookup; grammar rules to
        document command language; grammar rules to define api
        request and message formats; and api request and message
        formats extended to provide multi-version api support,
        from api levels 17 to 49 --- all of which taken alone
        would justify a 1.0 release according to our roadmap ---
        and, on the other hand, the recent and still existing
        timestamp bug which needs to be fixed prior to 1.0. 

      * Most of the files in the lib directory have been 
        relicensed from GPLv3+ to LGPLv3+ .  Note that the
        trading-shim sources, and in particular the files in
        the src and sql directories, remain under GPLv3+
        licensing.

      * The order submission bug has been fixed; it was due to
        incorrect reordering of grammar action symbols for the
        submit case.

      * The layout of the sql directory has changed, to reflect
        initial work on postgresql support (no, the shim does not
        yet connect to a psql database).  There are create table
        statements in both sql/mys and sql/pgs, and users are
        welcome to compare the two directories to get some idea
        of upcoming changes to the database, and work left to
        be completed.

      * Remove obsolete help text, and modify help command to print
        the same grammar text as the help option provides.  The
        syntax of the help command is now simply:

            help;

        Note that there are differences in the grammar rules for the
        command language and those for the api request and command
        formats; broadly speaking, the first is for the benefit of
        users, while the latter two are for the use of developers
        and the shim itself.

      * 101119 is a test release only, to allow recent posters to the
        mailing list to try out some of the features of the newest
        code.  In the file time.c there are trace procedures for those
        looking at time stamps, and cmd listener error handling is
        improved.  That being said, the database is in a state of
        rapid change, with more to come, and there is a known problem
        with orders, with submits treated as modifies.  So, don't use
        this for production, but please do provide feedback to the
        list about timestamps.

      * There are many changes to the database.  In brief, the Symbol,
        Expire, and Expiry tables are gone, and product data is loaded
        more or less directly from the load files, to the source
        tables, to the eventual product targets of --- and the names
        have now changed, as well --- of UndCon, FutCon, and OptCon,
        where the previous product table names were Static, Future,
        and Option.  These changes are part of reducing the number
        of auto increment uid attributes in the database, and
        modifying the create table statements to bring them closer to
        ansi sql.

      * Fix cmd listener error recovery by restoring empty command (";")
        rule and modifying word count update logic.  For the releases
        over most of the last two weeks --- from 101109 and prior to
        today's tarball, 101119 --- syntax errors would leave the cmd
        listener stuck, and these changes fix this problem.

      * Exchange sets are now used in place of locality in valued-based
        contract lookup, making the results precise for the available
        product set.  Either no symbol found or ambiguous symbol
        expression warnings are now emitted as needed.

        The idea of locality is no longer used at all in the shim
        beyond the related notion of currency, which serves much
        the same purpose in contract lookup and has the advantage
        of being precise with respect to IB's api contract data.

        Both the Locality variable, and the underlying table
        from the database, have been removed from the shim.
        Attempts to set "Locality" at initialization, as if the
        configuration variable Locality still existed, will give
        a parse error.  The Locality table remains in the database,
        though only in a vestigal role.

      * Fix a serious performance bug whereby, due to a error
        in calculation units in vars.c, for the time_slot variable
        used in definition of the select() delay, the select()
        system calls were being made 1000 times as frequently as
        intended.  This bug stems from the modification of the
        timestamp object from microsecond to nanosecond units for
        the fractional part, the purpose of which was to enable
        their use in construction of unique object ids.  The bug
        was introduced with 101103, and is fixed with 101111.

      * Fix table consistency errors for market scanner that
        prevented startup for api levels 25--34.

      * Fix Portfolio primary exchange attribute choice that
        led to scanner errors at startup when that attribute was
        blank.

      * Begin elimination of the auto_increment attribute from
        database create table statements, as part of achieving
        ANSI sql syntax in preparation for a port to PostgreSql.
        The database version has increased, and more such version
        changes will follow.  Warning: the symbols load files
        may change as well, and there are knock-on effects as
        well for the blob file.

      * Recent releases of the IB tws on OSX hardware have been
        observed to take many seconds after startup to first accept
        connections.  Symptoms for users of the shim include 421
        warnings, as the handshake is retried, and 523 errors,
        for handshake failure.  Note that users are responsible
        in such problems for setting the PollTime configuration
        variable from its default of 3 seconds to whatever higher
        value is needed; and note also that this problem seems
        to disappear once the IB tws has fully come to its
        senses.

      * Deleted subscriptions are now cached, and the cache map
        used as an alternative source for contract -> tick id
        mappings, to avoid the 411 symbol_is_not_subscribed error
        messages that otherwise result when a user action with
        respect to a subscription follows an error-driven
        subscription deletion.

      * The time stamp fields in the log have been converted from
        two unsynchronized counters of seconds and microseconds, to
        a unified clock, still with secs and usec fields, where the
        usec field now carries to seconds as one would expect, and
        where the unified timestamps are strictly monotonic, for
        both 32bit and 64bit hardware.

        Note that although the previous implementation was intended
        to provide monotonicity, due to a print formatting bug this
        was lost every 70-odd minutes.  The new implementation is
        strictly monotonic over the course of a day, that is up
        until midnight, as originally intended.

        To achieve monotonicity and synchronized clock display,
        there is a necessary change to the semantics of log
        timestamps in the event of clock skew, which change is easier
        to explain once given a description of the DualClock object
        implementation:

        Internally there are two clocks, gtd and tsc, for the two
        sources of time information: calls of gettimeofday(), and
        reads from the processor time stamp counter, respectively.
        The two are complementary, since gettimeofday() provides
        long term accuracy with lower resolution at the immediate
        cost of a system call, while the processor read gives cheap,
        high precision values that are potentially less accurate
        over time.  The former occurs once each time slot, immediately
        after the select() call and any resulting read() IOs, and
        the latter much more frequently, including once for each
        object construction, to provide the timestamp identifier.

        Previously the gtd and tsc clocks were free running and
        asynchronous, with the tsc clock free to climb, at least for
        64bit hardware, effectively without limit.  Then as now,
        updates to the gtd clock were once each time slot, and to the
        tsc clock, much more often, as events were parsed.  Clock
        skew, if any, was for the most part ignored.

        Now the gtd and tsc clocks are synchronized at startup, and
        time slot updates to the gtd clock must not roll back prior
        to the most recent tsc reading, else time go back, which
        would lose monotonicity.
        
        Given two counters, and the three goals of timestamp
        monotonicity; convenient reads of the time stamp counter
        without the cost of frequent checks against gettimeofday();
        and fidelity of the gtd clock to gettimeofday() --- then, in
        the event of clock skew, something has to give, and here
        it's the last of the three goals above.

        So, at the beginning of each time slot, the gtd clock is
        updated with the maximum of the gettimeofday() check and
        the most recent tsc reading.

        It follows that if the tsc runs very fast, or the system
        clock, very slow, the log display timestamp values will be
        running fast relative to gettimeofday(), in order to
        maintain monotonicity.

        This log artifact is unlikely in practice.  The time stamp
        counter and system gtd clock start out closely synchronized,
        and even if there is some slight clock skew, the shim spends
        most of each time slot, after a relatively short time
        processing events, in then waiting for IO, during all of
        which time the system gettimeofday() clock is free to catch
        up with the last tsc reading for parse object construction.

        Nevertheless, in the event that the internal gtd clock is
        running fast due to system clock skew to such a degree that
        the wait time during each time slot is not enough to catch
        up, as when a sysadmin sets the system clock back by brute
        superuser force, the shim includes code to indicate such
        occurrence in the log:

        A new warning message, 445: clock_skewed_single_step,
        has been added, and, in the event that the clock single-steps
        ahead for a time slot, is logged, along with the gettimeofday
        value being ignored, in case clients want to know the
        variance.  Note that unlike other warnings, no display is
        made to the stderr, since the useful information consists
        of the juxtaposition in the log of the event timestamp with
        the event payload gtd information.

        So, to sum up, the log now has a synchronous, monotonic
        seconds-microseconds clock, and in the event it runs ahead of
        the system gettimeofday clock, it logs warnings accordingly,
        although this does not typically occur.

      * Convert server log level argument symbols from initial cap
        names to all lower case, to simplify the description of the
        command grammar.

      * Restore ClientId > 8 support for risk mode, which was
        temporarily disabled by the addition of ClientId = 0 support,
        so that all non-negative values outside the range 1-7 and
        less than 2^32 are available for risk mode.  Fix a parser
        error for open order downloads at startup for api levels at
        or above 38.

      * The shim now provides multi-version support for api levels 17
        through the current IB tws server api level of 49, so that
        message formats as parsed vary to accord with the output of
        the IB tws for that api level.  This provides access to a
        number of previously inaccessible message attributes, such as
        the commissions information included with the newer versions of
        the open order message.  This work brings the shim close to
        its 1.0 release.

        Feel free to set the ApiLevel configuration variable as you
        choose from 17 (more than five years old) on up, and keep in
        mind that message attribute indices will vary for a given
        message attribute with the api level, so that client apps
        that pick apart messages will need to be updated if you
        choose to use a higher api level than the previous shim
        default of 23.

        In addition, risk mode is no longer clamped to api level 23,
        and works throughout the range of 17 through the current 49.
        Individual request versions have also been updated to the
        newest event format in most cases, except for req market data
        which is at 8 rather than 9, req contract data which is at 3
        rather than 6, and place order, at 15 as opposed to 30.

        This completes planned api coverage except for request all
        open orders, which will be ready in the next few days; market
        scanner subscriptions, which are in progress and will be
        finished soon; end fundamental data (req fundamental works
        fine), which is of uncertain testability; and the large gap
        for options processing, which is currently quite limited, and
        will not be extended until work is complete on the next
        major phase, contract fill.  Note that fa features are not,
        absent customer interest, scheduled for implementation.

      * As part of multi-version support, unify the two message
        version tables to one, with as one consequence that
        the parsing now recognizes obsolete message versions
        and prints a warning, e.g.:

        Message  7 version: expected  5, received  7 for Portfolio
        Problem: 427 api msg version mismatch, api level too old

        Increasing the ApiLevel configuration variable to a
        sufficiently high number, closer to 49 than the current
        default of 23, will avoid such messages, as will setting
        the MsgSlack config variable to a sufficiently high value
        from its default setting of 0.

      * Eliminate two defunct subscription-related tables ---
        TickConfig and ScanFilter --- and increment the database
        version to 1.86.  Increment shim version as well, from
        0.95 to 0.96, to reflect improved api level multi-version
        support.  The default blob file has not been updated, so
        this is a known problem --- ask on the list if you're
        interested.  Those with a working database can always
        create new blob files, so this is only a problem for
        new users making a trial run.

      * As part of ongoing work on extending api level version
        coverage, add a generic tick type list argument to the
        market data command, enlarge the set of tick indices and
        labels, and extend the number of market tick message
        types.

        In brief: you may now add a generic tick type list,
        e.g.: [100, 101, 104, 106] , as the last argument of your
        market data commands, after the contract expression, and
        before the semicolon; or not, as you wish.

        The trailing dummy TickConfig uid has been eliminated from
        the market data request command, the related table
        has been dropped from the database, and the database
        version has incremented to 1.85.

        Note: there are probably still more tick types, tick
        indices, and tick message types to be discovered, and
        added to the shim.  Unfortunately the printed form of the
        api guide has fallen somewhat out of date.

      * Abstract out explicit named data member constructor ops
        as construct types, with the result that ctor.c goes away,
        and the role of the code there is played by types defined
        in constructs.h and args.h, and used in ctor.h

      * Modify scanner to elide whitespace, and delete explicit
        whitespace from command language.

      * The syntax of the tick (req market data) command has been
        simplified as of 2010.0722, via the deletion of the unused 
        TickConfig uid index, in the past typically a trailing 1
        in the example scripts; those scripts have been updated as
        well.  Note that the TickConfig table thus becomes vestigial,
        and that it will be deleted from the database the next time
        some other db change would trigger a version change.

      * Fix bug whereby SecPerHq, seconds per history query, config
        variable (internally, sec_per_q), was clamped to zero for
        data mode, due to confusion between default and null values.

      * Modify order command input to require an alphanumeric id
        for the order key and order mark (previously generic strings
        were accepted), and increment the database version to reflect
        the fact that old journal information may no longer be
        readable.  Other alphanumeric symbols are also checked,
        and this more-strict input checking may force other parse
        errors.  If such occur for you for other statement types,
        please be sure to inform the list; I'm not aware of any
        such cases.

        Note that the risk.rb script was modified to replace hyphens
        in the order key template with underscores.  Currently, a
        token is classified as an alphanumeric identifier if it looks
        like a traditional program variable, so that it matches the
        regex [a-zA-Z_][a-zA-Z0-9_]* .  Users who have order scripts
        that generate order keys and markers taking advantage of the
        less restrictive checking previously in place, so that they
        would include characters other than [a-zA-Z0-9_] , will have
        to modify those patterns accordingly.

        Users should prefer this release of 2010.07.07 to the
        previous one of 2010.06.30 :

        Restore error handling for the cmd listener, eliminating a
        bug whereby syntax errors could hang cmd input parsing.

        Fix a spurious fatal exception for first time startup
        against an empty database, whereby the AccountCode insert,
        although successful, was not correctly detected, leading
        to a one-time termination with the code:

            Problem: 546 ice: name-based database table lookup failed

        Fix risk mode sql warnings error display to restore the
        previous behavior whereby db queries are listed to the
        stderr only if a mysql warning actually occurred; the
        last release was spraying all db queries to the stderr,
        even though warnings had not occurred. 

      * Trim false dependencies from order journal message side
        table foreign key dependency lists, and reduce accordingly
        the sql traffic required to insert journal entries.

      * Validate shim for server-side api levels of 46, 47, and 49,
        and, not incidently, fix the 427 warnings for market data
        messages, such as the following:

        Message  2 version: expected  5, received  6 for    Quant
        Problem: 427 api msg version mismatch -- shim is outdated
        Message  1 version: expected  5, received  6 for    Price
        Problem: 427 api msg version mismatch -- shim is outdated

        that occurred with api level 49 IB tws servers.  Note: api
        level 48 is no longer supported by any working IB tws that
        I know of.  At least, the only two tws versions that might
        have supported it in the past, 905.1 and 905.2, from early
        May, are no longer supported.  The previous IB tws, 903.4,
        is at api level 47, and the successors, 905.3 and 905.4 are
        at api level 49.

        So, to the best of my knowledge, no IB tws server advertises
        server side api level 48, and although I don't yet know if
        one might accept a client side request for api level 48, any
        future shim support for such would be defined via interpolation
        of 47 and 49, since there is no working server for api level 48
        to learn from.

        More precisely, the following is a table of api level, tws
        version, and tws build date for the most recent IB tws
        downloads I have to work with --- and Russ checks for them on
        a daily basis:

            45  897.9  2009.09.14
            45  899.4  2009.10.26
            45  899.7  2009.11.04
            45  899.8  2009.11.09
            45  900.3  2009.12.04
            46  901.3  2010.01.04
            47  903.4  2010.03.12
            49  905.3  2010.05.27
            49  905.4  2010.06.01

      * Add OrderLag config variable to pace orders; variable is
        floating point, has units of seconds, ranges from 0.0 to 1.0,
        and defaults to 0.3, which is the value suggested by folklore.

        Note: you probably need to either accept the default, or set
        this variable to a non-trivial value, that is significantly
        greater than the request interval time value of 0.020 seconds,
        since 2102 errors have been repeatedly observed for back to
        back orders otherwise.  If you see, in the log, error messages
        from the IB tws of the form (nnn would be the order tick id):

            |3| 4| 2|  nnn|2102|Unable to modify this order as its \
                                still being processed.|

        then you need to accept a higher config value for the
        OrderLag variable.  Also, the shim computes this time delay
        from the last order, if any, so the first order in a run of
        related order commands can go out right away.

        In the past downstream applications were responsible for
        inserting such time delays themselves, and this new config
        variable just simplifies downstream app coding; the need for
        the time delay, which is due to the IB tws, not the shim, is
        unchanged.

      * Add new history bar sizes and increment database version.

      * Finish tuning history-related timed queue processing and the
        related configuration variables, and reimplement the past.rb
        script to process events, check for returned data, and
        terminate based on such results.  These changes do, however,
        break the test.rb script, which no longer terminates on its
        own; enter an exit when desired to make it quit.

      * Fix significant latency bug for history query answer
        processing, and add timers and maps to prevent history query
        pacing violations.  Delete PastFilter table, and increment
        database version.  Release related data-100528 symbols file,
        with new database version embedded in blob file, so that
        blob file will work out of the box with new shim.

        If you are using an older (Red Hat) g++, by way of example
        an obsolete version such as 4.1.2, try removing the -g flag
        in the Makefile, and the shim should compile.  That being
        said, you really need to use a better-supported g++ , so
        such use is not necessarily supported.  Red Hat does have
        a package with a newer compiler, for just such a problem, so
        please consider installing it.

      * Increment database version.  Included among a variety of
        changes is the addition of a 5th table for message side
        journalling, FailedOrder, which captures error messages
        occurring in response to a place order or cancel order request.

      * Restore history query drop logic for risk mode and the drop
        option, and copy drop actions to the log.  Note that the
        wait option mentioned on freshmeat last week is now the
        default, and so not needed; and that the drop option, its
        antonym, restores the old behavior of history query drops
        even in data mode.  The drop option is not needed with risk
        mode, since it is in effect in any case.  Note that for risk
        mode it makes sense to set the InFlight variable to a high
        value, e.g., as much as 48, to simplify history query pacing.

      * Modify the cancel history command to accept a tick id index
        in place of the contract expression previously used, to avoid
        abiguity when multiple queries for the same symbol are in
        flight.

      * Add various features to improve bulk history query processing,
        including the wait option in order to offload history query
        delay queueing from the client, and matching of history query
        requests with message answers in order to detect query cancels
        caused by pacing violations. 

      * Merge history agent class (occurs in Agents) and history rate
        limiter class (occurred in Governor), and as part of the merger,
        use the history agent tick map, which is also a priority queue,
        in place of the rate limiter tick queue, so simplifying the
        update of tick date stamps to reflect the arrival time of
        history messages.

      * Restore previous log format behavior whereby, one, commands
        are marked as comments, |4|11| 0|, when enqueued, rather
        than as command whitespace, |1| 0| 0|; and two, for requests,
        accurate echo to the log, consisting of src-tag-ver triple +
        wire format, that is tag-ver-..., where the ellipsis refers
        to the variable arguments of the request.  Note: recent
        releases did not echo accurately echo requests to the log
        due to a garbled buffer.

      * Fix midnight rollover bug in Timer class.

      * Problems with init-time account code processing have been
        fixed, and the configuration variable Account is now, when
        defined, checked against the upstream account data query
        answer.

        There was a known, non-fatal init-time bug where, if
        the "download open orders on connection" tws gui api
        config option was checked, and while querying for account
        data (in order to determine the account code), and since
        the account code was not bound to the Bindings singleton
        til after the query loop, order status records read in as
        part of that query loop could not be properly written to
        the journal, since such processing requires a valid account
        code key.

        And, as a possible, though not yet observed variant on
        this problem, it is also conceivable that the upstream tws
        gui could choose to download open orders before answering
        the account data query, and again order status messages
        would not be processed properly.

        In either case, open order messages could not be written
        to the journal, and so there would have been non-fatal,
        418 warnings to the stderr on startup, reflecting the
        failed inserts and triggered by a failed account code
        lookup parse action.

        The two changes that fix this problem are as follows:

            in the absence of a value for the Account variable,
            the shim binds the account data key to the Bindings
            singleton as soon as it is known, within the read
            loop, instead of waiting until after the loop; this
            solves the first instance of the account data ---
            order status dependency problem above; and

            since the shim will not write order status messages
            to the journal without a non-null account code, the
            Account variable, if defined via user configuration,
            is bound to the account code key even before the
            upstream is queried for the account code.

        This latter feature requires user action, by setting the
        Account variable as part of a config file or init option
        let statement, and if that value proves to be false, that
        is it conflicts with what the upstream later claims the
        account code key to be, the shim exits.

        Note that if the Account variable is set, journal inserts
        during the startup account key search are bracketed in a
        sql transaction, with those inserts committed once the
        search succeeds, and rolled back to remove the possibly
        incorrect account code values if the search fails.  In
        contrast, if the Account variable is left undefined, any
        journal inserts are committed as written, since then all
        data is derived from the upstream, which is trusted as
        authoritative.

        So, users may set the Account variable or not, as they
        choose; the shim will use it if it's available, and in any
        case will query the upstream for the account code key,
        checking it against the Account variable if that has been
        defined, or using it to (finally) define the session
        account code key; and, in case of account code search
        failure, journal inserts are rolled back, or left
        undisturbed, as the untrusted Account variable was set,
        and might have contaminated the journal, or not.
   
      * Fix bug in hash function for null string ("") case; this is the
        actual fix for gcc 4.3.2, and other changes up until this point,
        even if involving beneficial refactorings, and although possibly
        masking the problem by accident, did not fix it.

        This bug shows up with gcc 4.3.2, and 32 bit, but not for 64bit
        gcc 4.4.3, which is what I use for internal development.

        There is now a common code path for all block-doubled containers,
        so that they all use the existing, tested code for the standard
        handles.  There are other significant internal changes to the
        hash map class, though none are known to actually fix a
        demonstrated bug as with the hash code thinko mentioned above.

        Unusually large collisions sets spill to a splay tree --- tests
        show that the max extent size of eight cells for 64 bit hardware
        rules out all but one such case --- and extents as well as the
        hash array are allocated by a container-specific allocator from
        the block handle.

        Note that for 32 bit hardware, where MaxExtent decreases to
        four cells, there is a non-trivial number of spills, e.g., in
        the high single digit percentages, to the splay tree.

      * Remove AccountCode from set of tables used to build blob file,
        since the data is not portable, and modify late stage account
        init processing to succeed without that data in data mode when
        the blob option is defined, with the result that the blob
        file shipped with the new data tarball, data-100331.tgz, can
        be used as is, without patching, so that you can try out the
        shim without creating a database or patching the blob file,
        as originally intended for use with the blob option.  This
        fixes a problem with older blob files, where the shim expected
        and checked for correct AccountCode data from the blob file,
        so that you had to either patch the blob file, or create your
        own with the dump option, which latter step required you to
        create the database.

        Complete denormalization of account code tuples in the journal;
        both cmd and msg side tables use the account key string rather
        than the AccountCode uid index as the value of the acc attribute.
        As part of these changes, the database version has incremented.

        Note that the shim now complains and stops during initialization
        if the account configuration variable is defined and yet does not
        match the value of the account code from the upstream; if you
        don't want such checking, don't define the Account config variable.

      * Break multiple inheritance diamond and eliminate virtual
        public base from Compound and Argument classes of type tag
        inheritance graph via: elimination of inheritance of Base
        class from Argument class; direct inheritance by Constant
        class from Base class; and type conversion operator and
        member function forwarding for Argument class.

        This refactoring change may be related somehow to a
        init-time crash observed on debian 32 bit systems; in
        any case, it fixes the problem for the one reproducer
        system available locally, which has a relatively old g++ .
        Two older, observed-to-fail, and the new, development-proven
        g++ versions are, respectively:

            g++ (Debian 4.3.2-1.1) 4.3.2
            g++ (Debian 4.3.4-6) 4.3.4
            g++ (Debian 4.4.2-9) 4.4.3 20100108 (prerelease)

       Debian 5.0 (lenny) ships with gcc 4.3.2; I'm running debian
       testing.  By the way, a release history for recent versions
       of gcc is:

            gcc 4.3.2 aug 27, 2008
            gcc 4.3.3 jan 24, 2009
            gcc 4.3.4 aug  4, 2009
            gcc 4.4.0 apr 21, 2009
            gcc 4.4.1 jul 22, 2009
            gcc 4.4.2 oct 15, 2009
            gcc 4.4.3 jan 21, 2010
        
      * Add new option, "past", for data mode only, to alter the
        Feeder's treatment of history queries; instead of dropping
        premature queries, stall the feed queue until the history
        query time period has passed.  This makes the shim more
        convenient as a history harvester, at the cost of high
        latency for those requests unlucky enough to follow one or
        more history queries.  So, use this option only for
        dedicated history collection, in a separate data mode
        session.

      * In the table definition for OrderStatus:
            add two order status enum constants:
                ApiPending
                ApiCancelled
        and increment the database version to 1.80 to reflect this
        change.

      * Add the new cmd line option "blob" --- for use in data mode
        only --- which loads symbols and other database information
        from a previously created blob file that has been placed in
        the sym directory.  Note that the blob option is ignored in
        risk mode, since an actual sql database must be available for
        journalling.

        Note also that, although blob files are created via the dump
        option in the log directory, the user is responsible for
        copying such files to the sym directory in order to make
        them accessible to the shim as blob input.

        Once given a previously created, suitably complete blob file,
        the blob option allows the user to use the shim for market
        data collection without an accompanying mysql database.

        If you download, in addition to the shim sources, the new
        data symbols tarball data-100226.tgz, you will find it
        includes a ShimBlob file created via the dump option from
        a regular shim session reflecting a database loaded with the
        default symbols load files.  [Note: the mar 1 release fixes
        the symbols directory "sym" symbolic link in order to make
        the blob file visible.]

        I.e., as long as you only need to use the shim in data mode,
        and if the default symbol set suits your purposes, you can
        now avoid creating the mysql trading-shim database at all.

        Note finally that shim startup via the blob option is faster,
        by several seconds, than ordinary startup with symbols loaded
        from the mysql database.
  
      * Add the new cmd line option "dump", which lists the database
        symbols to a blob file in the log directory at startup.

      * Add the new cmd line option "init", add a "let" statement
        to the command language, and reorder initialization deps so
        that the shim, once given the init option, reads an initial
        let statement after the (now optional) config file input and
        before using the resulting config variable values.

        The init option subsumes the functionality of the older link
        option, and that option, and so the "dbms" and "feed" commands
        that went with it, have all been eliminated.

        Note: the syntax of the config file has changed to correspond
        to that of the init option input-text prefix section, and the
        older config file syntax will no longer work.  This is a
        feature; the config file now may include, in addition to the
        let statement, free whitespace and comments.
        
        An example of a let statement might be: 

            let DbmsHost = xps400,	# you may use comments 
                ApiLevel = 23;		# and whitespace freely

        The older config file syntax for the same information would
        have been:

        DbmsHost xps400
        ApiLevel 23

        There is much more information in the shimrc file, found within
        the top-level directory where the tarball is unpacked.

        Note also that tarball setup practice has changed slightly with
        respect to the name of that file: it used to be delivered with
        the name " .shimrc ", which name is still required before the
        shim will notice and read it at startup.  So, whereas the
        .shimrc used to be available by default under that name once a
        tarball had been unpacked, you'll now need to copy or rename the
         shimrc file to include the leading dot.

      * Increment database version, due to attribute name change of
        Locality table ISO country code from "code" to "icc", to
        solve collisions with other attributes also named "code"
        for select statement generation during shim initialization.

      * Add additional error reporting for failure during calls to the
        PageMultiple class constructor, in response to a mailing list
        report.

      * Remove container block copy virtualization due to failure of
        older versions of g++ to successfully compile the virtualized
        class templates.  This issue may in the future be resolved by
        use of autotools tests to choose sources, with the result that
        users with obsolete versions of g++ would then not enjoy all
        the newest features.

      * Refactor Buffer and derived classes to use exactly one char
        vector, and to dynamically expand that vector as needed,
        so that the large buffers needed for fundamentals data will
        be allocated only when needed, with a consequent reduction in
        memory footprint compared to earlier versions of this month.
        This provides the block-doubling features of vectors for
        buffers as well; e.g., during input, the InpBuf class starts
        with a small buffer size and yet dynamically discovers the
        buffer size used by read(2) or fgets(2).

        Also modify the scanner Token class (via derivation, using
        a vector token class), so that individual tokens may be of
        arbitrary length.

      * Add internal cons pool to memory allocator, and fix recently
        introduced/now removed obj stack bugs that hogged memory; be
        sure to use the 091204 release in place of the 091030 version.

      * Modify log format for floating point attributes to take
        precision from input data, with the two results that:
            + no display precision from message input is ever lost; and
            + log format columns do not line up as well as before.
        This change is in response to multiple user requests. 

        Modifying the log format procedures to adaptively discover
        a "good" constant column width across the span of possible
        security symbols, and so recover column alignment, while
        still displaying all input precision, has not yet been done,
        and will take a while.  Since column width determination
        will probably use the min tick information from contract
        data queries, implementation of this feature may well
        occur during work on contract/symbols fill, the major 
        milestone task that follows both the current task, memory,
        and the next, versioning.  Please comment on the mailing
        list if this is a problem.

      * Add the BbgBsym relation to capture BSYM sextuples, while
        eliding redundant subscription string information.  The table
        is currently empty --- distributing load files for 8 million
        records would be inconvenient --- and in any case, the info
        is not yet useable by the shim, since it has not yet been
        cross-indexed against IB symbols data.  See the google
        bbg-symbols project, http://code.google.com/p/bbg-symbols/ ,
        to track ongoing progress for the data analysis, and
        http://bsym.bloomberg.com/sym/ to get original copies of the
        BSYM data files.

        Add to the database the market sector designator finite
        domain used by bbg as a hierarchical category value for
        the BSYM symbology, add the BSYM security type domain, and
        define insert statements for the SectDes and BbgType
        relations.  Increase the database version to reflect the
        change.

      * Add support, given an account subscription for Reuters
        fundamentals data, for the related command, request and
        message events.  There is an example of the command syntax
        in the script exs/data.

        The ApiLevel configuration variable must be set to 41 or
        higher, and most other api features, including market data,
        history, and orders, become inaccessible; you'll need to
        run another session with the ApiLevel still at 23 in order
        to use these other api features.

      * Continue to make bug fixes and efficiency improvements for
        memory allocation.

      * Add ruby sample client directory rsc to distribution, with
        contents

      * Fix execution reports request format problem by incrementing
        the event version from 2 to 3, so that the select exec cmd
        again works; it had silently stopped working.  Thanks to Ken
        for finding this bug.

        Note in the following explanation that the request execution
        report format has been constant since api level 9, so that
        the actual format has been effectively constant over time;
        that the date-time attribute of the filter is actually a
        time value, even though comments in the sample client
        sources claim that a yyyymmdd-hh:mm:ss string is required;
        and that the only symptom for a variety of invalid inputs
        is no return exec reports, rather than some more specific
        error message.  Note also that event versions are per event,
        are often but not always small integers, and differ radically
        from, but should be correlated with, the api level, currently
        now in the 40s.
        
        The increase in event version is to compensate for a silent
        upstream slippage, whereby the IB tws used to accept exec
        report requests, a non-blank time filter attribute at event
        version 2, and now only accepts non blank times for event
        version 3.   This has been verified using the IB sample
        client sources; if the time attribute of the filter is blank
        (wildcard), or the version is 3, the request triggers
        execution reports, and otherwise not.

      * Set the stdin to be unbuffered.  This helps prevent command
        starvation livelock for downstream scripts.  E.g., previously
        ruby scripts had to use Pty.spawn to open a child shim
        process, and with this change they can now use Open3.popen3 .

      * For risk mode only, change the rule for choosing the tick
        id sequence starting point from the current practice of
        accepting the proposed value X from the unsolicited next id
        message received at startup, to using max(X, Y + 1), where,
        given the session client_id C, Y is the result of executing
        the query given by interpolating C into the following
        template:

            select max(order_tag)
              from CreateEvent
             where client_id = C;

        There is a new option, dups, by which users can retain the
        previous behavior of blindly accepting the start value
        proposed by the IB tws.  Users should avoid this option 
        unless they are sure they know what they're doing; the
        intended use is, in combination with the warn option, for
        testers using the paper account who want to experimentally
        determine the next id reset period used by the IB tws.

      * Change database fractional type declarations in create table
        statements from decimal(10,4) to double.  Modify the rules
        for token type conversion in input to allow negative integral
        values to be promoted to doubles; this eliminates syntax
        errors such as the following:

            For token text: [   -3058] and type target [un_pnl]
            predicted token type:  dec, scanner saw:  neg
            Problem: 401 unexpected token text: syntax error

      * Annotate initialization code with additional obj stack release
        operations below the previous single Marker at the top level,
        thereby cutting the startup resident set size in half, as
        interleaving these block pops with the dbms read block push
        ops prevents obj stack growth. 

      * Add warn option to print status information about sql
        command and query warnings to the stderr.  Due to various
        features of mysql warning diagnostics, the information is
        still somewhat limited, and ways to work around these
        limits are being considered.  This feature is primarily
        intended for those especially concerned with the
        completeness and quality of risk mode journalling.

      * Improve various aspects of memory management for 
        block-doubled containers.  The 090904 release removed
        the warning below that has been in effect since 090821.

      * Fix midnight seconds roll-over bug, in response to a
        bug report from Ken Feng.  Although there was a check
        for midnight, and the seconds since midnight counter
        was reset to zero at that time, the cached value for
        midnight itself, seconds since the epoch, was not.

      * I meant my earlier warnings below about the unfinished
        nature of the releases from Fri 090821 on.  For those who
        are using one of those --- they were only debug releases
        --- and do not want to roll back to Tue 090811, please
        consider downloading today's (Wed) 090826 release.  It
        fixes known bugs from previous releases over the last
        couple of days.

      * Modify the create table definition for OrderStatus to
        comprehensively match the IB-documented order states.
        The key corrections are the split for Submitted into
        two states, the other being PreSubmitted, and the
        inclusion of a new and relatively rare state, Inactive.

        The spelling of the start states is not significant, since
        they are not observed in messages, but since the other
        changes are being made at this time, it makes sense to
        start using the IB names at this time in order to avoid
        future confusion.  Note: the database version increments
        to 1.76 --- this change *does* affect the interpretation
        of journal records.  From the create table statement as
        now shipped:

--  status  enum('Pending', 'Submitted', 'Filled',              -- thru db
--               'Partial', 'Cancelled') not null,              -- ver 1.75

    status  enum('PendingSubmit',
                 'PendingCancel',
                 'PreSubmitted', 'Submitted',
                 'Cancelled', 'Filled', 'Inactive') not null,   --  2

      * Modify the output for 404 probe failed messages to indicate
        the name of the nonterminal symbol that failed to match
        on input, in part so that users will know whether the
        problem occurred while reading IB tws api messages, or
        commands from the downstream.

        MsgI is the name of the start symbol for the IB tws api
        protocol messages, while there are a number of nonterminals
        used for commands, including CmdK, AskK, SelK, CanK and so
        on.

        Warning: this --- 090821 --- is bleeding edge code released
        to help diagnose a specific user problem.  If you don't know
        what I'm talking about, don't bother downloading it; next
        week's will be much improved. [Added: in this vein, I'm
        pushing another tarball that fixes the Makefile]

      * Fix the handshake logic to retry on client id collisions,
        so that it is again possible for multiple data mode shim
        sessions to share the same account and IB tws gui session,
        and similarly, so that it is more convenient to have multiple 
        concurrent risk mode sessions.  This restores a feature
        that was disabled as part of preliminary work on versioning
        earlier this year, and fixes a bug reported by PaulC ---
        thanks Paul!  Also, modify log banner display to ensure that
        only the last, successful handshake status message is logged.

      * Validate shim against the current (897.6) Unix download
        of the IB tws.

      * Add session-specific support for relatively new contract
        data messages, those that include the underlying IB conid;
        the symbol description; and the primary exchange.

        By session-specific I mean that at the higher ApiLevel
        values (43 or 44) needed to process the expanded message
        text, other api features such as tick data and history
        won't work; a session with ApiLevel set in the .shimrc
        file to 43 or 44 is strictly for some of the newer api
        features such as the expanded contract data messages.
        For an all-round useful shim, leave ApiLevel at 23 for
        now.

        A more complete implementation of these version-specific
        features is in the works, but is not my current focus.
        This incomplete feature add is an out-of-band effort to
        help us expand support for new symbols, and other users
        may want to use it for the same reason.

      * Fix log format error whereby nested type tuples,
        specifically position/3 and cashflow/3, were not expanded
        when logged. 

      * Change selected grammar terminal symbol type tags from
        unsigned to signed integral types to accept -1 values
        for inapplicable volume attributes in the case of
        currency realtime bars.

      * Fix orders cancellation bug by eliminating inapplicable 
        action from grammar rule.

      * Restore account code foreign key checking, not null
        declaration for acc attribute, and unique key declarations
        in OrderStatus table.  This, along with various features
        of the shim, is to prevent duplicate record inserts while
        ensuring that OrderStatus records are written if at all
        possible as long as the shim has reached main loop
        processing.  The database version has been incremented.

      * The shim now provides *limited* support to allow users to
        receive the tick price event (index 1) open price subtype
        (index 14) message.  This requires that you run the shim at
        api version 30 or 31, both of which are significantly beyond
        the past and current fully supported api version 23.  To do
        this, simply change the ApiLevel variable in the .shimrc file
        from its default value as shipped of 23 to either 30 or 31.

        Other features of the higher-level api market data interface,
        in particular generic tick strings and snapshots, are not yet
        accessible since the command language does not yet accept
        arguments to select them.

        There are known problems for the shim with other message types,
        in particular history, when you run at this higher api level.
        Orders are not supported at all; the shim will refuse to start,
        so you can only run in --data mode.

        You should probably only use this shim at api levels 30 and 31
        for the limited purpose of collecting market data that includes
        tick subtype 14, that is open price.  It will of course provide
        robust operation as before at api level 23; the changes herein
        are table-based, so the code paths for previously supported
        features are unaffected.

      * As part of reducing memory usage due to interned strings:
          + convert the term root class Data to be a pure virtual base;
          + replace keyed data member with string for non-textual constants;
          + disambiguate term symbols from term strings;
          + fix hash table code calculation initialization bug uncovered by
            obj stack release, enabling release of string temporaries, and
            then modify hash calculation furthur so that symbol intern
            process does not require any heap-allocated temporaries.
          + fix objstack pool release bug whereby only the most recent
            block was reset; now the loop to clear earlier blocks is
            traversed properly, and memory allocation once startup has
            completed becomes nearly steady-state, with only commands
            permanently tabled.
        The objstack release bug is the key improvement to memory
        management so far in July, with other fixes prior to this bringing
        only marginal improvements.  Corrected objstack release gives
        dramatic improvements in memory usage.

      * fix --- not mask --- the orders-related bug reported by Nils
        in his post of ts-general/2009-July/000451.html; currently all
        commands are permanent.

      * Increase the price and quantity tick tick type fields width from
        one digit to two, in preparation for the various new tick types.
        Fill in the can-auto-execute flag field of price ticks.  Fix bug
        whereby market depth messages displayed text codes for operation
        and side instead of the integral values coming over the wire ---
        now the log format core of the message uses the same integer codes
        as the IB tws api, and only the trailing suffix has the translated
        text strings.

      * Fix memory management bug whereby tick ids were not allocated
        via the permanent memory pool.

      * Enable, for the app side of the shim, memory collection
        for time slot and temporary allocations; there are still
        significant improvements in memory management to be made
        for library components.  For now, consider limiting use of
        this newest version of the shim for data collection and
        with the paper account, as memory collection is tested and
        refined.  Shim memory usage now levels off at what seems
        to be a steady-state for VmSize after initialization, as
        it should, although after already consuming large amounts
        of memory during startup.

      * Use obj-stacks to allocate events and their arguments,
        and pop the contents as needed, and in particular within
        the main loop at each iteration, so that event memory is 
        released at the end of each time slot.  This should greatly
        reduce the long term memory usage of the shim.  Other
        improvements to memory management are planned, as well as
        tests and diagnostics to track memory consumption.

      * Log journal post operations for order command events.  E.g.,
        in the log excerpt below, there are two bind commands; a
        comment indicating a create order command enqueue; the
        immediate dequeue of that command, since the send command
        queue had been empty; the resulting place order request;
        and a comment indicating that the journal entry succeeded.

  0.015326|1| 8| 0|bind key:take-pos-aa-090626-20-17-36 to oid:1;|
  0.253403|1| 8| 0|bind key:exit-pos-aa-090626-20-17-36 to oid:2;|
  0.000367|4|11| 0|# create item test-pos ibc:265598@SMART take-pos- ...
  0.000060|1|18| 0|create item test-pos ibc:265598@SMART take-pos-aa ...
  0.000043|2| 3|15|3|15|217|AAPL|STK||0.0000||1|SMART||USD||BUY|2|LM ...
  0.015829|4|12| 0|# create: STK:SMART:AAPL:USD: (LMT, 2, 100.0000,  ...

        Note that comment types are indicated by their indices, as
        defined in the table included in the description of byte-level
        request queueing below.  E.g., the 4|11 and 4|12 comment
        entries indicate command enqueue and journal post events.

      * Log journal post operations for new order status, open order,
        portfolio, and execution report messages.  Successful post
        operations are indicated by entries such as the following:

          0.005106|4|12| 0|# journal OrderStatus(2);|
          0.038027|4|12| 0|# journal ActiveOrder(2);|
          0.026289|4|12| 0|# journal OrderResult(4);|
          0.005298|4|12| 0|# journal OrderStatus(1);|
          0.008111|4|12| 0|# journal ActiveOrder(1);|
          0.005363|4|12| 0|# journal OrderStatus(1);|
          0.007184|4|12| 0|# journal ActiveOrder(1);|

        Unsuccessful entries are logged as a 418 warning:

          0.000081|4|98| 0|# 418: could not post ambiguous order message|

      * Implement the diff option, to print the time stamp detail
        attribute in the log as a difference from the last event,
        rather than an absolute number of microseconds.  The diff
        format uses a floating point fraction and the units change
        from usec to seconds.  The diff option is probably most
        useful for downstream users who are trying to track request
        and message latency during development.

        Events are monotonically increasing in time with the one
        exception of repeating groups such as history query answers,
        where the header is constructed after the detail lines, and
        so the actual differences are non-negative with this one
        exception.  Since construction of the timestamp object is
        clamped to give non-negative values, the first history bar
        detail line of a history query answer will show a zero
        difference from the header.  Without the diff option, in
        constrast, this first detail line will still be timestamped
        earlier in time than its predecessor header line.

        The bin/log.filter script has changed to ensure that the diff
        field is common to both halves of the output, and the log
        format width for tick ids is reduced accordingly.

      * Relax foreign key, null checking, and uniqueness attributes
        for the OrderStatus journal table.  The account code attribute
        of the OrderStatus table, though useful in relating its tuples
        to other tables, is not included in IB tws api order status
        messages, and until now has been synthesized by the shim from
        account data queries at startup.  This has introduced an
        undesirable dependency for journalling on successful startup
        initialization, and in particular meant that the shim could
        not parse order status messages until after the completion of
        late stage initialization.

        Following the principal that the shim should post api messages
        to the journal in exactly the form the api provides while
        avoiding the insert failures that might occur due to attribute
        checking, instead leaving cleanup for downstream processing,
        this account code synthesis as part of OrderStatus journalling
        has been removed.

        The information is still available from other journal tables,
        in particular from both OrderReport, copied from execution
        reports, and CreateEvent, copied from create order commands,
        and where it is still foreign key checked, since the shim can
        exert tighter control over its command language than it can
        over the IB tws api messages.

        The database version increases to 1.74 as a result of this
        change.

      * Important change in command syntax:

        The terminating semicolon for commands is now *required* .
        Previously, any of a semicolon, a semicolon followed by a
        newline or a newline alone served to indicate the end of
        command.

        The newline is no longer sufficient, and in some cases
        when you leave it out, the command parser will simply
        continue to absorb text until it sees that semicolon.

        This change is necessary in order to allow multiline
        commands in the future, and also simplifies the fix for an
        existing problem where, for certain cases where multiple
        commands occurred on the same line, command processing
        would delay or hang.

      * Modify and refactor the shim to add a feature I call
        byte-level request queueing, with the following effects:

            + the pathway from cmd input to request send is
              shorter, and so simpler;
            + each event's lifetime may be constrained to fit the
              time slot during which it was constructed, a useful
              precondition for future work on obj stack release,
              to recover event memory;
            + commands are logged twice, first as a comment when
              enqueued, and then as a command when the resulting
              cmd/req info is dequeued, and the request is sent;
              in both cases the command text is identical to the
              input text, as opposed to the abstract or schematic
              form used previously, and because there are distinct
              timestamps for each time logged, they are in order
              with respect to adjoining log events;
            + requests follow the related command as before;
            + the range for log comment indices has been changed
              from three digits to two, that is from [100-500]
              to [0-99]; the source codes for cmds, reqs, msgs,
              and comments remain as 1, 2, 3, and 4, respectively.
              The set of currently defined log comment indices
              is enumerated by the LogK::schema() procedure in
              rule.c, and the actual changes at this time are as
              follows:

                    name  from   to   purpose
                    ----  ----  ----  ----------------------------
                    star:  100    0   log banner
                    head:  101    1   log banner
                    hand:  102    2   handshake, after log banner
                    time:  103   10   performance measurement
                    nque: [new]  11   command enque notification
                    post: [new]  12   journal post  notification
                    drop:  501   41   syntax error token discards
                    warn:  500   98   copy stderr warning message
                    fail: [new]  99   copy stderr fatal exception

        So, the first four lines of your log, with time stamp info
        elided by elipsis, and the backslash indicating continuation,
        would now appear as, e.g.:

        ...|...|...|4| 0| 0|# |****************|
        ...|...|...|4| 1| 0|# |0.94|090609|data|
        ...|...|...|4| 0| 0|# |****************|
        ...|...|...|4| 2| 0|# |23|1|44|20090609 12:49:28 EST\
                              |Connect with: cv 23, id 1, sv 44|

      * Relax symbol name typechecking for contract descriptions
        in messages, and foreign key checking for the tables ---
        ActiveTuple, ResultTuple, and ReportTuple --- used to
        record open order, portfolio, and execution messages.
        This change allows the shim to accept such messages when
        they refer to alien contracts in the tws gui portfolio.

        More precisely, convert the symbol data member reference
        in the Description and ConExpTuple classes from class
        obj::Symbol to class obj::String, and delete the foreign
        key declaractions for symbol names in the journal tables
        of interest.  Of the other contract value attributes,
        foreign key checking for security type, currency, exchange,
        and right remains, so that in particular, the Exchange
        table must still include an entry for the venue of any
        working orders or executions reported over the IB tws api.

        As a result of these and other changes, the database
        version increases from 1.72 to 1.73.

      * Add a simple state machine to the per-order RiskId object
        to reject invalid transitions --- modify before create,
        multiple creates, and multiple submits --- and change the
        rules for computing the order transmit flag for modify so
        that it reflects the current state, that is, transmit
        flag off if not yet submitted, and transmit flag on if
        already submitted, with the effect that modify may be
        used both before and, given the order still working, after
        submission.

      * Reorder the input streams so that messages are parsed before
        commands, and so reorder their occurrence in the log.  This
        is at no cost in command latency, since request sending
        occurs after all input has been parsed; gives an arguably
        more intuitive ordering to log text; and allows significant
        savings in elapsed time for the regression tests.

      * Replace home-brewed Ruby-based template macro counter with
        __LINE__ occurrences for generation.h and nterminals.h, and
        remove the scripts bin/generation.rb and bin/nterminals.rb
        from the bin directory.  This change does *not* allow mix
        and match; for the new sources in src, you must also have
        the new bin directory, that is bin less the two ruby macro
        scripts, else your generation.h and nterminals.h headers
        will be *trampled*.  This possible build failure mode occurs
        since the Makefile still has a ruby preprocessing rule, as
        there is still one occurrence of ruby preprocessing, for
        lib/rdtsc_time.i to rdtsc_time.h, where the translation is
        non-trivial.

      * Add join command line option to open log file in append mode,
        as opposed to default and previous use of truncate mode.

      * Fix bug whereby strcmp return value outside {-1, 0, 1}, as
        evidently occurs for OSX stdlib, leads to a hang during
        search.  Changes consist of: adding the appropriate enum
        type, making its definition visible everywhere, converting
        from the int return for strcmp() to this type via multiple,
        explicit conditional tests in lib/string.h, narrowing int
        type declarations for the comparison result to the enum
        type, adding an exception as the default case in the switch
        where the type is tested, and adding trace logic to indicate
        what has happened when a type violation occurs. 

      * Fix race whereby order variable binding was extracted from
        event loop processing, which occurs after event input loop
        and so possibly following successive order commands, so that
        order variable binding instead occurs during command
        parsing, as an attribute grammar action.  This is the orders
        problem that led to my warning for Friday's release, and
        for which the mailing list post by Paul C suggested the problem
        (thanks, Paul).  This release also includes changes to the
        exs/risk.rb test script to make it easier to modify and reuse.

      * Fix bug in product lookup search loop that could under some
        circumstances hang the shim; thanks to Richard Pruss for
        finding this. [there are still reports of problems?!] 

      * Modify the wait command to accept floating point input, to
        read that input with microsecond resolution, and to update
        wait delays at each iteration of the main loop, which
        given the approximately 20 millisecond duration of those
        iterations, means that the minimum non-zero wait delay is
        now 0.020 seconds rather than 1 second.  This feature is
        provided to allow users to delay otherwise back to back
        orders, where such delay is evidently required in order to
        avoid limitations of the IB tws, and yet to spend less time
        than a full second, which was an unecessarily long delay.
        Anecdotal evidence indicates that the appropriate delay is
        on the order of a third or quarter of a second.

      * Add loop to select'd read to select again if: for IB tws
        socket, buffer was exactly filled; or, for stdin cmd input,
        a line was read in.

      * Set line buffering as program default for stdout when
        cout option is provided.  For data mode, add bulk option
        to override this default, instead allowing system default
        buffering to prevail, and thereby recovering the original
        behavior of shim prior to this change.  The bulk option
        is ignored in risk mode.

      * Fix bug whereby only first interface in getaddrinfo() list was
        tried for socket connect --- now each IF is tried until first
        success (thanks to Richard Pruss).  Also add AI_ADDRCONFIG flag
        to ai_flags attribute so that only IFs of the same type ---
        v4 or v6 --- as the external IF will be included in the list.

      * Validate shim against the 894.4 IB tws; server api level is
        still at 44, and no new message version numbers were
        encountered.

      * Add sent flag to ChangeOrder table, change time stamp name from
        "time" to "c_time", and add "m_time" attribute to track the
        approximate time that the order creation, modification, or
        cancellation action for this ChangeOrder record was transmitted
        to the IB tws.

      * Add the "select time" command to provide access to the current
        time request and its related message; and add the "select bars"
        and "cancel bars" commands to provide access to the realtime
        bars feature of the IB tws api; examples of the latter commands
        may be found in exs/bars .

      * Unify the multiple schema previously defined for Request tags
        to a single case each, modify the command schema for consistency
        with each related request, and modify cmd -> req trace error
        reporting to simplify error analysis if mapping fails.  These
        changes led to, in particular, many small changes to the schema
        for order-related commands and requests, so users who upgrade
        should check order processing with the paper account thoroughly
        before entering production use.  These changes greatly simplify
        request tags, and were necessary refactoring to accompany the
        ongoing implementation of multiple api version support.

      * Disable journalling and risk mode for api levels prior to 23.
        Previously, there was no support for operation as a whole prior
        to api 23; now, as such support is added for earlier and later
        levels, and since journal messages have been elaborated over
        time, the problem of how to map journal messages with api-varying 
        formats to the fixed database structure arises; older versions
        have too little data, while newer ones have useful, though
        not yet captured, attributes.

        The general answer to this issue is that there will be some
        minimum api level for journalling, and the journal will capture
        at least this lcd, currently api 23.

        Newer attributes may be added to the journal, either with
        possibly null declarations, or else by increasing this minimum
        api level for journalling past 23.  The latter choice would be
        at the cost of removing support for orders prior to the new
        minimum level for journalling, since journalling is considered
        a necessary safety feature for risk mode.

      * Turn off journal foreign key checking for ib contract
        ids --- the ibc attribute --- and account codes, to
        ensure that journal entries occur even for portfolio
        entries caused by tws gui positions with unknown
        contracts, or fa subaccounts not yet entered into the
        database.

      * Provide version specific type tuples for the exogenous
        session level message types, 3 -- 9; this is a necessary
        prelimary step to prepare for support of other api version
        levels.

      * Preen and otherwise cleanup cash symbols data in the sym
        directory, with the result that cash symbols now work.  You
        will need to download the newest symbols data tarball in
        addition to the shim source tarball to take advantage of these
        changes, and in addition to compiling the new shim, also
        recreate or repopulate your database(s).

        Modify locality input to the product search for CASH securities
        so the value is clamped to "n/a", which is the correct region
        for CASH products in any case.  This allows users to leave the
        region set to some reasonable local value for stocks and other
        symbols, and yet still be able to work with currency symbols in
        the same session.  Add currency text to the log file rhs symbol
        tuple annotation for the CASH security type, so that the symbol
        currency pair is visually simalar to the notation for a cross.
        Also elide the route in this case, since it is not very
        interesting for currencies.  The contract value expression
        form is accepted for CASH contracts, and there are examples of
        this in exs/tick .

      * Eliminate the undocumented sVersion (server version)
        attribute from the configuration language.  Delete invalid
        reference to slow.sql from table.sql.

      * Extend the contract value expression syntax in commands to
        allow colon separated expressions for stocks and futures
        such as:

        select tick STK:SMART:AMAT:USD 1;		wait  3;
        cancel tick STK:SMART:AMAT:USD;			wait  1;

        select tick FUT:ECBOT:YM:USD:20090320 1;	wait  3;
        cancel tick FUT:ECBOT:YM:USD:20090320;		wait  1;

      * Add multiplier attribute to various database tables, forcing
        a version change; and modify the shim to: read this
        attribute from the tables where it occurs; and write it as
        appropriate in requests.

      * Make miscellaneous fixes to allow contract data queries for
        individual options contracts, using the ibc:12345678 form.

      * Add fill.rb script, to filter contract info for options to
        sql insert statements, as load files for the database.

      * Add contract expressions to the contract data query command;
        examples are in exs/info.

      * Merge functionality of contract info query commands into one
        single command, "select info", with examples in exs/info.
        There is no longer any need to use the separate "wild"
        command for wildcard queries; the original "select info" form
        now accepts as an argument a variety of contract expressions.
        The "wild" command line option has also been eliminated; all
        contract info queries now accept arbitrary symbols, whether
        or not those symbols already occur in the database. 
        
      * Delete: contract expression form "cid:" from the cmd language;
        ContractRow table use from the shim, and existence, from the db;
        and notion of default contract from the database.  The sql
        script slow.sql, which selected default contracts, becomes dead
        code.  Attributes previously named variously as route, exch, and
        ecn for the two cases of foreign keys to the Exchange uid and
        name have been renamed to route and exch as their values are
        instances of Exchange uid and name, respectively.  That is,
        previous use of the attribute name "ecn" has been disambiguated
        to route or exch as the attribute was a numeric or string key.
        Modify more.sql to ensure that options data, once loaded into
        OptSrc, is propagated through Expire and Option to Product.
        The LocalSet table goes away, as does the WatchSets table, which
        was deprecated and unused, and in addition ComboLeg and ComboSet.
        The Db version goes to 1.69, and the shim version to 0.91.  As
        always, examples of the currently supported forms of contract
        expressions are provided in exs/tick .

        While making significant changes to the database, go ahead
        and also add three new options-related exchanges: NASDAQOM,
        IBSX, and MIBSX; and, completely unrelated, prune and reorder
        AtomTag, to make it consistent with current log usage, where
        cmds, reqs, msgs, and comments are numbered 1 -- 4.

        For a db version change like this, where significant tables,
        in this case ContractRow and LocalSet, have been eliminated,
        there are in practice two unpleasant choices in coding the
        drop script src'd as the first step of create.sql, which
        (re) creates and (re) populates tables: leave the old table
        names in, and get mysql warnings if it's used to drop post-1.68
        databases; or eliminate the names, and have the script fail the
        first time it is used to drop an old, pre-1.69 database.  (After
        all, ContractRow has to go away, or else the tables it depends
        on can't be dropped.)

        It's easy enough to leave the names in for a while, and so patch
        over the problem, but they need to go away eventually, and so the
        following script, five.sql, src'd for now from create.sql:

	--  Drop the tables that were eliminated from the database
        --  in the changeover from db version 1.68 to 1.69.  This
        --  script will lead to sql warnings for updates between
        --  post 1.68 database versions, e.g., from 1.69 to 1.70,
        --  once the latter is released, and so the source statement
        --  that refers to it in create.sql is transitional, and will
        --  eventually be removed.  Feel free to source it yourself
        --  if you are transitioning from a pre-1.69 database to a
        --  much newer one, and the create.sql with the newer code
        --  no longer does this for you.

            drop table if exists LocalSet;
            drop table if exists WatchSets;
            drop table if exists ComboLeg;
            drop table if exists ComboSet;
            drop table if exists ContractRow;

        If, after reading the above, you prefer to just use setup.sql to
        blow away both the testing and trading databases first, that's
        also an option.  More drastic, and more up-front effort to logon
        to the server with sql admin privileges, but still, your choice.

      * Add two 2009 entries for FUT:GBL, as an example of how users
        can add data to the load files, and from there to the database;
        see the mailing list for details.

      * Fix bug in mapping attributes from OrderFlags records to
        orders; please consider an upgrade if you use the shim for
        live orders.  The problem arose in part because the various 
        boolean flags had the same type, and the code has been 
        modified so that each now has a distinct type tag, so that
        this problem will not creep back in.

      * Add two new forms for contract expressions, with the prefix
        type tags pid: and ibc: , where the immediate argument is a
        product id or IB contract id, respectively, either of which
        serves to specify an abstract product; and the trailing arg
        is of the form "@route". E.g., each of the following serves
        to define a market data subscription to STK:SMART:AMAT :

            select tick cid:       8       1;	# deprecated
            select tick pid:    2846@SMART 1;
            select tick ibc:  266093@SMART 1;

        See the example scripts in directory exs for more details.
        With this change, the shim contract id form, e.g., cid:8 ,
        is deprecated; in the future, once this form has been
        eliminated, the ContractRow table, default routing, and
        LocalSet all will become unnecessary.  Note, however, that
        LocalFut is still needed at this point, to provide the
        contract expiry; that's a whole 'nother issue.

      * For database, fix bin/get_id.sql script to update contract
        table name from Contract to ContractRow, fix currency code
        for Brazil, changing it from BRE to BRL, and update syms.txt
        by rerunning bin/get_id.sql to reflect that name change.

      * In code.h, rewrite template member function that
        included a call to the partially specialized 
        parent class lookat() member function to use
        an explicitly dependent call --- P::lookat() ---
        in place of the using declaration, which, although
        correct, and acceptable to modern C++ compilers,
        is rejected by g++ 4.0.1.  In addition, fix bug
        in lib/rdtsc_time.i for the case of ppc processors.
        These two changes allow the shim to compile again
        on antique Darwin systems.

      * The -O3 optimization flag in the Makefile has been
        turned back on by default, since it works with
        debian testing, and g++ 4.3.2-1; if you are using
        ubuntu 8.04, you'll need to delete that flag, since
        g++ 4.3.2 on ubuntu 8.04 is known to produce flawed
        executables.

      * The database version has increased to 1.67

      * Contract expressions, which used to be bare numeric
        literals standing for the Contract table record uid,
        now must be prefixed by a type tag, the symbol "cid"
        followed by a literal colon, ":".  E.g., for market
        data subscription and cancellation to the march 09 YM
        contract, the following are the old and new forms:

            select tick         8 1;		# old
            cancel tick         8;

            select tick   cid:  8 1;		# new
            cancel tick   cid:  8;

        Whitespace around the colon is optional.  This change
        applies to all occurrences of contract uids in the
        language, whether in subscriptions; contract or history
        queries; or create order commands.  The examples in
        directory exs have been updated to reflect this new
        syntax, so see those files for illustrative examples.

        This and much of the later work to be done in this series
        will extend contract expressions in the command language to
        new forms.  The files in the examples directory should be
        modified in step with changes to the language, so that
        if you are uncertain of what exactly the new syntax at
        any point is, look there first.

      * Bind equation lhs and rhs expressions, which used to
        be labelled tuples, now use the infix colon, so that
        the syntax is analogous to contract expression factors.
        E.g., for the bind commands as echoed from exs/risk.rb,
        the examples below are the old and new forms:

            bind key(take-pos-aa-081223-13-06-52) to oid(1);	# old
            bind key(exit-pos-aa-081223-13-06-52) to oid(2);

            bind key:take-pos-aa-081223-13-06-52 to oid:1;	# new
            bind key:exit-pos-aa-081223-13-06-52 to oid:2;

      * Modify regression scripts to change YM front month contract id

      * Modify the get account code logic during initialization
        to read any number of order status and open order messages
        until reaching the first account code quad.  The loop now
        terminates only if there is: a parse error; a read timeout,
        with no data at all returned; or a table lookup failure for
        the account code string extracted from the first account
        quad that occurs.

      * Use the create order mark/brand value as the order_ref
        attribute of the place order request, and, when that
        value is returned as part of the open order message, store
        it as part of the journal entry written to the ActiveOrder
        table.  As a result of this change the database version
        has increased to 1.66.  Note that, although the IB tws
        online docs indicate that the order_ref attribute is for
        institutional investors only, it works fine for us in
        testing.  Please let the list know if the IB tws refuses
        to accept the resulting place order commands for your
        account, and in that case we may need to add a command
        line flag to control brand->order_ref mapping; for now,
        however, we prefer that this labelling be uniform and
        automatic.

      * Add a new parameter to the create order command, a
        user-selected label or brand that is stored to the journal
        as part of the CreateEvent record.  As a result of this
        change the database version has increased to 1.65.

        You are free to set this brand as you wish; it is meant
        to label order groups, e.g., a position.  If, in the
        future, a flattener is added to the shim, that command
        will probably take an order group label as its argument.

        Fix a bug whereby, if the journal was cleared, the shim
        would reset order ids to 1 in conflict with the next id
        value as directed by the IB tws; the shim now honors the
        handshake-phase next-id message as it did in the past.

      * Modify initialization to cancel the account data subscription
        after getting the account code, with the following effect:
            + you will no longer see more account data unless you ask
              for it; and
            + you no longer need to start scripts with a "cancel acct"
              command to end the subscription.
        Of course you are free to use the "select acct" command to
        restart the account data subscription as and when desired.

      * This first (081215) tarball of a new series is meant to provide
        a convenient transition from the prior series, which ended with
        the 081126 tarball; the shim database version is still 1.64, as
        it was for the 081126 release.  Note that the next tarball will
        see a bump in the shim database version number.

      * Code in tick.c has been refactored to simplify the way tick ids
        are tracked, which makes order processing more robust.


date: Wed Nov 26 2008
date: Fri Nov 21 2008
date: Wed Nov 19 2008
date: Fri Oct  3 2008
date: Thu Oct  2 2008
date: Tue Sep 30 2008
date: Fri Sep 26 2008
date: Thu Sep 18 2008
date: Mon Sep 15 2008
date: Thu Sep 11 2008
date: Fri Sep  5 2008
date: Wed Aug 27 2008
date: Mon Aug 25 2008
date: Fri Aug 22 2008
date: Wed Aug 20 2008
date: Tue Aug 19 2008
date: Thu Aug 13 2008
date: Wed Aug 13 2008
date: Tue Aug 12 2008
date: Mon Aug 11 2008
date: Fri Aug  8 2008 Triple 8 release
date: Thu Aug  7 2008
date: Wed Aug  6 2008
date: Mon Aug  4 2008
date: Fri Aug  1 2008
date: Thu Jul 31 2008
date: Wed Jul 30 2008
date: Thu Jul 24 2008
date: Fri Jul 11 2008
date: Fri Jun 27 2008
date: Thu Jun 19 2008
      --- through ---
date: Tue Mar 25 2008

      * Cross-session order modification and cancellation now works;
        see the script exs/kill.rb for an example.  This is the big
        feature of this release.

      * Turn of -O3 in the Makefile, which fails horribly for
        g++ 4.3.2, with any number of absolutely essential reference
        parameters optimized away during the constant initialization
        of singletree construction.  Feel free to add this back in
        if you use g++ 4.1.2 or earlier in the g++ 4.x series, since
        the optimization has worked there for us during development.

      * NOTE the new config file var LogFiles, explained below;
        the log files created with the file option will by default
        now end up in a subdirectory log.  To restore the previous
        behavior, change the value of LogFiles from log to . (dot)

      * Update the message version vector to correspond to tws 889.2

      * We now have a decently populated 64bit build farm, and I've
        shifted from developing in a 32bit environment to working with
        --- and running distcc against --- 64bit machines.  Users should
        change the LIB variable in the Makefile to provide whatever
        mysql path information they might need; as shipped, that value
        is set for Centos/64bit, which also works for Redhat and
        Fedora.  Debian users please feel free to post patches to the
        list.

      * Add configuration log directory variable LogFiles with
        default value log.  Prepend this path to the various log
        file names (ShimText, cmdinput.txt, shim2tws.bin, and
        tws2shim.bin) so that file opens expect the files to be
        there.  Since the log file variable value may be a
        fully-qualified path name, this allows the user to use
        logging while still preventing side effects relative to
        the current directory.  With, in addition, a home directory
        configuration file, the shim may then be freely run with
        any current working directory, and in particular, from
        outside your home directory tree.  Note that there is still
        a dependency on the current user, whose identity determines
        which home directory shall be searched for the config file.

      * Modify the journal process to check new messages against
        those already received for equality, in order to avoid
        attempting to post duplicate messages to the database
        journal.  This applies to order status, open order, and
        execution report messages, but not portfolio.

      * Fix bug whereby the shim would hang when starting against
        an FA account-connected IB tws, and improve account
        code lookup logic so that the shim can start with both
        ordinary and FA accounts.  This does *not* mean that
        FA functionality is supported: it isn't.  Please contact
        us by email if you need such support.  Note in particular
        that the account info command does not work with FA
        accounts, and that there is no support for order
        allocation.  There are probably other issues as well.

      * Fix the bid-ask label: this is described below, and note
        that the fix of 10.1.08 was incomplete, and the new release
        of 10.2.08 should be used instead.

      * Fix the bid-ask label for the what-to-show attribute of
        history queries, so that history queries now work for all
        20 cases of HistoryTag (rth x epoch x what-to-show, 2*2*5).

        Generalize the attribute type of the history bar volume
        from natural to integer, so that the N/A value of -1
        is quietly accepted.

      * Modify the history query command (select past ...) to
        lift the bar size, or unit; and the bar count, or duration,
        values from the HistoryTag table (via PastFilter) to be
        actual top-level arguments, and modify the regression tests
        in directory exs to reflect these changes to the command.

        The bar size is a label, e.g., m1 or s30, while the duration
        is a dimensioned number, digits followed immmediately by the
        unit, e.g., 1d or 1w.

        The database version has increased once for each change,
        as the duration and unit arguments were added to the history
        query command.

        There are now only 20 possible HistoryTag indices, and as
        before, the initial PastFilter indices map one-to-one to
        HistoryTag, so if you want outside rth, date-string TRADES,
        just use a 1 to take the first HistoryTag.

      * Roll the front month for YM and replace occurrences of STK:AIG
        with STK:AMAT as well.

        As part of ongoing code changes triggered by the fix of the
        floating point truncation problem a couple of weeks ago,
        modify the scanner to allow any reasonable floating point
        format input for command and message decimal-type tokens.

      * Validate shim against IB tws 887.2, which is at server 
        version 42; previous versions of the shim, which worked
        against sv 40, e.g. IB tws 886.5, have problems with sv 42
        for contract data requests, and this release fixes the
        problem.  There is also a message version jump from 6 to 7
        for Execution reports.  Please do *not* feel obliged to
        upgrade your IB tws to 887.2; we much preferred 886.x,
        and view the 887.2 as a work in progress.  We are testing
        the shim against 887.2 purely from a "be generous in what
        you accept" standpoint, especially for those users who
        have no alternative to 887.2, either because they did not
        save their 886.x jar file, or because they only recently
        started using the IB tws.

      * Fix truncation problem for floating point terms, negation
        problem for integers, and boundary conditions for partial
        insert failures.  Tighten tokenization type checking, and
        I'd be interested if anyone sees any error messages as a
        result.

        As part of ongoing code cleanup and bug fixes related to
        this truncation problem, one in particular is likely and
        more or less harmless:
 
            For type target TYPE_NAME and token text: 
            Problem: 401 unexpected token text: syntax error

        These 401 errors are transient, and outside of the volume
        of syntax error noise they spray to the stderr, harmless.
        I would, however, appreciate it if users would submit
        the type names for which they occur to the list, so that
        I can clean up any occurrences that are left.

      * Journal inserts for msgs use blocking to group records
        within each select() time slot.  Data from order commands,
        i.e., CreateEvent and ChangeOrder records, is still
        inserted one by one, for safety's sake.

      * The ibc attribute of OrderResult and OrderReport has been
        altered to allow nulls, and this is probably a stable
        choice for the time being, now that I've been back and forth
        on this.  After some analysis, it's clear that we can't
        depend on mapping contract descriptions to ib con ids, so
        until and unless we *both* upgrade to api client version 35
        or above *and* drop backward compatability for cv levels 34
        and below --- which latter draconian restriction is very
        unlikely --- nulls are going to be a possible problem for
        downstream scripts reading the ibc attribute.  Sorry.
       
      * Cleanup use of manifest constants in memory allocator and also
        string processing code to allow execution on 64bit architectures
        such as amd64.  This has not yet been tested widely in the 64bit
        space, that is beyond amd64, and this is not yet the thorough
        refactoring of the memory allocator that is on the priority list.
        More extensive changes will probably not occur until cross-session
        order manipulation is working --- that comes first.

      * Modify initialization to allow, for the case where the
        config variable defaults are sufficient, startup without
        a .shimrc file.  Previously the .shimrc file was required,
        whether in the home or current directory.

      * Include minimal dump and load scripts in sql/bin by way
        of example.  You will almost certainly need to adapt these
        before use.

      * Generalize the Bulletin message exchange attribute type from
        exchange key, which previously provided exchange-specific
        type-checking, to be a string, reflecting its use in the api
        for comma-separated lists of exchanges, e.g., HKFE,SEHK.  This
        implies that news bulletins will probably need human analysis,
        which is probably already true, or else that downstream
        programs need to do that much more work classifying bulletin
        components. 

        The shim does not currently pick apart compound attributes in
        messages, by way of example AIG SMART contract details:

        ADJUST,ALERT,ALGO,AON,AVGCOST,BASKET,COND,CONDORDER,DAY,DEACTEOD,
        DIS,GAT,GTC,GTD,GTT,HID,ICE,IOC,LIT,LMT,LOC,MIT,MKT,MOC,MTL,
        NONALGO,OCA,OPG,OPGREROUT,PREOPGRTH,REL,RTH,SCALE,SCALERST,STP,
        STPLMT,SWEEP,TIMEPRIO,TRAIL,TRAILLMT

        and may never do so; it doesn't yet seem that important.  That
        being said, it does work with exchange sets in the database,
        so the sequences representing exchange sets in messages are not
        out of reach, just not of interest, at least not for now.

      * Cleanup code to allow compilation for g++ 4.3.1 (Debian);
        current development has been taking place on g++ 4.1.2 (Centos).

        Changes consisted of:
          + Adding additional include statements for standard headers,
            to reflect cleanups to other library headers that reduced
            include dependencies.  One result of this is that cstdlib
            is now pulled in by src/glossary.h
          + Renaming typedef names in classes: C++ does not allow a
            type name, once used in a scope, to be redefined, and
            previously g++ did not count the first use as an argument
            of the typedef itself against that limitation, while now
            it does.  The result is a fair number of new local names
            in library templates.
          + Parenthesizing conditional logic expressions that use
            both || and &&, and bracketing nested conditional
            logic statements that have an inner but not outer
            else.  -Wall now complains about each of these forms,
            and since it overrides -Wpermissive the two are in
            effect mutually exclusive, so that given the practical
            value of -Wall there is not too much choice.
          + Improving the consistency of const for container template
            argument types.  Although C++ allows an argument type of T
            to be used where const T is required, the same is not
            true for a container C, which can't be used as an arg
            for a declaration of C.  g++ now catches more
            cases of such inconsistent declarations.

      * Internal testing of order journalling has found a number of
        bugs in the database representation of cash security type
        symbols, and this may cause transient error messages for
        cash-related Portfolio value events, which should be ignored
        for now.  This (stopgap) release: improves but does not
        completely fix database support for cash symbols; fixes a
        bug in the use of locality with respect to cash symbols;
        and improves the error message output, so that the common
        case for this problem may be more easily recognized, and
        ignored.

        More precisely: security type CASH records from table Source
        were not properly copied to the other tables that define the
        available products, and so were unavailable to the user (this
        is fixed); have incomplete currency pair information, due
        to holes in the data we've collected from the net (not yet
        fixed, due to restrictions by IB on redistribution of info
        collected via the api, though we plan to provide a fill
        script later); and all have "n/a" as the locality, so that,
        (1), that attribute is of no use in disambiguation, and,
        (2), should therefor be clamped internally by the shim to
        "n/a" for all cash symbols, ignoring the config file value
        that is used for symbols of other security types (this is
        fixed; there is no need for you to set the Locality to n/a
        for cash symbols, that is done for you now).  And, the
        common case referred to above occurs when you have some
        residue of foreign currency in your account due to previous
        transactions using the gui, so that at every startup there
        is a portfolio value message for that currency, leading to
        an error message such as:

            For msg: Portfolio
            and sym: CASH:EUR:USD
            Problem: 418 could not post ambiguous order message

        As long as such messages are for portfolio foreign currency
        residues, they may be safely ignored for now.  And, due
        to the holes in database cash symbol information, where
        currency pairs only have the base, and not the payment 
        or quote currency, you will not be able to use the shim
        to trade currencies yet.

      * Try out IB tws 886.4, and make minor changes to message table
        required to accept new message version for contract info, and
        in the process make a significant change to the message parser,
        as follows:

        The shim will now attempt to parse messages with a version one
        higher than it expects, in which case in addition to accepting
        the message and relaying it to its output channels, it will
        also display the transient error message:

        Problem: 427 api msg version mismatch -- upgrade soon

        At this point you should consider downloading a new version of
        the shim, in order to pick up the newest message tables.  Note
        that you can continue to use whatever client versions the shim
        previously supported, so that your downstream does not need to
        change, as long as it ignores the message version, as it should,
        about which more on the list.

      * Order journalling for the shim is now complete.  You don't
        need to refer to it --- the same information has been and
        still is available through the output log channel printed
        by the shim --- but it's there if you need it.

        In short, CreateEvent and ChangeOrder records track order
        requests by the shim towards the IB tws, while OrderStatus,
        ActiveOrder, OrderResult, and OrderReport records reflect
        the order status, open order, portfolio value, and execution
        report messages from the IB tws to the shim.

        For more detail, see the "Triple 8 release" message on the 
        mailing list.

      * Add code to dynamically add AccountCode entries to that
        table as needed at startup, following an account info
        query to the IB~tws.  This means that you no longer need
        to define account codes via the sql/mod/Account.sql load
        file, or manually insert entries after connecting to the
        edemo account.  You may still want to specialize the type
        attribute of shim-added entries for paper or demo accounts
        from 'paper/demo' to the appropriate one of the two cases,
        since the account string does not disambiguate those types.

        NOTE: Symbol, Product and *Contract* numbering has changed !!!!
        The sort order during database load for symbols and products
        has been changed to simplify product lookup, and as a result,
        most contracts have new numbers.  E.g., AIG (ibc 4301) moved
        from 2 to 6, although the front month YM is still at 144.

      * Modify the connection logic to handle client id collisions,
        so that it is more convenient for multiple shims to connect
        to the same account.  This, along with changes to the
        database post logic to support concurrent access by multiple
        shims, and the previous elimination of any need for downstream
        client writes to the database, should allow multiple shims
        to share the same tws account up to the IB configuration limit
        of eight connections.

      * Fix bug where open order messages sent by the tws at startup
        could not be parsed due to table lookup failures, leading to
        the fatal exception:
        Problem: 524 "could not obtain IB account info at startup".

        As part of the fix, the database reads have now been moved
        prior to the IB tws handshake, and the banner message has
        been changed accordingly to be similar to the following:

          The trading shim has connected to the database server
          and loaded 36962 products; connected to the IB tws as
          client id nn; and queried for the account code, which
          seems to be  DUnnnn.  Program initialization has been
          completed.

        This restores the order of initialization to that which was
        used prior to the addition of all the new database products,
        and the increase in startup time to around 7 seconds on a
        modern desktop.  As a result, all but the first line of
        the the banner is not printed until the end of initialization,
        after the 7 second delay.  The stages still correspond to
        the ordering of the banner clauses, so that connection
        exceptions replace the related clause.

      * Increase the database version to reflect changes in the
        definition of the default route for products; where before
        they were collected in Symbol, now they are partitioned
        between Static and Expire, which is where the exchange sets
        already are defined.  The previous arrangement led to
        multiple Product tuples for some of the ib contracts; the
        current arrangement gives a one-to-one mapping between
        Products and ib contract ids, and this is reflected in the
        addition of the ibc attribute to Product as a unique key.
        This gives an alternative, simpler, and arguably more
        correct means of product lookup for those journal records
        that have both a contract description and an ibc number,
        since the contract description is potentially ambiguous.

        The NumTries, DataSubs, and BookSubs configuration variables
        have been added to control the number of connection attempts,
        number of market data subscriptions, and number of market
        depth subscriptions per shim, e.g., with a DataSubs value of 12,
        eight shims sharing the same config file could make up to 96
        market data subscriptions.  The "once" cmd line option flag
        overrides the NumTries variable, so that say in risk mode,
        and where you want to have exact control over the ClientId,
        the shim will quit after the first connection attempt if
        there is a client id collision.  The HqPeriod variable,
        renamed from PastSlot, can be used to divide up the history
        query bandwidth; e.g., with a HqPeriod of 81 seconds,
        eight shims should be able to comfortably share one IB tws.

        Note that each shim still needs a separate text output file,
        if that is your log output channel, so you'll want to start
        each shim in a different directory.

      * Fix a bug whereby the account code record uid entered in
        CreateEvent order journal records was clamped to 1.

      * Fix bug in cancel command; risk.rb (temporarily, for now)
        demonstrates functionality.

      * Re-enable access to the SetLogLevel request via the command
        set loglevel L, where L is in { System, Error, Warning, Info,
        Detail }, corresponding to the IB tws log levels of 1 -- 5, 
        respectively.  The set loglevel cmd corresponds to the old
        verb command, which was eliminated as part of an overall
        redesign of the command language.

      * Modify log format for requests, so that it follows the wire
        format as seen by the IB tws.  More precisely, leaving aside
        the log prefix, and the trailing product description tuple
        where applicable, the body of a log format request event
        is exactly the same as is transmitted over the socket to the
        tws *except* that nulls are replaced with vertical bars.  

      * Fix bug in market data request layout for futures.

      * The security-type, route, symbol name triple that is appended
        to selected log text output, such as market depth, has now been
        expanded to include: expiry for futures and options; and also
        right and strike price for options.  That is, the dot-separated
        routed product description is still one vertical bar separated
        field, and now has a variable number dot delimited subfields.

      * Efforts in this series have led to: 
          + improvements to order journalling;
          + fixes for order modify, see exs/risk.rb for details;
          + a substantially increased database symbol set, and
          + a temporary dead end for the port to cygwin.
        In more detail:

      * Provide, as part of order journalling, table inserts for
        CreateEvent and ChangeOrder records.  This is the first
        visible change of the redesign of order logging to be
        monotonic.

      * Fix modify order, and other order related bugs, so that the
        script risk.rb provides a working example of bracket orders.

      * Restore shim-database version consistency checking; shim will
        refuse to start up if database version is incorrenct.

      * Order keys now must be unique, so the exs/bind script
        does not work after the first use, and you should see
        exs/risk.rb instead for an illustration of how order keys
        may be generated.  This requirement is a necessary step
        for future support of cross-session order manipulation.

      * Update message version numbers as needed to allow
        use of the shim against IB~tws 884.6.

      * Modify program stderr banner to indicate two step startup:
        the first, the connect to database and IB~tws, and second,
        the successful load of all the abstract [not yet routed]
        Products.  There is a 4 or 5 second pause from the first part
        of the banner to the product load message, indicating database
        read completion for all tables used by the shim, including
        the 37963 records referred to in the load message.

      * The database has been extensively redesigned, and the shim
        updated correspondly.  A symbol is now uniquely defined by
        the security type, name, currency, and region; a static 
        product is a symbol not needing an expiry, right, or strike,
        along with the route set for that symbol, i.e. a stock, index,
        cash, or bond with the corresponding route set; future and
        option products are as you would expect, though with the route
        broken out into the table Expire; and abstract products are
        pairs consisting of the security type and a keys into one of
        Static, Future, or Option, that is abstract products are
        tagged keys.  The products are said to be abstract since they
        have route sets, not routes, where a route set is each of the
        routes accepted by IB for the symbol.  Contracts are concrete
        products, that is abstract products specialized by a valid
        route.  Please ask on the list if you have questions about
        the new database design.

      * Provide default route contracts for 30640 symbols.  Due to
        conflicts between net and IB~tws contract info for futures,
        there is still a manual step needed to enable futures, via
        mod/LocalFut.sql, to ensure that expiry values are correct.

      * Add index, stock, cash, bond, and future symbols, see the
        load files in sql/sym for details.

      * Rename the Country table as Locality, to reflect the IB practice
        of adding three letter codes to refer to regions of the world.
        The database version increments here, in part because a surprising
        number of attributes have changed width from two to three characters.

      * Add full set of exchanges used by IB to database, and revise stock,
        commodity, and product map load files to be insensitive to exchange
        renumbering.  Populate Exchange security type sets from underlying
        route set data.

      * Note that the script sql/bin/get_id.sql allows you to inspect
        the contract and product numbering for your database.  In
        directory sql, run:

            mysql -u code -h [host] testing < bin/get_id.sql > cids

        and inspect the resulting file to see what your current contract
        numbering is.

      * Add the csep (alias) csv option to escape double quotes and use
        commas instead of vertical bars as the message log format field
        separator.

      * For cygwin: add -lz to Makefile library link list, for the benefit
        of mysql; add Makefile mk.cygwin.patch to root directory, for
        typical /usr/local/mysql install on cygwin -- change as needed for
        your local setup; add lib/math.c, for llrint, and corresponding
        patch math.cygwin.patch, to replace library call to llrint with
        inline assembler; add lib/math.h, as part of a revision of header
        file structure to handle the use of macros in newlib to implement
        certain standard math library procedure; and remove cruft from
        src/functional.h.

        Cygwin doesn't supply llrint, and the shim needs it, so apply the
        patch math.cygwin.patch to lib/math.c, or just edit the #if to
        change the 0 to a 1.

        Cygwin doesn't supply getaddrinfo, and the shim needs it, so unless
        your Windows provides getaddrinfo, the shim still may not compile
        completely for a cygwin box.  In that case you can:  patch lib/inet.c
        with inet.cygwin.patch, and check that the shim now compiles,
        starts up, and runs up until the failed attemp to connect to the
        IB~tws; and then back the patch out, and come up with getaddrinfo
        somehow.

        Possibilities:

        + newer Windows have getaddrinfo, so it may just be a matter of
          the Makefile not knowing where it is.  If you can figure out
          what header file is needed, and add it to lib/inet-h.h, that
          may do the trick.

        + there are service packs for some older Windows that add in
          getaddrinfo, and if you can come up with such a service pack,
          that may do the job.

        + I realize other projects, e.g., Postgresql, have a replacement
          getaddrinfo.  Unfortunately those:
            - include thousands of lines of project-specific header file
              cruft, and
            - boil down to:
                . forwarding to the system getaddrinfo, if possible, or
                . falling back on the older, obsolescent-under-POSIX.1
                  gethostbyname(), which brings me to a shorter, more
                  direct solution:

        + cygwin + newlib do provide gethostbyname, and you can modify
          the code in lib/inet.c and lib/socket.h to perform the 
          connection using that function.

        + wait; from the announcement of the cygwin 1.7 development cycle,
          http://cygwin.com/ml/cygwin-apps/2008-04/msg00185.html :

            - IPv6 support.  New API getaddrinfo, ...
                                     ^^^^^^^^^^^
            - New APIs: asnprintf, dprintf, _Exit, vasnprintf, vdprintf,
              confstr, posix_madvise, posix_memalign, exp10, exp10f,
              pow10, pow10f, lrint, lrintf, rint, rintf, llrint, ...
                                                         ^^^^^^

        In any case, please let those of us on the list know about your
        experience.  Cygwin has proven to be our most difficult porting
        problem yet, and with the getaddrinfo problem, it's not a fully
        solved problem, so your feedback is especially important.

        Without your posting to close the loop, the time we've put into
        porting the shim to your architecture is wasted.


date: Wed Mar 19 2008
date: Tue Mar 18 2008
date: Wed Mar 12 2008
date: Tue Mar  4 2008
date: Fri Feb 29 2008	Leap year release
date: Wed Feb  6 2008
date: Fri Feb  1 2008
date: Thu Jan 31 2008
date: Thu Jan 24 2008
date: Thu Jan 17 2008
date: Tue Jan 15 2008
date: Fri Jan 11 2008
date: Fri Jan  4 2008
date: Thu Jan  3 2008
date: Wed Jan  2 2008
date: Fri Dec 21 2007
date: Wed Dec 19 2007
date: Fri Dec 14 2007
date: Fri Dec  7 2007
date: Fri Nov 30 2007
date: Wed Nov 28 2007
date: Mon Nov 26 2007
date: Wed Nov 14 2007

      * Add new PastFilter records to improve bulk history query processing;
        see the script exs/past.30.rb for details.

      * Provide bind operator to give symbolic names to order template
        records; see the script exs/bind for an example.

      * Fix latency bug in reader as part of adding Reader_Fsm class, and
        splitting parse state between Reader_Fsm (top level) and existing
        Cursor (recursive descent) classes.  Eliminate routing override for
        futures, that used symbol home; this will now require that you fix
        contract routes for futures in LocalSet.sql, that is, this change
        both: improves flexibility for futures routing, beyond the symbol
        home; and exposes flaws, if any, in your contract data, where the
        route was previously ignored for futures.
        Fix duplication-of-events-to-syslog bug.

      * Modify command syntax, according to examples in directory exs,
        so that tws-request-generating commands have a common verb-selector
        form.  Modify the wildcard contract info command to accept a
        symbolic argument in place of the symbol id.  Perform table lookup
        for symbol and exchange strings, with the result that unknown
        exchange names will lead to syntax errors; this is by design.
        Please inform the list of new exchange names, so that we can update
        the database.

      * Fix for problem with orders in mapping of lineitems to orderids,
        which led to individual lineitems having multiple orderids.  This
        problem dates back about a month in this series, to a re-architecting
        of the code for cmd-req mapping, and has been alluded to both
        on the mailing list and, in this file, in earlier stanzas of this
        series.  Related changes to the cmd-req mapping code will continue,
        but the fix is being pushed out now for the benefit of those who
        have been waiting.
      * There are a number of changes to the rules controlling the order of 
        command interpretation which should make it easier to write command
        scripts.

        There has always been a distinction between internal commands, which
        are executed immediately in order to change the state of the shim,
        and upstream commands, which map to requests once interpreted, and
        are delayed on a queue according to the IB tws api rate limit of at
        most one request per 20 msec.  Now, the category of queued commands
        has been enlarged slightly, and the rules for queuing changed
        somewhat.

        The changes are, in brief, as follows: two internal commands, exit
        and wait, are now queued with the upstream commands; and all queued
        commands are executed strictly in order.  The new category of upstream
        and queued internal commands are referred to collectively as program
        commands, to indicate their use in scripting, while immediate
        internal commands will be called process commands.

        The rules for dequeuing upstream commands used to be somewhat
        involved, depending on a number of capacity restrictions imposed
        by the IB tws api.  Commands that reached the queue head and whose
        related requests were not yet ready for transmission due to market
        data or market depth subscription limits were requeued, in the
        [typically vain] hope that subscription cancellations would free
        up market data lines.  Such requeuing necessarily changed the order
        of execution, and made it difficult for downstream programs to track
        the feed queue state, and to know what subscriptions would be
        transmitted next.

        Now, when excess market data or market depth subcription commands
        reach the queue head, they are summarily dropped.  After some
        consideration, it's become clear that there is nothing else useful
        that the shim can do with them.  It's up to the downstream to manage
        subscription resources; the shim will honor subscription limits, but
        is not otherwise responsible.

        So, all program commands are executed in order, either evaluated
        for their internal side effects (wait and exit); or mapped to the
        corresponding request, and that sent; or dropped with a message
        if capacity limits don't allow upstream transmission.  Exit, wait,
        and all the upstream commands are classified as program commands.  

        The process commands are interpreted immediately, out-of-band;
        they are: wake, quit, help, list, and ping.  Note that wait is no
        longer a process command, but is now executed in order, as it
        reaches the head of the feed queue.

        The existing internal command "quit" has been classified as a
	process command; it takes effect immediately, even if there are
	commands in the feed queue not yet mapped to requests.  You would
	typically use quit interactively, to achieve immediate termination.
	As mentioned above, exit, which is a program command, should be
	used for scripting, and whenever you want to make sure that
	previously entered program commands are executed.  So, whenever you
	use "quit", there is a race between program termination, and any
	previously queued commands; while, whenever you use "exit", there
	is a possibly large delay as previous wait commands count down.

        The reclassification of the wait command so that it is queued has
        increased its functionality.  It not only delays termination, (e.g.,
        for subscribe, wait 3600, exit)  but also delays upstream commands
        that follow, e.g., create order, wait, submit order.  This is
        primarily useful for test scripts, or to hide any races in the
        IB tws order processing.

      * many minor changes to command processing, including some renumbering
        of commands
      * Note (080117): cancel is currently broken.  comment and whitespace
        handling for input language is improved, but still not completely
        correct.
      * The cmd input language of the shim has been modified to include
        comments, to allow scripting.

         +  Testing and data collection is simpler, since script names can
            be provided on the cmd line.

         +  You may now use the shim as a "shebang" style interpreter,
            since execve will pass the script name in as the last argument.
            Executable scripts beginning with an awk-style shell bang line
            will run as you would expect: 

                #./shim -f

        So, there is now an optional file spec, which, when it occurs,
        is *prior* to the required mode, what used to always be the first
        cmd line argument.

        E.g., with the shim executable in the current directory, both of the
        command lines below are valid, with the first accepting command
	input from stdin, and the second from a file named "input.tsd".
	(The symbol "---script" is a synonym for "-f", and either will
        work.)

       ./shim                     --data file save cout
       ./shim ---script input.tsd --data file save cout

      * Add OrderDelta table, and increment database version to 1.31
      * The format of the wire (order) command has changed:
         + move timeout attribute from wire (order) command to xmit command;
           this attribute is not yet in use, and the xmit cmd is not fully
           implemented, so the only effect is to decrease the number of args
           in the order command.
         + move request type (Submit, Modify, etc) to the front of the
           argument list, ahead of the line item index and order type
           (LMT, MKT, etc) so that request type is now first, with the
           result that the lineitem and modifiable order attributes are
           together, so that the parser can build up an OrderDelta
           object.
        As always, the examples in bin/includes describe the current format.
      * delete ExecFilter table from database, and increment the db version
      * implement the exec command, to query for execution reports for the
        current day; format is:
            exec Sym Time;
        where Sym is the name of the underlying, and Time is of the form
        hh::mm::ss
      * refactor cmd-to-req mapping, that is text of feed.c, and incidently:
        fix bug in info (request known contract details) command; and
        add wild [card] (request abstract contract query) command;
        see bin/test for examples
      * fix bugs in news and open orders message parsing; this daily is
        much to be preferred to others from earlier this month.
      * update YM front month in regression test (see bin/includes do_query)
      * begin modifications to implement 64-bit order ids; this is not
        a full release, but is rather so that interested parties can
        follow along in the code.
      * fix bug in order cancellation, see mailing list for details
      * re-enable the test modes, e.g., --msgs, as part of error text
        deconstruction; and re-enable reg_echo test, and its warnings
       (error msg trace) sub-test; other sub-tests are still broken,
        and so commented out.
      * Fix next command, add regression test for it, and cleanup send
        command and request agent lookup
      * This release begins a new series for no better reason than that a
        month has gone by since the last.  That delay is due to juggling,
        including:

          + An outdated roadmap, and incomplete efforts to update it,
            though we're not yet ready to release the new one.  From
            the current ROADMAP text:

        The previous version fell behind actual development, and so has been
        elided.  Furthur roadmap information will likely be part of the
        manual, although it's not there yet.  For now, ask on the list if
        you're curious.

          + Ongoing work on history, also not completely finished.
            Currently several critical bugs have been fixed, the
            related database tables have been changed, and the syntax
            of the history (past) command has been changed to require
            the end date on the command line, making it much more
            flexible.  (Previously the end date was stored in the HistoryTag
            table, for use in labelling query answer sets; that feature
            proved not to be of much use, and so has been sacrificed.)
            The history command currently accepts only textual dates, e.g.:
                Ymd_T(20070918  11:00:00)
            and is still unable to accept epoch-based dates.  Also, the
            insert feature for the HistoryBar table is currently broken.

          + The next significant feature planned for our internal roadmap,
            is order tracking, to provide a timeout and resulting log
            message if orders are not processed by the IB tws.  There have
            been significant internal changes to the code as preparation
            for this feature, but there is no change to shim functionality
            yet visible.

          - Currently an Ubuntu user has posted to the list noting that
            the shim will not compile for him.  We don't know why ---
            it compiles fine for us on our Debian box --- and we're 
            interested in any more information others may have about
            their own problems, if any, using the shim on Debian systems.
            Those who wish to test on Debian should start with this
            newest release.

            
date: Thu Oct 11 2007
date: Fri Sep 28 2007
date: Mon Sep 24 2007
date: Fri Sep 21 2007
date: Thu Sep 20 2007
date: Fri Sep 14 2007
date: Mon Sep 11 2007
date: Fri Sep  7 2007
date: Fri Aug 31 2007
date: Wed Aug 29 2007
date: Mon Aug 27 2007
date: Thu Aug 23 2007
date: Wed Aug 22 2007
date: Tue Aug 21 2007
date: Mon Aug 20 2007
date: Mon Aug 13 2007
date: Fri Aug 10 2007
date: Thu Aug  9 2007
date: Fri Aug  3 2007
date: Thu Aug  2 2007

      * add {s,d,w,m,y} unit suffix to PastFilter database record, and
        use the suffix in formulating history requests; see sql/load.sql
        load script and bin/history regression test script for usage
        example.
      * fix contract lookup for history messages to use tick rather than
        the [supposedly] related command.  This fixes a long-standing bug
        whereby the messages resulting from multiple, close together
        history commands would be labelled with the contract of the last
        command.  Note that the related problem, of the history tag, and
        so bar size, also being determined by the last of the commands, is
        not yet fixed, so that multiple successive history queries should
        still have a common tag, or else be fed to the shim with
        intervening 11 second delays.
      * fix history inserts, for both epoch and ymd_t message formats,
        so that either kind of HistoryTag record can be used to control
        history results.  Re-enable periodicity, which had been commented
        out earlier in this series.
      * add the PastSlot key value name, for use in controlling the interval
        between history queries; useful values are 10, the absolute minimum,
        and 11, the recommended value if multiple history (past) commands
        are to be enqueued in the shim.
      * finish modification of history query command input to provide
        end date on command line rather than from database.  This finishes
        a change started near the beginning of this series, so that you
        should download this release if you have tried any others since
        the beginning of August.  The new syntax moves the end date value
        from the database to the command line, adding an additional argument
        to the end of the history command.  E.g., the commands:

            past add  15  6 now;'
            past add 180  6 Ymd_T(20070918  11:00:00);'

        ask for 1 minute history bars for AIG up to the moment, and YM tue
        of last week at 11am.  Posting of epoch based dates does not work
        for now, so use PastFilter records (the uid 6 above) that select
        the textual date format.  Note that the end date must be a date
        string, the Ymd_T case, or else a blank (indicated by "now"); the
        IB~tws api does not accept epoch-based dates in the request format.
        As part of this change, delete the end attribute from PastFilter,
        and the database version increments as a result.
      * update YM front month in regression test (see bin/includes do_query)
      * add help mode cmd banner (simulated help help), and fix several
        other help mode bugs.
      * change help mode to accept non-send command input, i.e.:
           help, wait, wake, quit;
      * implement command-specific help command for most of the other
        commands; try: help help; for details.
      * expand FutDetail load from quarterly to monthly values, incidently
        renumbering the related contracts.
      * add a few symbols to Miscellany.sql, ProductMap.sql, and
        LocalSet.sql,
        in response to a user request; this gives (slightly) better European
        coverage.  Feel free to send in more new entries for the symbol
        tables.  And thanks to the user who sent us the data.
      * add architecture-sensitive logic to build the clock timer header
        file for Mac OSX ppc and x86_64
      * modify much of the error trace output to use cerr in place of
        stderr, in order to avoid format string integral-type word length
        problems between i86 and ppc, and merge the 8.31 mac ppc fork with
        the main line; please use this release in place of the 8.31 tarball
        if you downloaded it.
      * the Fri 8.31 release is for MacOs only, and there only for
        experimental use --- it is a *known* problem that it will not compile
        for Linux
      * the Wed 8.29 release is not for general use -- please don't download
        it unless you know what you are doing.
      * In the database load files, change the symbol SUNW to JAVA
      * The shim for Mac OS X is broken, and we're working on it.  The
        critical problem is the inline assembler to read the processor
        time stamp counter, which we're planning to extend to handle PPC.
        On a related but minor note, the version of g++ on Mac OS X 10.4.10
        complains about %d (int) formats for natural, that is unsigned
        numbers, and the instances of such in the code have been fixed
        with the 070823 release.  Also, old versions of g++ are unable to
        properly compile some of the template related code --- this is a
        known problem for, e.g.,  g++ 3.2.3 --- so you will need to upgrade
        your mac to g++ 4.0, if it's not already there. 
      * add DAX december future to LocalSet, for the benefit of European
        users.
        Please, if you are from across the pond, let us know how we can
        improve the default contract set in LocalSet.  We're not ignoring
        you, we just have trouble testing this stuff, since we don't have
        access to all the European symbols.  Please join the list, and speak
        up.  Although the future works, a tick subscription for the plain
        index doesn't; maybe problems with our IB tws account subscriptions,
        don't know; please let us know if it works for you.
      * rate-limit history queries to at most one every 10 seconds
      * modify the "pane" (synonym "window") option to close the display window
        when the shim exits; if you want the window to hang around, run the
        bin/tail.window script directly, prior to startup.
      * comment out most unfinished parts of the manual in the ps, pdf, and
        html forms; the *.tex sources for the partial chapters are still
        included in the tarball, so minor changes to manual.tex will give
        you the whole thing.
      * edit Ch. 3 of the manual, "Installation, Setup, and Startup", and
        add screenshots showing tail'd log output in a separate konsole
        window, via the pane (window popup) option
      * rename the "look" cmd, which serves as the dbms append operator,
        and reads in new records, as "read".  Eliminate bulk as a command
        and option name, leaving the bulk subscription command name as "load",
        and the name of the cmd line option forcing an init-time load, also 
        to be "load".
      * make numerous minor changes to cmd line option handling logic,
        including updates to help text, so that ./shim --help is the best
        current documenation of the available options
      * make file output to the ShimText file of the current directory the
        default, so that there is no longer any need to request it via the
        Save option, and disable syslog output as default, requiring instead
        that the logd (or its synonym, syslog) option be provided explicitly
        requesting this feature. Default was syslog and no ShimText, and is
        currently the opposite, ShimText and no syslog.
      * trim space cost of manual in several ways, dropping tarball size back
        to 3+ meg; again, the manual is still the main offender here
      * add filename and timeout key-value pairs to config file language,
        password key-value pair, ditto, and password value to dbms command;
        see Ch. 3 of manual for details
      * fix bug in init option (prompted initialization commands at startup)
      * modify Makefile to change compilation default from distcc to g++;
        reversing the patch mk.patch will recover the original distcc definition
        for CC.
      * work on bind command
      * continue refactoring JobQue 
      * reduce routing latency


date: Tue Jul 31 2007
date: Fri Jul 27 2007
date: Tue Jul 24 2007
date: Mon Jul 23 2007
date: Fri Jul 20 2007
date: Thu Jul 19 2007
date: Wed Jul 18 2007
date: Tue Jul 17 2007
date: Fri Jul 13 2007
date: Wed Jul 11 2007
date: Mon Jul  9 2007
date: Fri Jul  6 2007
date: Fri Jun 29 2007
date: Mon Jun 25 2007
date: Fri Jun 22 2007
date: Thu Jun 21 2007
date: Tue Jun 19 2007
date: Mon Jun 18 2007

      * relicense sources under the GPL version 3
      * saving event text to a local file named ShimText is now automatic,
        rather than being controlled by the save option, and that file is
        set for line rather than full buffering, so that latency is much
        reduced.
      * in addition to posting history messages to the database, the post()
        step also, 1: writes the message to a file named after the contract, and
        having the extension hql, e.g. FUT.SMART.YM.hgl, and, 2: if the history
        filter db record is periodic and has a non-null string script attribute,
        runs that script with the file name on the command line.  Currently the
        script bin/periodic demonstrates this feature, via the shim calling
        the hql2ps script, and if you have a postscript window open watching
        the resulting output file, here FUT.SMART.YM.ps, you'll see the image
        be updated every time a history query is answered (this follows from
        the periodicity of the query, which existing feature was broken earlier
        in the series, and is fixed here).  As part of the implementation for
        the periodicity-script feature, detail lines, and in particular history
        detail, now include a datestamp; whether or not this time value matters,
        it in any case simplifies downstream log line parsing, e.g. for the
        hql2ps script.
      * fix command parsing error recovery and trace/error message output;
        command input should be able to correctly parse the very next correct
        message after one including a syntax error, and the explanatory message
        for that error should be reasonably understandable; it is both more
        concise and less cryptic than before.
      * much work has been done on connection parameter input; earlier tarballs
        in this series [prior to the last date above] do not necessarily include
        all of the following features; as is generally the case, you should 
        prefer the last version of a series.
      * there is now a default .shimrc file in the tarball root directory
      * regression scripts now pull connection parameters via autoload
        from the bin/kv_pairs file
      * update old-style ipv4 ip-quad based connection logic to use
        getaddrinfo(), which is simple, general, reliable, and flexible.
        See RFC 2553 if you are interested.
      * connection parameters may be provided via any of the following means,
        in increasing order of priority: 
         + 0 the hardwired connection parameters in data.c
         + 1 a .shimrc file in $HOME                    (see below for details)
         + 2 "   "      "   "  the current directory      "    "    "    "
         + 3 dbms and feed commands prompted for, once given the init option;
           see the init_opt function in the regression test scripts for an
           example of the command syntax.  There is a total timeout of 20
           seconds for the two commands.
      * the shim accepts a reasonably flexible format for .shimrc files:
         + a .shimrc file consists of name-value pairs such as follows:
DbmsName        mysql
DbmsHost        somehost
UserName        shim
TableSet        testing
FeedName        tws
FeedHost        otherhost
FeedPort        7496
         + pairs may be in any order
         + not all pairs need be provided; if the .shimrc file is incomplete,
           missing values are filled in from the defaults defined in data.c
         + the FeedHost value may be an ip address or a dns name
         + each pair must be on a line by itself
         + the file must have left-aligned pairs only, that is without
           comments, blank lines, or leading whitespace.
         + the Password key is currently ignored
         + for the forseeable future, only "mysql" and "tws" will be accepted
           as the DbmsName and FeedName values.  Sorry, no Oracle for the time
           being, see the manual for more on this.
      * as mentioned above, the init option is now supported.  it triggers
        prompts for both dbms and feed connection parameters at startup;
        for examples of the the dbms and feed commands, see the the
        regression scripts, in particular bin/includes.
      * as part of ongoing work to allow more flexible contract lookup, the
        info cmd, which is being used as a guinea pig for the new features,
        produces various kinds of cruft on the stderr -- don't worry about it.
      * modify the syntax to the info (req contract data) command, deleting 
        the dummy subscription add/del arg prior to the contract id, adding
        the trailing flag for include-expired contracts, and bumping the
        version up to eliminate a nag message; see the bin/includes regression
        test for examples.  The fixes depended on an earlier change, to 
        separate ReqConData from the four subscription request types; note also
        that the info cmd used to be named data, and though was renamed to
        remove ambiguity with data mode, in any case still requests contract
        information.
      * the --help feature screens are increasingly out-of-date, and will be
        updated as part of future work on the commands chapter of the
        tutorial.  In particular, the --cmds, --reqs, and --msgs test modes
        have been effectively disabled.
      * add new stage to singletree construction; before/after is:
         + old: component, constants, stateful singletons
         + new:   ", early   "      ,   "        "       , late constants,
           with connection initialization delayed to the 4th stage
      * merge CmdQue and Feeder, and change from request queueing to command
           queing, so that tick id assignment is delayed as long as possible,
           to ensure that ids are sequential even when queued events are
           reordered.   Tick id construction has also been refactored.
      * refactor bulk subscriptions
      * fix bugs in news command
      * fix doc latex bugs due to changes in latex hyperref package
      * initialization bugfix


date: Fri Jun  1 2007
date: Tue May 29 2007
date: Fri May 25 2007
date: Fri May 18 2007
date: Thu May 17 2007
date: Tue May 15 2007
date: Mon May 14 2007
date: Fri May 11 2007
date: Fri May  4 2007
  
      * supplement hardcoded dbms params with input from cwd .shimrc name value
        pair file, e.g.

DbmsHost	server
UserName	shim
DataBase	testing

        for now, no comments, blank lines, or password -- to be added later
      * split Term derivation hierarchy out from obj to ast namespace, and
        eliminate afd subdirectory and contents by use of templates for kind.[ch]
        Due to extensive changes to sources, you probably want to follow along
        on this one.
      * modify shim to work with tws 872.6, which includes a forced upgrade
        to accept new, higher version status and portfolio messages.  The
        regression tests for these seem to work, although I'm hoping for a
        new set of api sources soon, so that I can check exactly what IB is
        doing to the wire format for these messages.  The version update must
        refer to some change in the api, I just don't know what at this point.
        You'll probably find that you have to follow along on this change, sorry.
      * by the way, the reason the tarball has jumped so in size is mostly
        due to inclusion of the latex2html output, so that now you have both
        the pdf and html documentation
      * more work on manual; chapters 1 and 2 have reached final draft status;
        also, now I can get back to coding the shim; hopefully, I can balance
        coding with documentation for awhile
      * more work on manual; see Table 1.1 in order to track added work
      * fix template istantation problems via code inlining.  The code
        already compiled fine on Centos, and now does fine on Debian as well:
          g++ (GCC) 3.4.6 20060404 (Red Hat 3.4.6-3)
          g++ (GCC) 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)
      * include online docs in tarball (latex2html output of manual.tex);
        they are also available at http://www.trading-shim.org/doc/
      * modify setup.sql script to set utf8 as the default table encoding
      * modify release packaging to preserve sym links, which is useful
        for a number of reasons, including more up-to-date docs, but
        which may be disconcerting to some long time users who are used
        to the previous layout.  The sym links currently are:
          src/bin     ->    ../bin
          doc/hdr/*.h -> ../../src/*.h
      * insert new chapter 6 about protocol record formats into manual
      * the program name (shim), mode (data or risk), and version prefix
        have been dropped from the output log file format, removing three
        fields from system log lines and the ShimText log file, and the
        attribute counts in bin/log.filter have been modified accordingly.
      * the version banner has been modified from, e.g.:
          4|100| 5|# |4|100|5|*******************|
          4|100| 5|# |4|100|5|version|0.30|999999|
          4|100| 5|# |4|100|5|*******************|
        to:
          4|100| 5|# |4|100|5|****************|
          4|100| 5|# |4|100|5|0.31|999999|data|
          4|100| 5|# |4|100|5|****************|
      * modify scanners to allow multibyte characters to pass through as text
      * include documentation.  NOTE: it is --- and by its nature will
        continue to be for quite a while --- an *incomplete draft*.  There
        is much more writing to be done, and continued changes to the code
        needed to reflect the specs as they are documented.  You will
        need the latex2html and graphviz packages to build the docs on your
        own, but a pdf is included as well.
      * many internal changes, including many partially completed features,
        so source diffs may be hard to follow at this point; the docs are
        the main reason for the release.


date: Wed Feb 28 2007
date: Tue Feb 27 2007
date: Mon Feb 26 2007
date: Wed Feb 21 2007
date: Thu Feb  8 2007
date: Fri Feb  2 2007
      * revise singletree construction to eliminate shared edge to Timer,
        and modify singletree inclusion hierarchy doc (singles.dot) accordingly
      * add explanation of naming conventions, in "name_rules.h"
      * verify, via regression tests, that shim runs against TWS 867.4
      * fix gcc 4.x syntax error to allow Debian users to follow bleeding edge,
        else they would have to hang back with a mid-Jan release.  The newest
        code, e.g. eval command, version-specific requests, etc., is very much
        in a state of flux.  There will be a better upgrade candidate coming
        out before the end of this series.
      * modify library typedefs to *begin* to accomodate OSX
      * Add open command to request open orders
      * Fix bug in command logging, and missing next command in data mode 
      * Add cmds option to help mode to list command syntax and semantics
        Note: data command has breakage, and help command is not yet working
        Note: repairs to command logging break echo cmds regression test --
            false positive, needs some work on test logic


date: Wed Jan 17 2007
date: Fri Jan 12 2007
date: Fri Jan  5 2007
date: Thu Jan  4 2007
date: Wed Jan  3 2007

      * Add parent-id related (atomic) bracket order regression test to
        test scripts; see risk.sql, and the modified function wasteful
        in bin/includes, for the sample sql and shell script code.
      * Fix parent-id related bugs in order wire formatting
      * Provide contract data query command, request, and message support,
        along with a regression test to demonstrate the new functionality.
      * Replace manifest constant for currency in Contract of "USD" with
        index to Currency table, and revise database as needed to support
        this change, with the consequence that database recreation is
        mandatory.  As a result of this fix, users should feel free to
        update the currency literal values for the contract descriptions
        in sql/mod/LocalSet.sql with other valid currency abbreviations,
        that is strings that are from the Currency table and supported by
        the tws.
      * Add request translation for market depth, to implement the existing
        book add subscription command.  For those readers who wonder about
        interpretation of the resulting data:
          Shim log messages share a common initial prefix consisting of
        date, host, session, and timing information.  This common prefix
        is then followed by a variable portion consisting of information
        derived from the tws api message, what I'll call the payload.
          The payload, again as is generally the case, consists first of
        the tws message text attributes, in order and with the printable
        graphical text as received, with the one exception that tick ids
        are rewritten with the related contract indices.  The ascii nulls
        used as attribute terminators are also rewritten to vertical bars,
        and whitespace is added in some places to improve field alignment.
        I call this middle portion of the log text the tws message body.
          The final portion of the payload and log text is an optional
        portion I call the augmented text, providing character translation
        for some of the numeric codes found in the tws message body.
          So, for market depth, the tws message body consists of the
        message tag, 12; version, 1; contract id, in place of the tick id, 
        e.g., 166 for YM; position index, for, say, a query on ten rows of
        market depth, a natural number between 0 and 9; the operation code
        and side code, about which more in a moment; and the price and quantity.
          Following the message body, the augmented text provides translations
        for: the side code, 0 or 1 for ask or bid; the operation code, 0, 1,
        or 2 for insert, update, or delete; and for the contract id, a symbol
        tuple, currently just the security type, route, and symbol.  By way of
        example, the initial three rows of YM as the subscription starts might
        be:

        12|1|       166| 0|0|1| 12533.0|     2|bid|insert|FUT.SMART.YM.
        12|1|       166| 1|0|1| 12532.0|     9|bid|insert|FUT.SMART.YM.
        12|1|       166| 2|0|1| 12531.0|    19|bid|insert|FUT.SMART.YM.

        12|1|       166| 1|0|0| 12536.0|     2|ask|insert|FUT.SMART.YM.
        12|1|       166| 2|0|0| 12537.0|     3|ask|insert|FUT.SMART.YM.
        12|1|       166| 2|0|0| 12537.0|     3|ask|insert|FUT.SMART.YM.

        The above text could be extracted from system log text using the
        bin/tsd.filter script, or if the save option was used, directly
        from the file ShimText found in the current directory.
        Note that market depth cancellation is not yet working; this should
        be fixed soon.
      * Add OPG (at-the-opening, for MOO and LOO orders) and FOK (fill-or-kill)
        to the time-in-force finite domain type in both the dbms and the shim,
        also reordering the numeric codes so that the dbms and shim must be
        upgraded in lockstep. Note: although these codes are accepted by the
        GUI, we have not yet verified API support.


date: Fri Dec 22 2006
date: Thu Dec 21 2006
date: Wed Dec 20 2006
date: Tue Dec 19 2006
date: Fri Dec 15 2006
date: Thu Dec 14 2006
date: Fri Dec  8 2006
date: Thu Dec  7 2006
date: Tue Dec  5 2006
date: Mon Dec  4 2006

      * Modify save option to, in addition to creating a message image file,
        also create a request image file.  The files are named tws2shim.bin
        and shim2tws.bin, and appear in the current directory.
      * implement news bulletin request/cancel
      * add new tick types 13--37 from api 9.00
      * implement the acct command, AccountData request, and KeyValue (6)
        message, and provide a simple type tuple for the Portfolio (7)
        message, in order to provide access to AccountData information
        via the log stream.
      * attempt shim with tws 865.5, with success -- no syntax errors
      * increase handshake timeout from 1 to 2 seconds to reduce the
        likelyhood of handshake failures at startup.
      * rename the test database from trading to testing.
      * patch bin/includes to emphasize the need to edit the literal
        bindings of the UserName, HostName, and DbmsName variables.
      * attempt shim with tws 865.2, with success -- no syntax errors
        FYI, the 864.* syntax errors occurred with plain old price and
        size market data, and the bug[s] manifested in a number of ways,
        including insertions, deletions, and substitutions, as if some
        race condition was garbling the data.  Since the shim is singly
        threaded, and the binary traces used to diagnose the problem
        (created using the save option) were written directly to disk
        immediately after the socket read whenever there was a positive
        return value, and using the same buffer,  I'm convinced the race
        condition was in the tws, and not the shim.  [See the logic for
        the Intake::sys_read() and InpBuf::read() functions, in "src/time.c"
        and "lib/call.c", respectively, if you wish to check this claim.]
        It's particularly suspicious that 862.9 and, at least up intil now,
        865.2 have not exhibited the problem, at least for us with our
        testing environment.  Bottom line: go ahead and upgrade to 865.2,
        and if instead you stick with an 864.* series tws, don't be surprised
        if there are errors.
      * attempt shim with tws 864.4, 864.6 and 864.7
        the status of 864.4, 864.6 and 864.7 is uncertain -- either the
        language has changed, or we have a bug, or the 864 series has a
        bug (most likely, a combination of these).
      * replace bad tarball from yesterday (and use the -X argument to tar
        to skip transient .swp files, the disappearance of which can
        evidently cause tar to leave big chunks out of the tarball)
      * build the Symbol (previously named Definition) table from the tables
        Underlying and ProductMap, and Underlying from distinct tables
        Stock, Currency, and Miscellany
      * use symbolic, or key value, literals in place of numeric contract
        literals in defining the LocalSet table
      * eliminate the Security table from the dbms; there is now a direct
        dataflow from Underlying to Symbol to Contract
      * add some non-us home symbols to Stock.sql, Miscellany.sql,
        and ProductMap.sql as examples of how these tables can be modified
        for the use of those outside the US


date: Tue Nov 14 2006
date: Mon Nov 13 2006
date: Thu Nov  9 2006
date: Tue Nov  7 2006

      * the save and pipe command line options have been implemented; note
        that full testing of the save feature will require the playback
        feature to be implemented, and so that will probably be added soon
      * regression tests require pdksh rather than ksh, due to variable parsers
      * now compiles on Debian with g++ --version:
          g++ (GCC) 4.1.2 20061028 (prerelease) (Debian 4.1.1-19)
      * the wire command verb has been renamed to the more intuitive "order"
      * absolute [limit price provided from downstream] LMT orders work; much
        other order functionality added as well.
      * order Create, Submit, and Modify work as intended.
      * repairs to achieve successful gcc4 compile (code still works under gcc3): 
        + dumped operator new() defs in the global namespace (sorry, no choice)
        + add template<> text to all type tag explicit specializations
        + move all template type procedure defs from alias to true namespaces
            e.g., from tag:: to namespace ObjectTypeSystem ... , and yes, this
            makes the sources harder to read; again, sorry, no choice
    *** WARNING: if you are attemping to modify the sources, you absolutely
            want to upgrade here.  The patch is large, more than 2k lines in
            both src and lib, and yet has virtually no semantic impact, that
            is the procedural code is very similar to Friday's release.  In
            any case, you'll have no choice once gcc4 upgrades become more
            prevalent, e.g., probably in December for Centos.


date: Fri Nov  3 2006
date: Thu Nov  2 2006
date: Wed Nov  1 2006

      * rename --exec mode as "--risk"
      * remove default load of SubRequest, and add load option for force load
      * fix tick mode to eliminate the wire command from its listener start symbol
        known bug: unknown command leaves command listener confused
      * implement Cancel form of wire command, to be used with Submit form
      * fall back to 862.9, due to intermittent syntax errors
        during 864.4 market data message processing
      * Port shim to tws 862.9, breaking, for now, backward compatability
      * integrate regression tests to one large functional test against
        a common process instance of the shim, exposing some problems
        due to the allocation and numbering of tick ids, again a known problem 
      * add tick and past commands as market data and history query alternative
        to loading subscriptions from the database; as a known bug, they
        interact badly with one another for now
      * Port shim to tws 864.4


date: Thu Oct 26 2006
date: Wed Oct 25 2006
date: Tue Oct 24 2006

      * Port shim to tws 862.9, breaking, for now, backward compatability
        with the previous development platform, 855.5
      * Improve the quality of the regression test scripts, for the increased
        convenience of those who wish to adapt them for external use.
      * Fix intermittent but fatal handshake problem; please consider upgrade
      * Significant changes to the order protocol:
        + the read command has been renamed to be the wire command
        + the wire command is the event that triggers an order, and
            includes most of the variable part of an order; the
            corresponding parts of the old ApiOrder table, now
            OrderContext, have been refactored out.
        + there are new tables LineItem, OcaGroup, and Position
        + the OrderJournal (old OrderStatus) table has been expanded,
            and writes to it by the shim include the new, variable parts
            of the wire command.
      * See the bin/order regression test to see how the new protocol works,
          and as always, in directory sql, read the various .sql scripts
          called by the master create.sql script, to find the create table
          definitions.


date: Wed Oct 18 2006
date: Tue Oct 17 2006
date: Mon Oct 16 2006
date: Fri Oct 13 2006

      * The order regression script provides a vehicle for testing orders;
      it enters a pair of buy and sell orders at market.  Order status
      updates, order modification, and order cancellation do NOT work.
      THIS CODE IS FOR TESTING ONLY.  USE ONLY WITH THE DEMO OR PAPER
      ACCOUNTS.  DO NOT RUN THIS VERSION AGAINST A REAL ACCOUNT.  You
      have been warned.  Mon -- includes bug fix for failure to read
      OrderStatus; Tue -- tick id base is now set dynamically by NextId
      at startup, before any market data requests or orders are sent;
      Wed -- fix type definitions for OrderStatus, input and output of
      OrderStatus records, and not so incidentally the place order
      transaction.


date: Fri Oct  6 2006

      * Various internal changes on the way to orders


date: Mon Sep 25 2006

      * Improved history:
        + robust history parsing
        + practical history insert of query results into the dbms
        + periodic (repeating) history queries


date: Fri Jun 30 2006

      * Numerous dbms changes, leading to a bump in dbms version:
        + changes to the dbms setup to maintain the ACID property
        + numerous changes to history-related tables 

      * Continued bugfixes, including:
        + an important fix to the dbms load routines
        + portability fixes giving successful compilation on FreeBSD 6.1


date: Fri Jun 15 2006

      * Mode names have changed; type "shim --help" for details

      * Continued bugfixes


date: Wed Jun  7 2006

      * Parser redesign, and continued bugfixes, including a hang on
        reader syntax errors.


date: Fri May 19 2006

      * There are many bugfixes, some for serious problems.  You should
        consider upgrading to this version of the trading-shim.

      * The shim has been validated up through tws version 857, and is
        known *not* to work with 858.  You must use a tws version prior
        to 858, e.g. 853.8, 854.8, 855.5, 856.5, or 857.5.  Most recent
        development has been against 855.5.

      * [Added note:] As of Tue Oct 24 2006, the shim works instead
        with 862.9, and the changes are *not* backwardly compatable.

      * The bind database operator has been renamed as look.


date: Wed Apr 26 2006

      * There is a limited capability to insert history results into
        the database, so that it is not necessary to extract and group
        history detail records from the log.  See the regression test
        bin/history for an example; note that, for now, the previous
        history set must be copied out to another table before each
        additional history query.

      * The FAQ navigation link has been populated with entries for
        other software packages used in development or operation of
        the shim.

      * Add One Chicago symbols to the default database load scripts;
        see load.sql for details.

      * Bug fixes:
        + market data price and size message quantity value.
        + hash table block-doubling code.


date: Mon Apr 10 2006   Initial release of the trading shim

      * database creation and connectivity is functional
      * handshake and connection to the upstream tws api server is functional
      * the input command channel is functional, and will be extended
      * the query result stream to the system logger is functional,
        and will be extended
      * the tick stream via the api market data request is functional
      * limited history data support is provided, and will be extended
      * orders have been blocked out, until more design and
        implementation work is completed
      * infrastructure includes a website, mailing lists, and bugtracker

        The release is comprised of, considering the C++ sources, 25k
        lines, split between 19k in the application domain, and 6k of
        components.  There is an extraordinarily large ratio of type
        information to executable source; the application domain .c
        files consist of only 7500 lines.

        The components include containers, a type-safe implementation of
        object stacks, and logic variables bound via matching.  Program
        initialization constructs a single-tree, a singleton object
        including all program state and constants, so that initialization
        dependencies are explicit and state coupling is drastically reduced.

        Objects are categorized by their type names to indicate the related
        binding pattern, for simple objects either functional, logical,
        monadic, or stateful, and in the case of compound stateful objects,
        either state machine or push-down automata.

	Most objects are pure constants, fully defined at the moment of
	creation and unchanging after that.  Others provide the carefully
	limited use of imperative state where this would allow concise, easily
	understood code; this careful balance of declarative and imperative
	programming is the hallmark of multiparadigm style.

	The careful control of shared state, whether via elimination, in the
	use of functionally or logically constant objects; limitation in scope,
        by the replacement of all global state by singleton parameters; or 
	limitation in time, by the use of factory patterns, drastically reduces
	the need for testing, to the point where functional testing can replace
        much of the unit tests that would otherwise be required.

	More precisely, container components still need careful testing, as one
	would expect, at least an order of magnitude more than application
	domain code.  In the application domain, stateful singletons still give
	trouble, and absorb the bulk of debugging effort.  The pure constant
	products put out by factory objects, however, have proved to be
	debug-once code; after the constructors are working the first time,
	they remain trouble-free.

        As a result, a limited number of fast, efficient functional tests can
        be run as part of every compile-link-run cycle, without serious time
        cost.  This meticulous use of one or two functional tests in a test
        harness has been absolutely critical in allowing just one programmer to
        produce 25k lines of code in less than 11 months.

	When I have departed from this approach, as when I commented out the
	orders tests during history query implementation, the regression in
	functionality has been as certain as sunrise.  At this point I'm a firm
	believer in the use of simplified code paths tested by a few fast,
	powerful regression tests, the tests few and fast so that developers
	can afford to run them hundreds or even thousands of times a day,
	powerful by their focus on high-level functionality, and effective
	because the underlying multiparadigm code hits the sweet spot between
	declarative simplicity and imperative efficiency.


HomeSearchInterior documentationProgram documentationFAQDownloadPDFsScreen shots ]  [ Contact UsSite MapBugzillaMailman |  Mailing List Archives: (MainLicenseAboutPressCopyright ]

Copyright (c) 2006 ... 2009 trading-shim.com, LLC, Columbus, OH