[ts-gen] The shim now requires order keys and markers to be identifiers

Bill Pippin pippin at owlriver.net
Wed Jul 7 18:01:30 EDT 2010

In brief, order command syntax is now more restrictive, and
requires that traditional identifier symbols be used for order
markers and keys.  Non-conforming journal records are no longer
readable by the newest shim, the database version has been
incremented to reflect this fact, and downstream scripts that
generate order commands may need to be modified to ensure that
order marker and key symbols have the correct syntax.

Such symbols must match the regex: [A-Za-z_][A-Za-z_0-9]* , and
so if your existing code and data already accords with this rule,
you're fine.  That being said, if you are working with the
shim-100630 release, you should shift to today's, that is
shim-100707 .

In more detail, Jordi Sanfeliu posted about a non-critical bug with
order commands, whereby syntax checking was not sufficiently strict,
in his postings:


In the third, above, he notes, speaking of the order command syntax

> ... I guess that it has a lower priority ...

Since the shim noted the problem with meaningful syntax warning
messages, and there were other tasks I wanted to work on, I've
delayed addressing this problem until now.  That being said, the 
newest report, with a segfault eventually occurring as a result of
continued processing with the invalid order marker, does raise the
priority.  By the way, although I was able to reproduce the journal
problems as reported, I haven't seen hide nor hair of a segfault,
so any new information about that beasty, if any, is of interest.

In the following, first, a summary of the original report by Jordi;
second, a description of the fix and resulting changes to the
command language; third, an extract from the NEWS file; and fourth,
a response to other issues from Jordi's second post above.

Existing problem:

Jordi noted, correctly, that, for order keys and markers in
order commands, the shim accepted numbers as well as traditional
programming language identifiers, those being symbols matching the
regex: [A-Za-z_][A-Za-z_0-9]* .  E.g., for an order marker of 1234,
though there would be syntax error warning messages such as the one
below, the order was nevertheless submitted.

>   For token text: [1234] and type target [mark]
>   predicted token type: str, scanner saw: pos
>   Problem: 401 unexpected token text: syntax error

Note that the above message is a 400-level warning, not a 500-level
fatal exception, and that the additional trace text normally following
with syntax errors, where the entire statement is rejected, has not

Later on, the stricter input checking applied to database records
would lead to errors rereading journal records, and so there were
problems modifying such orders.  The short term solution, of course,
was ... don't do that!  That is, respect the syntax error warning,
and use symbols, not numbers, as order keys and markers.

Modified command:

The fix is obvious, and indeed the shim now requires that order keys
and markers match the regex: [A-Za-z_][A-Za-z_0-9]* ; and otherwise,
the related order command is rejected.

So, for the current release, a order marker such as 1234 in an order
command would lead the shim to generate stderr trace information such
as the following: 

    For token text: [1234] and type target [mark]
    predicted token type: sym, scanner saw: pos
    Problem: 401 unexpected token text: syntax error
    Syntax error:
    Grammar rule: Compound create  
    Cursor state: match; text is:
    create item 1234 ibc:265598 at SMART order1 LMT 100 350.0 0.0;
    Terminal tag: name:mark index:3
    Drop:  ibc:265598 at SMART order1 LMT 100 350.0 0.0;


Note in the above that the syntax error is summarized, indicating
which command was rejected, with a caret message indicating where
in its text parsing stopped, and indicating in no uncertain terms
that the command was dropped.

NEWS file prefix:

      * Modify order command input to require an alphanumeric id
        for the order key and order mark (previously generic strings
        were accepted), and increment the database version to reflect
        the fact that old journal information may no longer be
        readable.  Other alphanumeric symbols are also checked,
        and this more-strict input checking may force other parse
        errors.  If such occur for you for other statement types,
        please be sure to inform the list; I'm not aware of any
        such cases.

        Note that the risk.rb script was modified to replace hyphens
        in the order key template with underscores.  Currently, a
        token is classified as an alphanumeric identifier if it looks
        like a traditional program variable, so that it matches the
        regex [a-zA-Z_][a-zA-Z0-9_]* .  Users who have order scripts
        that generate order keys and markers taking advantage of the
        less restrictive checking previously in place, so that they
        would include characters other than [a-zA-Z0-9_] , will have
        to modify those patterns accordingly.

        Users should prefer this release of 2010.07.07 to the
        previous one of 2010.06.30 :

        Restore error handling for the cmd listener, eliminating a
        bug whereby syntax errors could hang cmd input parsing.

        Fix a spurious fatal exception for first time startup
        against an empty database, whereby the AccountCode insert,
        although successful, was not correctly detected, leading
        to a one-time termination with the code:

            Problem: 546 ice: name-based database table lookup failed

        Fix risk mode sql warning error display to restore the
        previous behavior whereby db queries are listed to the
        stderr only if a mysql warning actually occurred; the
        last release was spraying all db queries to the stderr,
        even though warnings had not occurred. 

Unrelated issues:


    * Glad you liked the history command docs; you're the first to
      comment on them.

    * The one-time fatal exception you note in your 2nd post above,
      about name-based database table lookup having failed, is
      explained in the NEWS excerpt above.  The newest release
      includes a fix. 

    * The command parsing hang you describe in your 2nd post above,
      ditto: just download and compile today's release.

    * Today's most recent release also includes a fixed symbolic link
      for the symbols "sym" directory.  It may well be broken as
      unpacked --- it points up and over to a directory that may
      not exist yet --- but once you unpack the correct data tarball
      parallel to, and in the same directory as, the shim tarball,
      it works just fine.

    * There is no recent, comprehensive post explaining the order
      commands.  The risk.rb script is intended to demonstrate in
      brief how they can be used.  Beyond that, consider:

          1.  the IB java or C++ sample client sources, e.g.,
              EClientSocket.java and EReader.java, for attributes
              of the place order request;

          2.  IB's APIprintable.pdf guide, and in particular that
              the place order request parent id attribute groups
              child orders into an implicit OCA group, maintained
              and controlled by the IB tws, where, for the shim,
              the link structure is mirrored by the Template table
              parent attribute self-edges;

          3.  the order related tables (Template and its foreign
              key dependency children) and their create table
              definitions in sql/xact.sql, to understand how values
              for the IB order attributes are defined via order
              templates; and

          4.  the various shim command language order verbs, of
              which "create" binds an order key to a contract,
              and most encode a value for the transmit flag, as

                  verb      xmit
                  ------    ----
                  create    off
                  modify    off
                  submit    on
                  modsub    on
                  cancel    N/A

      There are also some posts I made back in August of last year
      on the order key and mark, implicit OCA groups, and the template
      table that may be of interest:


Good luck, and glad to see your progress with the shim.



More information about the ts-general mailing list