[ts-gen] Problem with "make: *** [test] Broken pipe" during build test

pippin at owlriver.net pippin at owlriver.net
Mon Dec 18 13:46:46 EST 2006

An anonymous Debian user [ADU] has fixed earlier problems with the regression
test script text, and is running a "make test", which will run the regression
tests after building the executable.  He sees the following:

> make -j8  shim
> make[1]: Entering directory `~/shim'
> make[1]: `shim' is up to date.
> make[1]: Leaving directory `~/shim'
> pdksh bin/regress
> set tws to Detail

> The trading shim has finished program initialization, including the
> construction of successful connections to the database and IB tws.

> YM history query
> make: *** [test] Broken pipe
> $

The short answer is that there may be a database version -- shim mismatch.
Download a new tarball just on general principles, and recreate the database
tables using the sql/create.sql script, by changing into the directory sql,
running the mysql interpreter, and source'ing create.sql.

I'll run through the debugging process here, though, in the hopes that it may
be of use to readers.

Reading from the beginning, make tells us that the executable has already
been built, and is up to date with respect to the available sources, then
runs bin/regress, in response to the "test" target that was given to it.
After that the bin/regress script starts the shim, and begins echoing
information to the screen, so that we know how far it's gotten.  It sets the
tws debug level to the highest level, 5, or "Detail", and tells us so, with
"set tws to Detail".  Then the shim tells us that it has finished startup
(keep in mind that the various programs that are writing to the screen are
running in parallel, so that trace output is asynchronous).  Next bin/regress
tells us that it is about to run a history query: "YM history query", and
then make complains that its connection to bin/regress (the test target) is
broken (make: *** [test] Broken pipe).

So, the script bin/regress died after trying to run a history query.  It
helps to know here that the bin/regress script, and the associated functions
in bin/includes, use ksh coprocessing in order to write to the background
dbms and shim processes; and that a ksh shell will die when it tries to write
to, or read from, a dead coprocess.  So, a reasonable suspicion at this point
is that the shim has died while trying to perform the history query it was
asked to do; that the bin/regress script died in consequence; and that make
has told us all it knows, that bin/regress went away.

Our ADU comments:

> Shockingly unhelpful error message, IMO

Yes; all we know at this point is that the regression test has failed.  The
best approach here, given that something has gone wrong, is to peel off
layers, and drop down to running the shim directly, so that error messages
aren't being filtered by overly protective intermediate programs.

Let me take the layers in turn.  To separate make and bin/regress, I could
say either of the following, as the shim needs to be built or not:

    make && bin/regress         # or simply

This probably wouldn't do much good, as the ksh script would still be in
the way.  Going deeper, and looking at bin/includes, in particular the
function do_query, we see the text:

> function do_query                                       # run a history query
> {
>     hit_shim 'YM history query'     'past add 166 1;'
> }

and, furthur on:

> function hit_shim { print -u7 "$2"; log_echo "$1"; }
> function log_echo
> {
>     echo "test:" $1
>     echo $$ $1 | logger -p local0.info
> }

We can ignore all the trace and logging cruft; all we care about is what goes
to the shim, since we want to talk to it directly, and here that is the text

    past add 166 1;

Past is the command verb for history queries; add means to make, rather than
cancel, a query; 166 is the contract id of interest, here for AIG; 1 is
meant to be the index of a history configuration record, controlling details
such as the bar size and number of rows; and the semicolon is an absolutely
necessary command terminator, without which the shim command channel will
keep reading, waiting for you to finish what you are saying.  By the way,
you can look at the sql/subs.sql script to see the create table statement for
the PastConfig table, and at sql/load.sql to see what has been pre-inserted
into it to support the regression tests.

If you run

    ./shim --data 

and type the past command, you will probably see some kind of error message
from the shim.  It will probably be very low level, but I suspect that it
will result from a failed database read, for information that is not there.

Try recreating the database, by changing into the directory sql, and running
"create.sql" from within the mysql interpreter; and then rerun the regression
test.  Note that much of the time a new release will require table recreation
as well, so you'll often need to do this when you get a new tarball.

Although there is code in the shim to check the database version, it is
commented out right now for reasons too involved with program internals for
me to go into; if you want to check the dbms version yourself, the following
query from within mysql will tell you what trading-shim database version your
dbms tables are at:

    Welcome to the MySQL monitor.  Commands end with ; or \g.
    Your MySQL connection id is 3864 to server version: 5.0.18

    Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

    mysql> select * from Version;
    | uid | current | text                   |
    |  19 | current | shim dbms version 1.19 |
    1 row in set (0.01 sec)

If the values for the uid and text attributes show a lower version than 1.19,
a good first step to fix your problems is to recreate and reload initial
values into the database tables, via the create.sql script.

Also, if you are curious about market data subscriptions, you can
edit the regression script to comment out the function call "do_query", which
runs the history query, e.g.:

    start_up                                                ; sleep 2
    #o_query                                                ; sleep 2
    reg_subs                                                ; sleep 1

and try to run bin/regress again, to see if you get market data subscription
messages in your system log output.

By the way, what you'd like to see from a make test is, for the current
release, something like this:

    make -j8  shim
    make[1]: Entering directory `/home/pippin/src/tws/src/tsd'
    make[1]: `shim' is up to date.
    make[1]: Leaving directory `/home/pippin/src/tws/src/tsd'
    test: set tws to Detail

    The trading shim has finished program initialization, including the
    construction of successful connections to the database and IB tws.

    test: YM history query
    test: ask for bulletins
    test: get subscriptions

    test: unit done
    test: cmds done
    test: warn done
    test: msgs done

I've added the prefix "test: " to the regression script in the newest release
to help make it clear what text comes from which programs.  The last test,
msgs, won't work properly because it depends on IB price data that we can't
distribute, but the others should run ok given correct functioning of
database, shim, and tws.

Please let me know if you still have problems.



More information about the ts-general mailing list