[ts-gen] shim-080326.tgz mysql scripts

Bill Pippin pippin at owlriver.net
Fri Apr 4 16:19:39 EDT 2008


Paul C notes:

> ... the latest tarball resulted in an error when recreating mysql
> databases with ./sql/create.sql.  I just commented out the line ...
> and it seems to be fine. 

Thanks again for catching a missing entry in my tarball creation
scripts.  I've rolled a new tarball, and when you unpack it, you'll
now find sql/sym/A0Csh.sql to exist.  Other symbol files for stocks,
indices, and likely bonds, futures and options, will show up there,
probably in the near future.

And from an earlier post, listing log file data and giving latencies
from one event to the next:

...
     0.000130s  #       |2| 1| 5|       219|Z|FUT|200806|0.00||1|LIFFE||
-->  0.124042s  #       |3| 2| 5|       219|8|          54938|0|size.vol
     0.000061s  #       |3| 1| 5|       219|6|  5969.0|     0| |price.su
     0.000043s  #       |3| 1| 5|       219|7|  5905.0|     0| |price.su
     0.000042s  #       |3| 1| 5|       219|1|  5915.5|     4| |price.ou
     0.000029s  #       |3| 2| 5|       219|0|              4|0|size.bid
...

Yes, the log file already has all the timing information you could need
as long as you are willing to work at the granularity of individual
events.  I suppose a number of awk scripts have already been written
for this purpose, so it makes sense to put one in the source package.

The script bin/diff, copied into this message following my signature,
converts a logfile such as the following, where the event timestamps
give the time in microseconds since processor startup:

18913|43896|   1389865|4|100| 0|# |4|100|0|****************|
18913|43896|   1389871|4|101| 0|# |4|101|0|0.76|070831|data|
18913|43896|   1389874|4|100| 0|# |4|100|0|****************|
18913|43896|   1389856|4|102| 0|# |4|102|0|23|1|39|20080404 12:11:36 EST
18913|43896|   1614138|3| 9| 1|1|
18913|43897|   1614276|1| 0| 0|#!./shim -f|
18913|43897|   1614303|3| 4| 2|      -1|2104|Market data farm connection
18913|43897|   1614514|1| 0| 0||
18913|43897|   1614587|1|11| 0|select book 195  5;		|
18913|43897|   1614604|1| 2| 0|wait 3600;|
18913|43897|   1614640|2|10| 3|195|195|5|
18913|43898|   3069427|3| 4| 2|      -1|2104|Market data farm connection
18913|43898|   3371536|3|12| 1|       195| 0|insert|bid| 12635.0|     7|
18913|43898|   3371556|3|12| 1|       195| 1|insert|bid| 12634.0|    14|
18913|43898|   3371574|3|12| 1|       195| 2|insert|bid| 12633.0|    22|
18913|43898|   3371591|3|12| 1|       195| 3|insert|bid| 12632.0|    61|
18913|43898|   3372325|3|12| 1|       195| 4|delete|bid| 12631.0|    40|
18913|43898|   3373061|3|12| 1|       195| 4|insert|bid| 12631.0|    40|
18913|43898|   3373079|3|12| 1|       195| 0|insert|ask| 12636.0|    13|
18913|43898|   3373096|3|12| 1|       195| 1|insert|ask| 12637.0|    11|
18913|43898|   3373113|3|12| 1|       195| 2|insert|ask| 12638.0|    26|
18913|43898|   3373130|3|12| 1|       195| 3|insert|ask| 12639.0|    43|
18913|43898|   3373147|3|12| 1|       195| 4|delete|ask| 12640.0|    57|
18913|43898|   3373169|3|12| 1|       195| 4|insert|ask| 12640.0|    57|

into one, as in Paul's log data above, with relative differences in
seconds:

18913|43896|  1.389865|4|100| 0|# |4|100|0|****************|
18913|43896|  0.000006|4|101| 0|# |4|101|0|0.76|070831|data|
18913|43896|  0.000003|4|100| 0|# |4|100|0|****************|
18913|43896| -0.000018|4|102| 0|# |4|102|0|23|1|39|20080404 12:11:36 EST
18913|43896|  0.224282|3| 9| 1|1|
18913|43897|  0.000138|1| 0| 0|#!./shim -f|
18913|43897|  0.000027|3| 4| 2|      -1|2104|Market data farm connection
18913|43897|  0.000211|1| 0| 0||
18913|43897|  0.000073|1|11| 0|select book 195  5;		|
18913|43897|  0.000017|1| 2| 0|wait 3600;|
18913|43897|  0.000036|2|10| 3|195|195|5|
18913|43898|  1.454787|3| 4| 2|      -1|2104|Market data farm connection
18913|43898|  0.302109|3|12| 1|       195| 0|insert|bid| 12635.0|     7|
18913|43898|  0.000020|3|12| 1|       195| 1|insert|bid| 12634.0|    14|
18913|43898|  0.000018|3|12| 1|       195| 2|insert|bid| 12633.0|    22|
18913|43898|  0.000017|3|12| 1|       195| 3|insert|bid| 12632.0|    61|
18913|43898|  0.000734|3|12| 1|       195| 4|delete|bid| 12631.0|    40|
18913|43898|  0.000736|3|12| 1|       195| 4|insert|bid| 12631.0|    40|
18913|43898|  0.000018|3|12| 1|       195| 0|insert|ask| 12636.0|    13|

Thanks,

Bill
________________________________________________________________________

#!/bin/awk -f

# Given a trading-shim logfile as input:
# replace the microsecond-valued time ticks with the difference of that
# tick and the previous one in seconds, thereby indicating the interval
# from one event to the next.

BEGIN { FS = "|"; last = 0 }
/.*/  {

    diff = ($3 - last)/1000000.0;
    last =  $3;

    for (i = 1; i<NF; i++)
        if (i != 3)
            printf "%s|", $i
        else
            printf "%10.6f|", diff

    if ($NF == "")			# terminating | gives
        printf "\n"			# trailing null field
    else				# the security triple
        printf "%s\n", $NF		# lacks trailing vbar
}



More information about the ts-general mailing list