[ts-gen] Diagnostic session transcript

R P Herrold herrold at owlriver.com
Thu May 17 12:33:34 EDT 2007

I had occasion to do ome diagnostic work for a list member one 
on one rcently.  His environment was: Debian (Ubuntu),

I have interspersed *** N *** markers, so I can refer to 
points where interesting things to talk about have occurred

item 1:  Note the ERROR, and 'your_dbms_host' -- As Bill 
mentioned in his ;reading the code recently, lets use 'egrep' 
and read the code:

[herrold at centos-4 shim]$ tar zxf shim-070228.tgz
[herrold at centos-4 shim]$ cd shim-070228
[herrold at centos-4 shim-070228]$ egrep -R 'your_dbms_host' *
[herrold at centos-4 shim-070228]$

This tells us where the value came from.

item 2, 3, 4:  During the session, as I was working from 
memory as to where substitutions existed, I had to go look a 
bit.  day to day, doing development and testing, I use a 
patching script written many months ago and incrementally 
updated when things change, to put in local values 
automatically; so I do not remember sometimes ;)

item 5:   ... and now it works; very quick, very easy.

item 6:  Just observing that the shim is reporting in the 
local TZ ...

item 7:  We use grep again to read the sources

item 8:  Here, I was concerned, as we have repeatedly noted in 
the documentation, that one MUST test on a paper accout to 
avoid burning real money ;)  The script: bin/unsafe does an 
entry and immediate exit, and is intentionally NOT in 'make 

item 9:  pastebins are great tools for remote debugging

item 10:  This is a puzzler, and we had not previously had a 
test to be dure that database writes are working;  we have 
been reworking the setup instructions to address assigning 
'least privilege' to the mysql database userid the shim uses, 
and we are investigating why it works on CentOS, but not 
Debian presently.

item 11.  Reflect that I have not wrapped the shim up into 
.deb (nor .rpm) packaging yet, so that 'dpkg' could protect me 
from dependencies disappearing silently -- if I had, I would 
have been warned with a mesage when I ran:
 	'apt-get upgrade'
... of course, I see in my back history on this box, with the 
recent Debian release:
 	apt-get -y --force-yes --ignore-hold dist-upgrade

Not much could protect me there, when I am telling the package 
management front end that I know what I am doing, and know how 
to repair the damage.  Fortunately, I recognized it readily, 
and a fresh conpile repaired the damage  ;)

item 12. shim-070228 was the last external release in a while 
before the recent (silent) resumption of releases [known not 
to yet recover Debian function].  We are aware that Debian is 
not working correctly post 070228, but we wanted to get the 
rather nice documnetation in process 'out there.' for some 
other people asking private questions.

PLEASE -- use the mailing list, so that all may benefit from 
your questions, and our answers -- we are quite aware of the 
need to keep the mailing list turnaround prompt and carefully 
written.  Google is indexing it, and we have a search box at 
the site.

As I say, Bill has written rather nice preliminary 
documentation at a high level, which appears in HTML at:
and in PDF form at:

item 13.  covered at item 11

item 14.  covered at item 12

And that pretty well wraps up my comments on this diagnostic 
session.  Remember to write.

-- Russ Herrold


Transcript of an IRC support session on 9 May 2007, commencing at 16:16 
US Eastern time (GMT 20:16)

15:54 =kiwifx> ping
16:00 =orc_orc> pong
16:00 =orc_orc> hi there
16:04 =kiwifx> hi
16:04 =kiwifx> good afternoon
16:05 =orc_orc> or morning as the case may admit
16:05 =kiwifx> indeed

 	... after some 'getting acquainted' preliminaries, the support
 	session commenced:

16:16 =orc_orc> to your problem getting the shim going, which distro
 	are you using?
16:17 =kiwifx> linux?
16:17 =orc_orc> yes
16:17 =kiwifx> ubuntu
16:17 =orc_orc> kk
16:17 =kiwifx> dapper drake
16:17 =orc_orc> so Debian
16:17 =kiwifx> yes
16:17 =kiwifx> works fine

16:17 =orc_orc> and you have a mysql server running and can compile
 	the shim, right?
16:17 =kiwifx> and connect to IB and get connection strings etc.
16:17 =orc_orc> kk
16:18 =orc_orc> so then run:  make test
 	and tell me what it shows or paste at www.pastebin.ca, and
 	let me know the URL to look at
16:19 =orc_orc> your connection is actually to the local TWS instance
 	of course, and the TWS then talks, through a FIX variant upstream
 	and offsite from you
16:20 =kiwifx> make[1]: `shim' is up to date.
16:20 =kiwifx> make[1]: Leaving directory `/usr/local/shim-070228'
16:20 =kiwifx> bin/regress
16:20 =kiwifx> ERROR 2005 (HY000): Unknown MySQL server host
 	'your_dbms_host' (1)

 	*** 1 ***

16:20 =kiwifx> test: set tws to Detail
16:20 =kiwifx> test: get account data
16:20 =kiwifx> The trading shim has finished program initialization,
 	including the
16:20 =kiwifx> construction of successful connections to the database
 	and IB tws.
16:20 =kiwifx> test: YM history query
16:20 =kiwifx> test: AIG arket depth
16:20 =kiwifx> test: YM market depth
16:20 =kiwifx> test: no market depth
16:20 =kiwifx> make: *** [test] Broken pipe

16:20 =orc_orc>  Unknown MySQL server host 'your_dbms_host'
16:20 =kiwifx> i know, but that's wrong, as it does connect

 	*** 2 ***

16:20 =orc_orc> that means you did not patch data.c or the
 	includes in bin/includes
16:20 =orc_orc> two [dbms] connections in play here
16:20 =kiwifx> oh
16:21 =kiwifx> i patched data.c
16:21 =orc_orc> the shim which works, and the scripts which need the edit
16:21 =orc_orc> yes - that is it all right
16:21 =kiwifx> i see
16:21 =orc_orc> the shell scripts talk to the database as well
16:21 =kiwifx> do i edit regress?
16:21 =orc_orc> bin/includes as I recall -- lemme look

 	*** 3 ***

16:22 =orc_orc> I patch all on an automated basis with each build so
 	I forget
16:22 =kiwifx> got it
16:22 =orc_orc> UserName=rph_shim
16:22 =orc_orc> HostName=xps400.first.lan
16:22 =orc_orc> DbmsName=rph_testing
16:23 =orc_orc> that section

 	*** 4 ***

16:23 =orc_orc> my initials are rph, and that lets me have a namespace
 	that Bill does not touch, and vice versa
16:24 =orc_orc> xps400 is a Dell xps400, not surprisingly
16:24 =kiwifx> nice
16:24 =kiwifx> ok i'll run again
16:25 =kiwifx> bin/regress
16:25 =kiwifx> test: set tws to Detail
16:25 =kiwifx> The trading shim has finished program initialization,
 	including the
16:25 =kiwifx> construction of successful connections to the database
 	and IB tws.
16:25 =kiwifx> test: get account data
16:25 =kiwifx> test: YM history query
16:25 =kiwifx> test: AIG arket depth
16:25 =kiwifx> test: YM market depth
16:25 =kiwifx> test: no market depth
16:25 =kiwifx> test: get subscriptions
16:25 =kiwifx> #eg_echo

 	*** 5 ***

16:25 =orc_orc> ok -- so, as root: tail -400 /var/log/messages
16:25 =kiwifx> root at shim:/var/log# tail -f /var/log/syslog
16:25 =kiwifx> May 10 09:00:59 shim : shim|data|0.28|13892|32457|
 	   9063042|3| 1| 5|       157|TXN|STK||0.00||1|SMART||USD|| 
16:25 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32457|
 	   9063054|3| 1| 5|       158|TYC|STK||0.00||1|SMART||USD|| 
16:25 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32457|
 	   9063113|3| 1| 5|       159|UTX|STK||0.00||1|SMART||USD|| 
16:25 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32457|
 	   9063128|3| 1| 5|       161|VRSN|STK||0.00||1|SMART||USD|| 
16:25 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32457|
 	   9063140|3| 1| 5|       163|WFMI|STK||0.00||1|SMART||USD|| 
16:25 =orc_orc> yayyyyy
16:25 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32457|
 	   9063151|3| 1| 5|       165|WMT|STK||0.00||1|SMART||USD|| 
16:25 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32457|
 	   9063162|3| 1| 5|       166|WYNN|STK||0.00||1|SMART||USD|| 
16:25 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32457|
 	   9063174|3| 1| 5|       171|TRIN-NYSE|IND||0.00||1|SMART||USD|| 
16:26 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32457|
 	   9063185|3| 1| 5|       177|YM|FUT|200703|0.00||1|ECBOT||USD|| 
16:26 =kiwifx> May 10 09:01:00 shim : shim|data|0.28|13892|32459|
 	  11011472|2| 7| 0|quit;| 
16:26 =kiwifx> so, that was the issue. 
16:26 =orc_orc> yes indeed

[Note: the local Time Zone for kiwifx  is 'across' the International date 
line from New York, and into 'tommorrow. As such he has the last 'tick' 
data of the day, from 16:01 NY time on the 'prior' day, which ended 24 
minutes before his paste; the 'YM' would still have been open for trading, 
and I am uncertain as to why it has a value at his '09:01:00']

 	*** 6 ***

16:26 =kiwifx> now, how to get that data into the database?
16:26 =orc_orc> it already is
16:26 =orc_orc> we [that is, the shim] inserted it for you
16:26 =kiwifx> which table?
16:27 * orc_orc looks
16:27 =orc_orc> what I am doing is: cd src, and grep for insert
16:28 =orc_orc> post.c:char_0 head("insert into HistoryBar\n"
16:28 =orc_orc> post.c:char_0 text("event: history insert");
16:28 =orc_orc> looks like HistoryBar

 	*** 7 ***

16:28 =kiwifx> empty
16:29 =orc_orc> yeah -- I'll look further -- other scripts do this for me
16:29 =kiwifx> k
16:31 =orc_orc> ok -- are you in a PAPER account???
16:31 =kiwifx> yes
16:31 =orc_orc> run:  bin/unsafe

 	*** 8 ***

16:31 =orc_orc> then on the DB, run:
 	select * from HistoryBar limit 5 ;
16:32 =orc_orc> we move stuff in and out of the test scripts,
 	depending on where we are developing
16:32 =kiwifx> running now
16:33 =orc_orc> should look like this: http://www.pastebin.ca/479607

 	*** 9 ***

16:33 =kiwifx> The trading shim has finished program initialization,
 	including the
16:33 =kiwifx> construction of successful connections to the database
 	and IB tws.
16:33 =kiwifx> test: protected entry:
16:33 =kiwifx> test: limit exit order
16:33 =kiwifx> test: send transaction
16:33 =kiwifx> test: open orders info
16:33 =kiwifx> test: fire order entry
16:33 =kiwifx> test: reverse position
16:33 =kiwifx> test: YM history query
16:33 =kiwifx> test: get account data
16:33 =kiwifx> test: AIG arket depth
16:33 =kiwifx> test: YM market depth
16:33 =kiwifx> test: no market depth
16:33 =kiwifx> test: get subscriptions
16:33 =kiwifx> # run_risk                     ; sleep 2
16:33 =kiwifx> # wasteful                     ; sleep 1
16:33 =kiwifx> # end_shim
16:33 =kiwifx> # end_dbms
16:33 =kiwifx> # exit
16:33 =kiwifx> #eg_echo
16:34 =kiwifx> no
16:34 =orc_orc> 16:33 =kiwifx> test: YM history query
16:34 =kiwifx> still empty
16:34 =orc_orc> that should have popualted the DB
16:34 =orc_orc> there may be a perms issue, but I doubt it ... hmmm
16:34 =kiwifx> maybe
16:34 =kiwifx> no insert priv.
16:34 =kiwifx> hang on, i'll do an insert manually
16:34 =orc_orc> lemme get a test case for you -- one moment
16:35 =orc_orc> > select count(*) from HistoryBar ;
16:35 =orc_orc> does that return zero?
16:36 =kiwifx> well, it returns 1 now as I just did a test insert
 	using the same connection parameters
16:36 =kiwifx> permissions should be ok
16:36 =orc_orc> very curious -- I was going to do that same test insert
16:36 =kiwifx> great programmers ....

 	*** 10 ***

16:36 =orc_orc> let me pop over the Debian unit and repeat this test
16:37 =kiwifx> k
16:38 =orc_orc> hmmm -- debian has upgraded a SO away from me since
 	I built my test shim case there:
 	 ./shim: /usr/lib/libmysqlclient.so.14: version
 		`libmysqlclient_14' not found (required by ./shim)
16:38 =orc_orc> ... rebuilding
16:39 =kiwifx> k

 	*** 11 ***

16:39 =orc_orc> what is date of tarball you are using?
16:39 =orc_orc> kk
16:39 =kiwifx> 070228
16:40 =orc_orc> yeah -- that is a good one before we ripped a lot up

 	*** 12 ***

16:54 =orc_orc> ping
16:59 =kiwifx> ping
17:00 =orc_orc> hey
17:00 =orc_orc> have to run -- roof maintenance
17:00 =kiwifx> any joy?
17:00 =orc_orc> stuff just finished build -- one moment
17:01 =orc_orc> one moment -- I am patching debian unit manually
 	as our code has been restructured and I missed a patch
17:01 =kiwifx> ok

 	*** 13 ***

17:04 =kiwifx> ok Russ.  Have a good evening. chat later.
17:04 =orc_orc> will advise on [Debian history] insert issue -- it has
 	worked here in the past -- will sort it
17:04 =orc_orc> good luck
17:04 =kiwifx> cheers

 	*** 14 ***

More information about the ts-general mailing list