[ts-gen] MacOS Shim problems
pippin at owlriver.net
Fri May 15 23:12:04 EDT 2009
About your problems getting the shim running on a Mac, in earlier posts
I believe you've described entering 127.0.0.1 as a trusted IP address
in the IB tws api configuration, and as the value of Feedhost in your
.shimrc file, and you've configured mysql so that auto increment works
properly, and as a result the shim produces its startup banner properly.
Grant for purposes of discussion that through this process you've
overcome IPv6 configuration problems, if any, and that the shim has
indeed successfully and correctly connected to the database and the
upstream IB tws api server, that the banner means just what it says,
and that socket communications are working just fine.
Perhaps I should offer congratulations at this point, since maybe you
are close to getting the shim working. Most people have been in the
past when they got this far.
Now, your symptoms; you ask:
> why does shim freeze somewhere between that initialize message and
> the prompt from shim coming up [?]
Probably the shim is not so much frozen, as twiddling its thumbs,
that is cycling around its main loop without recognizing that there
is input on the stdin, or upstream socket, perhaps because there
Also, in a later post, you ask why the shim seems to ignore its
> set acct on;
> The last two commands have no effect, the only way out is <cntl>-c.
The first command is incorrect syntax, but as explained below, that
still does not explain your symptoms; the exit; should work just fine.
In fact, this is the hardest problem for me to explain, and so below
I start from the basics.
First I'll go over some trace file configuration options.
The cmd line options cout, file, and logd send information to
stdout; the log file named, by default, log/ShimText; and the
system log. For testing, you should be using the file option,
and I believe you are already doing so.
You should be tailing the log file in another windown. The
bin/tail.window script is designed to help you do so, though you
need to have the kde konsole package installed. If this is not
available on the Mac, go ahead and look at the text of the script,
and consider if you want to adapt it to the Mac. Or, just a
tail -f will be better than nothing. Keep in mind that, due
to --- not yet fixed --- buffering problems with the cout option,
you want to be tail'ing the log file, and that just watching the
stdout is insufficient.
Next, if you add the save option as an argument of the command line
startup text for the shim, input commands, requests, and messages will
be traced to the files log/cmdinput.txt , log/shim2tws.bin , and
log/tws2shim.bin . The cmdinput.txt file is an exact image of any
command input; shim2tws.bin, a (binary) image of the request stream;
and tws2shim.bin, again a binary image, though of the message stream.
These files are useful in diagnosis after the fact to see what
events occurred. You can get a very rough translation of the
binary files by piping the binary images through the
bin/req.filter and bin/msg.filter scripts, respectively.
The record breaks are performed according to approximate heuristics,
and so the presence or absence of newlines does not by itself
indicate an error in the shim's operation, rather an unimportant
glitch in the (crude) translation scripts.
So, given the save option event logging, please consider the
1. Does every command you enter via the stdin show up in
the cmdinput.txt file?
2. Once given (1), do the related requests show up in
the shim2tws.bin file?
3. Once given (2), do the related messages show up in
the tws2shim.bin file?
As for the first test, the shim should just plain work, and
yet it's here that you say the shim is failing. This is hard
for me to understand, unless there are syntax errors in your
command input, in which case the shim should still produce some
kind of error message.
Still, first things first. The syntax given by an earlier post
for the account data subscription command was incorrect, and
would not work. An example of the correct syntax is given in
exs/acct, which has the text below, though without the 4-space
# Trigger the ReqAccountData request.
select acct; wait 2;
cancel acct; wait 2;
The shim should still have printed a syntax error, though, in
response to the invalid command [set acct], and the parser would
have recovered anyway after dropping the invalid cmd, so that
the exit; should have worked just fine.
Are you perhaps running one of the test scripts in order to
start the shim? E.g., if you run:
./shim --data file save < exs/tick
exs/tick --data file save
then you should get market data subscriptions, and the script
will provide the terminating exit; command, and the shim will
be completely unaware of your keyboard's input, since the tick
script has been opened as its standard input. I know this
question seems foolish, and again, the shim would exit just
fine, so this is admittedly grasping at straws.
By the way, if you want to test with the 'select acct;' command,
you should know that the IB tws seems to ignore additional account
data requests for a few minutes after filling the first request,
and the shim has already made one of those at startup in order to
determine the account code. A better test is probably the exs/tick
script, which has the nice property of producing a steady stream of
market data messages; or you could instead use the exs/book script,
which produces a rapid stream of market depth messages.
A more broadly-based test script is exs/test.rb , which sends a
number of data-mode commands to the shim, including history.
This script should produce copious log file output.
Beyond suspicious behavior found in the various trace and log files,
or some thinko such as starting the shim with the stdin attached to
some file not your keyboard, I'm at a loss for why the shim would
be unable to see the stdin you believe you are giving it . If you
are comfortable with C, you are of course free to add printfs to
the code. I'll discuss this point more in my next post.
By the way, the window to which you tail log file output may well
not be, and for clarity should not be, the same one where you start
the shim; and in that case, you probably realize that it's the
window where you start the shim that must receive keyboard input.
That is, each console window has it's own "stdin" that only gets
input if it has the keyboard focus. I'm sure you already recognize
this, but again, I'm grasping at straws here.
Otherwise I'm not sure what's wrong. We're unable to duplicate your
environment at this time --- we hope to do so reasonably soon, but
are not sure when --- and I have no idea why some of the oldest, most
stable code in the shim would fail to work properly.
More information about the ts-general