[ts-gen] shim --risk architecture [was: Hooking into shim directly via C/C++]

Ken Feng kfmfe04 at gmail.com
Fri Aug 28 13:40:55 EDT 2009

Hi Nils,

Thank you for posting this!  Your listing is actually extremely
useful.  It resolves the difficult question of:

"How do I poll for shim for --risk messages/responses (pull/wait)
while allowing some outside processes to send a new order (push)?"

Your response seems to be - use a shimpipe to connect to the shim and
listen for --risk messages/responses, while allowing new orders to be
sent, push-style, into the shimpipe, I think through the use of
select() - will look at the code in more detail later and try it out.
With your architecture, I can make shimpipe store information about
new orders that have been pushed through.

An alternative architecture, if we can also have multiple --risk shims
running, is to use two --risk shims.  One that is listening for
responses, and a second one that sends new orders.  However, I could
not figure out how to do this if I want the first shim to know about
new open orders before getting responses from the shim.

Maybe if someone has some ideas, they can post how this can be done
with two --risk shims or somehow one multiplexed --risk shim (even
better!) from Ruby.

For instance:

1. Send Buy 100 IBM
2. Listen for responses
3. Send Buy 200 IBM (may come in as a push anytime from an outside process)
4. Know about 3. before listening for responses

Thanks again.

- Ken

On 8/28/09, Nils Gebhardt <mail at ngebhardt.de> wrote:
> On Thu, 2009-08-27 at 09:05 +0900, Ken Feng wrote:
>> Hi Nils,
>> I am glad to hear that you were able to get this to work!
>> Would you care to share your code/invocations or at least show us some
>> pseudo-code of what you did?  Did you popen() and then fork() your
>> process?  What did you do to resynchronize the two processes, if you
>> needed to wait for a process to complete?
>> Any hints would be greatly appreciated.
> Hi Ken,
> when this wunderful order journaling - what I find really useful - came
> out  I was wondering what is the best way to invoke shim.
> Since I need the shim process up and running to catch status messages,
> I couldn't have a shim process per task but rather some sort of shim
> 'server' accepting new commands any time while still listening for tws
> on the other hand.
> I choose a select loop on a fifo/named pipe instead of opening a port,
> because it allows me to do the simple things from simple bash/awk
> scripts - I just echo all commands into the fifo.
> So I did
> 1) "mkfifo shimpipe"
> 2) a small C programm with something like
>   shim = popen("/pathto/shim --risk file save", "w");
>   int fd = open("shimpipe", O_RDONLY | O_NONBLOCK );
>   //int fd = open(FIFO_PATHNAME, O_RDWR );
>   fd_set rfds;
>   int rc;
>   struct timeval tv;
>   for(;;){
>   FD_ZERO(&rfds);
>   FD_SET(fd, &rfds);
>   tv.tv_sec = 0;
>   tv.tv_usec = 0;
>   rc = select(fd + 1, &rfds, NULL, NULL, &tv);
>   if (FD_ISSET(fd, &rfds)){
>   if (rc > 0) {
>     char buf[255];
>     ssize_t got = read(fd, buf, sizeof(buf));
>     if (got < 0) {
>       perror("read");
>       return 1;
>     }
>     else if (got == 0) {
>       close(fd);
>       fd = open("shimpipe", O_RDONLY );
>     }
>     else {
>       buf[got]='\0';
>       fprintf (shim, buf);
>       fflush(shim);
>     }
>   }
>   }
>   }
>   pclose(shim);
> ----%------
> 3) echo "select acct;" > shimpipe
> or whatever you want
> select() behaves a bit strange on fifos, since it doesn't listen on the
> fifo once it is marked as EOF.
> Therefore I close and reopen it. Better might be to open it as writer as
> well. On the other hand, EOF ensures here that each command is passed
> immediately and the chance to loose data between close/reopen
> and returning into the select loop should be zero for my very low volume
> traffic...
> I have no further concurrency problems. No need to call fork(), all
> messages are written immediately,
> output is not delivered exclusivly to the caller, but has to be greped
> (or piped through some awk scripts)
> from the log. Might be copied to a fifo as well if necessary.
> This is all really a bit simplicistic and probably not the solution to
> your problems. As said above, this approach
> helps me to:
> - simultanously listening to and passing asynchronous commands and tws
> messages
> - still have a scripting access to shim
> regards
> Nils

More information about the ts-general mailing list