Resource Selection

Each component of a market access system provides opportunities for redundancy to provide flexibility and improve reliability, from the network connection, through the tws process, dbms server and database, to the shim process itself. Issues related to configuration planning for redundant hardware resources, and in particular fault tolerant systems [11] are, however, outside the scope of this manual.

I'll focus instead on the flexibility and complexity that stem from multiple accounts, databases, and shim process instances. Note first that the shim has two primary roles, one of collecting market data and history queries, the other of submitting orders and watching the positions that result. These two roles are referred to as modes, about which more later, and for now it's enough for them to have names, data and risk respectively. Recall that for each real account you open with IB, you may also open a paper trading account, and that given such an account pair, and in the event you want to have both open at one time, you will need to run two instances of the IB tws, since each tws process connects to just one account. (Even though using both accounts at once may not be the common case, it may well be useful on occasion, e.g., if development and operational use coincide.) In this case the account information for the paper and real accounts would probably correspond to test and operational data, and distinct order journals should be used for each case. Such operation leads to two shim processes, one for each of the modes, each using a dedicated account and database, and each driven by separate downstream scripts, giving us e.g., the process-database diagram of Figure 3.1. The configuration problem here boils down to ensuring that programs talk to the correct counterpart and share an otherwise private database in common between themselves.

Figure 3.1: Configuration choices

Consider the case where data is being collected under the paper account, and live trading is going on with the real account. The downstream scripts will naturally talk to the correct shim process since they or a parent script probably took responsbility for starting those shims in the first place, which leaves two issues. There is a need for, one, database synchronization between downstream and the shim, and two, configuration control over shim connections to database server and IB tws. Both can be met if information is shared between downstream and shim, either via a shared configuration file, or an event sent from downstream to shim, and in fact either may be used, about which more in § 3.2.3. The result with respect to Figure 3.1 is that only solid edges would connect processes and databases, and that the dashed edges would be elided for this example.

The scenario above motivates one up-front resource choice and a couple of design decisions. There is a real need for multiple IB tws accounts, and even if multiple real accounts prove inconvenient to obtain, this need can be met at least in part by real-paper account pairs. There is a need also for multiple copies of a single database structure, which can be met with sql create-table scripts run in the context of distinct database names; and an easily shared format for the trading shim configuration file, since downstream programs need to read it too. Shim configuration is described in § 3.2.3, and cloning the database, in § 3.3.2.

Bill Pippin 2010-01-14