[ts-gen] Debugging intermittent lost posts [Was: Do I really need to ...]

Bill Pippin pippin at owlriver.net
Tue Sep 15 15:35:29 EDT 2009


Nils,

I've just pushed a release with improved diagnostic information
about sql warnings, and I suggest that you download this release
and use the newly added warn option before trying to debug the
OrderStatus problem. 

The rest of this post falls into two parts, one, an explanation of
the warn option, and two, the reasoning as to why I've added it for
your use.

First, as noted above, this new release includes a "warn" option
which enables trace printing to the stderr for mysql warning counts.
Although the orders tests that I run do not show warnings occurring,
there is an innoccuous trace for the query answer insert for history
queries, and so by way of example I list the [lightly edited] trace
message for a HistoryBar insert below:

    Query:
    insert into HistoryBar
    (bar, ibc, sym, route, bid, time, open, high, low, close, vol,  ...
    values
    (10, 53893055, 48161, 79, 10, STR_TO_DATE('20090915  09:30:00', ...
    (10, 53893055, 48161, 79, 10, STR_TO_DATE('20090915  10:00:00', ...
    (10, 53893055, 48161, 79, 10, STR_TO_DATE('20090915  11:00:00', ...
    (10, 53893055, 48161, 79, 10, STR_TO_DATE('20090915  12:00:00', ...
    (10, 53893055, 48161, 79, 10, STR_TO_DATE('20090915  13:00:00', ...
    (10, 53893055, 48161, 79, 10, STR_TO_DATE('20090915  14:00:00', ...

    has 6 warnings;
    sql state is 00000

As you can see, the trace gives the insert statement text, the
warnings count, and the ANSI sql state variable text.

In this case, by afterwards cut-n-pasting the query text by hand via
the mysql interpreter and then using the show warnings command, I find
that there was a forced type conversion for has_gaps values of 'false'
to 0, and that the query succeeded (sql state of 00000?), so that
the warnings here are messy but otherwise of no importance.

The above example is only to help you understand the text printed in
response to mysql warnings, and is otherwise besides the point; our
interest here is in warnings for inserts to OrderStatus.  So, I'd like
to suggest, Nils, that you use the warn option, and post to the list
both any query/warning trace outputs that you see for the OrderStatus
table, and also a heads up if you don't see such, yet are still losing
inserts.

Now, about this new focus on sql warnings as the iceberg tip here:
On reflection while driving home last night, it occurred to me that
the problem you are seeing is most likely due to failed dbms writes,
rather than problems at any earlier stage of processing, although
since I still have not observed this problem, I can't be certain.
My reasoning follows as to why I believe the problem is with the
generated insert statement, or at least the dbms server's handling
thereof:

    * the problem is intermittent, so the overall post pathway is
      correct for some cases;

    * since routing is typically either always correct, or always
      wrong, this gives me high confidence that the status messages
      are correctly routed to the Poster; 

    * recognized problems with msg->tuple and tuple->text mapping,
      if any, would be signalled via exception warning messages to
      the stderr;

    * the uniqueness check is, as noted in a previous post, clamped
      to succeed;

    * the insert ignore form of the sql insert command converts
      not just dup key, but also some other errors, including
      null violations, to warnings; and

    * although the shim has code to detect errors while fetching
      results for queries, and errors in evaluating sql statements,
      it did not previously have code to detect sql warnings.

Presumably you would have noticed the exception warning messages for
object mapping failures, and so, in conclusion, the only previously
unchecked pathway I know of for journal message insert failure is for
warnings during the sql insert, and given how long we've been looking
for the problem with OrderStatus, it's unlikely it could be anywhere
else.  Not impossible, just unlikely.

Feel free to look elsewhere, as discussed in my previous post about
trace points, especially if you are still having problems, and yet
there are no insert statement warning traces; yet in any case, please
use the warn option and let us know what you find.

Thanks,

Bill


More information about the ts-general mailing list