tracepoint-collect can generate traces with time inversion

[idigdoug]

After each buffer flush, tracepoint-collect write a PERF_RECORD_FINISHED_ROUND event to the file. The FINISHED_ROUND event asserts that there are no unflushed events, i.e. that if the last flushed event has timestamp X, the rest of the file will have no events with timestamp less than X.

This assertion is only mostly true. tracepoint-collect does not guard against several possible race conditions, and libtracepoint-control doesn't provide any help in guarding against the possible race conditions. Basically:

• Flush all events from CPU 1. Last flushed event has timestamp X.
• New event arrives on CPU 1 with timestamp X + 1.
• New event arrives on CPU 2 with timestamp X + 2.
• Flush all events from CPU 2. Last flushed event has timestamp X + 2.
• Write a FINISHED_ROUND event, indicating that there will be no subsequent events with timestamp less than X + 2.
• Later on, we flush CPU 1 and we flush that event with timestamp X + 1, creating a time inversion.

I'm not sure it's possible to completely eliminate the race conditions, but it should be possible to do better than the current implementation. One possibility would be to pass a roundEndTimestamp parameter to flush and make flush responsible for writing FINISHED_ROUND as appropriate. Flush would work as follows:

• If roundEndTimestamp is not 0, iterate through all buffers, flushing any event with timestamp less than roundEndTimestamp, then write a FINISHED_ROUND event.
• Iterate through all buffers, flushing all events and keeping track of the latest timestamp of the flushed events.
• Return that timestamp to the caller (or 0 if no events were flushed after FINISHED_ROUND).
• Caller uses the returned timestamp as the roundEndTimestamp in the next call to flush.