Under heavy load parameter database download fails

[davefiddes]

Problem experienced on a ZombieVerter VCU that appears to be under heavy load (for unclear reasons but very repeatable). Fetching the device database using CAN SDO will just stop part way through.

There appear to be two distinct failure modes:

• The param DB download just stops after a few seconds. Looking at the CAN trace it seems like the client has a requested the next segment but no reply is forthcoming from the device. Adding some trace variables show the request has made it into CanSdo::ProcessSDO but no frame is emitted by the call to

  libopeninv/src/cansdo.cpp

  Line 234 in 087010a

  canHardware->Send(0x580 + nodeId, data);

• The param DB download is terminated early. This appears to be because the required 7 bytes needed to fill a CAN SDO frame are not available. The code here

  libopeninv/src/cansdo.cpp

  Line 150 in 087010a

  if (PRINT_BUF_EMPTY())

  triggers and terminates the download. The client registers this but objects because it is malformed JSON. This can be triggered by adding uDelay(1000) after t.Run() in the main loop.

A wireshark capture of the first failure is attached:
db-download-failure.pcapng.gz
This is best viewed using the CANOPEN dissector. It omits the stream of CAN frames being sent by the VCU to CAN ID 0x100 and 0x300 every 10ms.

Issue #39 was found while trying to characterise this. This is a separate and internally sourced issue.

[davefiddes]

So the random concurrency failure has been addresses by PR #41. The logic flaw which causes database downloads to fail because the buffer is not being filled fast enough remains.

I would suggest that some sort of end-of-printing marker (e.g. NUL byte) is used to signal that the final part of the JSON has been generated. I think this may fit with a fix for issue #39.

[davefiddes]

I've given this some more thought and even had a go at producing a fix. It gets very complex. The root of the problem is having to postpone the SDO reply until sufficient data is available. This involves a timer context having to pick up the send which gets messy.

I don't think the problem is likely to occur on a healthy device. If the overload condition preventing the printing does happen on a temporary basis then recovery is just a case of trying again once the download times out.