Returning data with InsufficientInput

[197g]

This is to discuss if there is something to be done by fdeflate or not. Also, compare handling in weezl.

In png the InsufficientInput error can occur when flushing data from the sequence of zlib-copmressed IDAT chunks. When that happens, decoding will return an error. However, this error depends on the number of bytes the Read feeds to the decoder at once. We only flush (by setting end_of_input) when we have completely read the last IDAT chunk. Having smaller amounts of data at once will therefore read more often with a read call set end_of_input false.

A problem now occurs if the zlib stream is 'corrupt', i.e. missing a proper end. It can still contain sufficient amounts of data for pixel decoding to work, partially or fully. Given the above usage, the last fdeflate read call will definitely return an error. That is fine by itself, the mitigation strategy is on png. However, the amount of bytes ever indicated by output_consumed depends on input data chunks, and the respecitvely available output chunks! Tather tragically, if we feed data in smaller chunks (we setup the output chunk to a fixed additional length) the decoder has more opportunities to write output data we actually receive. In large-chunk calls however more of the data is suppressed with the final read that errors, so bigger portions of the data stream are also lost.

Compare to weezl:

• an error return in fdeflate does not carry any of the input / output consumption information, in weezl the offsets are always available.
• weezl suppresses InsufficientInput (which is a non-error LzwStatus::NoProgress) if any progress was made in the read.

[fintelia]

The decoder already tracks the amount of output bytes even when errors are encountered, so there's certainly a case to be made that we should change the API to expose that.

A smaller change that may fix most of the issue for InsufficientInput is simply have png not set end_of_input for the final block of data. There's no actual difference in the decoding behavior between the two, the decoder always flushes as much data as possible into the output buffer regardless of whether it is set. Once we have all the uncompressed output, we can then optionally check for a truncated stream or whether the IDATs contain too much uncompressed data.