SIGN IN SIGN UP

BUG/MEDIUM: slz: bound the bits wasted by the 9-bit literals

Commit 002e838 ("bug: always make sure to limit fixed output to less than
worst case literals") made sure that switching from EOB to FIXED to emit
a reference is only done when the reference pays for the way back, based
on the fact that once in FIXED state a reference is always smaller than
the bytes it replaces.

This was unfortunately not enough: the literals interleaved with those
references can still fail. Indeed, octets 144 to 255 cost 9 bits instead
of the 8 they would cost in a stored block. The <bit9> counter measures
exactly that, but it was reset after every reference, so a stream
alternating just under 52 such literals with a cheap reference never
reached the threshold that sends them as a stored block, and kept
inflating for as long as the pattern lasted. 51 random literals >= 144
followed by 4 bytes copied from 8 bytes earlier (i.e. almost the exact
same pattern as previously tested) produce 1073161 bytes out of 1048576
(+2.34%) where the API promises at most 1048663, i.e. 24 kB more than a
caller sizing its output buffer from that promise would expect for a
single call.

Bit9 isn't sufficient to track the debt cross references, so let's add a
second <debt> counter to the stream's unused space. It accumulates the
bits actually wasted by the literals emitted in huffman mode, and each
reference records what it saved over the same bytes sent as literals,
bounded to zero. Above SLZ_MAX_DEBT (200 bits, i.e. 25 bytes) the encoder
stops trusting bit9: pending literals are stored and references have to
compensate for the full round trip, which lets the literal runs merge into
a full 65535-byte block and stop the growth.

The crafted stream now produces 1048677 bytes (+14 bytes over the
documented maximum instead of +24509). Other inputs such as text or
silesia corpus do not show any change since it's quite hard to fall
into this case.

Note that the threshold is deliberately much larger than the 52 bits of
a switch to amortize oscillations without needlessly sending literals.

This needs to be backported to 1.2.

This is libslz upstream commit 039fdf8aac3acfdcaa27ae30b387c942bc58ef84.

Note: the impact in haproxy will start with tune.bufsize above 43kB
      for the default 1kB reserve. A simple workaround consists in
      always keeping the reserve (tune.maxrewrite) at least 1/32 of
      bufsize.
W
Willy Tarreau committed
6f75c98fcbdc36cf37cf7239ff571b90e9a65274
Parent: fbfe177