SIGN IN SIGN UP

enhancement(sinks): Batch buffer rework (#2866)

* Add new VecBuffer wrapper around Vec

This new buffer type keeps track of its maximum size, and pre-allocates
the buffer.

* Add BatchSettings parameter to all buffers

* Move detection of batch "fullness" to trait Batch

This also drops fn len, as it is no longer necessary.

* Make the Batch::push trait method failable

This requires reworking the logic in BatchSink::start_send, and so
demands some close scrutiny.

* Introduce StatefulBuffer wrapper and clean up BatchSink

* Drop superfluous Batch::is_full trait method

* Merge BatchBytesConfig and BatchEventsConfig and adjust sinks

* Account for the separating "," in JsonArrayBuffer

* Use a simplified fake-ish builder pattern to build BatchSettings

* Split up batch size and timeout settings

* Manually track number of bytes written in Buffer

* Handle individual items that are too large to buffer

If the first event to be added to a buffer is too large, this could
cause a sink pipeline to permanently block. Instead, drop the output
event with an error message but return as if it was added.

* Drop test batch_sink_allows_the_final_item_to_exceed_the_buffer_size

We explicitly do not want batches to exceed the buffer size.

Signed-off-by: Bruce Guenter <bruce@timber.io>
B
Bruce Guenter committed
d2fe5279087ceb3dddd561c3b3d01959fad59006
Parent: 3d11c34
Committed by GitHub <noreply@github.com> on 6/29/2020, 6:08:28 PM