enhancement(sinks): Batch buffer rework (#2866)
* Add new VecBuffer wrapper around Vec This new buffer type keeps track of its maximum size, and pre-allocates the buffer. * Add BatchSettings parameter to all buffers * Move detection of batch "fullness" to trait Batch This also drops fn len, as it is no longer necessary. * Make the Batch::push trait method failable This requires reworking the logic in BatchSink::start_send, and so demands some close scrutiny. * Introduce StatefulBuffer wrapper and clean up BatchSink * Drop superfluous Batch::is_full trait method * Merge BatchBytesConfig and BatchEventsConfig and adjust sinks * Account for the separating "," in JsonArrayBuffer * Use a simplified fake-ish builder pattern to build BatchSettings * Split up batch size and timeout settings * Manually track number of bytes written in Buffer * Handle individual items that are too large to buffer If the first event to be added to a buffer is too large, this could cause a sink pipeline to permanently block. Instead, drop the output event with an error message but return as if it was added. * Drop test batch_sink_allows_the_final_item_to_exceed_the_buffer_size We explicitly do not want batches to exceed the buffer size. Signed-off-by: Bruce Guenter <bruce@timber.io>
B
Bruce Guenter committed
d2fe5279087ceb3dddd561c3b3d01959fad59006
Parent: 3d11c34
Committed by GitHub <noreply@github.com>
on 6/29/2020, 6:08:28 PM