SIGN IN SIGN UP

Make the line/match join in FindAllMatchingLines linearithmic

Matches and lines are two sets joined on byte offset, but the join was
written as a nested scan, so every match was tested against every line:
O(lines * matches) per file per query. On machine generated files this
dominated everything else. Profiling a downstream consumer, this one
function was 55.65% flat / 64.81% cumulative of a 20s CPU profile taken
over three ordinary queries, against 4.56% for the substring search
underneath it. A 33,400 line CloudFormation file with 17,313 matches
costs 578 million comparisons.

lineOffsets is already built and sorted, so invert the loops: binary
search each match to the first line it overlaps and walk forward while
the match still covers the line. That is O(matches * log lines + lines),
and turns those 578 million comparisons into roughly 260 thousand.

Matches can span a line boundary, so the predicate stays an interval
overlap and a match is attached to every line it touches, not just the
one its start offset falls in. Deliberately not a two pointer merge:
MatchLocations is a map, so positions do not arrive globally sorted.

Behaviour change: within a line, Locs were emitted in map iteration
order and so varied run to run. They are now sorted by start offset,
which is what callers want and makes the function testable.

Benchmarks, 30k lines with 15k matches, and a 200 line file:

  pathological  324,613,092 ns -> 4,979,233 ns   (65x)
  ordinary           15,392 ns ->     4,300 ns   (3.6x)

Allocated bytes are flat and allocation count drops by a third, so the
speed is not bought with memory.

Verified with a property test that keeps the old implementation as
findAllMatchingLinesSlow and checks the two agree over 2000 random
inputs x 6 limits x 5 context settings, covering CRLF, empty and
trailing empty lines, first and last line matches, overlapping and
zero length matches, boundary spanning matches and short term
filtering. Mutation tested to confirm the suite catches dropping the
multi line walk, disabling the short term skip, an off by one in the
search predicate, and removing the sort. Binaries built either side of
the change produce byte identical output over 6 queries x 3 output
formats.

Also hoists the loop invariant shouldFilterShortTerms call out of the
per line loops in FindMatchingLines and FindMatchingLinesMulti. Those
two still have the same nested scan, bounded by their 100 line cap.
B
Ben Boyter committed
afa6ae2172bf7bf03f4e7d302c9d1e4661eba442
Parent: 2d2e49a