Make the line/match join in FindAllMatchingLines linearithmic
Matches and lines are two sets joined on byte offset, but the join was written as a nested scan, so every match was tested against every line: O(lines * matches) per file per query. On machine generated files this dominated everything else. Profiling a downstream consumer, this one function was 55.65% flat / 64.81% cumulative of a 20s CPU profile taken over three ordinary queries, against 4.56% for the substring search underneath it. A 33,400 line CloudFormation file with 17,313 matches costs 578 million comparisons. lineOffsets is already built and sorted, so invert the loops: binary search each match to the first line it overlaps and walk forward while the match still covers the line. That is O(matches * log lines + lines), and turns those 578 million comparisons into roughly 260 thousand. Matches can span a line boundary, so the predicate stays an interval overlap and a match is attached to every line it touches, not just the one its start offset falls in. Deliberately not a two pointer merge: MatchLocations is a map, so positions do not arrive globally sorted. Behaviour change: within a line, Locs were emitted in map iteration order and so varied run to run. They are now sorted by start offset, which is what callers want and makes the function testable. Benchmarks, 30k lines with 15k matches, and a 200 line file: pathological 324,613,092 ns -> 4,979,233 ns (65x) ordinary 15,392 ns -> 4,300 ns (3.6x) Allocated bytes are flat and allocation count drops by a third, so the speed is not bought with memory. Verified with a property test that keeps the old implementation as findAllMatchingLinesSlow and checks the two agree over 2000 random inputs x 6 limits x 5 context settings, covering CRLF, empty and trailing empty lines, first and last line matches, overlapping and zero length matches, boundary spanning matches and short term filtering. Mutation tested to confirm the suite catches dropping the multi line walk, disabling the short term skip, an off by one in the search predicate, and removing the sort. Binaries built either side of the change produce byte identical output over 6 queries x 3 output formats. Also hoists the loop invariant shouldFilterShortTerms call out of the per line loops in FindMatchingLines and FindMatchingLinesMulti. Those two still have the same nested scan, bounded by their 100 line cap.
B
Ben Boyter committed
afa6ae2172bf7bf03f4e7d302c9d1e4661eba442
Parent: 2d2e49a