Should MAXSCOREBulkScorer sometimes apply filters after essential clauses.

[jpountz]

Currently, MAXSCOREBulkScorer always applies filters in parallel with essential clauses (by intersecting the filter with a heap containing essential clauses in a leap frog fashion).

When the sum of costs of essential clauses is less than the cost of the filter, it would likely be more efficient to "post filter" hits of essential clauses, ie. pull hits of essential clauses into a DocAndScoreBuffer and then remove hits from this buffer that don't match the filter.

[prudhvigodithi]

So is my understanding correct ?

Step 1: Essential clauses only → DocAndScoreBuffer
Step 2: Remove non-matching docs from buffer using filter

If so this makes sense. When essential clauses are selective, paying the cost of leap-frogging with a broad filter on every candidate is wasteful.

[prudhvigodithi]

If my above understanding is correct, going over the code the key change is here https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/MaxScoreBulkScorer.java#L149. 1st allow scoreInnerWindowSingleEssentialClause() and scoreInnerWindowMultipleEssentialClauses(), then call the post filter and finally the scoreNonEssentialClauses.

There is also a todo comment https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/MaxScoreBulkScorer.java#L168

    // TODO: Sometimes load the filter into a bitset and use the more optimized execution paths with
    // this bitset as `acceptDocs`

If i'm not wrong the filter can be converted to bits then filter would be applied during collection itself avoiding even the post-filter step. When this isn't available, the post-filter approach would still be an improvement over the current leap-frog.