feat: unify filter sidebar and query input into single where clause

[Official-Krish]

Summary

The filter sidebar and the query input box previously held independent state — selecting a value in the sidebar applied it to the search but never appeared in the query input, so the two could silently drift out of sync. This PR makes the where clause the single source of truth for both.

How it works:

• Selecting a filter in the sidebar rewrites the matching facet clause directly in the query input (e.g. clicking error in the level facet writes level:"error" into the box)
• The sidebar reads its checked state back from the where text, so editing the query also updates the sidebar
• Free-text and complex query content is preserved — only the specific facet clauses are rewritten
• Works for both query dialects (Lucene and SQL)
• The separate filters URL param is removed; a one-time migration moves any legacy persisted filters into the where clause on first load

New internals in common-utils/filters.ts:

• parseWhereClauseToFilterState — parse a where string back into FilterState
• filterStateToWhereClause — render FilterState to a where string
• replaceFilterClauses — surgically replace only the facet clauses in a where string, preserving everything else
• dateTimeValueExpr extracted to common-utils/core/dateTimeValue.ts to avoid duplication between filters.ts and queryParser.ts

Before	After
Clicking a sidebar filter applies it to the search, but the query input stays empty.	Clicking a sidebar filter writes the condition into the query input (e.g. level:"error").

How to test on Vercel preview

Preview routes: /search

Steps:

1. Open the Search page and select any log/trace source
2. Click a value in the filter sidebar (e.g. a service name or log level)
3. Verify the corresponding condition appears in the query input box (e.g. ServiceName:"my-service" or level:"error")
4. Manually type a condition into the query input (e.g. level:"warn") and press enter
5. Verify the matching value is highlighted/checked in the filter sidebar
6. With existing query text like my error message level:"error", click a different level in the sidebar
7. Verify only the level: clause is rewritten — the free text my error message is preserved

References

Fixes #2751

[changeset-bot]

🦋 Changeset detected

Latest commit: e4e377f

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@hyperdx/common-utils	Minor
@hyperdx/api	Patch
@hyperdx/app	Patch
@hyperdx/otel-collector	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

@Official-Krish is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

[greptile-apps]

Greptile Summary

The PR unifies search-sidebar filters and query text around the where clause, adding Lucene/SQL parsing, targeted predicate rewriting, legacy-filter migration, and query-language translation.

• Derives sidebar state from the current where text and writes sidebar mutations back into it.
• Preserves free-form query content while replacing representable facet clauses.
• Adds migration, parse diagnostics, unrepresentable-query warnings, and extensive round-trip regression tests.

Confidence Score: 4/5

The PR is not yet safe to merge because same-field predicates that the sidebar cannot faithfully represent remain active when users select replacement values.

Repeated SQL predicates are preserved and then combined with the newly selected value, while Lucene modifier terms and negated ranges likewise remain alongside replacement predicates; the sidebar can therefore display a selection that does not reflect the effective query and can unexpectedly return no rows.

Files Needing Attention: packages/common-utils/src/filters.ts

Important Files Changed

Filename	Overview
packages/common-utils/src/filters.ts	Adds the core Lucene/SQL parsing and rewriting machinery, but unrepresentable and repeated same-field predicates can remain active alongside sidebar replacements.
packages/app/src/DBSearchPage.tsx	Makes where canonical, migrates legacy filters, and wires query parsing, language translation, and sidebar rewrites into search-page state.
packages/app/src/searchFilters.tsx	Adds adapters between where-clause facet state and the existing sidebar filter API.
packages/common-utils/src/tests/filterRoundTrip.test.ts	Adds broad regression coverage for parsing, replacement, migration, Boolean grouping, modifiers, ranges, and repeated SQL predicates.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  W[Where clause] --> P[Parse facet state]
  P --> S[Filter sidebar]
  S --> R[Replace facet clauses]
  R --> W
  L[Legacy filters] --> M[Merge into where]
  M --> W

Loading

_{Reviews (4): Last reviewed commit: "fix(search): unified sidebar-where claus..." | Re-trigger Greptile}

[pulpdrew]

Hey @Official-Krish, thanks for the PR, this is definitely a feature we'd love to see!

However, there are some things that I think will need to be addressed.

Bugs in WHERE string generation (Lucene)

1. NOT term AND ServiceName:"api" → click a filter → term AND (ServiceName:"api" OR ServiceName:"accounting"). The NOT is stripped, so the query is inverted.
2. ServiceName:"api" OR SeverityText:"error" → click a filter → ServiceName:"api" AND SeverityText:"error". OR silently becomes AND.
3. ServiceName:("api" OR "web") AND term → click a filter → ("api" OR "web") AND term AND (ServiceName:"api" OR ServiceName:"web"). The field name is dropped, so a field-scoped search becomes a full-text search, and the clause is duplicated.
4. Duration:[* TO 100] AND ServiceName:"api" → click a filter → ServiceName:"web". The range clause is deleted because it starts at character offset 0 (falsy-offset check).
5. Duration:{10 TO 20} → click a filter → Duration:[10 TO 20]. Exclusive bounds silently become inclusive.
6. msg:"hello"~2 AND ServiceName:"api" → click a filter → the ~2 proximity modifier is dropped (same for ^3 boost).
7. level:error (unquoted, the idiomatic Lucene form) → click warn in the sidebar → level:error AND level:"warn". Always returns zero rows.
8. Click a sidebar value for an attribute key containing (, ", { or [ (e.g. LogAttributes['a(b)']) → the emitted text fails to parse, so the query breaks and cannot be fixed from the sidebar. A key containing a space (LogAttributes['my key']) parses but becomes free text plus a nonexistent field.
9. "timeout)" OR "error" → click a filter → "timeout)" OR "error" AND level:"x". The paren inside the quoted value defeats the top-level-OR check, so the parens that f89fc39c9 added are omitted and AND binds tighter.
10. term1 OR term2 → click a filter four times → (term1 OR term2) AND .... One space is added per interaction, growing the query text and URL without bound.

Bugs in WHERE string generation (SQL)

11. ServiceName = 'a' OR ServiceName = 'b' → click a filter → ServiceName = 'a' OR ServiceName = 'b' AND SeverityText IN ('error'). Same precedence bug f89fc39c9 fixed for Lucene; the filter applies only to the 'b' branch.
12. ServiceName = 'a' -- temp note → click a filter → ServiceName = 'a' -- temp note AND SeverityText IN ('error'). The new predicate lands inside the comment: the checkbox looks applied but nothing is filtered.
13. msg = 'x AND y IN (' (unbalanced paren in a string) or `it's` = 1 (quote in a quoted identifier) → click the same filter twice → ... AND ServiceName IN ('a') AND ServiceName IN ('b'). Paren counting runs before the in-string guard, so conjunct splitting stops and clauses pile up. Unchecking stops working; new selections have no effect.
14. Same duplication for any column needing backticks: click a value on a service-name column twice → `service-name` IN ('a') AND `service-name` IN ('a', 'b'). The matcher compares quoted keys against unquoted ones.
15. ServiceName IN (SELECT name FROM t) AND foo = 1 → the sidebar renders a checkbox labelled SELECT name FROM t; click any filter → foo = 1 AND ServiceName IN ('b'). The subquery is destroyed.

Backwards incompatibility (existing URLs and saved searches)

16. Open a bookmarked URL / saved search with whereLanguage=lucene, where=ServiceName:"api" and filters=[SeverityText IN ('error')] → migration produces SeverityText:"error". The where clause's own filter is destroyed and written back to the URL.

UX regressions

17. It's common to start querying in lucene, potentially adding filters, then switch to SQL when a more complex condition is needed. Previously, all filters persisted when switching languages. Now all filters are lost until the user re-selects them or re-writes the WHERE input, which is stuck in the previous language. Ideally we should not lose filters when switching languages, the filters should transfer over to the new language.
18. Type service:" (any incomplete query) → all checkboxes clear, and clicking a filter does nothing at all while still triggering a re-query. No error or explanation is shown.
19. NOT ServiceName:"api" or term AND NOT ServiceName:"api" → the sidebar shows api as checked, i.e. the opposite of what the query does.
20. ServiceName:"api" OR SeverityText:"error" → the sidebar shows both as checked, implying an AND.

Performance regressions

21. Type in the search input → the whole filter sidebar re-renders per keystroke. handleSetFilters now depends on the watched where value, so all eight mutators get new identities and defeat memo(DBSearchPageFiltersComponent). The search page is noticeably laggy when typing, with these changes.

[Official-Krish]

Thanks for the review @pulpdrew! I'll work through these issues, push a revised implementation that addresses them, and update the PR shortly.

[greptile-apps]

Replacement retains old SQL predicates

When a user changes a sidebar value for a field with repeated SQL predicates such as host IN ('a') AND host IN ('b'), the duplicate count excludes that field from replacement and appends the new predicate instead. The query becomes host IN ('a') AND host IN ('b') AND host IN ('c'), so the old restrictions remain active and the sidebar selection can return zero rows.