Skip to content

feat(logs): preserve field order from VictoriaLogs response#666

Merged
arturminchukov merged 2 commits into
mainfrom
563-preserve-field-order
Jun 17, 2026
Merged

feat(logs): preserve field order from VictoriaLogs response#666
arturminchukov merged 2 commits into
mainfrom
563-preserve-field-order

Conversation

@arturminchukov

@arturminchukov arturminchukov commented Jun 4, 2026

Copy link
Copy Markdown
Member

Related issue: #563

Describe Your Changes

  • Serialize log fields via fastjson Del + MarshalTo instead of a Go map, so the labels JSON keeps VL's field order instead of an alphabetical sort
  • Stop parsing _stream into fields (VL already expands stream labels as top-level fields); keep the raw _stream string for the stable log id
  • Remove the now-unused orderedLabels type and pkg/utils/stream_fields_parser
  • Update test fixtures to real VL top-level expansion; drop invalid_stream case

Checklist

The following checks are mandatory:


Summary by cubic

Preserves VictoriaLogs field order in labels JSON and stops parsing _stream. Builds stable log IDs from raw _time, _msg, and _stream_id bytes (keyed on _stream_id) to match VL responses and improve stability (#563).

  • Refactors
    • Serialize labels with fastjson (Del unwanted fields, then MarshalTo) so log fields keep VL order; _stream/_stream_id are kept as-is.
    • Split responses and tests into response_logs.go, response_stats.go, response_hits.go; keep shared types/error parsers/labelsToJSON in response.go.
    • Remove pkg/utils/stream_fields_parser; update fixtures and drop pkg/plugin/test-data/invalid_stream.

Written for commit e0db154. Summary will update on new commits.

Review in cubic

@arturminchukov arturminchukov requested a review from dmitryk-dk June 4, 2026 13:18
@arturminchukov arturminchukov self-assigned this Jun 4, 2026
@arturminchukov arturminchukov requested a review from hagen1778 June 4, 2026 13:18
Comment thread pkg/plugin/response.go
Comment on lines -118 to -127
stream := value.GetStringBytes(streamField)
rowStream = string(stream)
stf, err := utils.ParseStreamFields(rowStream)
if err != nil {
return newResponseError(fmt.Errorf("%s", err), backend.StatusInternal)
}
for _, field := range stf {
labels[field.Label] = field.Value
}
if !value.Exists(messageField) && len(stf) >= 0 {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed stream parsing, cause stream labels always exist in the log as separate labels, so we don't need to parse them manually.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 14 files

Re-trigger cubic

@dmitryk-dk dmitryk-dk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As for me we shouldn't change tests files because those are real cases which was fixed via the real users issue.
If we just need to leave _stream fields we can update only this logic and tests related to it

@hagen1778 hagen1778 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to refactor datasource_test.go in future. I think it is too verbose and repetetive - that's the reason why it got so many changes in this PR

Comment thread pkg/plugin/response.go Outdated
Comment thread pkg/plugin/response.go Outdated
Comment thread pkg/plugin/response.go Outdated
@arturminchukov arturminchukov force-pushed the 563-preserve-field-order branch from abc55b0 to 25b5a2e Compare June 10, 2026 15:04
@arturminchukov

Copy link
Copy Markdown
Member Author

1st commit:

  • Split the response.go into response_logs.go, response_stats.go, and response_hist.go with their tests without any changes in functionality.

2-nd commit:

  • removed parsing _stream field.
  • changed function buildLogID - now it requires byte arguments and changed _stream field to _stream_id field.
  • fixed tests according to missing parse stream functionality (just removed unnecessary test, and fixed those tests that expected parsing _stream field)

@dmitryk-dk @hagen1778 Could you please review it again?

@hagen1778 hagen1778 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread pkg/plugin/datasource_test.go Outdated
@arturminchukov arturminchukov force-pushed the 563-preserve-field-order branch from 25b5a2e to 8707e50 Compare June 11, 2026 12:06

@dmitryk-dk dmitryk-dk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good!
One thing what i do not like is that we removed from the tests the real test case
but if you sure it is expected than ok

_, err := w.Write([]byte(`{"_time":"2024-02-20", "_stream":"{application=\"logs-benchmark-Apache.log-1708437847\",hostname=}"}`))
			if err != nil {
				t.Fatalf("error write reposne: %s", err)
			}

@arturminchukov

Copy link
Copy Markdown
Member Author

Overall looks good! One thing what i do not like is that we removed from the tests the real test case but if you sure it is expected than ok

_, err := w.Write([]byte(`{"_time":"2024-02-20", "_stream":"{application=\"logs-benchmark-Apache.log-1708437847\",hostname=}"}`))
			if err != nil {
				t.Fatalf("error write reposne: %s", err)
			}

I removed this test case, cause I removed the parsing _stream field, so this test case is no longer relevant.

- Split response.go into response_logs.go, response_stats.go, response_hits.go
- Split response_test.go likewise (response_logs/stats/hits_test.go)
- Keep shared types, error parsers and labelsToJSON in response.go
- Build labels via MarshalTo so log fields keep the order returned by
  VictoriaLogs instead of being sorted alphabetically (#563); _stream and
  _stream_id are kept as-is, _stream is no longer parsed/expanded
- buildLogID now takes []byte and hashes the raw _time/_msg/_stream_id
  bytes (id keyed on _stream_id), avoiding string<->[]byte round-trips
@arturminchukov arturminchukov force-pushed the 563-preserve-field-order branch from 8707e50 to e0db154 Compare June 16, 2026 13:38
@arturminchukov arturminchukov merged commit fcc06d3 into main Jun 17, 2026
6 checks passed
@arturminchukov arturminchukov deleted the 563-preserve-field-order branch June 17, 2026 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants