feat(logs): preserve field order from VictoriaLogs response#666
Conversation
| stream := value.GetStringBytes(streamField) | ||
| rowStream = string(stream) | ||
| stf, err := utils.ParseStreamFields(rowStream) | ||
| if err != nil { | ||
| return newResponseError(fmt.Errorf("%s", err), backend.StatusInternal) | ||
| } | ||
| for _, field := range stf { | ||
| labels[field.Label] = field.Value | ||
| } | ||
| if !value.Exists(messageField) && len(stf) >= 0 { |
There was a problem hiding this comment.
Removed stream parsing, cause stream labels always exist in the log as separate labels, so we don't need to parse them manually.
dmitryk-dk
left a comment
There was a problem hiding this comment.
As for me we shouldn't change tests files because those are real cases which was fixed via the real users issue.
If we just need to leave _stream fields we can update only this logic and tests related to it
hagen1778
left a comment
There was a problem hiding this comment.
It would be nice to refactor datasource_test.go in future. I think it is too verbose and repetetive - that's the reason why it got so many changes in this PR
abc55b0 to
25b5a2e
Compare
|
1st commit:
2-nd commit:
@dmitryk-dk @hagen1778 Could you please review it again? |
25b5a2e to
8707e50
Compare
dmitryk-dk
left a comment
There was a problem hiding this comment.
Overall looks good!
One thing what i do not like is that we removed from the tests the real test case
but if you sure it is expected than ok
_, err := w.Write([]byte(`{"_time":"2024-02-20", "_stream":"{application=\"logs-benchmark-Apache.log-1708437847\",hostname=}"}`))
if err != nil {
t.Fatalf("error write reposne: %s", err)
}
I removed this test case, cause I removed the parsing |
- Split response.go into response_logs.go, response_stats.go, response_hits.go - Split response_test.go likewise (response_logs/stats/hits_test.go) - Keep shared types, error parsers and labelsToJSON in response.go
- Build labels via MarshalTo so log fields keep the order returned by VictoriaLogs instead of being sorted alphabetically (#563); _stream and _stream_id are kept as-is, _stream is no longer parsed/expanded - buildLogID now takes []byte and hashes the raw _time/_msg/_stream_id bytes (id keyed on _stream_id), avoiding string<->[]byte round-trips
8707e50 to
e0db154
Compare
Related issue: #563
Describe Your Changes
Checklist
The following checks are mandatory:
Summary by cubic
Preserves VictoriaLogs field order in labels JSON and stops parsing
_stream. Builds stable log IDs from raw_time,_msg, and_stream_idbytes (keyed on_stream_id) to match VL responses and improve stability (#563).fastjson(Delunwanted fields, thenMarshalTo) so log fields keep VL order;_stream/_stream_idare kept as-is.response_logs.go,response_stats.go,response_hits.go; keep shared types/error parsers/labelsToJSONinresponse.go.pkg/utils/stream_fields_parser; update fixtures and droppkg/plugin/test-data/invalid_stream.Written for commit e0db154. Summary will update on new commits.