-
Notifications
You must be signed in to change notification settings - Fork 470
chore(llmobs): [MLOB-4813] log warning if litellm imported before being patched #15530
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
|
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 248 ± 2 ms. The average import time from base is: 252 ± 3 ms. The import time difference between this PR and base is: -3.8 ± 0.1 ms. Import time breakdownThe following import paths have shrunk:
|
Performance SLOsComparing candidate nicole-cybul/litellm-patching-issue (c1fea00) with baseline main (01850f2) 📈 Performance Regressions (2 suites)📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.116µs (SLO: <10.000µs 📉 -48.8%) vs baseline: 📈 +21.4% Memory: ✅ 38.633MB (SLO: <41.000MB -5.8%) vs baseline: +5.2% ✅ ospathbasename_noaspectTime: ✅ 1.091µs (SLO: <10.000µs 📉 -89.1%) vs baseline: +0.6% Memory: ✅ 38.633MB (SLO: <41.000MB -5.8%) vs baseline: +5.0% ✅ ospathjoin_aspectTime: ✅ 6.120µs (SLO: <10.000µs 📉 -38.8%) vs baseline: +1.1% Memory: ✅ 38.555MB (SLO: <41.000MB -6.0%) vs baseline: +4.8% ✅ ospathjoin_noaspectTime: ✅ 2.288µs (SLO: <10.000µs 📉 -77.1%) vs baseline: +0.1% Memory: ✅ 38.555MB (SLO: <41.000MB -6.0%) vs baseline: +5.2% ✅ ospathnormcase_aspectTime: ✅ 3.446µs (SLO: <10.000µs 📉 -65.5%) vs baseline: -1.0% Memory: ✅ 38.535MB (SLO: <41.000MB -6.0%) vs baseline: +4.8% ✅ ospathnormcase_noaspectTime: ✅ 0.569µs (SLO: <10.000µs 📉 -94.3%) vs baseline: -0.5% Memory: ✅ 38.633MB (SLO: <41.000MB -5.8%) vs baseline: +5.0% ✅ ospathsplit_aspectTime: ✅ 4.888µs (SLO: <10.000µs 📉 -51.1%) vs baseline: +1.3% Memory: ✅ 38.673MB (SLO: <41.000MB -5.7%) vs baseline: +5.1% ✅ ospathsplit_noaspectTime: ✅ 1.588µs (SLO: <10.000µs 📉 -84.1%) vs baseline: +0.2% Memory: ✅ 38.574MB (SLO: <41.000MB -5.9%) vs baseline: +5.1% ✅ ospathsplitdrive_aspectTime: ✅ 3.660µs (SLO: <10.000µs 📉 -63.4%) vs baseline: -0.2% Memory: ✅ 38.437MB (SLO: <41.000MB -6.3%) vs baseline: +4.9% ✅ ospathsplitdrive_noaspectTime: ✅ 0.701µs (SLO: <10.000µs 📉 -93.0%) vs baseline: -0.3% Memory: ✅ 38.555MB (SLO: <41.000MB -6.0%) vs baseline: +4.5% ✅ ospathsplitext_aspectTime: ✅ 4.550µs (SLO: <10.000µs 📉 -54.5%) vs baseline: +0.5% Memory: ✅ 38.555MB (SLO: <41.000MB -6.0%) vs baseline: +4.6% ✅ ospathsplitext_noaspectTime: ✅ 1.383µs (SLO: <10.000µs 📉 -86.2%) vs baseline: -0.4% Memory: ✅ 38.614MB (SLO: <41.000MB -5.8%) vs baseline: +4.9% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.335µs (SLO: <20.000µs 📉 -83.3%) vs baseline: 📈 +15.6% Memory: ✅ 35.055MB (SLO: <35.500MB 🟡 -1.3%) vs baseline: +4.9% ✅ 1-count-metrics-100-timesTime: ✅ 197.603µs (SLO: <220.000µs 📉 -10.2%) vs baseline: -2.1% Memory: ✅ 35.075MB (SLO: <35.500MB 🟡 -1.2%) vs baseline: +5.0% ✅ 1-distribution-metric-1-timesTime: ✅ 3.171µs (SLO: <20.000µs 📉 -84.1%) vs baseline: -1.2% Memory: ✅ 35.036MB (SLO: <35.500MB 🟡 -1.3%) vs baseline: +4.8% ✅ 1-distribution-metrics-100-timesTime: ✅ 211.624µs (SLO: <230.000µs -8.0%) vs baseline: -1.9% Memory: ✅ 35.075MB (SLO: <35.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ 1-gauge-metric-1-timesTime: ✅ 2.144µs (SLO: <20.000µs 📉 -89.3%) vs baseline: -1.7% Memory: ✅ 35.055MB (SLO: <35.500MB 🟡 -1.3%) vs baseline: +4.7% ✅ 1-gauge-metrics-100-timesTime: ✅ 137.171µs (SLO: <150.000µs -8.6%) vs baseline: +0.6% Memory: ✅ 35.016MB (SLO: <35.500MB 🟡 -1.4%) vs baseline: +4.9% ✅ 1-rate-metric-1-timesTime: ✅ 2.989µs (SLO: <20.000µs 📉 -85.1%) vs baseline: -1.5% Memory: ✅ 35.055MB (SLO: <35.500MB 🟡 -1.3%) vs baseline: +4.9% ✅ 1-rate-metrics-100-timesTime: ✅ 213.213µs (SLO: <250.000µs 📉 -14.7%) vs baseline: -0.4% Memory: ✅ 35.114MB (SLO: <35.500MB 🟡 -1.1%) vs baseline: +5.1% ✅ 100-count-metrics-100-timesTime: ✅ 20.065ms (SLO: <22.000ms -8.8%) vs baseline: -0.3% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.9% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.221ms (SLO: <2.550ms 📉 -12.9%) vs baseline: -0.4% Memory: ✅ 35.095MB (SLO: <35.500MB 🟡 -1.1%) vs baseline: +5.0% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.409ms (SLO: <1.550ms -9.1%) vs baseline: -0.2% Memory: ✅ 34.701MB (SLO: <35.500MB -2.2%) vs baseline: +4.7% ✅ 100-rate-metrics-100-timesTime: ✅ 2.192ms (SLO: <2.550ms 📉 -14.0%) vs baseline: +0.3% Memory: ✅ 34.760MB (SLO: <35.500MB -2.1%) vs baseline: +5.0% ✅ flush-1-metricTime: ✅ 4.518µs (SLO: <20.000µs 📉 -77.4%) vs baseline: +0.3% Memory: ✅ 35.055MB (SLO: <35.500MB 🟡 -1.3%) vs baseline: +4.9% ✅ flush-100-metricsTime: ✅ 174.857µs (SLO: <250.000µs 📉 -30.1%) vs baseline: +0.7% Memory: ✅ 35.016MB (SLO: <35.500MB 🟡 -1.4%) vs baseline: +4.3% ✅ flush-1000-metricsTime: ✅ 2.181ms (SLO: <2.500ms 📉 -12.7%) vs baseline: ~same Memory: ✅ 35.881MB (SLO: <36.500MB 🟡 -1.7%) vs baseline: +4.8% 🟡 Near SLO Breach (14 suites)🟡 djangosimple - 30/30✅ appsecTime: ✅ 19.629ms (SLO: <22.300ms 📉 -12.0%) vs baseline: +0.5% Memory: ✅ 68.203MB (SLO: <70.500MB -3.3%) vs baseline: +4.9% ✅ exception-replay-enabledTime: ✅ 1.362ms (SLO: <1.450ms -6.1%) vs baseline: -0.4% Memory: ✅ 66.228MB (SLO: <67.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ iastTime: ✅ 19.608ms (SLO: <22.250ms 📉 -11.9%) vs baseline: -0.2% Memory: ✅ 68.084MB (SLO: <70.000MB -2.7%) vs baseline: +4.9% ✅ profilerTime: ✅ 15.432ms (SLO: <16.550ms -6.8%) vs baseline: +0.4% Memory: ✅ 56.393MB (SLO: <57.500MB 🟡 -1.9%) vs baseline: +4.9% ✅ resource-renamingTime: ✅ 19.540ms (SLO: <21.750ms 📉 -10.2%) vs baseline: +0.7% Memory: ✅ 68.105MB (SLO: <70.500MB -3.4%) vs baseline: +4.8% ✅ span-code-originTime: ✅ 19.949ms (SLO: <28.200ms 📉 -29.3%) vs baseline: +1.2% Memory: ✅ 68.203MB (SLO: <71.000MB -3.9%) vs baseline: +5.0% ✅ tracerTime: ✅ 19.604ms (SLO: <21.750ms -9.9%) vs baseline: ~same Memory: ✅ 68.164MB (SLO: <70.000MB -2.6%) vs baseline: +5.0% ✅ tracer-and-profilerTime: ✅ 21.777ms (SLO: <23.500ms -7.3%) vs baseline: -0.2% Memory: ✅ 69.245MB (SLO: <71.000MB -2.5%) vs baseline: +4.8% ✅ tracer-dont-create-db-spansTime: ✅ 19.653ms (SLO: <21.500ms -8.6%) vs baseline: +0.3% Memory: ✅ 68.144MB (SLO: <70.000MB -2.7%) vs baseline: +4.8% ✅ tracer-minimalTime: ✅ 16.790ms (SLO: <17.500ms -4.1%) vs baseline: +0.2% Memory: ✅ 67.790MB (SLO: <70.000MB -3.2%) vs baseline: +4.6% ✅ tracer-nativeTime: ✅ 19.429ms (SLO: <21.750ms 📉 -10.7%) vs baseline: ~same Memory: ✅ 68.164MB (SLO: <72.500MB -6.0%) vs baseline: +4.6% ✅ tracer-no-cachesTime: ✅ 17.618ms (SLO: <19.650ms 📉 -10.3%) vs baseline: ~same Memory: ✅ 68.066MB (SLO: <70.000MB -2.8%) vs baseline: +4.7% ✅ tracer-no-databasesTime: ✅ 19.146ms (SLO: <20.100ms -4.7%) vs baseline: +0.3% Memory: ✅ 67.869MB (SLO: <70.000MB -3.0%) vs baseline: +4.9% ✅ tracer-no-middlewareTime: ✅ 19.284ms (SLO: <21.500ms 📉 -10.3%) vs baseline: ~same Memory: ✅ 68.223MB (SLO: <70.000MB -2.5%) vs baseline: +4.9% ✅ tracer-no-templatesTime: ✅ 19.671ms (SLO: <22.000ms 📉 -10.6%) vs baseline: +1.4% Memory: ✅ 68.158MB (SLO: <70.500MB -3.3%) vs baseline: +4.9% 🟡 errortrackingdjangosimple - 6/6✅ errortracking-enabled-allTime: ✅ 16.303ms (SLO: <19.850ms 📉 -17.9%) vs baseline: -0.5% Memory: ✅ 69.776MB (SLO: <70.000MB 🟡 -0.3%) vs baseline: +4.8% ✅ errortracking-enabled-userTime: ✅ 16.310ms (SLO: <19.400ms 📉 -15.9%) vs baseline: -0.2% Memory: ✅ 69.855MB (SLO: <70.000MB 🟡 -0.2%) vs baseline: +5.0% ✅ tracer-enabledTime: ✅ 16.371ms (SLO: <19.450ms 📉 -15.8%) vs baseline: +0.2% Memory: ✅ 69.816MB (SLO: <70.000MB 🟡 -0.3%) vs baseline: +5.0% 🟡 errortrackingflasksqli - 6/6✅ errortracking-enabled-allTime: ✅ 2.068ms (SLO: <2.300ms 📉 -10.1%) vs baseline: +0.1% Memory: ✅ 55.758MB (SLO: <56.500MB 🟡 -1.3%) vs baseline: +4.9% ✅ errortracking-enabled-userTime: ✅ 2.074ms (SLO: <2.250ms -7.8%) vs baseline: +0.1% Memory: ✅ 55.817MB (SLO: <56.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ tracer-enabledTime: ✅ 2.064ms (SLO: <2.300ms 📉 -10.3%) vs baseline: ~same Memory: ✅ 55.679MB (SLO: <56.500MB 🟡 -1.5%) vs baseline: +4.7% 🟡 flasksimple - 18/18✅ appsec-getTime: ✅ 3.381ms (SLO: <4.750ms 📉 -28.8%) vs baseline: -0.2% Memory: ✅ 55.457MB (SLO: <66.500MB 📉 -16.6%) vs baseline: +4.8% ✅ appsec-postTime: ✅ 2.866ms (SLO: <6.750ms 📉 -57.5%) vs baseline: +0.4% Memory: ✅ 55.715MB (SLO: <66.500MB 📉 -16.2%) vs baseline: +4.9% ✅ appsec-telemetryTime: ✅ 3.390ms (SLO: <4.750ms 📉 -28.6%) vs baseline: +0.7% Memory: ✅ 55.471MB (SLO: <66.500MB 📉 -16.6%) vs baseline: +5.0% ✅ debuggerTime: ✅ 1.867ms (SLO: <2.000ms -6.7%) vs baseline: ~same Memory: ✅ 47.975MB (SLO: <49.500MB -3.1%) vs baseline: +5.0% ✅ iast-getTime: ✅ 1.854ms (SLO: <2.000ms -7.3%) vs baseline: ~same Memory: ✅ 44.494MB (SLO: <49.000MB -9.2%) vs baseline: +4.5% ✅ profilerTime: ✅ 1.907ms (SLO: <2.100ms -9.2%) vs baseline: ~same Memory: ✅ 48.782MB (SLO: <50.000MB -2.4%) vs baseline: +4.8% ✅ resource-renamingTime: ✅ 3.349ms (SLO: <3.650ms -8.2%) vs baseline: -0.2% Memory: ✅ 55.374MB (SLO: <56.000MB 🟡 -1.1%) vs baseline: +4.8% ✅ tracerTime: ✅ 3.365ms (SLO: <3.650ms -7.8%) vs baseline: ~same Memory: ✅ 55.447MB (SLO: <56.500MB 🟡 -1.9%) vs baseline: +4.9% ✅ tracer-nativeTime: ✅ 3.369ms (SLO: <3.650ms -7.7%) vs baseline: ~same Memory: ✅ 55.359MB (SLO: <60.000MB -7.7%) vs baseline: +4.7% 🟡 flasksqli - 6/6✅ appsec-enabledTime: ✅ 2.062ms (SLO: <4.200ms 📉 -50.9%) vs baseline: +0.2% Memory: ✅ 55.758MB (SLO: <66.000MB 📉 -15.5%) vs baseline: +4.7% ✅ iast-enabledTime: ✅ 2.076ms (SLO: <2.800ms 📉 -25.9%) vs baseline: +0.5% Memory: ✅ 55.797MB (SLO: <62.500MB 📉 -10.7%) vs baseline: +4.9% ✅ tracer-enabledTime: ✅ 2.057ms (SLO: <2.250ms -8.6%) vs baseline: -0.2% Memory: ✅ 55.738MB (SLO: <56.500MB 🟡 -1.3%) vs baseline: +4.8% 🟡 httppropagationextract - 60/60✅ all_styles_all_headersTime: ✅ 80.763µs (SLO: <100.000µs 📉 -19.2%) vs baseline: -0.7% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ b3_headersTime: ✅ 14.197µs (SLO: <20.000µs 📉 -29.0%) vs baseline: ~same Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.0% ✅ b3_single_headersTime: ✅ 13.254µs (SLO: <20.000µs 📉 -33.7%) vs baseline: -0.5% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.1% ✅ datadog_tracecontext_tracestate_not_propagated_on_trace_id_no_matchTime: ✅ 63.703µs (SLO: <80.000µs 📉 -20.4%) vs baseline: -0.2% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.9% ✅ datadog_tracecontext_tracestate_propagated_on_trace_id_matchTime: ✅ 69.373µs (SLO: <80.000µs 📉 -13.3%) vs baseline: +5.0% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.7% ✅ empty_headersTime: ✅ 1.629µs (SLO: <10.000µs 📉 -83.7%) vs baseline: +1.3% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ full_t_id_datadog_headersTime: ✅ 22.537µs (SLO: <30.000µs 📉 -24.9%) vs baseline: +0.3% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.6% ✅ invalid_priority_headerTime: ✅ 6.566µs (SLO: <10.000µs 📉 -34.3%) vs baseline: +0.8% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.7% ✅ invalid_span_id_headerTime: ✅ 6.536µs (SLO: <10.000µs 📉 -34.6%) vs baseline: -0.2% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ invalid_tags_headerTime: ✅ 6.518µs (SLO: <10.000µs 📉 -34.8%) vs baseline: +0.5% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.5% ✅ invalid_trace_id_headerTime: ✅ 6.521µs (SLO: <10.000µs 📉 -34.8%) vs baseline: +0.4% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.4% ✅ large_header_no_matchesTime: ✅ 27.593µs (SLO: <30.000µs -8.0%) vs baseline: -1.0% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.6% ✅ large_valid_headers_allTime: ✅ 28.777µs (SLO: <40.000µs 📉 -28.1%) vs baseline: ~same Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.6% ✅ medium_header_no_matchesTime: ✅ 9.883µs (SLO: <20.000µs 📉 -50.6%) vs baseline: +0.6% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ medium_valid_headers_allTime: ✅ 11.344µs (SLO: <20.000µs 📉 -43.3%) vs baseline: +0.5% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ none_propagation_styleTime: ✅ 1.728µs (SLO: <10.000µs 📉 -82.7%) vs baseline: +0.2% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.2% ✅ tracecontext_headersTime: ✅ 34.770µs (SLO: <40.000µs 📉 -13.1%) vs baseline: -0.2% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ valid_headers_allTime: ✅ 6.525µs (SLO: <10.000µs 📉 -34.8%) vs baseline: -0.1% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.7% ✅ valid_headers_basicTime: ✅ 6.107µs (SLO: <10.000µs 📉 -38.9%) vs baseline: ~same Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.8% ✅ wsgi_empty_headersTime: ✅ 1.623µs (SLO: <10.000µs 📉 -83.8%) vs baseline: +1.1% Memory: ✅ 34.721MB (SLO: <35.500MB -2.2%) vs baseline: +4.3% ✅ wsgi_invalid_priority_headerTime: ✅ 6.594µs (SLO: <10.000µs 📉 -34.1%) vs baseline: +0.6% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.0% ✅ wsgi_invalid_span_id_headerTime: ✅ 1.617µs (SLO: <10.000µs 📉 -83.8%) vs baseline: +0.2% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ wsgi_invalid_tags_headerTime: ✅ 6.587µs (SLO: <10.000µs 📉 -34.1%) vs baseline: +0.2% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.6% ✅ wsgi_invalid_trace_id_headerTime: ✅ 6.618µs (SLO: <10.000µs 📉 -33.8%) vs baseline: +1.2% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.7% ✅ wsgi_large_header_no_matchesTime: ✅ 28.777µs (SLO: <40.000µs 📉 -28.1%) vs baseline: ~same Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.6% ✅ wsgi_large_valid_headers_allTime: ✅ 29.806µs (SLO: <40.000µs 📉 -25.5%) vs baseline: ~same Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.1% ✅ wsgi_medium_header_no_matchesTime: ✅ 10.283µs (SLO: <20.000µs 📉 -48.6%) vs baseline: +0.2% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.3% ✅ wsgi_medium_valid_headers_allTime: ✅ 11.642µs (SLO: <20.000µs 📉 -41.8%) vs baseline: ~same Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.0% ✅ wsgi_valid_headers_allTime: ✅ 6.584µs (SLO: <10.000µs 📉 -34.2%) vs baseline: +0.5% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ wsgi_valid_headers_basicTime: ✅ 6.160µs (SLO: <10.000µs 📉 -38.4%) vs baseline: +1.2% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% 🟡 httppropagationinject - 16/16✅ ids_onlyTime: ✅ 21.854µs (SLO: <30.000µs 📉 -27.2%) vs baseline: +4.7% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.7% ✅ with_allTime: ✅ 27.619µs (SLO: <40.000µs 📉 -31.0%) vs baseline: +0.4% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ with_dd_originTime: ✅ 24.715µs (SLO: <30.000µs 📉 -17.6%) vs baseline: +0.3% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ with_priority_and_originTime: ✅ 24.016µs (SLO: <40.000µs 📉 -40.0%) vs baseline: +0.4% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.6% ✅ with_sampling_priorityTime: ✅ 20.998µs (SLO: <30.000µs 📉 -30.0%) vs baseline: +0.9% Memory: ✅ 34.859MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.9% ✅ with_tagsTime: ✅ 25.734µs (SLO: <40.000µs 📉 -35.7%) vs baseline: -0.3% Memory: ✅ 34.898MB (SLO: <35.500MB 🟡 -1.7%) vs baseline: +5.2% ✅ with_tags_invalidTime: ✅ 27.299µs (SLO: <40.000µs 📉 -31.8%) vs baseline: -0.3% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ with_tags_max_sizeTime: ✅ 26.226µs (SLO: <40.000µs 📉 -34.4%) vs baseline: +0.3% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +4.7% 🟡 otelspan - 22/22✅ add-eventTime: ✅ 40.328ms (SLO: <47.150ms 📉 -14.5%) vs baseline: ~same Memory: ✅ 39.550MB (SLO: <47.000MB 📉 -15.9%) vs baseline: +4.9% ✅ add-metricsTime: ✅ 261.044ms (SLO: <344.800ms 📉 -24.3%) vs baseline: -0.2% Memory: ✅ 43.868MB (SLO: <47.500MB -7.6%) vs baseline: +5.1% ✅ add-tagsTime: ✅ 317.810ms (SLO: <321.000ms 🟡 -1.0%) vs baseline: +0.3% Memory: ✅ 43.719MB (SLO: <47.500MB -8.0%) vs baseline: +4.9% ✅ get-contextTime: ✅ 80.542ms (SLO: <92.350ms 📉 -12.8%) vs baseline: +0.2% Memory: ✅ 39.776MB (SLO: <46.500MB 📉 -14.5%) vs baseline: +4.5% ✅ is-recordingTime: ✅ 38.003ms (SLO: <44.500ms 📉 -14.6%) vs baseline: ~same Memory: ✅ 39.515MB (SLO: <47.500MB 📉 -16.8%) vs baseline: +4.9% ✅ record-exceptionTime: ✅ 58.853ms (SLO: <67.650ms 📉 -13.0%) vs baseline: +0.1% Memory: ✅ 40.071MB (SLO: <47.000MB 📉 -14.7%) vs baseline: +5.1% ✅ set-statusTime: ✅ 44.281ms (SLO: <50.400ms 📉 -12.1%) vs baseline: ~same Memory: ✅ 39.539MB (SLO: <47.000MB 📉 -15.9%) vs baseline: +5.1% ✅ startTime: ✅ 38.281ms (SLO: <43.450ms 📉 -11.9%) vs baseline: +2.7% Memory: ✅ 39.482MB (SLO: <47.000MB 📉 -16.0%) vs baseline: +4.8% ✅ start-finishTime: ✅ 82.781ms (SLO: <88.000ms -5.9%) vs baseline: ~same Memory: ✅ 37.297MB (SLO: <46.500MB 📉 -19.8%) vs baseline: +4.5% ✅ start-finish-telemetryTime: ✅ 84.269ms (SLO: <89.000ms -5.3%) vs baseline: +0.1% Memory: ✅ 37.316MB (SLO: <46.500MB 📉 -19.8%) vs baseline: +4.8% ✅ update-nameTime: ✅ 38.990ms (SLO: <45.150ms 📉 -13.6%) vs baseline: +1.2% Memory: ✅ 39.663MB (SLO: <47.000MB 📉 -15.6%) vs baseline: +4.9% 🟡 ratelimiter - 12/12✅ defaultsTime: ✅ 2.367µs (SLO: <10.000µs 📉 -76.3%) vs baseline: +0.5% Memory: ✅ 34.977MB (SLO: <35.500MB 🟡 -1.5%) vs baseline: +4.6% ✅ high_rate_limitTime: ✅ 2.433µs (SLO: <10.000µs 📉 -75.7%) vs baseline: +0.7% Memory: ✅ 35.134MB (SLO: <35.500MB 🟡 -1.0%) vs baseline: +5.1% ✅ long_windowTime: ✅ 2.361µs (SLO: <10.000µs 📉 -76.4%) vs baseline: +0.2% Memory: ✅ 35.154MB (SLO: <35.500MB 🟡 -1.0%) vs baseline: +4.9% ✅ low_rate_limitTime: ✅ 2.356µs (SLO: <10.000µs 📉 -76.4%) vs baseline: +0.7% Memory: ✅ 35.075MB (SLO: <35.500MB 🟡 -1.2%) vs baseline: +4.9% ✅ no_rate_limitTime: ✅ 0.821µs (SLO: <10.000µs 📉 -91.8%) vs baseline: -0.4% Memory: ✅ 35.095MB (SLO: <35.500MB 🟡 -1.1%) vs baseline: +4.9% ✅ short_windowTime: ✅ 2.480µs (SLO: <10.000µs 📉 -75.2%) vs baseline: -0.7% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +3.9% 🟡 recursivecomputation - 8/8✅ deepTime: ✅ 309.280ms (SLO: <320.950ms -3.6%) vs baseline: +0.2% Memory: ✅ 35.979MB (SLO: <36.500MB 🟡 -1.4%) vs baseline: +4.9% ✅ deep-profiledTime: ✅ 328.738ms (SLO: <359.150ms -8.5%) vs baseline: ~same Memory: ✅ 39.813MB (SLO: <40.500MB 🟡 -1.7%) vs baseline: +4.9% ✅ mediumTime: ✅ 6.987ms (SLO: <7.400ms -5.6%) vs baseline: ~same Memory: ✅ 34.760MB (SLO: <35.500MB -2.1%) vs baseline: +4.9% ✅ shallowTime: ✅ 0.947ms (SLO: <1.050ms -9.9%) vs baseline: +1.2% Memory: ✅ 34.878MB (SLO: <35.500MB 🟡 -1.8%) vs baseline: +5.0% 🟡 samplingrules - 8/8✅ average_matchTime: ✅ 136.344µs (SLO: <290.000µs 📉 -53.0%) vs baseline: -0.3% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ high_matchTime: ✅ 175.233µs (SLO: <480.000µs 📉 -63.5%) vs baseline: +1.3% Memory: ✅ 34.760MB (SLO: <35.500MB -2.1%) vs baseline: +4.9% ✅ low_matchTime: ✅ 98.321µs (SLO: <120.000µs 📉 -18.1%) vs baseline: -1.1% Memory: ✅ 603.503MB (SLO: <700.000MB 📉 -13.8%) vs baseline: +4.9% ✅ very_low_matchTime: ✅ 2.661ms (SLO: <8.500ms 📉 -68.7%) vs baseline: -0.4% Memory: ✅ 71.004MB (SLO: <75.000MB -5.3%) vs baseline: +4.7% 🟡 sethttpmeta - 32/32✅ all-disabledTime: ✅ 10.610µs (SLO: <20.000µs 📉 -46.9%) vs baseline: +0.5% Memory: ✅ 35.586MB (SLO: <36.000MB 🟡 -1.1%) vs baseline: +4.7% ✅ all-enabledTime: ✅ 41.006µs (SLO: <50.000µs 📉 -18.0%) vs baseline: +1.8% Memory: ✅ 35.547MB (SLO: <36.000MB 🟡 -1.3%) vs baseline: +4.7% ✅ collectipvariant_existsTime: ✅ 40.679µs (SLO: <50.000µs 📉 -18.6%) vs baseline: -0.5% Memory: ✅ 35.389MB (SLO: <36.000MB 🟡 -1.7%) vs baseline: +4.0% ✅ no-collectipvariantTime: ✅ 40.163µs (SLO: <50.000µs 📉 -19.7%) vs baseline: -0.3% Memory: ✅ 35.271MB (SLO: <36.000MB -2.0%) vs baseline: +3.6% ✅ no-useragentvariantTime: ✅ 38.677µs (SLO: <50.000µs 📉 -22.6%) vs baseline: -1.2% Memory: ✅ 35.330MB (SLO: <36.000MB 🟡 -1.9%) vs baseline: +4.0% ✅ obfuscation-no-queryTime: ✅ 40.608µs (SLO: <50.000µs 📉 -18.8%) vs baseline: -0.1% Memory: ✅ 35.488MB (SLO: <36.000MB 🟡 -1.4%) vs baseline: +4.3% ✅ obfuscation-regular-case-explicit-queryTime: ✅ 75.988µs (SLO: <90.000µs 📉 -15.6%) vs baseline: +0.5% Memory: ✅ 35.586MB (SLO: <36.500MB -2.5%) vs baseline: +4.3% ✅ obfuscation-regular-case-implicit-queryTime: ✅ 76.619µs (SLO: <90.000µs 📉 -14.9%) vs baseline: +0.2% Memory: ✅ 35.645MB (SLO: <36.500MB -2.3%) vs baseline: +4.7% ✅ obfuscation-send-querystring-disabledTime: ✅ 154.460µs (SLO: <170.000µs -9.1%) vs baseline: +0.4% Memory: ✅ 35.724MB (SLO: <36.500MB -2.1%) vs baseline: +5.0% ✅ obfuscation-worst-case-explicit-queryTime: ✅ 148.570µs (SLO: <160.000µs -7.1%) vs baseline: ~same Memory: ✅ 35.606MB (SLO: <36.500MB -2.5%) vs baseline: +4.6% ✅ obfuscation-worst-case-implicit-queryTime: ✅ 154.947µs (SLO: <170.000µs -8.9%) vs baseline: ~same Memory: ✅ 35.665MB (SLO: <36.500MB -2.3%) vs baseline: +4.9% ✅ useragentvariant_exists_1Time: ✅ 39.596µs (SLO: <50.000µs 📉 -20.8%) vs baseline: +0.2% Memory: ✅ 35.330MB (SLO: <36.000MB 🟡 -1.9%) vs baseline: +3.8% ✅ useragentvariant_exists_2Time: ✅ 40.711µs (SLO: <50.000µs 📉 -18.6%) vs baseline: -0.4% Memory: ✅ 35.468MB (SLO: <36.000MB 🟡 -1.5%) vs baseline: +4.1% ✅ useragentvariant_exists_3Time: ✅ 40.142µs (SLO: <50.000µs 📉 -19.7%) vs baseline: -0.2% Memory: ✅ 35.547MB (SLO: <36.000MB 🟡 -1.3%) vs baseline: +4.5% ✅ useragentvariant_not_exists_1Time: ✅ 39.645µs (SLO: <50.000µs 📉 -20.7%) vs baseline: -0.4% Memory: ✅ 35.429MB (SLO: <36.000MB 🟡 -1.6%) vs baseline: +4.0% ✅ useragentvariant_not_exists_2Time: ✅ 39.593µs (SLO: <50.000µs 📉 -20.8%) vs baseline: -0.3% Memory: ✅ 35.311MB (SLO: <36.000MB 🟡 -1.9%) vs baseline: +3.8% 🟡 span - 26/26✅ add-eventTime: ✅ 18.098ms (SLO: <22.500ms 📉 -19.6%) vs baseline: -0.5% Memory: ✅ 37.017MB (SLO: <53.000MB 📉 -30.2%) vs baseline: +5.0% ✅ add-metricsTime: ✅ 88.469ms (SLO: <93.500ms -5.4%) vs baseline: +0.8% Memory: ✅ 41.079MB (SLO: <53.000MB 📉 -22.5%) vs baseline: +5.0% ✅ add-tagsTime: ✅ 140.850ms (SLO: <155.000ms -9.1%) vs baseline: -1.0% Memory: ✅ 41.085MB (SLO: <53.000MB 📉 -22.5%) vs baseline: +5.1% ✅ get-contextTime: ✅ 16.949ms (SLO: <20.500ms 📉 -17.3%) vs baseline: -0.3% Memory: ✅ 36.761MB (SLO: <53.000MB 📉 -30.6%) vs baseline: +4.8% ✅ is-recordingTime: ✅ 17.142ms (SLO: <20.500ms 📉 -16.4%) vs baseline: -0.5% Memory: ✅ 36.685MB (SLO: <53.000MB 📉 -30.8%) vs baseline: +4.4% ✅ record-exceptionTime: ✅ 36.542ms (SLO: <40.000ms -8.6%) vs baseline: ~same Memory: ✅ 37.329MB (SLO: <53.000MB 📉 -29.6%) vs baseline: +4.7% ✅ set-statusTime: ✅ 18.667ms (SLO: <22.000ms 📉 -15.2%) vs baseline: +0.2% Memory: ✅ 36.703MB (SLO: <53.000MB 📉 -30.7%) vs baseline: +4.8% ✅ startTime: ✅ 17.321ms (SLO: <20.500ms 📉 -15.5%) vs baseline: +3.2% Memory: ✅ 36.762MB (SLO: <53.000MB 📉 -30.6%) vs baseline: +4.6% ✅ start-finishTime: ✅ 50.986ms (SLO: <52.500ms -2.9%) vs baseline: -0.2% Memory: ✅ 34.839MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.9% ✅ start-finish-telemetryTime: ✅ 52.272ms (SLO: <54.500ms -4.1%) vs baseline: ~same Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +4.9% ✅ start-finish-traceid128Time: ✅ 53.963ms (SLO: <57.000ms -5.3%) vs baseline: -0.6% Memory: ✅ 34.721MB (SLO: <35.500MB -2.2%) vs baseline: +4.6% ✅ start-traceid128Time: ✅ 17.244ms (SLO: <22.500ms 📉 -23.4%) vs baseline: -0.1% Memory: ✅ 36.723MB (SLO: <53.000MB 📉 -30.7%) vs baseline: +5.0% ✅ update-nameTime: ✅ 17.356ms (SLO: <22.000ms 📉 -21.1%) vs baseline: +0.6% Memory: ✅ 36.901MB (SLO: <53.000MB 📉 -30.4%) vs baseline: +5.0% 🟡 tracer - 6/6✅ largeTime: ✅ 29.118ms (SLO: <32.950ms 📉 -11.6%) vs baseline: -0.2% Memory: ✅ 35.940MB (SLO: <36.500MB 🟡 -1.5%) vs baseline: +5.1% ✅ mediumTime: ✅ 2.863ms (SLO: <3.200ms 📉 -10.5%) vs baseline: +0.2% Memory: ✅ 34.800MB (SLO: <35.500MB 🟡 -2.0%) vs baseline: +5.0% ✅ smallTime: ✅ 331.902µs (SLO: <370.000µs 📉 -10.3%) vs baseline: +1.5% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.8%
|
Kyle-Verhoog
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just one suggestion on the log line 🚢
Co-authored-by: kyle <kyle@verhoog.ca>
Description
Because the litellm integration patches functions that may already have been imported before the patching occurs, I am adding a log that will warn customers if
LLMObs.enable()is called after litellm is imported but before it is patched.The warning suggests a few ways that users can fix this tracing issue, including moving
LLMObs.enable()before importing litellm.Note that this is not a perfect solution. We only really care about whether the user imported any patched methods like
litellm.completion. If they are doing a genericimport litellmstatement, they should not have any tracing issues and this warning will be irrelevant. I am not aware of a way to know which specific methods the user has imported, so checking forlitellminsys.modulesseems like the next best approach.Testing
Currently, tracing is broken for a situation like the following where the code is run with
python3 app.py.Running this code now, produces the following warning message:
Implementing any of these alternatives (as shown below) will silence the warning message.
Risks
Additional Notes