Added information about guardrails#6
Conversation
cableman
force-pushed
the
feature/guardrails
branch
from
9a5d7bb to
0ee7642
Compare
|
Added a note related to this here os2ai/helm-deployments#28 (comment) I'll review this with my feedback in mind. |
|
@SigneA-hm this should be reviewed and merged |
|
I'll withdraw from this review and let's @hypesystem take the wheel. |
hypesystem
left a comment
hypesystem
left a comment
There was a problem hiding this comment.
Nice, I like the documentation overall, it gives me a good idea of the general approach.
I've added some notes, mostly where I think some more abstract/broader context is warranted, so readers are more likely to understand not just what we are doing, but why 😄
| default_max_context_tokens: 8192 | ||
| max_context_tokens_by_model: | ||
| openai/some-deployed-model: 32768 | ||
| pop_trailing_tool_messages: false |
There was a problem hiding this comment.
I think some comments on what the different parts of this config does would make sense, if we want it here. Why are the settings what they are, how might a user change them?
There was a problem hiding this comment.
Maybe a link to the configuration reference could do a lot of the heavy lifting here. I'm still in doubt as to what default_on does from reading this (and honestly litellm's docs too https://docs.litellm.ai/docs/proxy/guardrails/quick_start#default-on-guardrails)
| a 25% headroom factor for tokens LiteLLM/the provider may add later. | ||
| 3. __Update `max_tokens` / `max_completion_tokens`__ in the request so the model can't be asked for more than fits. | ||
| 4. __Trim input messages__ (`litellm.trim_messages`) if `current_input_tokens > max_input_tokens`, dropping older | ||
| messages from the head until it fits. |
There was a problem hiding this comment.
This seems like a potentially dangerous strategy, if this means losing the initial context of a chat, which is often more significant framing than the details of interaction happening in later messages?
Out of scope for now, I guess, but we could talk about summarizing the old messages instead, maybe?
| - `_repair_tool_call_pairings` — strip orphan `role: tool` messages and orphan `tool_calls` entries that the trimmer | ||
| may have created. | ||
| - (Optional, opt-in via `pop_trailing_tool_messages`) pop trailing `role: tool` messages and re-run the repair, then | ||
| append `"Please continue"` if the new terminus is an assistant message. |
There was a problem hiding this comment.
"Please continue" feels like it could skew the output, especially if the language in the context window otherwise isn't English?
|
|
||
| The message trimming guardrail can be configured in the litellm | ||
| values [file](https://github.com/os2ai/helm-deployments/blob/develop/applications/litellm/litellm-values.yaml#L108) | ||
| configuration file in the helm chart. |
There was a problem hiding this comment.
A note on why we need message trimming would make sense after this paragraph, just very briefly. E.g. what happens if oversized message histories are not trimmed, and how does the guardrail avoid it? In a sentence of two.
| ### Why the trailing-tool pop is opt-in | ||
|
|
||
| The "normal" agent-loop shape ends on a `role: tool` message: | ||
|
|
||
| ```mermaid | ||
| flowchart LR | ||
| U[User] --> A["Assistant{tool_calls}"] | ||
| A --> T["Tool{result}"] | ||
| T --> C([model is asked to continue here]) | ||
| ``` | ||
|
|
||
| Most providers (OpenAI, Anthropic, Google, Mistral via the official APIs) __accept__ this shape — that's how tool | ||
| calling works. Popping the tool message and substituting `"Please continue"` deprives the model of the result it was | ||
| supposed to reason from, so the default is __off__. |
There was a problem hiding this comment.
I don't understand this section. Specifically Popping the tool message and substituting "Please continue" deprives the model of the result it was supposed to reason from, so the default is __off__. doesn't really tell me what happens in the cases where the setting is enabled vs disabled, and what exactly the default behavior is.
What is the effect of depriving the model of the result it was supposed to reason from? (Am I understanding it correctly that this refers to "the result of the tool call", and if so, could we call it that?)
|
|
||
| ## How Message Trimming works | ||
|
|
||
| `async_pre_call_hook` runs on every chat completion request. The flow: |
There was a problem hiding this comment.
Before the step-by-step flow, I think a sentence stating what the pros and cons of our approach is would make sense.
E.g. "Sending a too large message to the model can be fatal for the entire conversation, so we take a conservative approach in estimating a safe completion budget for the message" and then explain what a safe completion budget is, why we calculate it as is? I think that would give a lot of good context for evaluating the approach.
hypesystem
left a comment
hypesystem
left a comment
There was a problem hiding this comment.
As discussed in today's meeting, here's a suggested text explaining the context between Open WebUI and LiteLLM and how guardrails are attached.
| nav_order: 7 | ||
| --- | ||
|
|
||
| # LiteLLM guardrails |
There was a problem hiding this comment.
Suggestion: as the first bit of this section, we could add a brief explanation of how guardrails work and are applied, which would also clear up some confusion about e.g. the default_on parameter as discussed in os2ai/Feedback#1 (comment)
A suggested text you can adopt or modify:
LiteLLM guardrails are a way to execute code on and around input and output sent to and from
models managed by LiteLLM.
The Open WebUI instance sends requests to LiteLLM, but does not support setting guardrails on
a per-request basis. For this reason, guardrails that should execute on messages from Open
WebUI should be set as always on, by setting the `default_on` setting to `true` when declaring
the guardrail (see below).4 participants
mermaiddiagrams in this docs site