Tool/Function calling UI

[gadenbuie]

Tool Call Current State

Tool calling is currently entirely limited to the back-end chat interface. Tools are registered with ellmer or chatlas (see below), but shinychat does not do anything in the UI to indicate that a tool call is being made.

For background, here are how ellmer and chatlas register and store tool definitions.

ellmer

library(ellmer)

get_current_time <- function(tz = "UTC") {
  format(Sys.time(), tz = tz, usetz = TRUE)
}

tool_get_current_time <- tool(
  get_current_time,
  .description = "Gets the current time in the given time zone.",
  tz = type_string(
    "The time zone to get the current time in. Defaults to `\"UTC\"`.",
    required = FALSE
  )
)

chat <- chat_openai(model = "gpt-4o", echo = "all")
chat$register_tool(tool_get_current_time)

chat$chat("How long ago exactly was the moment Neil Armstrong touched down on the moon?")
#> > How long ago exactly was the moment Neil Armstrong touched down on the moon?
#> < [tool request (call_WkRmaly9E7kgpMB5RPWVzijh)]: get_current_time(tz = "UTC")
#> < [tool request (call_0OqdPpugMw3wjX9IEz1xVwxd)]: get_current_time(tz = 
#> < "America/New_York")
#> > [tool result  (call_WkRmaly9E7kgpMB5RPWVzijh)]: 2025-02-27 17:53:17 UTC
#> > [tool result  (call_0OqdPpugMw3wjX9IEz1xVwxd)]: 2025-02-27 12:53:17 EST
#> < Neil Armstrong touched down on the moon on July 20, 1969, at 20:17 UTC.
#> < 
#> < As of now, which is February 27, 2025, at 17:53 UTC, it has been 
#> < approximately 55 years, 7 months, and 7 days since that historic moment.
#> <

Internally, tool() creates a ToolDef instance with name, description and arguments properties.

chatlas

import requests
from chatlas import ChatOpenAI
from dotenv import load_dotenv

load_dotenv()

def get_current_temperature(latitude: float, longitude: float):
    """
    Get the current weather given a latitude and longitude.

    Parameters
    ----------
    latitude
        The latitude of the location.
    longitude
        The longitude of the location.
    """
    lat_lng = f"latitude={latitude}&longitude={longitude}"
    url = f"https://api.open-meteo.com/v1/forecast?{lat_lng}&current=temperature_2m,wind_speed_10m&hourly=temperature_2m,relative_humidity_2m,wind_speed_10m"
    response = requests.get(url)
    json = response.json()
    return json["current"]

chat = ChatOpenAI(model="gpt-4o-mini")
chat.register_tool(get_current_temperature)
chat.chat("What's the weather like today in Duluth, MN?", echo="all")
#> 👤 User turn:
#> 
#> What's the weather like today in Duluth, MN?
#> 
#> 🤖 Assistant turn:
#> 
#>  # tool request (call_YRma1FOUHVPGHkfylqdHw886)
#>  get_current_temperature(latitude=46.7833, longitude=-92.1062)
#> 
#> << 🤖 finish reason: tool_calls >>
#> 
#> 
#> 👤 User turn:
#> 
#>  # tool result (call_YRma1FOUHVPGHkfylqdHw886)
#>  {'time': '2025-02-27T17:45', 'interval': 900, 'temperature_2m': 3.2, 'wind_speed_10m': 21.6}
#> 
#> 🤖 Assistant turn:
#> 
#> Today in Duluth, MN, the temperature is approximately 3.2°C with a wind speed of 21.6 km/h.

Chat.register_tool() has signature Chat.register_tool(func, *, model=None), i.e. it takes a function func and determines input parameters from the function docstring. For more complicated tools, you can pass a pydantic model to model.

Internally, .register_tool() creates a Tool instance with properties .func, .schema and .name. The schema stores the function description.

Tool calls in shinychat

Here are a few design sketches for what tool calling might look like:

Step	Block	Inline
Tool call starts
Tool call completes
Extra info1

^{1. The extra info being outlined here is happening with two different mechanisms. In the block-style UI, the tool call completion hook returns custom UI with Shiny UI elements to show the results. In the inline-style UI, we could use popovers to display additional information about the call, e.g. showing the parameters used, etc.}

There are two points in the tool-calling lifecycle where we need status updates:

1. When the tool is called
2. When the tool call completes

I'm envisioning that tools would be registered with shinychat in similar to how they're registered with chatlas or ellmer. Internally (or alternatively) shinychat could provide classes that extend chatlas.Turn or UI methods for ellmer::TurnDef.

Ideally, the minimum amount of work required would be to register the tools, from there our default methods could create the UI as needed, using only information in the tool definition. For example, the name of the get_current_weather tool above could be converted into the UI label "Get current weather".

Depending on the use-case, I can also see wanting to pick between a block display or an inline display. The block works best for larger or bigger tasks, and the inline display could be used in situations where each turn is likely to include many tool calls.

Both of these variants would be encapsulated in functions available to users that we call with normal defaults. For example, we could default to showing "Get current weather", but a user could provide their own method that uses our block-display design but changes the title to "Get weather in Duluth, MN".

The completion method would, by default, find the tool call UI added at the start of the tool call and simply mark it as completed by updating its attributes.

That said, we also want it to be possible for the completion callback to send entirely custom UI that replaces the initial tool call UI. This is shown in the last row of the table for the block method, where instead of simply marking the flight search tool call as complete, the data received in the tool call is presented using custom UI.

Static Case

To start with the most simple and straight-forward case, we'll consider a simple chat with a single tool call.

User: What time is it in London?
Assistant: 
  Sure, I can look up the time.
  <tool_request id="123" name="get_current_time" tz="UK/London">
User: <tool_result id="123">2025-03-31 11:12:13</tool_result>
Assistant: It's 11am in London.

ellmer handles the tool request in the first assistant message, automatically invokes the tool, and returns the result. Note that in the chat turns, ellmer stores the tool result as a user turn, because we ran code locally to invoke the tool. Note also that, in a live session, control of the chat isn't returned to the user until after the second assistant message.

Structurally, the turns look like this at the end of this chat:

• Turn(role = "user")
  • ContentText (user message)
• Turn(role = "assistant")
  • ContentText (assistant message)
  • ContentToolRequest
• Turn(role = "user")
  • ContentToolResult
• Turn(role = "assistant")
  • ContentToolText

Currently, when launching live_browser(client) on a chat client object, ellmer calls contents_markdown() on the turns. For contents_markdown(<ContentText>), this is a simple transformation that extracts the text entered by the user or returned from the LLM. On the other hand, contents_markdown(<ContentToolRequest>) and contents_markdown(<ContentToolResult>) are no-ops and hide the tool request/result from the chat UI.

I propose that we introduce a new generic -- contents_shinychat() -- that we use instead and that can be used to create a default display for ContentToolRequest or ContentToolResult. We'd also have a contents_shinychat(<Chat>) method, wherein we'd reorganize the turns to coalesce tool results into a single assistant message. We would also suppress the tool request display and only show the tool results.

sequenceDiagram
    participant R as R Session
    participant client
    participant ellmer
    participant Terminal as Terminal (emit)
    participant UI as UI (yield)
    
    
    rect rgba(255, 165, 0, 0.05)
    Note over client: ContentText (user)
    client-->>UI: contents_shinychat(turn)
    Note over UI: User message
    end
    
    rect rgba(100, 100, 255, 0.05)
    Note over client: ContentText (assistant)
    client-->>UI: contents_shinychat(turn)
    Note over UI: Asisstant message
    Note over client: ContentToolRequest
    Note over client: ContentToolResult
    client-->>UI: contents_shinychat(ContentToolResult)
    Note over UI: Tool result display
    end

Loading

Live Case

When running live, the Content* objects are not directly used in the UI. In general, they're created by ellmer and recorded in the chat's turns, but the response from the LLM (API) are streamed to the UI via yielded strings.

sequenceDiagram
    participant R as R Session
    participant client as  client<br>(record)
    participant ellmer
    participant Terminal as Terminal<br>(emit)
    participant UI as UI<br>(yield)
    
    UI->>client: User input submitted
    Note over client: ContentText (user)
    client->>ellmer: chat_append(client$stream())
    activate client
    activate ellmer
    ellmer-->>Terminal: emit(chunk)
    Note over Terminal: Text of assistant response
    ellmer-->>UI: yield(chunk)
    Note over UI: Display of assistant response
    ellmer->>client: 
    Note over client: ContentText (assistant)
    deactivate client
    rect rgba(100, 100, 255, 0.05)
    ellmer->>client: Record tool request
    activate ellmer
    Note over client: ContentToolRequest
    ellmer-->>Terminal: emit(request)
    Note over Terminal: echo="all"
    ellmer-->>UI: yield(request)
    Note over UI: on_tool_request(request)
    Note over UI: contents_shinychate(request)
    ellmer->>R: Invoke tool
    activate R
    R-->>UI: chat_append_stream()
    Note over UI: Text appended during tool call
    R->>ellmer: tool result 
    deactivate R
    ellmer-->>Terminal: emit(result)
    Note over Terminal: echo="all"
    ellmer-->>UI: yield(result)
    Note over UI: on_tool_result(result)
    Note over UI: contents_shinychat(result)
    deactivate ellmer
    deactivate ellmer
    ellmer->>client: 
    Note over client: ContentToolResult
    end

Loading

Here's a description of the process depicted in the sequence diagram. I've added bold to the steps that we would be modifying to make this approach work.

1. User Input: The user provides input through the UI. This input is sent to the client as ContentText from the user.

2. Client to ellmer: The client calls client$stream(input) and ellmer makes the API request to the LLM and returns a generator as a response. Calling shinychat::chat_append(stream) directs yielded strings to the UI.

3. Assistant Response (Text): ellmer emits chunks of the assistant's response to the Terminal (assuming echo="all") and yields chunks to the UI for display, showing the text of the assistant's response in both the Terminal and the UI.

4. Assistant Response (Content): ellmer records the assistant message with ContentText and ContentToolRequest objects in the assistant turn.

5. Tool Request: ellmer currently doesn't yield the tool request, but I propose we add a yield_all option to $stream() and $stream_async(). When TRUE, we yield all non-text content at the end of the assistant turn.

6. Tool Request Display: shinychat receives the yielded ContentToolRequest and transforms it with contents_shinychat() before appending to the current chat message.

7. Tool Invocation: ellmer invokes the tool in the R session.

8. Text Appended During Tool Call: During the tool invocation (inside the tool function body), tool authors can use chat_append_stream() to append content to the UI. This content is ephemeral unless it ends up recorded in the tool result.

9. Tool Result: The tool function returns a result.

   Currently, this can be any jsonifiable object but doesn't have a special class. In addition to returning a regular R object, I propose we also allow the tool to return a ContentToolResult object, which might be a custom user-defined class that inherits from ContentToolResult.

   In addition to the properties currently used by ContentToolResult -- id, value, error -- we would add detail (additional data), call_tool (the tool def of the calling tool) and call_args (the arguments used when calling the tool).

   Currently, the tool results are converted to ContentToolResult and stored as a new user turn, but with yield_all = TRUE, ellmer would yield the contents of the turn into the generator.

10. Tool Result Display: Again, shinychat would receive the yielded ContentToolResult, call contents_shinychat() on the ContentToolResult and append the formatted result to the chat message.

11. On Tool Callbacks: With this approach, shinychat can also own the on_tool_request() and on_tool_result() callbacks. The immediate need is to to remove, replace or hide any UI from the ContentToolRequest when we receive a ContentToolResult. These callbacks do not need to be user-facing at this point.

12. The final goal is that, once we have a paired ContentToolRequest and ContentToolResult, to have the final live state be equivalent to the chat state in the static case.

Things we need

• A way to list attached tools from chat, e.g. chat$get_tools() or chat.get_tools(). Would be useful if we want to take a chat client and register its tools with shinychat. Not sure if this is strictly required, but could be a nice addition regardless. This now exists in ellmer and chatlas.

• shinychat gains contents_shinychat() generic, with methods for Chat, ContentToolRequest, ContentToolResult, etc., otherwise falling back to contents_html() or contents_markdown().

• Expand data included in the ContentToolResult object: id, call_tool, call_args are added by ellmer when the tool is invoked. value is the value that's sent to the LLM. detail is a list that collects any other data that someone would want to add to the tool result (analogue to CustomEvent.detail in JavaScript).

• ellmer gains support for tools to return ContentToolResult objects. If a tool returns a ContentToolResult, ellmer fills in id, call_tool and call_args. Otherwise, ellmer creates the ContentToolResult.

• ellmer gains yield_all in Chat$stream() and Chat$stream_async(). When TRUE, we yield non-text assistant responses after the assistant turn completes (text is already yielded) and we yield the contents of the user turn added by tool invokation.

• With the above in place, tool authors could return custom ContentToolRequest objects with custom contents_shinychat methods for formatting for display in Shiny.

• Additionally, ToolDef should gain an annotations property that allows tool definitions to carry additional properties that would be used in display. annotations were recently added to the MCP schema. contents_shinychat() would hook into these annotations for displaying the tool name or knowing which tools modify their environment, etc.

• Relatedly, shinychat needs a way to append to the current chat without knowing the ID of the chat. This would allow tool authors to write tool functions that append to shinychat when it's available or do something else when used without a shinychat UI. That might look something like this:

  my_tool <- function(...) {
     chat_ui <- local_shinychat()
     chat_ui$append("Progress: 0%") # no-op if called outside `chat_append()`
  
     # ... do stuff ...
     chat_ui$replace("Progress: 50%")
     # or maybe
     chat_ui$append("Progress: 50%", operation = "replace")
  
     # finally...
     chat_ui$replace("All done!")
     result
  }

[gadenbuie]

Some prior art:

• Add an interface for surfacing tool calls chatlas#69

[gadenbuie]

From @wch in posit-dev/chatlas#69 (comment):

For example, imagine a tool where you give it the name of a city, and the tool (1) looks up the coordinates for the city, and (2) look up the weather for those coordinates. If the conversation goes like this. Note that the parts in square brackets wouldn't be shown -- I'm just using them here to annotate what's going on under the hood.

User:
  I'm in New York. Is today a nice day for a walk?

Assistant:
  I'll look up the weather for you.
  [Tool call begins]
  Looking up coordinates for New York...
  New York is at 40.71° N, 74.01° W
  Looking up weather...
  Current conditions in New York: 62 degrees and sunny.
  [Tool call ends]
  Yes, today is a nice day for a walk in New York.

With the current code in this PR, you can't display the middle two lines of the tool call phase. But here's how the rest would look:

async def get_current_temperature(city_name: str):
    lat, lon = await find_coordinates(city_name)
    temp, sun = await find_weather(lat, lon)

    return ToolResult(
        {"temperature": temp, "sun": sun},
        user=f"Current conditions in {request["city_name"]}: {temp} degrees and {sun}\n\n",
    )

chat_model.register_tool(
    get_current_temperature,
    on_request=lambda request: f"Looking up coordinates for {request["city_name"]}...\n\n",
)

It would be nice if the tool itself had access to the stream and could yield to it, like:

async def get_current_temperature(city_name: str):
    f"Looking up coordinates for {request["city_name"]}...\n\n"
    lat, lon = await find_coordinates(city_name)
    yield f"{request["city_name"]} is at {lat}, {lon}\n\n"
    yield "Looking up weather...\n\n"
    temp, sun = await find_weather(lat, lon)
    yield f"Current conditions in {request["city_name"]}: {temp} degrees and {sun}.\n\n"

    return ToolResult({"temperature": temp, "sun": sun})

chat_model.register_tool(get_current_temperature)

Some other options instead of using yield:

• A function get_current_chat() that uses some sort of context.

  async def get_current_temperature(city_name: str):
      chat = get_current_chat()
      chat.emit(f"Looking up coordinates for {request["city_name"]}...\n\n")
      ...

  And if the tool function is called in a non-chat context, it could just return a dummy chat object where chat.emit() is a no-op. Note that this is not the Shiny-level chat object -- this is at the chatlas layer, so that this tool would show progress in both a Shiny app and when used at the console.

• Chatlas could pass in the chat object if there tool has a parameter named _chat.

      async def get_current_temperature(city_name: str, *, _chat: Chat):
          ...

• Chatlas could pass in a generic metadata object if there tool has a parameter named _meta, and that object could have chat on it. And arbitrary other stuff could be put on the _meta object.

      async def get_current_temperature(city_name: str, *, _meta):
          _meta.chat.emit(f"Looking up coordinates for {request["city_name"]}...\n\n")
          ...

One final idea, which I kind of like: It could also be useful to let the user define other data to pass in. You could imagine a tool call where you want to provide the tool with some information, but you don't want to send that information to the LLM.

For example, in something like Sidebot, suppose you have a data set and you want the LLM to use tool calls to run SQL queries on the data. You want to send the user request and schema and the LLM, but you don't want to send it the data.

# Define the tool in a scope outside of the app's server code
def run_query(query: str, data: pd.DataFrame, chat: Chat):
    ...


def server(input, output):
    # Suppose the value of df can change over time
    df = pd.DataFrame(...)

    chat_model = ChatAnthropic(...)

    @chat_model.register_tool(run_query, extra_args = {
        "data": lambda: df
        "chat": lambda: chat_model
    })

In this case, the LLM only sees and uses the query parameter, and developer has full control over the extra args that are passed to the tool, namely data and chat.

In the example above, I used lambda for both of the args, because I was thinking that df could change over time. However, chat_model doesn't change over time, and maybe you'd want to support both dynamic and static arguments. In that case we could make it so the developer would mark the dynamic arguments, say, with a function called dynamic():

    @chat_model.register_tool(run_query, extra_args = {
        "data": dynamic(lambda: df)
        "chat": chat_model
    })

[cpsievert]

I've been thinking recently that it should be the tool's responsibility to display content. This way, the 80% use case could be handled by a tool decorator provided by PyShiny/shinychat.

In Python, it could look like this.

NOTE: Out of curiosity, I took a quick and dirty stab at implementing this

from chatlas import ChatAuto
from shiny.express import ui

chat_client = ChatAuto()
chat = ui.Chat("chat")

@chat.display_tool
def get_current_temperature(lat, lon):
       return "72"

# We could make this a decorator too, so you can stack them
chat_client.register_tool(get_current_temperature)

....

And in shinychat, it could look something like this.

NOTE: I haven't yet fully thought through the implementation here, but my hunch is that it'd be straightforward
wrapper around chat_append_message()

library(shinychat)
library(ellmer)
library(bslib)

ui <- page_fillable(chat_ui("chat_id"))

server <- function(...) {
    chat_client <- chat_openai() 

     get_current_temperature <- function(lat, lon) "72" 

     chat_client$register_tool(tool(
         get_current_temperature |> chat_display_tool("chat_id"),
         "get_current_temperature"
      ))

      ....

}

And for custom tool UI, one can accomplish this today by calling chat.message_stream_context() (Python) / chat_append_message() (R) inside the tool. The chat_append_message() approach currently doesn't have the same ability as chat.message_stream_context() in that the former can only replace the entire chat message content, not portions of the content, but that's a general incompatibility we should address anyway.

We could also help make simple customizations a bit easier by allowing you to customize what gets shown before/after a tool call via display_tool() (e.g., those could take on_request/on_result callbacks that return UI).

A major wrinkle I can see with this approach so far is that display_tool() doesn't necessarily know how the result actually gets sent to the model. One way we could address this is to hang the relevant method/info on tool(). This way, if display_tool() gets a tool() it knows how to get the value the model receives. And, if display_tool() gets passed a normal function, we could assume you want the default way values get sent (i.e., upgrade functions passed to display_tool() to a tool())

[cpsievert]

As an update to my previous comment, we could also make it effectively a one-liner to get all your registered tools to display with our default UI this way:

chat.display_tools(chat_client.get_tools())

shinychat::chat_display_tools("chat_id", chat_client$get_tools())

This would require adding .get_tools() to ellmer and chatlas.

[cpsievert]

Oh, and assuming we go with this decorator approach, seems we'll also want a way to set tools:

tools = chat.display_tools(chat_client.get_tools())
chat_client.set_tools(tools)

tools <- shinychat::chat_display_tools("chat_id", chat_client$get_tools())
chat_client$set_tools(tools)

[gadenbuie]

I've started a POC for the decorator approach and implemented $get_tools() and $set_tools() in ellmer here: tidyverse/ellmer#383

[gadenbuie]

I'm realizing that an issue with the decorator approach is that it's going to inextricably link the tools in the chat_client with the "chat_ui" shinychat UI. While that's generally in line with how the chat client <-> UI works, it's not ideal, especially because it'd be hard to unwind if you take a chat client, clone it, and use it with another chat UI.

While I think there is room for more tight coupling between client+UI (#14), I think we need a solution that doesn't modify the user or third party's tool function and fits more naturally with this pattern of linking the client and UI:

  observeEvent(input$chat_user_input, {
    stream <- chat$stream_async(input$chat_user_input)
    chat_append("chat", stream)
  })

I think that for our first pass we should focus on rendering ContentToolRequest and ContentToolResult. These content types already exist and can be rendered into the chat UI with a smaller amount of work. And regardless of how we enable tool call UI, we'll need these hooks to work.

Once we have that feature in place, we can focus on adding tool call status updates (i.e. real-time updates emitted by the tool function).

[wch]

Some thoughts - I wrote similar things in posit-dev/chatlas#69 (comment) (which is kind of long):

It would be useful for a tool to be able to take in extra stuff from the application. For example, suppose the tool does some long-running computation and you want it to report the progress as it works. But the progress reporting could take different forms depending on the application. It could be a chat message, or it could be a toast (like shiny's Progress), or it could be a no-op.

The tool function would look like this.

do_compute <- function(x, report_progress = function(x) NULL) {
   # ... do some computation
   report_progress(25)
   # ... do some computation
   report_progress(50)
   # ... do some computation
   report_progress(75)
   # ... do some computation
   report_progress(100)

   return(result)
}

Currently, if you want to provide that function as a tool for the LLM, you'd have to wrap it up in another function:

server <- function(input, output, session) {
  chat <- chat_openai(model = "gpt-4o")
  
  my_report_progress <- function(percent) {
    # Do some sort of progress reporting here
  }
  
  my_do_compute <- function(x) {
    do_compute(x, my_report_progress)
  }
  
  chat$register_tool(tool(
    my_do_compute,
    "Do a long computation.",
    x = type_number(
      "A number to do a long computation on.",
    )
  ))
}

(Side note: is it possible for the my_report_progress function to inject stuff into the UI output stream, so it can add text for the user to see in the chat? I have found that to be a very useful thing to do in some things I'm working on.)

It's a little clunky to have to create a wrapper function for each tool, but it's not the end of the world. Another option would be able to pass in extra args that just aren't shown to the LLM. For example:

server <- function(input, output, session) {
  chat <- chat_openai(model = "gpt-4o")
  
  my_report_progress <- function(percent) {
    # Do some sort of progress reporting here
  }
  
  chat$register_tool(tool(
    my_do_compute,
    "Do a long computation.",
    x = type_number(
      "A number to do a long computation on.",
    ),
    extra_args = list(
      report_progress = my_report_progress
    )
  ))
}

I have used this technique in some TypeScript code where I define several tools, many of which use the same set of extra args, and it's nice to not have to wrap up every single one of those tools in a separate session-scoped function.

[gadenbuie]

Static Case

To start with the most simple and straight-forward case, we'll consider a simple chat with a single tool call.

User: What time is it in London?
Assistant: 
  Sure, I can look up the time.
  <tool_request id="123" name="get_current_time" tz="UK/London">
User: <tool_result id="123">2025-03-31 11:12:13</tool_result>
Assistant: It's 11am in London.

ellmer handles the tool request in the first assistant message, automatically invokes the tool, and returns the result. Note that in the chat turns, ellmer stores the tool result as a user turn, because we ran code locally to invoke the tool. Note also that, in a live session, control of the chat isn't returned to the user until after the second assistant message.

Structurally, the turns look like this at the end of this chat:

• Turn(role = "user")
  • ContentText (user message)
• Turn(role = "assistant")
  • ContentText (assistant message)
  • ContentToolRequest
• Turn(role = "user")
  • ContentToolResult
• Turn(role = "assistant")
  • ContentToolText

Currently, when launching live_browser(client) on a chat client object, ellmer calls contents_markdown() on the turns. For contents_markdown(<ContentText>), this is a simple transformation that extracts the text entered by the user or returned from the LLM. On the other hand, contents_markdown(<ContentToolRequest>) and contents_markdown(<ContentToolResult>) are no-ops and hide the tool request/result from the chat UI.

I propose that we introduce a new generic -- contents_shiny() -- that we use instead and that can be used to create a default display for ContentToolRequest or ContentToolResult. We'd also have a contents_shiny(<Chat>) method, wherein we'd reorganize the turns to coalesce tool results into a single assistant message. We would also suppress the tool request display and only show the tool results.

sequenceDiagram
    participant R as R Session
    participant client
    participant ellmer
    participant Terminal as Terminal (emit)
    participant UI as UI (yield)
    
    
    rect rgb(240, 245, 255)
    Note over client: ContentText (user)
    client-->>UI: contents_shiny(turn)
    Note over UI: User message
    end
    
    rect rgb(240, 245, 255)
    Note over client: ContentText (assistant)
    client-->>UI: contents_shiny(turn)
    Note over UI: Asisstant message
    Note over client: ContentToolRequest
    Note over client: ContentToolResult
    client-->>UI: contents_shiny(ContentToolResult)
    Note over UI: Tool result display
    end

Loading

Live Case

When running live, the Content* objects are not directly used in the UI. In general, they're created by ellmer and recorded in the chat's turns, but the response from the LLM (API) are streamed to the UI via yielded strings.

sequenceDiagram
    participant R as R Session
    participant client
    participant ellmer
    participant Terminal as Terminal (emit)
    participant UI as UI (yield)
    
    UI->>client: User input submitted
    Note over client: ContentText (user)
    client->>ellmer: chat_append(client$stream())
    activate client
    activate ellmer
    ellmer-->>Terminal: emit(chunk)
    Note over Terminal: Text of assistant response
    ellmer-->>UI: yield(chunk)
    Note over UI: Display of assistant response
    ellmer->>client: 
    Note over client: ContentText (assistant)
    deactivate client
    rect rgb(240, 245, 255)
    ellmer->>client: Record tool request
    activate ellmer
    Note over client: ContentToolRequest
    ellmer-->>Terminal: emit(request)
    Note over Terminal: Not implemented
    ellmer-->>UI: yield(on_tool_request(request))
    Note over UI: Display of tool request
    ellmer->>R: Invoke tool
    activate R
    R-->>UI: chat_append_stream()
    Note over UI: Text appended during tool call
    R->>ellmer: tool result 
    deactivate R
    ellmer-->>Terminal: emit(result)
    Note over Terminal: Not implemented
    ellmer-->>UI: yield(on_tool_result(result))
    Note over UI: Display of tool result
    deactivate ellmer
    deactivate ellmer
    ellmer->>client: 
    Note over client: ContentToolResult
    end

Loading

Here's a description of the process depicted in the sequence diagram. I've added bold to the steps that we would be modifying to make this approach work.

1. User Input: The user provides input through the UI. This input is sent to the client as ContentText from the user.

2. Client to ellmer: The client calls client$stream(input) and ellmer makes the API request to the LLM and returns a generator as a response. Calling shinychat::chat_append(stream) directs yielded strings to the UI.

3. Assistant Response (Text): ellmer emits chunks of the assistant's response to the Terminal (assuming echo="all") and yields chunks to the UI for display, showing the text of the assistant's response in both the Terminal and the UI.

4. Assistant Response (Content): ellmer records the assistant message with ContentText and ContentToolRequest objects in the assistant turn.

5. Tool Request Display: We don't current do anything to emit or yield the tool request, but I propose we, at least, yield the results of calling an on_tool_request() callback that would, for Shiny's purposes, call contents_shiny() on the tool request.

6. Tool Invocation: ellmer invokes the tool in the R session.

7. Text Appended During Tool Call: During the tool invocation (inside the tool function body), tool authors can use chat_append_stream() to append content to the UI. This content is ephemeral unless it ends up recorded in the tool result.

8. Tool Result: The tool function returns a result.

   Currently, this can be any jsonifiable object but doesn't have a special class. In addition to returning a regular R object, I propose we also allow the tool to return a ContentToolResult object, which might be a custom user-defined class that inherits from ContentToolResult.

   In addition to the properties currently used by ContentToolResult -- id, value, error -- we would add detail (additional data), call_tool (the tool def of the calling tool) and call_args (the arguments used when calling the tool).

9. Tool Result Display: Currently, the tool results are converted to ContentToolResult and stored in the turns, but we would add an on_tool_result() callback, the results of which would be yielded to the UI.

[gadenbuie]

Key changes/tasks

• shinychat gains contents_shiny() generic, with methods for Chat, ContentToolRequest, ContentToolResult, etc., otherwise falling back to contents_html() or contents_markdown().

• Expand data included in the ContentToolResult object: id, call_tool, call_args are added by ellmer when the tool is invoked. value is the value that's sent to the LLM. detail is a list that collects any other data that someone would want to add to the tool result (analogue to CustomEvent.detail in JavaScript).

• ellmer gains support for tools to return ContentToolResult objects. If a tool returns a ContentToolResult, ellmer fills in id, call_tool and call_args. Otherwise, ellmer creates the ContentToolResult.

• ellmer gains support for deciding what values are yielded into the response generator when a tool is called or returned.

  • One option is an on_tool_request() and on_tool_result() callback. These would be generally useful to shinychat for running some code during the tool calling lifecycle without modifying the tool functions. The challenge with this approach is that these callbacks need to return a string that's yielded into the response generator.

  • Another option is to call a contents_*() function on the content and yield/emit the string result. In this approach, an option would set the contents function to call for each, e.g.

    chat_append <- function(id, response, ...) {
       local_options(list(
          ellmer.format_yield = contents_shiny,
          ellmer.format_emit = contents_text
       ))
       
       # ... rest of chat_append() ...
    }

  • A combination of both. Tool request/result callbacks could be registered and run sequentially on request/result, discarding the return values. Separately, the yield/emit formatters would decide what content is yielded/emitted.

• With the above in place, tool authors could return custom ContentToolRequest objects with custom contents_shiny methods for formatting for display in Shiny.

• Additionally, ToolDef should gain an annotations property that allows tool definitions to carry additional properties that would be used in display. annotations were recently added to the MCP schema. contents_shiny() would hook into these annotations for displaying the tool name or knowing which tools modify their environment, etc.

• Relatedly, shinychat needs a way to append to the current stream without knowing the ID of the chat. This would allow tool authors to write tool functions that append to shinychat when it's available or do something else when used without a shinychat UI. That might look something like this:

  my_tool <- function(...) {
     chat_ui <- local_shinychat()
     chat_ui$append("Progress: 0%") # no-op if called outside `chat_append()`
  
     # ... do stuff ...
     chat_ui$replace("Progress: 50%")
     # or maybe
     chat_ui$append("Progress: 50%", operation = "replace")
  
     # finally...
     chat_ui$replace("All done!")
     result
  }

[cpsievert]

5. Tool Request Display: We don't current do anything to emit ... the tool request

We do emit the tool request/response when echo="all" is set, and we've been talking recently about making echo="all" the default.

We need to be able to affect, temporarily, what values are yielded by ellmer

What if we just made this an option on $stream()? Something like $stream(include_tools = TRUE)? In Python this could be an overload so that when .stream(include_tools = False) you know you're getting strings, but .stream(include_tools = True) might include the tool content objects. We might even consider eventually making include_tools = True the default eventually to mirror the echo="all" (i.e., by default, we emit and yield tool requests/results).

shinychat gains contents_shiny() generic, with methods for Chat, ContentToolRequest, ContentToolResult

I don't love that shinychat would extend upon an ellmer construct in this way. What if ellmer put these methods on as.tags() instead? That way, chat_append() as it currently exists could render them, and they could also statically render, even outside of the chat component.

I know that still leaves the problem of "replaying" pre-existing turns in the UI. We could address that by having contents_markdown.ContentToolRequest()/contents_markdown.ContentToolResult() delegate to their relevant as.tags() methods.

ellmer gains support for tools to return ContentToolResult objects. If a tool returns a ContentToolResult, ellmer fills in id, call_tool and call_args. Otherwise, ellmer creates the ContentToolResult.

I like this. I would also say this would be a good place to control how the value gets transformed before getting sent to the model (i.e., something like the format_as argument in #69).

Relatedly, shinychat needs a way to append to the current stream without knowing the ID of the chat.

It's possible for two streams to be running simultaneously, so I don't really see a way around having to inform the tool about the relevant chat component.

[gadenbuie]

We do emit the tool request/response when echo="all" is set, and we've been talking recently about making echo="all" the default.

Yeah, that was just a typo that I've fixed now. Thanks.

What if we just made this an option on $stream()?

Yeah I like that. I think it makes sense to yield the unformatted contents and have chat_append() handle the formatting calls, too. I'll look in to this.

What if ellmer put these methods on as.tags() instead? ... We could address that by having contents_markdown.ContentToolRequest()/contents_markdown.ContentToolResult() delegate to their relevant as.tags() methods.

contents_shiny() (or contents_shinychat() is more likely what we'd call it), solves a bunch of problems for us. ellmer might be open to contents_markdown() returning HTML, but personally I would put that behavior in contents_html(), which clearly signals that HTML will be returned. If include_tools becomes an argument of $stream(), I could also see it being an argument of contents_markdown(), in which case you'd want the tool content to be rendered as markdown.

contents_html() would be a tempting tool for shinychat, but unlike contents_markdown() which defers to contents_text(), contents_html() returns NULL by default if there isn't an appropriate method. So contents_shinychat() would gives us a way to call contents_html(), falling back to contents_markdown() because our chat UI can handle the mix. That also ensures that people could implement a contents_html() generic for their tool result and we'd pick it up.

I don't love that shinychat would extend upon an ellmer construct in this way.

I would agree with you if there were some part of the proposal that was reaching into ellmer internals. I also don't like having to decide how ellmer formats yielded content that's added by the tool calls, but include_tools and just yielding the ContentTool* objects is a good solution.

Also, contents_shinychat() can include calls to as.tags(). To me, putting as.tags() inside of contents_markdown() does a lot more linking and intermingling of concerns than shinychat having an appropriate content formatter.

ellmer gains support for tools to return ContentToolResult objects. If a tool returns a ContentToolResult, ellmer fills in id, call_tool and call_args. Otherwise, ellmer creates the ContentToolResult.

I like this. I would also say this would be a good place to control how the value gets transformed before getting sent to the model (i.e., something like the format_as argument in #69).

In my proposal, <ContentToolResult>@value is still the value that's sent to the LLM and it's transformed in the current way, which is via tool_string() (internal in ellmer, returns @value if a string or json or jsonifies the value otherwise). I don't think we should change this, but if we did it'd be to make tool_string() a generic that takes a ContentToolRequest.

Again, I don't think we should change the status quo in ellmer. Instead, we can allow tool authors to append additional data via @detail if they need more than @value.

Relatedly, shinychat needs a way to append to the current stream without knowing the ID of the chat.

It's possible for two streams to be running simultaneously, so I don't really see a way around having to inform the tool about the relevant chat component.

It's possible for two streams to be running as a result of chat_append(id, stream)? Can you give an example of how that would happen?

I need to revise the wording from "current stream" to "append to the current chat message". I.e. sending additional content directly to the chat UI, but not by putting it into the response generator.

[cpsievert]

I don't think we should change this, but if we did it'd be to make tool_string() a generic that takes a ContentToolRequest.

Oh, I was more so thinking about the Python case. There we don't have I(), so we'll need some way to opt-out of the json.dump().

It's possible for two streams to be running as a result of chat_append(id, stream)?

I was again thinking more so of the Python case, which uses extended tasks (here's an example). It's a bit more a of theoretical concern for R since it's promises based (and thus, blocking within session), but ExtendedTask does make it a possibility.

[gadenbuie]

Relatedly, shinychat needs a way to append to the current stream chat message without knowing the ID of the chat.

It's possible for two streams to be running simultaneously, so I don't really see a way around having to inform the tool about the relevant chat component.

It's possible for two streams to be running as a result of chat_append(id, stream)?

I was again thinking more so of the Python case, which uses extended tasks (here's an example). It's a bit more a of theoretical concern for R since it's promises based (and thus, blocking within session), but ExtendedTask does make it a possibility.

Even then, you're calling chat.append_message_stream() and chat2.append_message_stream(). You'd have to have two streams to the same chat UI happening simultaneously for this to be problematic/impossible.