diff --git a/examples/evaluation/use-cases/EvalsAPI_Image_Inputs.ipynb b/examples/evaluation/use-cases/EvalsAPI_Image_Inputs.ipynb index b27a9434d3..4f95bdfa23 100644 --- a/examples/evaluation/use-cases/EvalsAPI_Image_Inputs.ipynb +++ b/examples/evaluation/use-cases/EvalsAPI_Image_Inputs.ipynb @@ -245,7 +245,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "To create the run, we pass in the eval object id, the data source (i.e., the data we compiled earlier), and the chat message input we will use for sampling to generate the model response. While we won't dive into it in this cookbook, EvalsAPI also supports stored completions containing images as a data source. \n", + "To create the run, we pass in the eval object id, the data source (i.e., the data we compiled earlier), and the chat message input we will use for sampling to generate the model response. Note that EvalsAPI also supports stored completions and responses containing images as a data source. See the [Additional Info: Logs Data Source](#additional-info-logs-data-source) section for more info.\n", "\n", "Here's the sampling message input we'll use for this example." ] @@ -509,6 +509,22 @@ "print(json.dumps(dict(first_item), indent=2, default=str))" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Additional Info: Logs Data Source\n", + "\n", + "As mentioned earlier, EvalsAPI supports logs (i.e., stored completions or responses) containing images as a data source. To use this functionality, change your eval configurations as follows: \n", + "\n", + "Eval Creation\n", + " - set `data_source_config = { \"type\": \"logs\" }`\n", + " - revise templating in `grader_config` to use `{{item.input}}` and/or `{{sample.output_text}}`, denoting the input and output of the log\n", + "\n", + "Eval Run Creation\n", + " - specify the filters in the `data_source` field that will be used to obtain the corresponding logs for the eval run (see the [docs](https://platform.openai.com/docs/api-reference/evals/createRun) for more information)" + ] + }, { "cell_type": "markdown", "metadata": {},