Adding Customer id & tags #261

yifan1207 · 2025-05-19T03:02:49Z

📝 Summary

Adding cusomter id and tags columns in DB and adding set_metadata() function to enable users to update and set metadata of a trace including
Supported keys:
- customer_id: ID of the customer using this trace
- tags: List of tags for this trace
- has_notification: Whether this trace has a notification
- overwrite: Whether to overwrite existing traces
- rules: Rules for this trace
- name: Name of the trace
🎯 Purpose
give user ability to track customer usage of agents
give user ability to add tags to better carategotize and filter their traces
Merge with Judgment PR https://github.com/JudgmentLabs/judgment/pull/339

🎥 Demo of Changes

🧪 Testing

✅ Checklist

Self-review
Video demo of changes
Unit Tests and CI/CD tests are passing
Reviewers assigned

📌 Linear Issue

✏️ Additional Notes

gemini-code-assist

Hello @yifan1207, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

Summary of Changes

Hello team,

Gemini here, providing a summary of this pull request. This PR introduces functionality to associate customer IDs and tags with traces. The primary goal, as stated in the description, is to allow users to track customer usage of agents and categorize/filter traces more effectively. This is achieved by adding customer_id and tags fields to the trace data structure and providing methods to set and manage this metadata.

The changes involve modifying the core Trace and Tracer classes to accommodate these new fields and introducing a flexible set_metadata function, along with convenience methods like set_customer_id and set_tags. The metadata is included when traces are saved. A new demo script is also added to illustrate how to use this new feature.

Highlights

Add Customer ID and Tags to Traces: The Trace object now includes optional customer_id (string) and tags (list of strings) attributes to store user-defined metadata.
Implement set_metadata Function: A new set_metadata method is added to both the Trace and Tracer classes. This method allows setting customer_id, tags, and other supported trace metadata fields (like name, has_notification, overwrite, rules) using either a key-value pair or keyword arguments. The Tracer class method acts as a proxy to set metadata on the currently active trace.
Add Convenience Methods for Metadata: Convenience methods set_customer_id and set_tags are added to the Tracer class, providing simpler ways to set these specific metadata fields on the current trace.
Include Metadata in Saved Traces: The save method of the Trace class is updated to include the customer_id and tags when sending trace data to the trace manager client.
Add print_metadata Helper: A print_metadata method is added to both Trace and Tracer classes to easily print the current trace's metadata for debugging and verification.
New Demo Script: A new demo file (src/demo/customer_id_and_tags.py) is included to demonstrate how to use the new set_metadata function to add customer IDs and tags to a trace.

Changelog

Click here to see the changelog

src/demo/customer_id_and_tags.py
- Added new demo script to showcase setting customer ID and tags on a trace using the new set_metadata function.
src/judgeval/common/tracer.py
- Added customer_id and tags attributes to the Trace class.
- Implemented set_metadata method on the Trace class to update various trace metadata fields.
- Added set_tags convenience method on the Trace class.
- Updated the save method of the Trace class to include customer_id and tags in the saved payload.
- Added print_metadata method to the Trace class for debugging.
- Added proxy methods set_customer_id, set_tags, set_metadata, and print_metadata to the Tracer class to operate on the current trace.
- Replaced rprint with standard print in the log method.
src/judgeval/data/trace.py
- Added customer_id field to the Trace Pydantic model definition.
trace_metadata.json
- Added new example output file showing saved trace metadata including customer ID and tags.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Trace flows like a stream,
Metadata, a helpful gleam,
Customer, tag, name,
Guiding the code's game,
Making sense of the dream.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces the ability to add customer ID and tags to traces, along with a set_metadata function to manage trace attributes. The changes look generally good and enhance the usability of tracing.

Summary of Findings

Boolean Metadata Handling: The conversion of has_notification and overwrite metadata values using bool(v) can lead to misinterpretation if string inputs like "false" are provided. This should be made more robust.
Consistency for None Values in set_metadata: The set_metadata method in TraceClient handles setting attributes to None differently based on whether the (key, value) pair or **kwargs style is used. This inconsistency should be addressed.
Payload Change Confirmation: The parent_name field has been removed from the trace_data payload in TraceClient.save(). Confirmation is needed that this change is coordinated with backend API updates.
Attribute Initialization: The has_notification attribute could be initialized in TraceClient.__init__ for consistency with other new metadata fields like customer_id and tags.

Merge Readiness

The pull request introduces valuable features for trace metadata. Before merging, I recommend addressing the identified high and medium severity issues, particularly concerning the boolean metadata handling and the consistency of None value processing in set_metadata. Also, please confirm the coordination of the parent_name payload change with the backend. Once these points are clarified and resolved, the PR should be in good shape. As an AI, I am not authorized to approve pull requests; please ensure further review and approval from team members.

src/judgeval/common/tracer.py

JCamyre

Looks great Yifan! Couple small requests

src/judgeval/common/tracer.py

JCamyre · 2025-05-22T03:32:07Z

trace_metadata.json

@@ -0,0 +1,11 @@
+{


Delete this file

SecroLoL · 2025-06-12T23:56:15Z

src/demo/customer_id_and_tags.py

I think we removed the src/demo/ folder a few PRs ago.

I'm gonna move this script to the cookbooks repo!

I also think we should remove this file in that case

JudgmentLabs/judgment-cookbook#12

JCamyre

Couple quick questions, then good to merge :)

JCamyre · 2025-06-19T02:20:40Z

src/judgeval/common/tracer.py

+        - tags: List of tags for this trace
+        - has_notification: Whether this trace has a notification
+        - overwrite: Whether to overwrite existing traces
+        - rules: Rules for this trace


How does this field relate to Minh's rules?

ohh like allow users to edit their rule as part of metadata if they need. should we allow this?

Not necessary!

src/judgeval/common/tracer.py

JCamyre · 2025-06-19T02:25:38Z

src/judgeval/common/tracer.py

+        """
+        self.set_metadata(tags=tags)
+
+    def print_metadata(self):


Thinking either making this a private method _print_metadata or deleting entirely.
Do you see an application for this outside of debugging?

I think people might want to use this to see what it is, just like a method od convenience?

I don't think so - when people are tracing their agents in production they don't want this extra info to clutter their logs.

okok delted

JCamyre · 2025-06-19T02:28:17Z

src/judgeval/common/tracer.py

        except Exception as e:
            warnings.warn(f"Failed to send evaluation runs batch: {e}")

+    def _send_traces_batch(self, traces: List[Dict[str, Any]]):


Confused here - isn't this from Minh's PR? Not sure why it is showing as new here.
Also, does it make sense to send these in parallel

ohh I think yeaa they are directly from minh's PR? I was making some edit to add compatibility to the semi saving trace span so that bscailly updating metadata would also use the queue to edit

Interesting that it is showing that you added the entire function. Which file did you copy this function from?

ohh my bad now I see it basically I can just use minh's queing functions deleted these!

JCamyre

Couple last things!

src/judgeval/common/tracer.py

JCamyre · 2025-06-20T05:15:08Z

src/judgeval/common/tracer.py

+        - tags: List of tags for this trace
+        - has_notification: Whether this trace has a notification
+        - overwrite: Whether to overwrite existing traces
+        - rules: Rules for this trace


Not necessary!

src/judgeval/common/tracer.py

JCamyre · 2025-06-20T05:17:06Z

src/judgeval/common/tracer.py

+        """
+        self.set_metadata(tags=tags)
+
+    def print_metadata(self):


I don't think so - when people are tracing their agents in production they don't want this extra info to clutter their logs.

JCamyre · 2025-06-20T05:17:39Z

src/judgeval/common/tracer.py

        except Exception as e:
            warnings.warn(f"Failed to send evaluation runs batch: {e}")

+    def _send_traces_batch(self, traces: List[Dict[str, Any]]):


Interesting that it is showing that you added the entire function. Which file did you copy this function from?

JCamyre · 2025-06-20T05:17:51Z

src/judgeval/common/tracer.py

            }
            self._span_queue.put(eval_data)

+    def queue_trace(self, trace_data: Dict[str, Any]):


JCamyre · 2025-06-20T05:18:01Z

src/judgeval/common/tracer.py

+        else:
+            warnings.warn("No current trace found, cannot set tags")
+
+    def print_metadata(self):


Think we should delete

JCamyre

LGTM!

src/judgeval/common/tracer.py

yifan1207 added 3 commits May 18, 2025 19:48

customer ID & tags

b60f66a

update exmaple file

c06b732

fixes

e1e5373

gemini-code-assist bot reviewed May 19, 2025

View reviewed changes

gemini-code-assist bot suggested changes May 19, 2025

View reviewed changes

src/judgeval/common/tracer.py Outdated Show resolved Hide resolved

src/judgeval/common/tracer.py Outdated Show resolved Hide resolved

src/judgeval/common/tracer.py Show resolved Hide resolved

src/judgeval/common/tracer.py Show resolved Hide resolved

JCamyre self-requested a review May 22, 2025 00:22

JCamyre added the In Review Task is currently being reviewed label May 22, 2025

JCamyre requested changes May 22, 2025

View reviewed changes

JCamyre changed the base branch from main to staging June 11, 2025 23:05

SecroLoL reviewed Jun 12, 2025

View reviewed changes

SecroLoL mentioned this pull request Jun 13, 2025

Add cookbook for tagging traces with metadata JudgmentLabs/judgment-cookbook#12

Open

merge

0ca7539

yifan1207 force-pushed the customer-id-tags branch from 4f7e7e8 to 0ca7539 Compare June 17, 2025 17:12

respond to commetns

aee1ed5

yifan1207 force-pushed the customer-id-tags branch from 367fb94 to aee1ed5 Compare June 17, 2025 18:45

yifan1207 added 7 commits June 17, 2025 11:45

Merge branch 'staging' into customer-id-tags

45a6fad

demo

4f77f3e

Merge remote-tracking branch 'origin/staging' into customer-id-tags

5249339

fix errors and update to new trac saving format

25da537

Merge branch 'staging' into customer-id-tags

5be2e7a

Merge branch 'staging' into customer-id-tags

99dbf78

Merge branch 'staging' into customer-id-tags

e2a4c98

JCamyre requested changes Jun 19, 2025

View reviewed changes

JCamyre requested changes Jun 20, 2025

View reviewed changes

yifan1207 added 2 commits June 20, 2025 02:02

remove print function

eb12bfb

remove unneded functions

f5be2d6

JCamyre approved these changes Jun 20, 2025

View reviewed changes

src/judgeval/common/tracer.py Show resolved Hide resolved

JCamyre merged commit 91ec68b into staging Jun 20, 2025
6 of 7 checks passed

Adding Customer id & tags #261

Adding Customer id & tags #261

Uh oh!

Conversation

yifan1207 commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Summary

🎯 Purpose

Merge with Judgment PR https://github.com/JudgmentLabs/judgment/pull/339

🎥 Demo of Changes

🧪 Testing

✅ Checklist

📌 Linear Issue

✏️ Additional Notes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Changelog

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Summary of Findings

Merge Readiness

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JCamyre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SecroLoL Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JCamyre left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JCamyre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yifan1207 commented May 19, 2025 •

edited

Loading

SecroLoL Jun 12, 2025 •

edited

Loading