Skip to content

Evaluation: Show cost #113

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Ayush8923 merged 7 commits into from
Apr 17, 2026
Merged

Evaluation: Show cost #113

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
bbadd89
added support for showing cost in UI
AkhileshNegi Apr 9, 2026
c0ca999
cleanup UI
AkhileshNegi Apr 10, 2026
35771f7
cleanup UI
AkhileshNegi Apr 10, 2026
27c788c
coderabbit
AkhileshNegi Apr 10, 2026
331085d
Merge branch 'main' into enhancement/evaluation-cost
AkhileshNegi Apr 17, 2026
7e09697
cleanup after review
AkhileshNegi Apr 17, 2026
15d5499
reusing tooltip
AkhileshNegi Apr 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion app/components/InfoTooltip.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
"use client";

import type { ReactNode } from "react";

interface InfoTooltipProps {
text: string;
text: ReactNode;
}

export default function InfoTooltip({ text }: InfoTooltipProps) {
Expand Down
34 changes: 32 additions & 2 deletions app/components/evaluations/EvalRunCard.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,12 @@ import {
AssistantConfig,
getScoreObject,
} from "@/app/components/types";
import { getStatusColor } from "@/app/components/utils";
import { getStatusColor, formatCostUSD } from "@/app/components/utils";
import { timeAgo } from "@/app/lib/utils";
import ConfigModal from "@/app/components/ConfigModal";
import ScoreDisplay from "@/app/components/ScoreDisplay";
import CostIcon from "@/app/components/icons/evaluations/CostIcon";
import InfoTooltip from "@/app/components/InfoTooltip";

export interface EvalRunCardProps {
job: EvalJob;
Expand Down Expand Up @@ -81,7 +83,7 @@ export default function EvalRunCard({
</div>
)}

{/* Row 3: Dataset + Config (left) | Actions (right) */}
{/* Row 3: Dataset + Config + Cost (left) | Actions (right) */}
<div className="flex items-center justify-between gap-4 mt-3">
<div
className="flex items-center gap-3 text-xs"
Expand Down Expand Up @@ -113,6 +115,34 @@ export default function EvalRunCard({
{assistantConfig.name}
</span>
)}
{job.cost?.total_cost_usd != null && (
<span className="flex items-center gap-1.5">
<CostIcon className="flex-shrink-0" />
{formatCostUSD(job.cost.total_cost_usd)}
<InfoTooltip
text={
<div className="space-y-1">
{job.cost.response && (
<div className="flex justify-between gap-3">
<span>Response generation</span>
<span>
{formatCostUSD(job.cost.response.cost_usd)}
</span>
</div>
)}
{job.cost.embedding && (
<div className="flex justify-between gap-3">
<span>Cosine similarity calculation</span>
<span>
{formatCostUSD(job.cost.embedding.cost_usd)}
</span>
</div>
)}
</div>
}
/>
</span>
Comment on lines +119 to +144
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Invalid DOM nesting: <div> elements inside <span>.

The outer wrapper on line 127 is a <span> (phrasing content), but it now contains <div> children (the tooltip trigger on line 130 and the tooltip body on line 151). React will emit a validateDOMNesting warning and some browsers may reflow unexpectedly. Change the wrapper to a <div> (or make the trigger/tooltip <span>s).

Proposed fix 
-              <span className="flex items-center gap-1.5">
+              <div className="flex items-center gap-1.5 relative">
                 <CostIcon className="flex-shrink-0" />
                 {formatCostUSD(job.cost.total_cost_usd)}
                 ...
-              </span>
+              </div>
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
<span className="flex items-center gap-1.5">
<CostIcon className="flex-shrink-0" />
{formatCostUSD(job.cost.total_cost_usd)}
<div
className={`inline-flex items-center justify-center w-4 h-4 rounded-full text-xs font-normal cursor-help ${isCostTooltipOpen ? "bg-[#171717] text-white" : "text-[#737373]"}`}
onMouseEnter={(e) => {
const rect = e.currentTarget.getBoundingClientRect();
const tooltipWidth = 280;
const centerX = rect.left + rect.width / 2;
const clampedLeft = Math.min(
Math.max(centerX - tooltipWidth / 2, 8),
window.innerWidth - tooltipWidth - 8,
);
setCostTooltipPos({
top: rect.top - 8,
left: clampedLeft,
});
setIsCostTooltipOpen(true);
}}
onMouseLeave={() => setIsCostTooltipOpen(false)}
>
i
</div>
{isCostTooltipOpen && (
<div
className="fixed z-50 px-3 py-2 rounded-md text-xs whitespace-normal pointer-events-none space-y-1 bg-[#171717] text-white w-[260px] shadow-md -translate-y-full"
style={{
top: costTooltipPos.top,
left: costTooltipPos.left,
}}
>
{job.cost.response && (
<div className="flex justify-between gap-3">
<span className="text-[#a3a3a3]">
Response generation
</span>
<span>{formatCostUSD(job.cost.response.cost_usd)}</span>
</div>
)}
{job.cost.embedding && (
<div className="flex justify-between gap-3">
<span className="text-[#a3a3a3]">
Cosine similarity calculation
</span>
<span>
{formatCostUSD(job.cost.embedding.cost_usd)}
</span>
</div>
)}
</div>
)}
</span>
<div className="flex items-center gap-1.5 relative">
<CostIcon className="flex-shrink-0" />
{formatCostUSD(job.cost.total_cost_usd)}
<div
className={`inline-flex items-center justify-center w-4 h-4 rounded-full text-xs font-normal cursor-help ${isCostTooltipOpen ? "bg-[`#171717`] text-white" : "text-[`#737373`]"}`}
onMouseEnter={(e) => {
const rect = e.currentTarget.getBoundingClientRect();
const tooltipWidth = 280;
const centerX = rect.left + rect.width / 2;
const clampedLeft = Math.min(
Math.max(centerX - tooltipWidth / 2, 8),
window.innerWidth - tooltipWidth - 8,
);
setCostTooltipPos({
top: rect.top - 8,
left: clampedLeft,
});
setIsCostTooltipOpen(true);
}}
onMouseLeave={() => setIsCostTooltipOpen(false)}
>
i
</div>
{isCostTooltipOpen && (
<div
className="fixed z-50 px-3 py-2 rounded-md text-xs whitespace-normal pointer-events-none space-y-1 bg-[`#171717`] text-white w-[260px] shadow-md -translate-y-full"
style={{
top: costTooltipPos.top,
left: costTooltipPos.left,
}}
>
{job.cost.response && (
<div className="flex justify-between gap-3">
<span className="text-[`#a3a3a3`]">
Response generation
</span>
<span>{formatCostUSD(job.cost.response.cost_usd)}</span>
</div>
)}
{job.cost.embedding && (
<div className="flex justify-between gap-3">
<span className="text-[`#a3a3a3`]">
Cosine similarity calculation
</span>
<span>
{formatCostUSD(job.cost.embedding.cost_usd)}
</span>
</div>
)}
</div>
)}
</div>
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@app/components/evaluations/EvalRunCard.tsx` around lines 127 - 178, The DOM
nesting is invalid because the outer element using className "flex items-center
gap-1.5" (currently a <span>) contains <div> children for the tooltip trigger
and body; update the wrapper in EvalRunCard (the element that wraps CostIcon,
formatCostUSD, the tooltip trigger that calls
setCostTooltipPos/setIsCostTooltipOpen, and the conditional tooltip that reads
costTooltipPos and job.cost) to be a <div> (or alternatively change the inner
tooltip <div>s to <span>s) so the structure is semantically valid and React's
validateDOMNesting warnings are resolved.

)}
</div>
<div className="flex items-center gap-3 flex-shrink-0">
<button
Expand Down
23 changes: 23 additions & 0 deletions app/components/icons/evaluations/CostIcon.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
interface IconProps {
className?: string;
style?: React.CSSProperties;
}

export default function CostIcon({ className, style }: IconProps) {
return (
<svg
className={`w-3.5 h-3.5 ${className ?? ""}`}
fill="none"
viewBox="0 0 24 24"
stroke="currentColor"
strokeWidth={2}
style={style}
>
<path
strokeLinecap="round"
strokeLinejoin="round"
d="M12 6v12m-3-2.818.879.659c1.171.879 3.07.879 4.242 0 1.172-.879 1.172-2.303 0-3.182C13.536 12.219 12.768 12 12 12c-.725 0-1.45-.22-2.003-.659-1.106-.879-1.106-2.303 0-3.182s2.9-.879 4.006 0l.415.33M21 12a9 9 0 1 1-18 0 9 9 0 0 1 18 0Z"
/>
</svg>
);
}
16 changes: 16 additions & 0 deletions app/components/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,21 @@ export interface AssistantConfig {
is_deleted: boolean;
}

export interface EvalCostEntry {
model: string;
cost_usd: number;
input_tokens?: number;
output_tokens?: number;
prompt_tokens?: number;
total_tokens: number;
}

export interface EvalCost {
response?: EvalCostEntry;
embedding?: EvalCostEntry;
total_cost_usd: number;
}

export interface EvalJob {
id: number;
run_name: string;
Expand All @@ -130,6 +145,7 @@ export interface EvalJob {
assistant_id?: string;
organization_id: number;
project_id: number;
cost?: EvalCost | null;
inserted_at: string;
updated_at: string;
}
Expand Down
15 changes: 15 additions & 0 deletions app/components/utils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,21 @@ export const getStatusColor = (
}
};

/**
* Formats a USD cost value for display
* @param cost - Cost in USD
* @returns Formatted cost string (e.g., "$0.0013", "$1.25")
*/
export const formatCostUSD = (cost: number): string => {
if (!Number.isFinite(cost)) {
return "N/A";
}
if (cost < 0.01) {
return `$${cost.toFixed(4)}`;
}
return `$${cost.toFixed(2)}`;
};
Comment thread
coderabbitai[bot] marked this conversation as resolved.

/**
* Calculates dynamic thresholds for color coding based on score distribution
* @param scores - Array of similarity scores
Expand Down
Loading