feat(integration): Add KubeflowCallback to enable automatic progress …

[abhijeet-dhumal]

What does this PR do?

Fixes #44486

Adds KubeflowCallback to enable automatic progress and metrics reporting for training jobs running on Kubeflow Trainer.

When training runs inside a Kubeflow TrainJob, the callback automatically:

• Reports training progress percentage (0-100%)
• Calculates and reports estimated time remaining (ETA)
• Pushes training metrics (loss, accuracy, etc.) to the TrainJob status

This enables real-time visibility into training progress via standard Kubernetes APIs:

kubectl get trainjob my-job -o jsonpath='{.status.trainerStatus}'
# {"progressPercentage": 67, "estimatedRemainingSeconds": 3600, "metrics": [...]}

Zero friction for users - no code changes required:

from transformers import Trainer, TrainingArguments

trainer = Trainer(model=model, args=TrainingArguments(...), train_dataset=ds)
trainer.train()  # Progress automatically reported when running in Kubeflow

Changes

• Add is_kubeflow_available() function that detects Kubeflow environment via KUBEFLOW_TRAINER_STATUS_URL env var
• Add KubeflowCallback class following the MLflowCallback/WandbCallback pattern
• Register in INTEGRATION_TO_CALLBACK and get_available_reporting_integrations()
• Add unit tests in tests/trainer/test_trainer_callback.py

Related PRs

Repository	PR	Description
kubeflow/trainer	#3227	Controller-side implementation (status server)
kubeflow/sdk	kubeflow/sdk#367	SDK update_runtime_status() utility

---

Additional context

This is part of KEP-2779 - a coordinated effort with the Kubeflow community to enable real-time training progress tracking on Kubernetes.

The callback has an optional dependency on kubeflow-sdk. The SDK handles:

• HTTP transport to the Kubeflow controller
• Authentication via projected service account tokens
• Throttling (max 1 update per 5 seconds)

When the TrainJobProgress feature gate is enabled in Kubeflow Trainer, the controller automatically injects these environment variables into training pods:

• KUBEFLOW_TRAINER_STATUS_URL - HTTPS endpoint for status updates
• KUBEFLOW_TRAINER_STATUS_CA_CERT - CA cert for TLS
• KUBEFLOW_TRAINER_STATUS_TOKEN - Projected SA token for auth

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[abhijeet-dhumal]

Waiting for Kubeflow Trainer feature implementation to be merged : kubeflow/trainer#3227 ...