-
Notifications
You must be signed in to change notification settings - Fork 5.4k
Add AI code review tools evaluation policy #8910
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| # AI Code Review Tools | ||
|
|
||
| The Kubernetes project may evaluate AI-powered code review tools on a | ||
| per-repo opt-in basis. This document describes the process for requesting, | ||
| evaluating, and deciding on the use of such tools. | ||
|
|
||
| ## Scope | ||
|
|
||
| This policy covers AI tools that automatically review pull requests, such as | ||
| CodeRabbit or GitHub Copilot code review. It does not cover other AI-powered | ||
| tooling such as CI/CD, security scanning, or code generation assistants. | ||
|
|
||
| ## Requesting a New Tool | ||
|
|
||
| A subproject lead listed in the repository's top-level [OWNERS] file files an | ||
| issue on [kubernetes/org]. The issue must: | ||
|
|
||
| - Identify the tool and link to its documentation | ||
| - Describe the use cases and what the subproject is trying to accomplish | ||
| - Explain why existing approved tools do not meet their needs | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there a list of already-approved tools somewhere?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. IIRC k8s repo admins can enable Copilot auto code reviews, but as @kannon92 mentioned in kubernetes/org#5930 (comment), it requires PR authors to have Copilot subscription first, or the maintainers need to manually request reviews from Copilot
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. OK. I'm just saying that if we have existing tools, we should link to a list somewhere, so that it'll save a lot of duplicate requests. |
||
| - List the specific repositories for the pilot | ||
|
|
||
| ## Privacy and Security Assessment | ||
|
|
||
| Upon receiving a request, the [GitHub Administration Team] conducts a privacy | ||
| and security assessment of the tool. The assessment documents: | ||
|
|
||
| - What GitHub permissions and OAuth scopes the tool requires | ||
| - What data the tool accesses and where it is sent | ||
| - What AI models are used to process the code | ||
| - Data retention and deletion policies | ||
| - Security certifications (SOC2, etc.) | ||
| - Whether access can be scoped to specific repositories | ||
|
|
||
| The assessment is documented in the [kubernetes/org] issue for transparency. | ||
|
|
||
| ## Approval | ||
|
|
||
| The [GitHub Administration Team] reviews the request and the privacy and | ||
| security assessment, and approves or rejects the request. If approved, the | ||
| [GitHub Administration Team] enables the tool on the requested repositories | ||
| and applies an org-wide default configuration. | ||
|
|
||
| ## Pilot Structure | ||
|
|
||
| - The pilot runs for a 90-day evaluation period starting from the date the | ||
| tool is enabled | ||
| - The tool is enabled only on the specific requested repositories, not org-wide | ||
| - An org-wide default configuration is applied; repositories may customize | ||
| within those bounds | ||
| - The sponsoring subproject is responsible for collecting feedback from | ||
| contributors and reviewers during the pilot | ||
|
|
||
| ## Evaluation and Decision | ||
|
|
||
| At the end of the pilot period, the sponsoring subproject provides a summary | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. how is this summary provided? is this going to be an artifact somewhere? is providing some structure to the summary a good idea so we can have consistency on the reports? |
||
| covering: | ||
|
|
||
| - Quality of reviews (signal vs noise) | ||
| - Contributor and reviewer feedback | ||
| - Any issues encountered | ||
|
|
||
| The [GitHub Administration Team], in consultation with the sponsoring | ||
| subproject, decides to: | ||
|
|
||
| - Continue and expand to additional repositories | ||
| - Continue with modifications | ||
| - Remove the tool | ||
|
|
||
| Expansion to additional repositories follows the same request process described | ||
| above. | ||
|
|
||
| ## Removal | ||
|
|
||
| If a pilot is unsuccessful or a tool is no longer desired, the | ||
| [GitHub Administration Team] will disable the integration. Subproject leads may | ||
| request removal at any time by filing an issue on [kubernetes/org]. | ||
|
|
||
| [GitHub Administration Team]: /github-management/README.md#github-administration-team | ||
| [OWNERS]: /contributors/guide/owners.md | ||
| [kubernetes/org]: https://github.com/kubernetes/org/issues | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what about open source tools leveraging AI ? we could have something built in the open just for issue triage for example.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The CNCF provides https://dosu.dev/ for this purpose already. ref: https://contribute.cncf.io/resources/services/hosted-tools/#tools
But imo I'm not sure this policy should be in scope for your question.