Skip to content

Commit d15632c

Browse files
mcgrofmcgrof
committed
bootlinux: add support for A/B kernel testing
Right now we use the same kernel for all target nodes. We want to compare and contrast different kenrels for different features. We add support for A/B testing by leveraging the baseline and dev groups provided to us by KDEVOPS_BASELINE_AND_DEV. This extends the bootlinux playbook by enabling us to allow a different kernel tree / ref to be used for the dev group. This just becomes a configuration thing. The targets are intuitive: make linux # Handles A/B compilation transparently make linux-baseline # Build and install baseline kernel only make linux-dev # Build and install development kernel only Generated-by: Claude AI Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
1 parent 77bba98 commit d15632c

7 files changed

Lines changed: 483 additions & 7 deletions

File tree

‎PROMPTS.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,3 +123,48 @@ source "workflows/mmtests/Kconfig.thpchallenge"
123123
source "workflows/mmtests/Kconfig.fs"
124124

125125
This separation is preferred as it helps us scale.
126+
127+
## Kernel development and A/B testing support
128+
129+
### Adding A/B kernel testing support for different kernel versions
130+
131+
**Prompt:**
132+
We want to add support for when users enable KDEVOPS_BASELINE_AND_DEV we want
133+
to extend workflows/linux/Kconfig with the a choise set of options to either a)
134+
use the same kernel ref or b) allow the user to specify a different ref tag.
135+
This will enable A/B testing with different kernel versions. When a different
136+
kernel refs are desirable we will want to extend the compilation step and
137+
installation of the Linux kernel in two steps. The first will be for the ref
138+
and target of A (baseline tag) and the second will be for the target ref of B
139+
(dev tag). However we want to fold these two steps in one for when
140+
KDEVOPS_BASELINE_AND_DEV is used and make install is used, it would happen
141+
transparently for us. The resulting linux kernel directory would end up with
142+
the "dev" ref at the end. In case a user wants to re-compile a target ref for
143+
baseline or dev we want to add (if we don't have already) a make linux-baseline
144+
and make linux-dev so that we can build and install the target ref tag on the
145+
baseline (A) or dev (B). The make linux target then would serially do make
146+
linux-baseline and make linux-dev. Extend documentation for all this and also
147+
add the respective prompt to PROMPTS.md once done. Avoid adding extra spaces to
148+
code or documentation at the end of each line. These end up in red color on
149+
diffs and hurt my eyes. Extend CLAUDE.md to understand styling for these rules
150+
about not wanting lines ending in white space for styling.
151+
152+
**AI:** Claude Code
153+
**Commit:** [To be determined]
154+
**Result:** Complete A/B kernel testing implementation with comprehensive configuration options.
155+
**Grading:** 70%
156+
157+
**Notes:**
158+
159+
The implementation successfully added:
160+
161+
1. **Makefile Implementation**: the AI failed to grasp the value of
162+
output yaml, and made ugly Makefile changes to extract variables.
163+
164+
2. **Ansible Integration**: The AI failed to write the required changes on
165+
the ansible playbook at first. A secondary prompt made it just move the
166+
definitions to the ansible playbook but failed to address serially compiling
167+
linux for the baseline group first followed by the dev group after.
168+
169+
3. **Documentation**: The AI is not grasping the preference to respect 80
170+
character lengths.

‎docs/kdevops-make-linux.md‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,161 @@ To verify the kernel on it:
1313
```bash
1414
make uname
1515
```
16+
17+
## A/B Kernel Testing
18+
19+
kdevops supports A/B testing with different kernel versions when
20+
`KDEVOPS_BASELINE_AND_DEV` is enabled. This allows you to compare performance
21+
or behavior between different kernel versions across baseline and development nodes.
22+
23+
### Configuration Options
24+
25+
When A/B testing is enabled, you can choose between two approaches:
26+
27+
#### Same Kernel Reference (Default)
28+
Use the same kernel tree and reference for both baseline and dev nodes:
29+
```
30+
A/B kernel testing configuration (BOOTLINUX_AB_SAME_REF) [Y/n/?]
31+
```
32+
33+
This is useful for testing configuration changes or different test parameters
34+
with identical kernels.
35+
36+
#### Different Kernel References
37+
Use different kernel references for baseline and dev nodes:
38+
```
39+
A/B kernel testing configuration
40+
1. Use same kernel reference for baseline and dev (BOOTLINUX_AB_SAME_REF)
41+
> 2. Use different kernel references for baseline and dev (BOOTLINUX_AB_DIFFERENT_REF)
42+
```
43+
44+
This enables testing between different kernel versions, commits, or branches.
45+
46+
When using different references, configure:
47+
- **Development kernel tree URL**: Git repository (defaults to baseline tree)
48+
- **Development kernel reference**: Branch, tag, or commit (e.g., "v6.8", "linux-next")
49+
- **Development kernel release/local version**: Custom version strings for identification
50+
51+
### Make Targets
52+
53+
#### Standard Linux Building
54+
```bash
55+
make linux # Build and install kernels for all nodes
56+
```
57+
58+
When A/B testing with different references is enabled, this automatically:
59+
1. Builds and installs baseline kernel on baseline nodes
60+
2. Builds and installs development kernel on dev nodes
61+
3. Leaves the working directory with the dev kernel checked out
62+
63+
#### Individual Node Targeting
64+
```bash
65+
make linux-baseline # Build and install kernel for baseline nodes only
66+
make linux-dev # Build and install kernel for dev nodes only
67+
```
68+
69+
These targets are available when `KDEVOPS_BASELINE_AND_DEV=y` and allow
70+
selective building and installation.
71+
72+
### Usage Examples
73+
74+
#### Testing Kernel Versions
75+
Compare v6.7 (baseline) vs v6.8 (development):
76+
77+
```bash
78+
# Configure baseline kernel
79+
menuconfig → Workflows → Linux kernel → Git tree to clone: linus
80+
Reference to use: v6.7
81+
82+
# Configure A/B testing
83+
menuconfig → Workflows → Linux kernel → A/B kernel testing
84+
→ Use different kernel references
85+
→ Development kernel reference: v6.8
86+
87+
make bringup # Provision baseline and dev nodes
88+
make linux # Install v6.7 on baseline, v6.8 on dev
89+
make fstests # Run tests on both kernel versions
90+
make fstests-compare # Compare results between versions
91+
```
92+
93+
#### Testing Development Branches
94+
Compare stable vs linux-next:
95+
96+
```bash
97+
# Baseline: stable kernel
98+
menuconfig → Reference to use: v6.8
99+
100+
# Development: linux-next
101+
menuconfig → A/B kernel testing → Development kernel reference: linux-next
102+
103+
make linux-baseline # Install stable kernel on baseline nodes
104+
make linux-dev # Install linux-next on dev nodes
105+
```
106+
107+
#### Bisection Support
108+
Test specific commits during bisection:
109+
110+
```bash
111+
# Update development reference for bisection
112+
menuconfig → Development kernel reference: abc123def
113+
114+
make linux-dev # Install bisection commit on dev nodes
115+
# Run tests and analyze results
116+
```
117+
118+
### Working Directory State
119+
120+
After running `make linux` with different references:
121+
- The Linux source directory contains the **development kernel** checkout
122+
- Both baseline and dev nodes have their respective kernels installed
123+
- Use `git log --oneline -5` to verify the current checkout
124+
125+
To switch the working directory to baseline:
126+
```bash
127+
git checkout v6.7 # Switch to baseline reference
128+
```
129+
130+
### Integration with Testing Workflows
131+
132+
A/B kernel testing integrates seamlessly with all kdevops testing workflows:
133+
134+
```bash
135+
# Run fstests with kernel comparison
136+
make linux # Install different kernels
137+
make fstests # Test both kernel versions
138+
make fstests-compare # Generate comparison analysis
139+
140+
# Run fio-tests with kernel comparison
141+
make linux # Install different kernels
142+
make fio-tests # Performance test both kernels
143+
make fio-tests-compare # Compare performance metrics
144+
145+
# Run sysbench with kernel comparison
146+
make linux # Install different kernels
147+
make sysbench # Database tests on both kernels
148+
```
149+
150+
### Best Practices
151+
152+
1. **Version Identification**: Use descriptive kernel release versions to distinguish builds
153+
2. **Sequential Testing**: Install kernels before running test workflows
154+
3. **Result Organization**: Use baseline/dev labels in test result analysis
155+
4. **Git Management**: Keep track of which reference is currently checked out
156+
5. **Systematic Comparison**: Use `*-compare` targets for meaningful analysis
157+
158+
### Troubleshooting
159+
160+
#### Build Failures
161+
- Ensure both kernel references are valid and accessible
162+
- Check that build dependencies are installed on all nodes
163+
- Verify git repository permissions and network connectivity
164+
165+
#### Version Conflicts
166+
- Use different `kernelrelease` and `localversion` settings for clear identification
167+
- Check `/boot` directory for kernel installation conflicts
168+
- Verify GRUB configuration after kernel installation
169+
170+
#### Node Targeting Issues
171+
- Confirm `KDEVOPS_BASELINE_AND_DEV=y` is enabled
172+
- Verify baseline and dev node groups exist in inventory
173+
- Check ansible host patterns with `make linux-baseline HOSTS=baseline`

‎playbooks/roles/bootlinux/defaults/main.yml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,3 +52,15 @@ kdevops_workflow_enable_cxl: False
5252

5353
bootlinux_cxl_test: False
5454
bootlinux_tree_set_by_cli: False
55+
56+
# A/B testing defaults
57+
bootlinux_ab_same_ref: True
58+
bootlinux_ab_different_ref: False
59+
60+
# Development kernel settings (used when bootlinux_ab_different_ref is True)
61+
bootlinux_dev_tree: ""
62+
bootlinux_dev_tree_ref: "master"
63+
bootlinux_dev_tree_kernelrelease: ""
64+
bootlinux_dev_tree_localversion: ""
65+
bootlinux_tree_custom_kernelrelease: False
66+
bootlinux_tree_custom_localversion: false

‎playbooks/roles/bootlinux/tasks/main.yml‎

Lines changed: 93 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,74 @@
6161
when:
6262
- not kdevops_baseline_and_dev|bool
6363

64+
- name: Determine if this is a dev node for A/B testing
65+
set_fact:
66+
bootlinux_is_dev_node: "{{ ansible_hostname | regex_search('^.*-dev$') is not none }}"
67+
when:
68+
- kdevops_baseline_and_dev|bool
69+
- bootlinux_ab_different_ref|bool
70+
71+
- name: Set development group full custom kernel release
72+
set_fact:
73+
target_linux_kernelrelease: "{{ bootlinux_dev_tree_kernelrelease if bootlinux_dev_tree_kernelrelease != '' else target_linux_kernelrelease }}"
74+
when:
75+
- kdevops_baseline_and_dev|bool
76+
- bootlinux_ab_different_ref|bool
77+
- bootlinux_tree_custom_kernelrelease|bool
78+
- bootlinux_is_dev_node|default(false)|bool
79+
80+
- name: Set development group local append version
81+
set_fact:
82+
target_linux_localversion: "{{ bootlinux_dev_tree_localversion if bootlinux_dev_tree_localversion != '' else target_linux_localversion }}"
83+
when:
84+
- kdevops_baseline_and_dev|bool
85+
- bootlinux_ab_different_ref|bool
86+
- bootlinux_tree_custom_localversion|bool
87+
- bootlinux_is_dev_node|default(false)|bool
88+
89+
- name: Set development kernel parameters for dev nodes
90+
set_fact:
91+
target_linux_git: "{{ bootlinux_dev_tree if bootlinux_dev_tree != '' else target_linux_git }}"
92+
target_linux_ref: "{{ bootlinux_dev_tree_ref }}"
93+
target_linux_config: "config-{{ bootlinux_dev_tree_ref }}"
94+
when:
95+
- kdevops_baseline_and_dev|bool
96+
- bootlinux_ab_different_ref|bool
97+
- bootlinux_is_dev_node|default(false)|bool
98+
99+
- name: Determine active kernel parameters for A/B testing with 9P
100+
set_fact:
101+
active_linux_ref: "{{ bootlinux_dev_tree_ref }}"
102+
when:
103+
- kdevops_baseline_and_dev|bool
104+
- bootlinux_ab_different_ref|bool
105+
- bootlinux_tree_custom_kernelrelease|bool
106+
- bootlinux_9p|bool
107+
run_once: true
108+
delegate_to: localhost
109+
110+
- name: Determine full custom kernel release for A/B testing with 9P
111+
set_fact:
112+
active_linux_kernelrelease: "{{ hostvars[groups['dev'][0]]['target_linux_kernelrelease'] if hostvars[groups['dev'][0]]['target_linux_kernelrelease'] is defined else target_linux_kernelrelease }}"
113+
when:
114+
- kdevops_baseline_and_dev|bool
115+
- bootlinux_ab_different_ref|bool
116+
- bootlinux_tree_custom_kernelrelease|bool
117+
- bootlinux_9p|bool
118+
run_once: true
119+
delegate_to: localhost
120+
121+
- name: Determine localversion kernel release for A/B testing with 9P
122+
set_fact:
123+
active_linux_localversion: "{{ hostvars[groups['dev'][0]]['target_linux_localversion'] if hostvars[groups['dev'][0]]['target_linux_localversion'] is defined else target_linux_localversion }}"
124+
when:
125+
- kdevops_baseline_and_dev|bool
126+
- bootlinux_ab_different_ref|bool
127+
- bootlinux_tree_custom_localversion|bool
128+
- bootlinux_9p|bool
129+
run_once: true
130+
delegate_to: localhost
131+
64132
- include_role:
65133
name: create_data_partition
66134

@@ -412,6 +480,20 @@
412480
- not bootlinux_9p|bool
413481
- snaik_oil_file.stat.exists
414482

483+
- name: Checkout correct git ref for A/B testing with 9P
484+
git:
485+
repo: "{{ target_linux_git }}"
486+
dest: "{{ bootlinux_9p_host_path }}"
487+
version: "{{ active_linux_ref | default(target_linux_ref) }}"
488+
force: yes
489+
tags: [ 'build-linux' ]
490+
when:
491+
- bootlinux_9p|bool
492+
- kdevops_baseline_and_dev|bool
493+
- bootlinux_ab_different_ref|bool
494+
run_once: true
495+
delegate_to: localhost
496+
415497
- name: Get nproc on the control node
416498
command: "{{ num_jobs }}"
417499
tags: [ 'build-linux', 'cxl-build' ]
@@ -429,21 +511,24 @@
429511
tags: [ 'build-linux' ]
430512
when:
431513
- bootlinux_9p|bool
432-
- target_linux_kernelrelease | length > 0
514+
- (active_linux_kernelrelease | default(target_linux_kernelrelease)) | length > 0
433515
run_once: true
434516
delegate_to: localhost
435517

436-
- name: Generate user kernelrelease {{ target_linux_kernelversion.stdout }}-{{ target_linux_kernelrelease }}
518+
- name: Generate user kernelrelease
437519
set_fact:
438-
target_user_kernelrelease: "{{ target_linux_kernelversion.stdout }}-{{ target_linux_kernelrelease }}"
520+
target_user_kernelrelease: "{{ target_linux_kernelversion.stdout }}-{{ active_linux_kernelrelease | default(target_linux_kernelrelease) }}"
439521
tags: [ 'build-linux' ]
440522
when:
441523
- bootlinux_9p|bool
442-
- target_linux_kernelrelease | length > 0
524+
- bootlinux_tree_custom_kernelrelease|bool
525+
- (active_linux_kernelrelease | default(target_linux_kernelrelease)) | length > 0
526+
- target_linux_kernelversion is defined
527+
- target_linux_kernelversion.stdout is defined
443528
run_once: true
444529
delegate_to: localhost
445530

446-
- name: Build {{ target_linux_tree }} {{ target_user_kernelrelease }} on the control node using {{ nproc_9p.stdout }} threads
531+
- name: Build {{ target_linux_tree }} with custom kernel release on the control node using {{ nproc_9p.stdout }} threads
447532
make:
448533
jobs: "{{ nproc_9p.stdout }}"
449534
chdir: "{{ bootlinux_9p_host_path }}"
@@ -452,11 +537,13 @@
452537
tags: [ 'build-linux' ]
453538
when:
454539
- bootlinux_9p|bool
540+
- bootlinux_tree_custom_kernelrelease|bool
455541
- target_linux_kernelrelease | length > 0
542+
- target_user_kernelrelease is defined
456543
run_once: true
457544
delegate_to: localhost
458545

459-
- name: Build {{ target_linux_tree }} {{ target_user_kernelrelease }} on the control node using {{ nproc_9p.stdout }} threads
546+
- name: Build {{ target_linux_tree }} on the control node using {{ nproc_9p.stdout }} threads
460547
make:
461548
jobs: "{{ nproc_9p.stdout }}"
462549
chdir: "{{ bootlinux_9p_host_path }}"

0 commit comments

Comments
 (0)