feat(image-test): run LISA test suites locally via 'image test'#292
Open
liunan-ms wants to merge 1 commit into
Open
feat(image-test): run LISA test suites locally via 'image test'#292liunan-ms wants to merge 1 commit into
liunan-ms wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR enables running LISA-based image test suites locally via azldev image test, expanding the existing test-suite model (previously only pytest was runnable) to support generating a LISA runbook, booting the image via QEMU, and executing LISA against it.
Changes:
- Extend test-suite configuration with an optional
[lisa]subtable (LisaConfig) including framework git source + test cases and related options, plus validation of the framework ref and presence of test cases. - Add a local LISA runner that clones the framework repo at a pinned commit, creates/reuses a per-suite venv, generates a qemu-platform runbook, generates an ephemeral SSH keypair, and invokes LISA.
- Regenerate schema, CLI docs, and scenario snapshots to reflect the new configuration surface.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| schemas/azldev.schema.json | Generated JSON schema updates to include GitSourceConfig and LisaConfig. |
| scenario/snapshots/TestSnapshotsContainer_config_generate-schema_stdout_1.snap | Snapshot refresh for schema generation output (container mode). |
| scenario/snapshots/TestSnapshots_config_generate-schema_stdout_1.snap | Snapshot refresh for schema generation output (non-container mode). |
| internal/projectconfig/testsuite.go | Adds LisaConfig, GitSourceConfig, and new validation logic for LISA suites. |
| internal/projectconfig/testsuite_test.go | Unit tests for new LISA validation behavior. |
| internal/projectconfig/loader_test.go | Loader test updated to parse and validate a LISA suite config. |
| internal/app/azldev/cmds/image/test.go | Routes lisa suites to the new local runner and updates help text. |
| internal/app/azldev/cmds/image/lisarunner.go | New implementation: clone framework, venv setup, runbook generation, qcow2 enforcement, and LISA invocation. |
| internal/app/azldev/cmds/image/lisarunbook.go | New runbook YAML model + generator for QEMU platform. |
| internal/app/azldev/cmds/image/lisarunbook_internal_test.go | Unit tests for runbook YAML generation. |
| docs/user/reference/cli/azldev_image_test.md | Regenerated CLI docs describing LISA suite behavior. |
liunan-ms
changed the title
liunan-ms
changed the title
Previously LISA-type suites were rejected by 'azldev image test' and only served as metadata for external orchestration. This adds a local runner that generates a LISA runbook from a suite's configured test cases, boots the image in a QEMU VM, and runs the tests. - Extend LisaConfig with a framework git source, test-cases, pip pre-install/extras, and extra-args; validate that the framework ref is a full commit SHA and that test-cases is non-empty. - Generate a qemu-platform runbook that inlines the image path and an ephemeral admin key, with keep_environment=no so VMs are torn down. - Auto-generate and clean up an ephemeral admin SSH key pair per run. - Require the image to be qcow2 (reject other formats) before booting. - Regenerate CLI docs, JSON schema, and scenario snapshots.
liunan-ms
force-pushed
the
liunan/add_lisa
branch
from
July 17, 2026 21:36
9e6b25e to
efd21bc
Compare
| return fmt.Errorf("git is required to clone LISA repos:\n%w", err) | ||
| } | ||
|
|
||
| lisaBaseDir := filepath.Join(env.WorkDir(), lisaDirName) |
Comment on lines
+287
to
+291
| // Always checkout the pinned ref, even when reusing an existing checkout, so an | ||
| // interrupted or externally-modified clone still runs against the correct revision. | ||
| if err := gitProvider.Checkout(env, repoDir, source.Ref); err != nil { | ||
| return "", fmt.Errorf("failed to checkout %#q:\n%w", source.Ref, err) | ||
| } |
Comment on lines
+28
to
+32
| For LISA tests, the test runner executes on the host and boots the image in a | ||
| QEMU VM. azldev clones the LISA framework, generates a runbook from the suite's | ||
| configured test cases, and runs it against the image. azldev generates an | ||
| ephemeral SSH key pair to access the booted VM and removes it once the suite | ||
| finishes. |
Comment on lines
+371
to
+378
| // buildLisaArgs constructs the LISA command-line arguments. The runbook path is passed | ||
| // via -r, and extra-args are appended after placeholder expansion. | ||
| func buildLisaArgs( | ||
| runbookPath string, | ||
| extraArgs []string, | ||
| imageConfig *projectconfig.ImageConfig, | ||
| options *ImageTestOptions, | ||
| ) []string { |
liunan-ms
marked this pull request as ready for review
July 17, 2026 23:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Previously LISA-type test suites were rejected by
azldev image testand only served as metadata for external orchestration. This adds a local runner that generates a LISA runbook from a suite's configured test cases, boots the image in a QEMU VM, and runs the tests.Changes
LisaConfigwith a framework git source,test-cases, pip pre-install/extras, andextra-args; validate that the framework ref is a full commit SHA and thattest-casesis non-empty.keep_environment=noso VMs are torn down after the run.Testing
mage unit— passmage check all— pass (golangci-lint, editorconfig, staticcheck, licenses, ruff, pyright)mage scenarioUpdate— snapshots regeneratedUsage in microsoft/azurelinux#18042