Skip to content

TEST_CREATION.md

Latest commit

 

History

History
357 lines (253 loc) · 8.51 KB

TEST_CREATION.md

File metadata and controls

357 lines (253 loc) · 8.51 KB

Creating Tests with the Revyl CLI

Back to README | Commands | Agent Skills

This guide covers the full CLI authoring loop for Revyl tests:

  • create tests from YAML
  • scaffold a test and then edit it locally
  • import reusable modules
  • push, run, open, and troubleshoot tests

Choose a Workflow

Use one of these flows depending on where your test starts:

  1. YAML-first CLI (recommended) Start from a local YAML file, validate it, create the remote test, and bootstrap .revyl/config.yaml automatically.
  2. Scaffold first Create an empty or module-seeded remote test with revyl test create, sync the generated YAML into .revyl/tests/, then edit and push locally.
  3. Dev loop to regression Use revyl dev test create after an exploratory session, then refine the synced YAML and push it back as a stable regression.

Prerequisites

Before creating tests, make sure you have:

  • authenticated with revyl auth login
  • an app/build available for the target platform
  • the correct app name for test.build.name

Use these commands to confirm app names before you author YAML:

revyl app list
revyl app list --platform ios
revyl app list --platform android

test.build.name must match a Revyl app name for the test's platform. If the name does not resolve, test push will fail.

Workflow 1: YAML-First CLI

This is the best path when you already know the flow you want to encode.

1. Write a YAML file

Example smoke-login-ios.yaml:

test:
  metadata:
    name: smoke-login-ios
    platform: ios
    tags:
      - smoke
  build:
    name: ios-test
  blocks:
    - type: instructions
      step_description: Sign in with valid test credentials.
    - type: validation
      step_description: The inbox is visible.

2. Validate the YAML

revyl test validate ./smoke-login-ios.yaml

Use --json when you want machine-readable output in CI or scripts:

revyl test validate ./smoke-login-ios.yaml --json

3. Create the test from the YAML file

revyl test create smoke-login-ios --from-file ./smoke-login-ios.yaml

What this does:

  • validates the YAML
  • copies it to .revyl/tests/smoke-login-ios.yaml
  • creates or updates the remote test through the normal push flow
  • writes .revyl/config.yaml if it does not already exist
  • stores _meta.remote_id, _meta.remote_version, checksum, and sync timestamps in the local YAML

This YAML-first bootstrap works even when the project does not already have .revyl/config.yaml.

4. Iterate locally and push changes

Edit the synced file:

.revyl/tests/smoke-login-ios.yaml

Then push:

revyl test push smoke-login-ios --force

Use --force when you want to overwrite the current remote version with your local YAML.

5. Run or open the test

revyl test run smoke-login-ios
revyl test open smoke-login-ios
revyl test report smoke-login-ios --json

Workflow 2: Scaffold First, Then Edit Locally

This path is useful when you want the CLI to create the remote test shell first.

1. Create the test scaffold

revyl test create smoke-login-ios --platform ios

Useful flags:

revyl test create smoke-login-ios --platform ios --no-open
revyl test create smoke-login-ios --platform ios --app <app-id>
revyl test create smoke-login-ios --platform ios --module login
revyl test create smoke-login-ios --platform ios --tag smoke --tag ios

This flow:

  • creates the remote test immediately
  • adds the alias to .revyl/config.yaml unless --no-sync is used
  • syncs the generated YAML into .revyl/tests/<name>.yaml
  • optionally opens the browser editor unless --no-open is used

2. Edit the local YAML

After scaffold creation, refine:

.revyl/tests/smoke-login-ios.yaml

Then push the updated YAML back to the server:

revyl test push smoke-login-ios --force

Use revyl test open smoke-login-ios when you want to continue editing in the browser instead of locally.

YAML Anatomy

Every test file has the same top-level structure:

test:
  metadata:
    name: my-test
    platform: ios
  build:
    name: ios-test
  blocks:
    - type: instructions
      step_description: Do something

Key fields:

  • test.metadata.name: the remote test name
  • test.metadata.platform: ios or android
  • test.build.name: the Revyl app name to run against
  • test.blocks: the ordered test steps

Common block types:

  • instructions Use for a single user action.
  • validation Use for assertions about the expected state after an action.
  • manual Use for framework-level actions such as wait, go_home, navigate, set_location, kill_app, or open_app.
  • module_import Use for reusable flows such as login or onboarding.
  • if, while, extraction, code_execution Use when you need conditional logic, variables, or dynamic behavior.

Manual block examples:

- type: manual
  step_type: wait
  step_description: "3"

- type: manual
  step_type: navigate
  step_description: myapp://inbox

open_app is valid, but Revyl opens the app automatically at test start, so using it as the first block is often unnecessary.

Authoring Rules for Stable Tests

Prefer these patterns:

  1. One action per instruction step.
  2. Keep assertions in separate validation blocks.
  3. Validate durable user-visible outcomes instead of transient loading copy.
  4. Use variables for credentials and secrets instead of hard-coding them in reusable tests.
  5. Keep module imports at the top of the flow when they establish shared setup.

Good:

- type: validation
  step_description: The inbox is visible.
- type: instructions
  step_description: Tap Compose.

Bad:

- type: instructions
  step_description: Verify the inbox is visible and tap Compose.

Reusing Modules

Modules let you keep common flows out of individual tests.

List modules:

revyl module list
revyl module list --search login

Print a ready-to-paste YAML snippet:

revyl module insert login

Example output:

- type: module_import
  step_description: "login"
  module_id: "65c5ac48-b980-43c7-a78e-e58b0daf183b"

You can paste that into test.blocks, or seed a scaffolded test directly:

revyl test create login-smoke --platform ios --module login

Example YAML with a module import:

test:
  metadata:
    name: login-smoke
    platform: ios
  build:
    name: ios-test
  blocks:
    - type: module_import
      step_description: login
      module_id: 65c5ac48-b980-43c7-a78e-e58b0daf183b
    - type: validation
      step_description: The inbox is visible.

Troubleshooting

A test with that name already exists

If create or push fails because the remote name already exists:

  • choose a new test name, or
  • inspect remote names first with revyl test remote, or
  • use the scaffold flow when your goal is to reuse an existing remote test name and sync it locally

build.name does not resolve

If push says the app could not be found:

revyl app list --platform ios
revyl app list --platform android

Then update test.build.name to match the actual Revyl app name exactly.

Test shows as stale or [missing-upstream]

This usually means the local file or alias points at a remote test ID that no longer exists or is not visible in the current org.

Start with:

revyl sync --tests --prune
revyl test remote

Then re-push the local YAML if needed:

revyl test push <test-name> --force

Target a non-default backend

Use REVYL_BACKEND_URL when you want test creation and push to hit a local, staging, or preview backend:

REVYL_BACKEND_URL=http://127.0.0.1:8000 revyl test push smoke-login-ios --force
REVYL_BACKEND_URL=http://127.0.0.1:8000 revyl module list

Prefer 127.0.0.1 over localhost if your machine resolves localhost to IPv6 and the backend only listens on IPv4.

If you also need browser-based commands such as revyl test open to point at a non-default frontend, set REVYL_APP_URL to the matching app host.

Recommended Loop

For most CLI users, this is the simplest repeatable flow:

# 1) Author locally
revyl test validate ./my-test.yaml

# 2) Create and bootstrap local state
revyl test create my-test --from-file ./my-test.yaml

# 3) Iterate on the synced file
revyl test push my-test --force

# 4) Run and inspect failures
revyl test run my-test
revyl test report my-test --json