From 93cbf1d8fb8ab3976aae8643d43a410c7644c4e6 Mon Sep 17 00:00:00 2001 From: Lili Deng Date: Fri, 12 Jun 2026 09:41:26 +0800 Subject: [PATCH] docs(prompts): align install / runbook / test-writer with LISA loader and CI lint - install-lisa: capture LISA_HOME / PYTHON / VENV after install, write them to /memories/session/lisa-install.md, set env vars, and add troubleshooting for venv-related 'lisa not found' / wrong Python. Require verifying the running LISA's root via python -c 'import lisa, pathlib; print(pathlib.Path(lisa.__file__).parent.parent)' and recording that path (not pwd) as LISA_HOME. - lisa_runbook_generator: add Step 0 to resolve LISA_HOME (session memory -> env var -> workspace probe -> ask user) before emitting YAML; require absolute extension: paths (Rule 8); reflect venv activation in the run command shown to the user; add troubleshooting for ModuleNotFoundError: No module named 'microsoft'. Document _fix_path_for_old_code_layout semantics so 'import_builtin_tests: true' is the primary pattern for microsoft tests. - lisa_test_writer: add Step 9 Lint Gate documenting the CI stack (black, isort, flake8, mypy, pylint) inline so generated test cases satisfy pyproject.toml / pylintrc, and require running the five local commands before reporting completion. --- .github/prompts/install-lisa.prompt.md | 161 ++++++++++++++++++ .../prompts/lisa_runbook_generator.prompt.md | 160 ++++++++++++++++- .github/prompts/lisa_test_writer.prompt.md | 68 ++++++++ 3 files changed, 387 insertions(+), 2 deletions(-) diff --git a/.github/prompts/install-lisa.prompt.md b/.github/prompts/install-lisa.prompt.md index 184a2087be..e1b9f11dfc 100644 --- a/.github/prompts/install-lisa.prompt.md +++ b/.github/prompts/install-lisa.prompt.md @@ -264,11 +264,139 @@ lisa --help --- +## Capture the Install Path (so runbooks can find `microsoft.testsuites`) + +`microsoft/testsuites/` is loaded dynamically by LISA. The mechanism (see +`lisa/parameter_parser/runbook.py::_fix_path_for_old_code_layout`) is: + +- LISA computes `` from the **running** install's + `lisa/__init__.py` location (i.e., whatever `pip` resolved the `lisa` + console-script to). +- A runbook's `extension:` path is rewritten to `/lisa/microsoft` + and registered as the Python package `microsoft` **only when** that path + is already under `/lisa/microsoft`. +- If the user has two LISA checkouts on disk and pip picked one but the + runbook points at the other, the rewrite silently doesn't fire and the + run dies with `ModuleNotFoundError: No module named 'microsoft'` even + though the directory exists. + +So the install step must guarantee — and **record** — the *single* repo +that the `lisa` command actually loads from. + +When install completes, do **all four** of the following: + +1. **Verify which repo the `lisa` command loads from** (this is the + authoritative `LISA_HOME`, never trust `pwd`): + + ```bash + # Linux / WSL + -c "import lisa, pathlib; print(pathlib.Path(lisa.__file__).parent.parent)" + ``` + + ```powershell + # Windows + & '' -c "import lisa, pathlib; print(pathlib.Path(lisa.__file__).parent.parent)" + ``` + + If the printed path is **not** the repo you just installed, there is a + stale install winning the resolution. Fix it before continuing: + + - ` -m pip uninstall -y lisa` until `pip show lisa` reports nothing, + then re-run `pip install -e .[azure]` from the desired repo, OR + - inside the desired repo, run ` -m pip install -e . --force-reinstall`, + OR + - simply delete the unused checkout from disk. + +2. **Print the absolute install path** at the end, on its own line: + + ```text + LISA_HOME= + PYTHON= + ``` + + `PYTHON` matters whenever a venv is involved (`quick-install-dev.ps1` + always creates one at `\.venv`; `quick-install.sh` does + when invoked with `--use-venv true`). Without it, opening a fresh shell + that hasn't activated the venv either fails with `lisa: command not + found` or silently picks up a system Python that has no LISA. + + Resolve `PYTHON` like this: + - Windows venv: `\.venv\Scripts\python.exe` + - Linux/WSL venv: `/.venv/bin/python` + - No venv (system / `pipx` / `--user` install): record `PYTHON=python` + so downstream tools know no activation is required. + +3. **Set the `LISA_HOME` environment variable** for the current shell. If a + venv was created, also tell the user how to activate it (or invoke the + venv python directly without activation): + + ```powershell + # Windows (PowerShell) + $env:LISA_HOME = '' + [Environment]::SetEnvironmentVariable('LISA_HOME', '', 'User') + + # If a venv exists, activate it in every new shell, OR call the venv python directly: + & '\.venv\Scripts\Activate.ps1' + # — or — + & '\.venv\Scripts\python.exe' -m lisa --help + ``` + + ```bash + # Linux / macOS / WSL + export LISA_HOME='' + echo 'export LISA_HOME=""' >> ~/.bashrc + + # If a venv exists: + source '/.venv/bin/activate' + # — or — + '/.venv/bin/python' -m lisa --help + ``` + + Do not bake `Activate.ps1` into `$PROFILE` automatically — mention it as + an option, let the user opt in. + +4. **Record it in session memory** (so other prompts in the same workspace + can read it without re-asking): + + - Path: `/memories/session/lisa-install.md` + - Content: + + ```text + LISA_HOME= + PYTHON= + VENV= + ``` + + The `lisa_runbook_generator` prompt reads this file and trusts step 1's + path as authoritative — if you skip step 1 and write `pwd` here, it WILL + bite you the moment a runbook needs microsoft testsuites. + +The `microsoft/testsuites/` directory then lives at +`$LISA_HOME/lisa/microsoft/testsuites` (note the doubled `lisa/lisa/` — +the outer `lisa/` is the repo, the inner `lisa/` is the Python package). + +--- + ## Troubleshooting ### "lisa" command not found - **Windows**: Restart PowerShell to refresh PATH - **Linux**: Ensure `~/.local/bin` is in PATH: `export PATH="$HOME/.local/bin:$PATH"` +- **venv install (any OS)**: the `lisa` console-script only exists inside the venv. + Either activate it (`.venv\Scripts\Activate.ps1` / `source .venv/bin/activate`) + or call it via the venv python without activation: + `/.venv/bin/python -m lisa --help` (Linux) / + `\.venv\Scripts\python.exe -m lisa --help` (Windows). + After activation, `(.venv)` should appear in the prompt and `where.exe lisa` + / `which lisa` should resolve into the venv directory. + +### Wrong Python is picked up (system Python instead of venv) +Symptom: `python -c "import lisa; print(lisa.__file__)"` errors out, or points +to a different LISA install than expected, even though install succeeded. +Fix: confirm the active interpreter with `python -c "import sys; print(sys.executable)"`. +It must match the `PYTHON` line recorded in `/memories/session/lisa-install.md`. +If not, activate the venv first, or invoke the venv python explicitly +(`/bin/python` / `\Scripts\python.exe`). ### Python version too old - Re-run the install script with `--python-version 3.12` (Linux) or `-PythonVersion "3.12"` (Windows) @@ -282,6 +410,39 @@ lisa --help - Ubuntu/Debian: `sudo apt-get install -y python3-dev build-essential libssl-dev libffi-dev` - RHEL/CentOS: `sudo dnf install -y python3-devel gcc openssl-devel libffi-devel` +### `ModuleNotFoundError: No module named 'microsoft'` when running a runbook + +Symptom: `lisa -r my_runbook.yml` logs +`loading Python extensions from ` then fails with +`ModuleNotFoundError: No module named 'microsoft'`. + +LISA found the directory — the failure is in the auto-rename step. The path +you gave to `extension:` must be under the **running LISA's** +`/lisa/microsoft`, otherwise the loader keeps a generic name +like `lisa_ext_0` and absolute imports such as +`from microsoft.testsuites.xfstests.xfstests import ...` cannot resolve. + +Fix in order of preference: +1. Re-run the **Capture the Install Path** verification step above to + discover the running LISA's actual root + (` -c "import lisa, pathlib; print(pathlib.Path(lisa.__file__).parent.parent)"`). +2. If the printed path is not the repo you intended, you have **two LISA + checkouts** — pip resolved the wrong one. Either `pip uninstall lisa` + in the unused checkout, or `pip install -e . --force-reinstall` from + the desired repo. +3. In the runbook, drop `extension:` and use + + ```yaml + import_builtin_tests: true + ``` + + instead. LISA then loads `/lisa/microsoft` itself — + the path can never disagree with the install. +4. Only if you actually need a custom (non-microsoft) extension, keep + `extension:` and make sure each entry is an absolute path under the + *running* LISA root, or a path relative to the runbook file that + resolves there. + ### VS Code F5 hangs / debugpy traceback in `importlib.metadata` Symptom: hitting F5 launches `C:\Program Files\Python312\python.exe` (system Python), then debugpy stalls inside `describe_environment` reading broken package METADATA. Cause: newer `ms-python` versions ignore `launch.json`'s `"python"` field and use the workspace interpreter, which defaults to system Python (no LISA installed there). diff --git a/.github/prompts/lisa_runbook_generator.prompt.md b/.github/prompts/lisa_runbook_generator.prompt.md index 0856746f27..fb5941ec47 100644 --- a/.github/prompts/lisa_runbook_generator.prompt.md +++ b/.github/prompts/lisa_runbook_generator.prompt.md @@ -4,6 +4,73 @@ You generate LISA runbook YAML files for running tests. Ask the user what scenar --- +## Step 0: Locate the **executing** LISA install (do this BEFORE generating YAML) + +LISA's runbook loader has a built-in path: when `extension:` points to +anything under `/lisa/microsoft`, LISA rewrites the path to +`/lisa/microsoft` and forces the package name to `microsoft`. +This is what makes cross-imports like +`from microsoft.testsuites.xfstests.xfstests import ...` work. + +`` is **not** an env var. It is computed at run time from +`Path(lisa.__file__).parent.parent` of the LISA install that the `lisa` +command actually loads. If the user has more than one LISA checkout, pip's +entry-point may point to a different one than the runbook expects, the +rewrite never fires, and you get `ModuleNotFoundError: No module named 'microsoft'` +even though the path on disk looks correct. + +Resolve **the executing LISA root** in this order — stop at the first hit: + +1. **Run the verification command** (preferred — it cannot be wrong): + + ```bash + # Linux / WSL + -c "import lisa, pathlib; print(pathlib.Path(lisa.__file__).parent.parent)" + ``` + + ```powershell + # Windows + & '' -c "import lisa, pathlib; print(pathlib.Path(lisa.__file__).parent.parent)" + ``` + + The printed path **is** `LISA_HOME`. Use it. If `` is unknown, + take it from `/memories/session/lisa-install.md` (written by the install + prompt) or from the active venv (`which lisa` / `where.exe lisa` then + resolve the script's shebang). + +2. **Session memory**: `/memories/session/lisa-install.md` lists `LISA_HOME`, + `PYTHON`, `VENV` from the install step. Cross-check with step 1 — if they + disagree, **trust step 1** and update the memory file. +3. **Ask the user** (only if both above fail): "Where is the LISA repo that + `lisa --version` runs from?". + +Use `LISA_HOME` as a literal absolute path inside the generated runbook +(see Rule 8). Do NOT emit placeholders like `` in the final +YAML. + +**venv reminder when presenting the run command:** if `VENV` is non-empty, +always show the user the run command **with the venv activated or invoked +explicitly**, e.g.: + +```powershell +# Activate then run (Windows) +& '\Scripts\Activate.ps1'; lisa -r path\to\runbook.yml +# — or, no activation needed — +& '\Scripts\python.exe' -m lisa -r path\to\runbook.yml +``` + +```bash +# Activate then run (Linux/WSL) +source '/bin/activate' && lisa -r path/to/runbook.yml +# — or — +'/bin/python' -m lisa -r path/to/runbook.yml +``` + +Skipping venv activation — or running a `lisa` from a different repo than +`LISA_HOME` — are the two common causes of `ModuleNotFoundError`. + +--- + ## Step 1: Clarify the Scenario Before generating YAML, ask: @@ -12,6 +79,8 @@ Before generating YAML, ask: 3. **What images?** (see Image Formats below) 4. **Special requirements?** (security profile, WSL, purchase plan, disk/NIC) 5. **Azure auth method?** (only when platform is azure — see Auth Methods below) +6. **Where will the runbook live?** — needed to compute the correct `extension:` + path (absolute is safest; see Rule 8). ### Image Formats @@ -112,7 +181,8 @@ Top-level sections (only `platform` + `testcase` are required): |---------|---------| | `name` | Descriptive run name | | `include` | Inherit from other YAML files: `- path: ./azure.yml` | -| `extension` | Extra test module paths: `- "/lisa/microsoft/testsuites"` | +| `extension` | Optional. Extra test-module roots. For microsoft testsuites, prefer `import_builtin_tests: true` (Rule 8 option B) instead of listing `extension:` here — it removes a class of path mistakes. If you must use `extension:`, follow Rule 8. | +| `import_builtin_tests` | Set to `true` to auto-load the microsoft testsuites bundled with the running LISA install. No path needed. | | `variable` | Parameters with `$(name)` substitution. Supports `is_secret: true`, `is_case_visible: true`, `file: ./secrets.yml` | | `platform` | Where to run (azure, ready, qemu, openvmm) | | `testcase` | What to run — filter by priority, name, area | @@ -373,4 +443,90 @@ Refer to `lisa/microsoft/runbook/` for available base runbooks. 4. **Use `include`** for existing base runbooks — don't duplicate. Check `lisa/microsoft/runbook/`. 5. **Security profiles** require Gen2 images and the marketplace **object format** — the 4-part string shorthand does not support them. 6. **Transformer phases** run in order: `init` → `expanded` → `environment_connected` → `expanded_cleanup` → `cleanup`. Use `phase: environment_connected` for transformers that need a provisioned VM (e.g., installing components on the node). `expanded` runs before environments are created. -7. Search `@workspace` for existing runbooks before generating — reuse patterns from `runbook/`. \ No newline at end of file +7. Search `@workspace` for existing runbooks before generating — reuse patterns from `runbook/`. +8. **Loading microsoft testsuites — pick exactly one of these patterns; never + invent a third one:** + + **(A) Preferred when the runbook lives _inside_ the LISA repo at + `//.yml` and references microsoft tests:** + use the canonical relative form, exactly like the shipped runbooks under + `lisa/microsoft/runbook/`: + + ```yaml + extension: + - "../testsuites" # when runbook is at lisa/microsoft/runbook/*.yml + # or + - "../../testsuites" # when runbook is one level deeper + ``` + + The relative path must resolve to `/lisa/microsoft/testsuites` + so LISA's `_fix_path_for_old_code_layout` rewrites the package to + `microsoft` and absolute imports work. + + **(B) Preferred when the runbook lives _outside_ the LISA repo (e.g., a + user-managed `runbooks/` folder somewhere else):** drop `extension:` + entirely and use the built-in tests flag: + + ```yaml + import_builtin_tests: true + ``` + + This makes LISA call `import_package(/lisa/microsoft, "microsoft")` + internally — same effect as (A), but with no path to get wrong. Use this + for the `lisa-bug-fix` / install-prompt-generated runbooks that sit in + user-chosen directories. + + **(C) Absolute path** — only for non-microsoft custom extensions, or as a + last resort when (A) and (B) don't fit. The path **must** point under + `/lisa/microsoft` (verified via Step 0), otherwise the + auto-rewrite to package name `microsoft` does not fire and absolute + imports break: + + ```yaml + extension: + - "/abs/path/to//lisa/microsoft/testsuites" + ``` + + Never emit `` placeholders, never use a relative path + anchored to CWD, and never point at a different LISA checkout than the + one `lisa --version` runs from — that's the bug we're trying to avoid. + +--- + +## Troubleshooting + +### `ModuleNotFoundError: No module named 'microsoft'` (or `microsoft.testsuites`) + +This fails *after* LISA logs `loading Python extensions from ...`, which +means LISA found the directory. The real cause is one of: + +- **Path is not under `/lisa/microsoft`**, so the loader's + auto-rewrite to package name `microsoft` does not fire. Confirm with the + Step 0 verification command and compare against your runbook's + `extension:` value. +- **Two LISA repos on disk**, pip's `lisa` entry-point points at one and + your runbook points at the other. Only `/lisa/microsoft` of + the running repo counts. Either: + - delete / `pip uninstall lisa` the other checkout, or + - re-`pip install -e .` from the repo you actually want, or + - use Rule 8 option (B) `import_builtin_tests: true`, which always loads + from the running LISA's repo. +- **Path is on a different filesystem from the running LISA install** (e.g., + WSL `/mnt/wsl/temp/lisa` vs `/root/lisa`). Same fix as above. + +Fix recipe: + +1. Verify the running LISA's root: + + ```bash + -c "import lisa, pathlib; print(pathlib.Path(lisa.__file__).parent.parent)" + ``` + +2. Compare with `LISA_HOME` from `/memories/session/lisa-install.md`. They + **must** be the same path. If not, fix the install (re-run the install + from inside the desired repo) before regenerating runbooks. +3. Switch the runbook to `import_builtin_tests: true` (Rule 8 option B) and + remove the `extension:` line — simplest reliable fix. +4. Re-run with `-d` and confirm the log shows + `loading Python extensions from /lisa/microsoft` (the directory + above `testsuites`, not `testsuites` itself — that's the rewrite firing). \ No newline at end of file diff --git a/.github/prompts/lisa_test_writer.prompt.md b/.github/prompts/lisa_test_writer.prompt.md index de1da86184..59539c5cf6 100644 --- a/.github/prompts/lisa_test_writer.prompt.md +++ b/.github/prompts/lisa_test_writer.prompt.md @@ -7,6 +7,7 @@ You are a senior LISA maintainer helping write correct, production-quality test ## Step 1: Mandatory Pre-Code Workflow **Do not generate code until the user confirms your Design Plan.** +**Do not report the task as complete until Step 9 (Lint Gate) passes.** 1. **Gather**: Search `#codebase`/`@workspace` for existing Tools (`lisa/tools/`), Features (`lisa/features/`), and similar TestSuites (`lisa/microsoft/testsuites/`). 2. **Verify**: Confirm API signatures from the codebase. **Never invent an API.** If a Tool/Feature is missing, ask the user before proceeding. @@ -126,3 +127,70 @@ class SriovValidation(TestSuite): pass ``` + +--- + +## Step 9: Lint Gate (MANDATORY before declaring done) + +CI runs `black`, `isort`, `flake8`, `mypy`, `pylint`, `unittest discover` on every push/PR (see `.github/workflows/continuous-integration-workflow.yml`). **A test case is not "done" until the new file passes all of them locally.** + +### Rules the generated code must satisfy + +**Formatting (`black` + `isort`, configured in `pyproject.toml`)** +- Line length **≤ 88** chars (hard limit). +- Black `target-version = py38` style: double quotes, trailing commas in multi-line collections/calls, magic-trailing-comma respected. +- `isort`: `multi_line_output = 3`, `line_length = 88`, `force_grid_wrap = 0`, `include_trailing_comma = true`, `use_parentheses = true`. Group order: stdlib → third-party → first-party (`lisa`, `microsoft`). + +**flake8 (`select = B, BLK, C90, E, F, I, W, N`)** +- `BLK` — must pass `black --check` (otherwise BLK100 fires). +- `I` — must pass `isort --check`. +- `C90` — McCabe complexity **≤ 15**. Split deep nests / many branches into helper methods. +- `B` (bugbear) — no mutable default args, no `assert` for runtime checks, no `except:` bare clauses. +- `N` (pep8-naming) — `snake_case` functions/vars, `PascalCase` classes, `UPPER_CASE` module constants. *Ignored*: `N818` (exception names need not end in `Error`). +- *Ignored*: `E203, W503, E713, E231, E702` — don't waste time satisfying these. + +**mypy (`strict = true, ignore_missing_imports = true`)** +- **Every** function/method needs full type hints, including `-> None`. +- No implicit `Optional` — write `Optional[X]` or `X | None` explicitly when a default is `None`. +- No bare `Any` returns from public methods unless absolutely needed. +- `**kwargs: Any` is the standard signature for `before_case` / `after_case`. +- Imports must be resolvable in the editable `mslisa` install; new third-party imports require an entry in `pyproject.toml`. + +**pylint (`pylintrc`)** +- No unused imports / variables. No wildcard imports. +- No `print(...)` — use `log.info / log.debug`. +- Docstrings required on `TestSuite` class and on `before_case` / `after_case` overrides. +- Don't over-broadly catch `Exception` unless re-raising or wrapping in `LisaException`. + +**Test infrastructure** +- File must be `unittest`-discoverable (real LISA test, not script). Don't add `if __name__ == "__main__":`. +- All `time.sleep(...)` calls → `check_till_timeout()` / `retry_without_exceptions()` (CI doesn't ban it directly, but reviewers will). + +### Local commands the agent must run before reporting done + +From `lisa\` directory in the repo (`\lisa\`): + +```powershell +$f = "microsoft\testsuites\\.py" +..\.venv\Scripts\python.exe -m black --check $f +..\.venv\Scripts\python.exe -m isort --check $f +..\.venv\Scripts\python.exe -m flake8 $f +..\.venv\Scripts\python.exe -m mypy --ignore-missing-imports $f +..\.venv\Scripts\python.exe -m pylint $f +``` + +If any command fails: +1. **Auto-fix formatting** with `python -m black $f` and `python -m isort $f` (then re-run `--check`). +2. **Read the actual error**, fix the source, re-run that single command. +3. Do **not** suppress with `# noqa` / `# type: ignore` unless the rule is genuinely wrong for this code (rare). When you do, add a one-line comment explaining why. +4. Only declare the task complete when **all five commands exit 0**. + +### Quick self-check before submitting + +- [ ] Class inherits `TestSuite`; method has `@TestCaseMetadata` +- [ ] All params and return type are annotated +- [ ] No `time.sleep`, no bare `except:`, no `print()` +- [ ] Imports grouped & sorted (stdlib / third-party / `lisa` / `microsoft`) +- [ ] Line length ≤ 88 +- [ ] `black --check`, `isort --check`, `flake8`, `mypy`, `pylint` all pass on the new file +