Skip to content

Reduce tool-denial failures in Daily Compiler Threat Spec Optimizer#40743

Merged
pelikhan merged 3 commits into
mainfrom
copilot/aw-resolve-tool-denial-issue
Jun 22, 2026
Merged

Reduce tool-denial failures in Daily Compiler Threat Spec Optimizer#40743
pelikhan merged 3 commits into
mainfrom
copilot/aw-resolve-tool-denial-issue

Conversation

Copilot AI commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

The Daily Compiler Threat Spec Optimizer was exhausting the Copilot SDK tool-denial guardrail and terminating early. The failure came from a brittle workflow tool policy: the agent requested normal repository inspection operations that did not match the workflow’s exact shell allowlist.

  • Workflow tool policy

    • Replaced a narrow list of exact shell commands with the read-only command stems the workflow actually needs (git, go, find, ls, cat, sed, awk, grep, head, pwd).
    • This makes Copilot SDK permission matching tolerant of routine command variations like different git log, git diff, and go test invocations.
  • Prompt guidance

    • Added an explicit workflow constraint telling the agent to use the approved shell commands for repository inspection instead of built-in file read/view tools.
    • This aligns the prompt with the workflow’s restricted Copilot SDK permission model.
  • Compiled workflow

    • Recompiled the generated .lock.yml so the updated allowlist is reflected in the runtime --allow-tool arguments.

Example of the effective change:

tools:
  bash:
    - git
    - cat
    - find
    - ls
    - sed
    - awk
    - grep
    - head
    - pwd
    - go

@github-actions

This comment has been minimized.

Copilot AI and others added 2 commits June 22, 2026 06:51
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix compiler threat spec optimizer tool denial issue Reduce tool-denial failures in Daily Compiler Threat Spec Optimizer Jun 22, 2026
Copilot AI requested a review from pelikhan June 22, 2026 07:02
@github-actions github-actions Bot mentioned this pull request Jun 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hey @Copilot 👋 — nice fix for the tool-denial guardrail exhaustion in the Daily Compiler Threat Spec Optimizer! Broadening the allowlist from exact shell invocations to command stems (git, go, grep, etc.) is a clean and principled approach that will tolerate the natural variation in how those commands get called.

One item to follow up on:

  • No test coverage for the updated tool policy — the diff only touches the .md spec and the compiled .lock.yml. If the project has a workflow compilation or validation script (e.g. something that verifies the .lock.yml reflects the spec, or a smoke-test that runs the workflow in a dry-run mode), it would be good to confirm those pass. If no such harness exists today, this PR is a good prompt to add one.

If you would like a hand, here is a prompt you can assign to your coding agent:

Check whether the gh-aw repository has any validation or compilation tests for .lock.yml workflow files.
If a validation script exists, run it against .github/workflows/daily-compiler-threat-spec-optimizer.lock.yml to confirm the updated bash tool allowlist is correctly reflected.
If no harness exists, propose a lightweight test (e.g. a Makefile target or CI step) that verifies the compiled .lock.yml matches its source .md spec for the daily-compiler-threat-spec-optimizer workflow.

Generated by ✅ Contribution Check · 153 AIC · ⌖ 15.6 AIC · ⊞ 5.9K ·

@pelikhan pelikhan marked this pull request as ready for review June 22, 2026 14:14
Copilot AI review requested due to automatic review settings June 22, 2026 14:14
@pelikhan pelikhan merged commit 7f30df9 into main Jun 22, 2026
@pelikhan pelikhan deleted the copilot/aw-resolve-tool-denial-issue branch June 22, 2026 14:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot couldn't run its full agentic review because no GitHub Actions runner was available. Make sure your repository has a runner available to run Copilot's review, or add a copilot-setup-steps.yml file specifying one with the runs-on attribute. See the docs for more details.

Updates the Daily Compiler Threat Spec Optimizer workflow to reduce Copilot SDK tool-denial failures by broadening the bash allowlist and adding prompt guidance aligned with the restricted tool model.

Changes:

  • Replaced exact-match bash command allowlist with command “stems” (e.g., git, go, cat, etc.) to tolerate argument variations.
  • Added a “Tooling Constraint” section to the workflow prompt to guide the agent to use allowed shell commands for inspection.
  • Recompiled the generated .lock.yml so runtime --allow-tool arguments reflect the updated policy.
Show a summary per file
File Description
.github/workflows/daily-compiler-threat-spec-optimizer.md Broadens the tool allowlist and adds prompt guidance about tool constraints.
.github/workflows/daily-compiler-threat-spec-optimizer.lock.yml Regenerates compiled workflow runtime args to match the updated allowlist.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 2/2 changed files
  • Comments generated: 3

Comment on lines 37 to +47
bash:
- git ls-files pkg/workflow/*.go
- git ls-files pkg/parser/*.go
- cat specs/compiler-threat-detection-spec.md
- "git log --since=\"2 days ago\" --oneline -- pkg/workflow pkg/parser actions/setup/js"
- "git diff -- pkg/workflow pkg/parser actions/setup/js"
- go test -v ./pkg/workflow/...
- git
- cat
- find
- ls
- sed
- awk
- grep
- head
- pwd
- go
Comment on lines +835 to +836
# --allow-tool shell(git:*)
# --allow-tool shell(go:*)
Comment on lines +71 to +73
## Tooling Constraint

This workflow uses a restricted Copilot SDK shell allowlist. For repository inspection, use the approved shell commands above (`git`, `cat`, `find`, `ls`, `sed`, `awk`, `grep`, `head`, `pwd`, `go`) instead of built-in file read/view tools, and avoid requesting commands outside that set.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants