[prompt-clustering] Copilot Agent Prompt Clustering Analysis — 2026-05-15

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-04-27 → 2026-05-15)
Total Tasks Analyzed: 998
Clusters Identified: 8
Overall Success Rate: 80.0% (798 merged / 196 closed / 4 open)

Eight thematic clusters emerge from NLP analysis of copilot agent task descriptions. The dominant cluster (~49% of tasks) covers general workflow code and bug fixes, while the smallest is a tight outlier of 20 WIP/placeholder lint-fix PRs with only a 30% merge rate. Cache-memory & experiment tasks have the highest success rate at 90.6%; AWF firewall/config work and WIP lint-fixes have the lowest.

Cluster Map

Daily Volume

Cluster Summary Table

Cluster	Theme	Tasks	Success	Avg Files	Avg +/−	Avg Commits	Top Terms
C5	General workflow code & bug fixes	492	82%	22.8	503/480	3.8	workflow, files, bug, job, path, run
C0	MCP servers / CLI / Playwright tooling	111	78%	33.5	366/204	3.6	mcp, cli, tool, mode, playwright, gateway
C2	Agent prompt / model optimization	90	87%	25.2	431/154	3.4	model, agent, daily, sub, token, workflow
C6	Shared workflow imports / consolidation	80	76%	40.2	730/320	4.4	shared, workflow, workflows, import, agentic, otlp
C4	Safe-output handlers (issue/discussion/PR)	76	79%	28.3	392/134	4.1	safe, safe output, output, safe outputs, outputs, issue
C3	AWF firewall / config / version bumps	65	68%	98.1	992/638	4.0	awf, firewall, version, config, awf config, default
C1	Cache memory & prompt experiments	64	91%	13.3	339/81	3.5	experiment, cache, variant, workflow, memory, prompt
C7	WIP / placeholder lint-fix PRs	20	30%	37.1	286/23	1.9	started description, thanks asking, asking work, work started, asking, date form

Complexity vs Success

Per-Cluster Breakdown (top 5 representative PRs each)

C5 — General workflow code & bug fixes

• Size: 492 tasks (49.3% of total)

• Outcome: 404 merged, 85 closed, 3 open → 82.1% merge rate

• Avg per PR: 22.8 files changed · +504/−480 lines · 3.8 commits · 1.5 reviews · 2.3 comments

• Top keywords: workflow, files, bug, job, path, run, failure, agent, engine, github

  • ⏳ #32298 — Propagate GH_AW_INFO_ENGINE_ID into setup steps so setup OTel spans emit `gh-aw.engine.i
  • ⏳ #32295 — Thread context.Context through all ResolveSHA call sites
  • ✅ #32294 — fix: use AWFProxyLogsDir constant in generateSquidLogsUploadStep
  • ⏳ #32280 — Pass OTEL_EXPORTER_OTLP_HEADERS env var to mcpg container; remove headers from gateway JSO
  • ✅ #32274 — ci: recompile workflows and fix compiler/test regressions

C0 — MCP servers / CLI / Playwright tooling

• Size: 111 tasks (11.1% of total)

• Outcome: 87 merged, 24 closed, 0 open → 78.4% merge rate

• Avg per PR: 33.5 files changed · +366/−204 lines · 3.6 commits · 1.4 reviews · 2.9 comments

• Top keywords: mcp, cli, tool, mode, playwright, gateway, tools, help, validation, mcp gateway

  • ✅ #32236 — docs: sync frontmatter-full.md and cli.md with schema/CLI surface changes from Add emoji frontmatter field #32200 and
  • ✅ #32185 — Make Sentry MCP queries explicit in smoke OTEL workflow
  • ✅ #32128 — Bump mcpg (DIFC proxy) to v0.3.8
  • ✅ #32011 — Normalize CLI help UX across mcp/logs/update and restore update flag parity
  • ✅ #31975 — Add --staged flag to compile for forced staged workflows

C2 — Agent prompt / model optimization

• Size: 90 tasks (9.0% of total)

• Outcome: 78 merged, 12 closed, 0 open → 86.7% merge rate

• Avg per PR: 25.2 files changed · +432/−154 lines · 3.4 commits · 1.6 reviews · 2.1 comments

• Top keywords: model, agent, daily, sub, token, workflow, run, inline, alias, inline sub

  • ✅ #32263 — Add daily linter-miner agentic workflow for auto-generating Go linters
  • ❌ #32240 — Add Codex fallback when token usage artifacts are missing
  • ✅ #32238 — Reduce token pressure in Daily Observability Report workflow
  • ✅ #32237 — Prefetch Copilot reflect data before agent startup in Daily Model Inventory Checker
  • ✅ #32102 — feat(architecture-guardian): offload violation classification to small inline sub-agent

C6 — Shared workflow imports / consolidation

• Size: 80 tasks (8.0% of total)

• Outcome: 61 merged, 19 closed, 0 open → 76.2% merge rate

• Avg per PR: 40.2 files changed · +731/−321 lines · 4.4 commits · 2.0 reviews · 4.5 comments

• Top keywords: shared, workflow, workflows, import, agentic, otlp, agentic workflow, frontmatter, agentic workflows, imports

  • ✅ #32318 — Extract Grafana MCP config into a shared component for Smoke OTEL Backends
  • ✅ #32269 — Deduplicate recursive runtime imports during workflow self-import expansion
  • ✅ #32248 — Add gh aw deploy to orchestrate remote workflow rollout via PR
  • ✅ #32247 — Use shared sentry mcp in smoke-otel-backends
  • ✅ #32233 — Add OpenTelemetry reference page and centralize observability docs

C4 — Safe-output handlers (issue/discussion/PR)

• Size: 76 tasks (7.6% of total)

• Outcome: 60 merged, 15 closed, 1 open → 78.9% merge rate

• Avg per PR: 28.3 files changed · +392/−134 lines · 4.1 commits · 1.4 reviews · 3.1 comments

• Top keywords: safe, safe output, output, safe outputs, outputs, issue, agent, workflow, handler, discussion

  • ✅ #32283 — fix(linter-miner): add go build/test to bash allowlist and harden error handling
  • ⏳ #32273 — Allow unsigned PR branch pushes when signed commits are disabled
  • ✅ #32249 — Add 4-hour chaos PR bundle fuzzer workflow for safe-output create-pull-request stress test
  • ✅ #32246 — Handle update_pull_request update-branch soft 422s as non-fatal in safe outputs
  • ❌ #32239 — Refactor safe-output configs to embed SafeOutputTargetConfig

C3 — AWF firewall / config / version bumps

• Size: 65 tasks (6.5% of total)

• Outcome: 44 merged, 21 closed, 0 open → 67.7% merge rate

• Avg per PR: 98.1 files changed · +992/−639 lines · 4.0 commits · 1.6 reviews · 5.8 comments

• Top keywords: awf, firewall, version, config, awf config, default, schema, bump, awf firewall, proxy

  • ❌ #32209 — Raise Daily Cache Strategy Analyzer ET budget above default 25M cap
  • ✅ #32161 — Remove firewall.effective-token-steering frontmatter key
  • ✅ #32126 — chore: bump AWF firewall to v0.25.46
  • ✅ #31986 — Increase default max-runs from 100 to 500 across compiler, schema, and docs
  • ❌ #31910 — Fix Pi AWF internal proxy routing so /reflect no longer fails during smoke runs

C1 — Cache memory & prompt experiments

• Size: 64 tasks (6.4% of total)

• Outcome: 58 merged, 6 closed, 0 open → 90.6% merge rate

• Avg per PR: 13.3 files changed · +340/−82 lines · 3.5 commits · 1.7 reviews · 1.8 comments

• Top keywords: experiment, cache, variant, workflow, memory, prompt, state, cache memory, run, experiments

  • ✅ #32255 — Enable experiment-driven model selection in smoke-copilot workflow
  • ✅ #32253 — Expand audit-workflows repo memory taxonomy
  • ✅ #32193 — bump mcpg image to v0.3.9 (fixes wazero cache EACCES on artifact upload)
  • ✅ #31992 — daily-experiment-report: install numpy/matplotlib when running on PyPy
  • ✅ #31927 — Add detail_level A/B experiment to daily architecture diagram workflow output

C7 — WIP / placeholder lint-fix PRs

• Size: 20 tasks (2.0% of total)

• Outcome: 6 merged, 14 closed, 0 open → 30.0% merge rate

• Avg per PR: 37.1 files changed · +286/−24 lines · 1.9 commits · 0.6 reviews · 0.3 comments

• Top keywords: started description, thanks asking, asking work, work started, asking, date form, form plan, plan progress, thanks, description date

  • ❌ #32190 — [WIP] Fix failing GitHub Actions job 'test'
  • ❌ #32189 — [WIP] Fix failing GitHub Actions job 'build-wasm'
  • ❌ #32188 — [WIP] Fix failing GitHub Actions job test
  • ❌ #32122 — [WIP] Refactor to create shared agentic workflows tool for analysis
  • ❌ #32121 — [WIP] Create shared/github-proxy-default.md for toolset bundling

Sampled Data Table — 10 most recent PRs per cluster (80 total)

PR #	Cluster	State	Title	Files	+/−
#32298	C5	⏳	Propagate GH_AW_INFO_ENGINE_ID into setup steps so setup OTel spans	7	+63/−0
#32295	C5	⏳	Thread context.Context through all ResolveSHA call sites	46	+472/−339
#32294	C5	✅	fix: use AWFProxyLogsDir constant in generateSquidLogsUploadStep	2	+158/−84
#32280	C5	⏳	Pass OTEL_EXPORTER_OTLP_HEADERS env var to mcpg container; remove head	245	+528/−1112
#32274	C5	✅	ci: recompile workflows and fix compiler/test regressions	9	+179/−105
#32268	C5	✅	Fix: gh aw status and gh aw logs bypass local filesystem when `--r	5	+396/−114
#32256	C5	✅	Fix chaos PR bundle fuzzer activation failure	1	+153/−81
#32254	C5	❌	Recompile workflows: remove deprecated properties and regenerate lock	457	+21746/−40152
#32235	C5	✅	Fix go fmt failure in CreatePullRequestsConfig	1	+1/−1
#32234	C5	✅	Add dedicated aw.yml package manifest reference page	3	+66/−2
#32236	C0	✅	docs: sync frontmatter-full.md and cli.md with schema/CLI surface chan	2	+255/−40
#32185	C0	✅	Make Sentry MCP queries explicit in smoke OTEL workflow	2	+17/−21
#32128	C0	✅	Bump mcpg (DIFC proxy) to v0.3.8	232	+1135/−1115
#32011	C0	✅	Normalize CLI help UX across mcp/logs/update and restore update flag p	16	+118/−42
#31975	C0	✅	Add --staged flag to compile for forced staged workflows	10	+121/−9
#31914	C0	✅	Align strict-mode compatibility: allow Playwright MCP warnings and hon	4	+45/−19
#31766	C0	✅	Fix 5 CLI consistency issues from automated inspection	6	+10/−5
#31706	C0	❌	restrict MCP_GATEWAY_PAYLOAD_DIR to 0700 to prevent bearer token enume	223	+348/−60
#31614	C0	✅	Auto-detect ARC/DinD and emit AWF --docker-host-path-prefix in gener	224	+2430/−500
#31605	C0	✅	Add centralized slash-command trigger strategy with compiler-generated	60	+1938/−516
#32263	C2	✅	Add daily linter-miner agentic workflow for auto-generating Go linters	2	+1914/−0
#32240	C2	❌	Add Codex fallback when token usage artifacts are missing	2	+169/−9
#32238	C2	✅	Reduce token pressure in Daily Observability Report workflow	2	+67/−44
#32237	C2	✅	Prefetch Copilot reflect data before agent startup in Daily Model Inve	3	+65/−19
#32102	C2	✅	feat(architecture-guardian): offload violation classification to small	63	+200/−232
#31911	C2	✅	Add scheduled pr-sous-chef workflow to advance open PRs toward maintai	2	+1798/−0
#31885	C2	❌	Move Daily Issues/News/Fact workflows off aw-gpu-runner-T4 to stop c	6	+55/−55
#31834	C2	✅	Use AWF /reflect for Copilot model discovery in daily-model-inventor	1	+29/−73
#31803	C2	✅	[WIP] Optimize daily-agentrx-trace-optimizer to use agent session data	2	+28/−32
#31785	C2	✅	Add daily AgentRx trace optimizer workflow	2	+1756/−0
#32318	C6	✅	Extract Grafana MCP config into a shared component for Smoke OTEL Back	3	+68/−24
#32269	C6	✅	Deduplicate recursive runtime imports during workflow self-import expa	3	+274/−96
#32248	C6	✅	Add gh aw deploy to orchestrate remote workflow rollout via PR	7	+558/−1
#32247	C6	✅	Use shared sentry mcp in smoke-otel-backends	2	+38/−30
#32233	C6	✅	Add OpenTelemetry reference page and centralize observability docs	6	+218/−130
#32200	C6	✅	Add emoji frontmatter field	463	+3763/−3273
#32173	C6	✅	Add observability.otlp.ignore-if-missing to downgrade missing OTLP c	8	+395/−9
#32158	C6	❌	Enable shared OTLP reuse across all agentic workflows	2	+15/−14
#32104	C6	❌	docs: add audience-appropriate output channel heuristic to create-agen	62	+84/−61
#32027	C6	✅	Support glob patterns in allowed label filters for safe-outputs	5	+35/−13
#32283	C4	✅	fix(linter-miner): add go build/test to bash allowlist and harden erro	3	+115/−177
#32273	C4	⏳	Allow unsigned PR branch pushes when signed commits are disabled	18	+268/−26
#32249	C4	✅	Add 4-hour chaos PR bundle fuzzer workflow for safe-output create-pull	2	+1598/−0
#32246	C4	✅	Handle update_pull_request update-branch soft 422s as non-fatal in s	3	+65/−2
#32239	C4	❌	Refactor safe-output configs to embed SafeOutputTargetConfig	21	+260/−124
#32224	C4	✅	feat(jsweep): add branch-prefix to create-pull-request safe-outputs, d	10	+175/−25
#32219	C4	✅	Allow @copilot mentions in PR Sous Chef safe outputs	228	+1890/−1587
#32065	C4	✅	Fix Claude MCP config path expansion for Safe Output Health Monitor	3	+9/−8
#32064	C4	✅	[aw] Enforce mandatory safe-output noop in PR Sous Chef runs	1	+2/−1
#32028	C4	✅	feat(create-agent-session): migrate from gh CLI to REST API for task c	3	+183/−188
#32209	C3	❌	Raise Daily Cache Strategy Analyzer ET budget above default 25M cap	2	+27/−26
#32161	C3	✅	Remove firewall.effective-token-steering frontmatter key	270	+1021/−1106
#32126	C3	✅	chore: bump AWF firewall to v0.25.46	233	+2660/−2655
#31986	C3	✅	Increase default max-runs from 100 to 500 across compiler, schema, a	234	+507/−507
#31910	C3	❌	Fix Pi AWF internal proxy routing so /reflect no longer fails during	1	+146/−84
#31833	C3	✅	Fix AWF timestamp command escaping that broke test workflow golden o	224	+494/−494
#31823	C3	✅	Render AWF token steering in agent and gateway JS parsers	4	+373/−4
#31808	C3	✅	feat(otel): capture true Execute-Agent-CLI start time so gh-aw.agent s	239	+710/−69
#31796	C3	✅	Bump AWF to v0.25.44 and add firewall.effective-token-steering compi	274	+3385/−3069
#31789	C3	❌	Bump AWF to v0.25.44 and expose token steering/model multipliers in co	237	+2921/−2595
#32255	C1	✅	Enable experiment-driven model selection in smoke-copilot workflow	2	+32/−27
#32253	C1	✅	Expand audit-workflows repo memory taxonomy	3	+207/−86
#32193	C1	✅	bump mcpg image to v0.3.9 (fixes wazero cache EACCES on artifact uploa	236	+1146/−1389
#31992	C1	✅	daily-experiment-report: install numpy/matplotlib when running on PyPy	2	+25/−13
#31927	C1	✅	Add detail_level A/B experiment to daily architecture diagram workfl	2	+227/−174
#31813	C1	✅	spec-extractor: switch round-robin to git-based README version stamps	2	+2/−4
#31807	C1	✅	Add reasoning_depth A/B experiment to daily-security-red-team workfl	21	+426/−271
#31602	C1	❌	Add CodeGraph shared MCP workflow for code search	7	+851/−7
#31594	C1	✅	docs: extend memory.md with built-in GitHub graph and git history stra	1	+23/−0
#31492	C1	❌	daily-doc-healer: add prompt_style A/B experiment and variant-gated	2	+208/−170
#32190	C7	❌	[WIP] Fix failing GitHub Actions job 'test'	232	+1703/−73
#32189	C7	❌	[WIP] Fix failing GitHub Actions job 'build-wasm'	230	+1686/−72
#32188	C7	❌	[WIP] Fix failing GitHub Actions job test	231	+1689/−73
#32122	C7	❌	[WIP] Refactor to create shared agentic workflows tool for analysis	0	+0/−0
#32121	C7	❌	[WIP] Create shared/github-proxy-default.md for toolset bundling	0	+0/−0
#32120	C7	❌	[WIP] Create shared/strict-copilot.md for strict and copilot-requests	0	+0/−0
#32096	C7	✅	fix: mask otlp headers	2	+101/−2
#32042	C7	✅	[WIP] Fix failing GitHub Actions job lint-js	2	+2/−6
#32041	C7	✅	[WIP] Fix failing GitHub Actions job for CLI completion	1	+0/−1
#32036	C7	✅	[WIP] Fix failing GitHub Actions job lint-js	1	+1/−1

Key Findings

1. Cache-memory & prompt-experiment work is the safest bet (90.6% merge rate) — these are small, scoped changes (avg 13 files, +340/−82 lines) where the agent is wiring up cache I/O, adding trending history, or tweaking experiment prompts. Low review burden, fast turnaround.
2. AWF firewall/config bumps are the riskiest non-WIP cluster (67.7% merge rate) — these PRs are also the largest by far (avg 98 files, +992/−639). Many touch wasm goldens, config schemas, version bumps across multiple workflows; large surface area inflates the close rate.
3. The general-workflow cluster (C5) absorbs ~49% of all tasks — a heterogeneous bucket of bug fixes, refactors, and small workflow code edits. Its 82% merge rate roughly matches the overall average; this is the agent's bread-and-butter category.
4. WIP placeholder PRs (C7) merge at only 30% — top terms (thanks asking, work started, description date, form plan) reveal these are auto-generated lint-fix attempts that stall. Worth investigating why these specific recoveries fail so often.
5. Shared-workflow consolidation (C6) has the highest file churn per task (avg 40 files) but a healthy 76% merge rate — sweeping migrations succeed when the pattern is well-defined.

Recommendations

1. Investigate the WIP lint-fix path (C7). Of 20 such PRs, 14 closed without merging. The agent appears to open WIP PRs with placeholder bodies but cannot complete them — examine #32003, #32004, #32036 for the failure mode.
2. Pre-scope AWF/firewall work (C3) into smaller PRs. Average 98 changed files per PR is the largest of any cluster and correlates with the lowest non-WIP success rate. Splitting wasm-golden regenerations, config schema changes, and version bumps would likely raise merge rate.
3. Borrow patterns from the cache/experiment cluster (C1) for other agent tooling. The combination of small, well-scoped, cache-aware prompts produces a 90% merge rate — useful as a template for new daily/trending agents.
4. Consider sub-clustering C5. It's so large (492 PRs) that its top terms (workflow, files, bug, job, path) are essentially the union of everything that didn't fit elsewhere. A second pass focusing just on C5 would surface finer structure.

Methodology

• Input: PR title + first paragraph (before ## Changes) for each of 1000 copilot PRs created 2026-04-27 → 2026-05-15. After filtering prompts <30 chars, 998 PRs remained.
• Vectorization: TF-IDF over unigrams + bigrams, min_df=3, max_df=0.6, domain stopwords removed (gh, aw, pkg, pr, etc).
• Clustering: KMeans, k chosen by silhouette score over k∈[4,8]; k=8 selected (silhouette 0.032).
• Caveat: Silhouette scores are low (typical of high-dimensional sparse TF-IDF on short texts) — clusters are interpretable but not perfectly separated. Cluster 5 in particular is a catch-all.

References: §25913157542

Generated by 📊 Copilot Agent Prompt Clustering Analysis · ● 14.5M · ◷

• expires on May 16, 2026, 11:07 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #32602.