Fix azqr tool selection for compliance prompts (#2819) - #3134
Open
KarishmaGhiya wants to merge 4 commits into
Open
Fix azqr tool selection for compliance prompts (#2819)#3134KarishmaGhiya wants to merge 4 commits into
KarishmaGhiya wants to merge 4 commits into
Conversation
Improve the Azure Quick Review (azqr) tool, extension namespace, and consolidated tool descriptions so prompts asking to scan a subscription for compliance issues or compliance recommendations reliably select the extension_azqr tool in All and Namespace modes. The descriptions now mirror user phrasing and explicitly disambiguate azqr from Azure Policy and Azure Advisor, which previously out-ranked it for these prompts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ee38628b-51ed-4861-91e1-7a07f3b5360b
KarishmaGhiya
requested review from
JasonYeMSFT,
RickWinter,
anannya03,
g2vinay,
hallipr and
xirzec
July 22, 2026 00:44
|
Azure Pipelines: Successfully started running 1 pipeline(s). There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves tool-selection reliability for Azure Quick Review CLI (azqr) compliance-focused prompts by rewording azqr-related tool/namespace descriptions across the sources used in All/Namespace modes and by the ToolDescriptionEvaluator, and adds a changelog entry for the fix.
Changes:
- Updated azqr command/namespace/consolidated tool descriptions to better match user prompt vocabulary (“scan … for compliance issues/recommendations”) and disambiguate from overlapping tools.
- Synced the ToolDescriptionEvaluator namespace fixture with the updated azqr description.
- Added an Azure MCP Server changelog entry describing the tool-selection fix.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/Azure.Mcp.Tools.Extension/src/ExtensionSetup.cs | Updates extension command-group description to better surface azqr compliance-scan intent. |
| tools/Azure.Mcp.Tools.Extension/src/Commands/AzqrCommand.cs | Rewords azqr tool metadata description used in All mode for better prompt matching/disambiguation. |
| servers/Azure.Mcp.Server/src/Resources/consolidated-tools.json | Updates consolidated/namespace-mode mapped tool description for azqr compliance scanning. |
| eng/tools/ToolDescriptionEvaluator/prompts/namespace-tools.json | Keeps evaluator fixture description in sync with updated azqr tool description. |
| servers/Azure.Mcp.Server/changelog-entries/karishmaghiya-fix-azqr-tool-selection.yaml | Adds changelog entry for the azqr tool-selection behavior fix. |
| Name = "azqr", | ||
| Title = "Azure Quick Review CLI Command", | ||
| Description = "Runs Azure Quick Review CLI (azqr) commands to generate compliance and security reports for Azure resources, identifying non-compliant configurations or areas for improvement. Requires a subscription id and optionally a resource group name. Returns the generated report file path. Note: azqr is different from Azure CLI (az).", | ||
| Description = "Runs Azure Quick Review CLI (azqr) to scan an Azure subscription (or resource group) for compliance issues and provide compliance recommendations. Generates a compliance and security assessment report that identifies non-compliant configurations and recommends improvements for your Azure resources. Use this whenever a user wants to scan, check, review, or assess a subscription for compliance issues or compliance recommendations, or wants recommendations to fix compliance and security problems. Requires a subscription id and optionally a resource group name. Returns the generated report file path. Note: azqr performs compliance and security scans and is different from Azure CLI (az), from Azure Policy assignments, and from Azure Advisor cost, performance, and reliability recommendations.", |
Contributor
There was a problem hiding this comment.
You can also just use the words "file paths" without specifying the file format.
| "id": "", | ||
| "name": "extension_azqr", | ||
| "description": "Runs Azure Quick Review CLI (azqr) commands to generate compliance/security reports for Azure resources.\r\nThis tool should be used when the user wants to identify any non-compliant configurations or areas for improvement in their Azure resources.\r\nRequires a subscription id and optionally a resource group name. Returns the generated report file's path.\r\nNote that Azure Quick Review CLI (azqr) is different from Azure CLI (az).", | ||
| "description": "Runs Azure Quick Review CLI (azqr) to scan an Azure subscription (or resource group) for compliance issues and provide compliance recommendations. Generates a compliance and security assessment report that identifies non-compliant configurations and recommends improvements for your Azure resources. Use this whenever a user wants to scan, check, review, or assess a subscription for compliance issues or compliance recommendations, or wants recommendations to fix compliance and security problems. Requires a subscription id and optionally a resource group name. Returns the generated report file path. Note: azqr performs compliance and security scans and is different from Azure CLI (az), from Azure Policy assignments, and from Azure Advisor cost, performance, and reliability recommendations.", |
Contributor
There was a problem hiding this comment.
You can also just use the words "file paths" without specifying the file format.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
alzimmermsft
approved these changes
Jul 22, 2026
g2vinay
reviewed
Jul 23, 2026
g2vinay
left a comment
g2vinay
left a comment
Contributor
There was a problem hiding this comment.
Run Tool Description Evaluator to ensure, things are working well.
…github.com/KarishmaGhiya/mcp into karishmaghiya/fix-azqr-tool-selection-2819
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #2819.
In All and Namespace modes, the E2E prompts for the Azure Quick Review CLI were not triggering the expected
extension_azqrtool. The prompts were instead being routed to other tools because the azqr descriptions did not surface the vocabulary users actually use, and did not disambiguate azqr from tools with overlapping "compliance"/"recommendations" language.Root cause
The three failing prompts all contain the words "compliance" and "recommendations":
Check my Azure subscription for any compliance issues or recommendationsProvide compliance recommendations for my current Azure subscriptionScan my Azure subscription for compliance recommendationsThe original
extension_azqrdescription ranked #3 for all three, losing to Azure Policy (whose description contains "governance and compliance management") and Azure Advisor (which owns generic "recommendations").Fix
Reworded the azqr tool description across all description sources so it:
Files changed
tools/Azure.Mcp.Tools.Extension/src/Commands/AzqrCommand.cs—[CommandMetadata]description (All mode tool description).servers/Azure.Mcp.Server/src/Resources/consolidated-tools.json—audit_azure_resources_compliancemapped tool description (Namespace/consolidated mode).tools/Azure.Mcp.Tools.Extension/src/ExtensionSetup.cs— extension command-group description.eng/tools/ToolDescriptionEvaluator/prompts/namespace-tools.json— evaluator fixture kept in sync.servers/Azure.Mcp.Server/changelog-entries/karishmaghiya-fix-azqr-tool-selection.yaml— changelog entry.Validation
A local embedding proxy (MiniLM cosine similarity against the real competitor descriptions) was used to confirm ranking direction:
Invoking Livetests
Copilot submitted PRs are not trustworthy by default. Users with
writeaccess to the repo need to validate the contents of this PR before leaving a comment with the text/azp run mcp - pullrequest - live. This will trigger the necessary livetest workflows to complete required validation.