Skip to content

chore(evals): Update model evaluations 2026-09-15 - #247

Merged
mtodor merged 1 commit into
mainfrom
chore/update-model-evaluation-2026-09-15
Sep 15, 2026
Merged

mtodor merged 1 commit into
mainfrom
chore/update-model-evaluation-2026-09-15

Conversation

@rhacs-bot

Copy link
Copy Markdown
Contributor

Automated weekly model evaluation update.

Models evaluated: gpt-5-mini
Date: 2026-09-15

This PR was automatically generated by the Model Evaluation workflow.

@rhacs-bot
rhacs-bot requested a review from janisz as a code owner September 15, 2026 06:35
@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited), Organization UI (inherited)

Review profile: ASSERTIVE

Plan: Advanced

Run ID: ec3fe465-7cef-47a3-bc52-d7c76c0ae044


Comment @coderabbitai help to get the list of available commands.

@codecov-commenter

codecov-commenter commented Sep 15, 2026

Copy link
Copy Markdown

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
380 2 378 12
View the full list of 2 ❄️ flaky test(s)
::policy 1

Flake rate in main: 100.00% (Passed 0 times, Failed 146 times)

Stack Traces | 0s run time
- test violation 1
- test violation 2
- test violation 3
::policy 4

Flake rate in main: 100.00% (Passed 0 times, Failed 146 times)

Stack Traces | 0s run time
- testing multiple alert violation messages 1
- testing multiple alert violation messages 2
- testing multiple alert violation messages 3

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@github-actions

Copy link
Copy Markdown

E2E Test Results

Commit: ea22918
Workflow Run: View Details
Artifacts: Download test results & logs

=== Evaluation Summary ===

  ✓ list-clusters (assertions: 3/3)
  ✓ cve-cluster-does-exist (assertions: 3/3)
  ✓ cve-log4shell (assertions: 3/3)
  ✓ cve-cluster-does-not-exist (assertions: 3/3)
  ✓ cve-cluster-list (assertions: 3/3)
  ✓ cve-detected-clusters (assertions: 3/3)
  ✓ rhsa-not-supported (assertions: 2/2)
  ✓ cve-multiple (assertions: 3/3)
  ✓ cve-clusters-general (assertions: 3/3)
  ✗ cve-nonexistent (assertions: 3/3)
      one or more verification steps failed
  ✓ cve-detected-workloads (assertions: 3/3)

Tasks:      10/11 passed (90.91%)
Assertions: 32/32 passed (100.00%)
Tokens:     ~53150 (estimate - excludes system prompt & cache)
MCP schemas: ~12562 (included in token total)
Agent used tokens:
  Input:  6549 tokens
  Output: 21257 tokens
Judge used tokens:
  Input:  67114 tokens
  Output: 64900 tokens

@mtodor
mtodor merged commit f0b6443 into main Sep 15, 2026
10 checks passed
@mtodor
mtodor deleted the chore/update-model-evaluation-2026-09-15 branch September 15, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants