chore: trigger NVSkills CI for cuopt-install - #1672
Open
ramakrishnap-nv wants to merge 8 commits into
Open
Conversation
Collaborator
Author
|
/nvskills-ci |
📝 WalkthroughWalkthroughThe cuOpt install skill adds six installation evaluation cases and refreshes its benchmark report, skill-card metadata, documentation spacing, and Sigstore signature bundle. ChangescuOpt install skill refresh
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
ramakrishnap-nv
added
improvement
CI Test Summary⏭️ All 5 test job(s) skipped. |
Collaborator
Author
|
/nvskills-ci |
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skills/cuopt-install/BENCHMARK.md`:
- Around line 74-87: Update the benchmark scoring documentation near the
dimension bands and overall verdict to define Overall as the arithmetic mean of
the five skill-assisted dimension scores, including the rounding rule and score
basis. State that the verdict gate uses these skill-assisted scores rather than
baseline values, while preserving the existing threshold and verdict rules.
- Around line 15-18: Update the benchmark report metadata in BENCHMARK.md to
identify the exact evaluated input file and task ID, specifically distinguishing
the Tier 3 dataset from the inst-eval-001-docker-server task. Reconcile the
Dataset digest and task count with the actual inputs evaluated, recording both
dataset details if both sources were used.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 71a63a7f-8e5b-40c8-83f9-3f5aaab8ee84
📒 Files selected for processing (3)
skills/cuopt-install/BENCHMARK.mdskills/cuopt-install/skill-card.mdskills/cuopt-install/skill.oms.sig
Comment on lines
+74
to
+87
| | Dimension | Question | Scored signals | | ||
| |---|---|---| | ||
| | Security | Is it safe to use? | `security` (100%) | | ||
| | Correctness | Is the answer correct? | `accuracy` (100%) | | ||
| | Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) | | ||
| | Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) | | ||
| | Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) | | ||
|
|
||
| - MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/cuopt-install/SKILL.md`) | ||
| - MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/cuopt-install/SKILL.md`) | ||
| - LOW SCHEMA/unexpected_file: Unexpected 'benchmark' in skill root (`skills/cuopt-install/benchmark`) | ||
| - LOW SCHEMA/author_format: Author must be of the form 'Name <email@host>' (`skills/cuopt-install/SKILL.md`) | ||
| - Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%. | ||
| - Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL. | ||
| - Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate. | ||
| - The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold. | ||
| - Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`). | ||
| - Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Document the overall-score formula and gate input.
The report defines dimension thresholds and the verdict gate, but it does not define how Overall is derived from the five dimensions or whether the thresholds use baseline or skill-assisted scores. The displayed values imply an arithmetic mean, and both baseline efficiency values are 0%; without this rule, readers cannot reproduce the Overall values or the PASS decision. Add the aggregation formula, rounding rule, and explicit skill-assisted-score basis.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skills/cuopt-install/BENCHMARK.md` around lines 74 - 87, Update the benchmark
scoring documentation near the dimension bands and overall verdict to define
Overall as the arithmetic mean of the five skill-assisted dimension scores,
including the rounding rule and score basis. State that the verdict gate uses
these skill-assisted scores rather than baseline values, while preserving the
existing threshold and verdict rules.
Collaborator
Author
|
/ok to test 751a554 |
Collaborator
Author
|
/ok to test 1495865 |
Collaborator
Author
|
/ok to test d22f2fd |
A single eval gives the NVSkills Tier 3 live-agent evaluation almost no signal about the skill. Add six cases covering the areas the skill actually documents: CUDA suffix matching and the NVIDIA package index, the one-way cuopt-cuXX -> libcuopt-cuXX dependency, the Compute Capability 7.0 hardware floor, asking the required questions on an underspecified install request, the pip-or-conda (not both) rule, and ModuleNotFoundError triage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skills/cuopt-install/evals/evals.json`:
- Around line 50-60: Update the expected_behavior array for
inst-eval-005-underspecified-install-request to explicitly require asking about
the target environment, covering local GPU, cloud instance,
container/Kubernetes, or remote host without a local GPU, alongside the existing
interface, CUDA version, and package-manager requirements.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 30d036f5-46bd-4605-b6b3-96f986a03354
📒 Files selected for processing (1)
skills/cuopt-install/evals/evals.json
Comment on lines
+50
to
+60
| "id": "inst-eval-005-underspecified-install-request", | ||
| "question": "Install cuOpt for me.", | ||
| "expected_skill": "cuopt-install", | ||
| "expected_script": null, | ||
| "ground_truth": "The request is underspecified, so the agent asks the required questions before giving a command: which interface (Python, C, or REST server), which CUDA version is installed (checked with nvcc --version or nvidia-smi), which package manager is preferred (pip, conda, or Docker), and what the target environment is (local GPU machine, cloud instance, container/Kubernetes, or a remote host without a local GPU). It does not guess a default and emit an install command, and it does not run the install itself.", | ||
| "expected_behavior": [ | ||
| "Asks which interface is wanted — Python, C, or REST server", | ||
| "Asks which CUDA version is installed, referencing nvcc --version or nvidia-smi", | ||
| "Asks whether pip, conda, or Docker is preferred", | ||
| "Does not silently assume defaults and emit a single install command" | ||
| ] |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Require the environment clarification in expected_behavior.
Line 54 requires the agent to ask about the target environment. Lines 55-59 do not assert that requirement. The evaluation can pass a response that omits a required question.
Proposed fix
"expected_behavior": [
"Asks which interface is wanted — Python, C, or REST server",
"Asks which CUDA version is installed, referencing nvcc --version or nvidia-smi",
"Asks whether pip, conda, or Docker is preferred",
+ "Asks about the target environment, such as a local GPU machine, cloud instance, container, or remote host",
"Does not silently assume defaults and emit a single install command"
]📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| "id": "inst-eval-005-underspecified-install-request", | |
| "question": "Install cuOpt for me.", | |
| "expected_skill": "cuopt-install", | |
| "expected_script": null, | |
| "ground_truth": "The request is underspecified, so the agent asks the required questions before giving a command: which interface (Python, C, or REST server), which CUDA version is installed (checked with nvcc --version or nvidia-smi), which package manager is preferred (pip, conda, or Docker), and what the target environment is (local GPU machine, cloud instance, container/Kubernetes, or a remote host without a local GPU). It does not guess a default and emit an install command, and it does not run the install itself.", | |
| "expected_behavior": [ | |
| "Asks which interface is wanted — Python, C, or REST server", | |
| "Asks which CUDA version is installed, referencing nvcc --version or nvidia-smi", | |
| "Asks whether pip, conda, or Docker is preferred", | |
| "Does not silently assume defaults and emit a single install command" | |
| ] | |
| "id": "inst-eval-005-underspecified-install-request", | |
| "question": "Install cuOpt for me.", | |
| "expected_skill": "cuopt-install", | |
| "expected_script": null, | |
| "ground_truth": "The request is underspecified, so the agent asks the required questions before giving a command: which interface (Python, C, or REST server), which CUDA version is installed (checked with nvcc --version or nvidia-smi), which package manager is preferred (pip, conda, or Docker), and what the target environment is (local GPU machine, cloud instance, container/Kubernetes, or a remote host without a local GPU). It does not guess a default and emit an install command, and it does not run the install itself.", | |
| "expected_behavior": [ | |
| "Asks which interface is wanted — Python, C, or REST server", | |
| "Asks which CUDA version is installed, referencing nvcc --version or nvidia-smi", | |
| "Asks whether pip, conda, or Docker is preferred", | |
| "Asks about the target environment, such as a local GPU machine, cloud instance, container, or remote host", | |
| "Does not silently assume defaults and emit a single install command" | |
| ] |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skills/cuopt-install/evals/evals.json` around lines 50 - 60, Update the
expected_behavior array for inst-eval-005-underspecified-install-request to
explicitly require asking about the target environment, covering local GPU,
cloud instance, container/Kubernetes, or remote host without a local GPU,
alongside the existing interface, CUDA version, and package-manager
requirements.
Collaborator
Author
|
/nvskills-ci |
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skills/cuopt-install/BENCHMARK.md`:
- Around line 37-42: Update the uplift values in both benchmark tables in
BENCHMARK.md to match the defined skill-score-minus-baseline calculation: Claude
Code correctness +8, Claude Code effectiveness +17, Codex discoverability +47,
and Codex efficiency +64. If these values derive from unrounded scores, document
the calculation and rounding method alongside the tables.
In `@skills/cuopt-install/skill-card.md`:
- Around line 69-74: Recompute the uplift deltas in the comparison table,
especially Claude Code Correctness and Effectiveness plus Codex Discoverability
and Efficiency, so each displayed integer matches the shown endpoints. If the
calculations use unrounded source values, explicitly state that in the table;
otherwise update the displayed deltas to the correct rounded values.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: ce2ea433-b196-4bf6-8405-c9faa072ef01
📒 Files selected for processing (3)
skills/cuopt-install/BENCHMARK.mdskills/cuopt-install/skill-card.mdskills/cuopt-install/skill.oms.sig
🚧 Files skipped from review as they are similar to previous changes (1)
- skills/cuopt-install/skill.oms.sig
Comment on lines
+37
to
+42
| | Overall | 56% → 90% (+34 points) | 63% → 91% (+28 points) | | ||
| | Security | 93% → 86% (-7 points) | 100% → 100% (±0 points) | | ||
| | Correctness | 86% → 94% (+9 points) | 89% → 100% (+11 points) | | ||
| | Discoverability | 27% → 96% (+69 points) | 47% → 94% (+46 points) | | ||
| | Effectiveness | 64% → 81% (+18 points) | 76% → 93% (+17 points) | | ||
| | Efficiency | 12% → 93% (+81 points) | 3% → 67% (+63 points) | |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- relevant BENCHMARK.md lines ---'
cat -n skills/cuopt-install/BENCHMARK.md | sed -n '1,100p'
printf '%s\n' '--- related benchmark files and references ---'
git ls-files | rg '(^|/)(BENCHMARK|benchmark|skill-card|.*cuopt-install.*)' || true
rg -n -i 'uplift|Overall|Correctness|Discoverability|Effectiveness|Efficiency|baseline|Claude Code|Codex' skills/cuopt-install . --glob '!skills/cuopt-install/BENCHMARK.md' --glob '!*.lock' | head -300Repository: NVIDIA/cuopt
Length of output: 50368
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- skill-card benchmark and methodology ---'
cat -n skills/cuopt-install/skill-card.md | sed -n '30,90p'
printf '%s\n' '--- JSON top-level structures ---'
python3 - <<'PY'
import json
from pathlib import Path
for name in ("skills/cuopt-install/benchmark/evals.json",
"skills/cuopt-install/evals/evals.json"):
path = Path(name)
print(f"\n{name}")
data = json.loads(path.read_text())
print("type:", type(data).__name__)
if isinstance(data, dict):
print("keys:", list(data)[:50])
for key, value in data.items():
if isinstance(value, (list, dict)):
size = len(value)
print(f" {key}: {type(value).__name__}[{size}]")
else:
print(f" {key}: {value!r}"[:300])
elif isinstance(data, list):
print("length:", len(data))
if data:
print("first item keys:", list(data[0]) if isinstance(data[0], dict) else type(data[0]).__name__)
PY
printf '%s\n' '--- numeric and uplift-related JSON text ---'
rg -n -i 'uplift|baseline|skill_score|baseline_score|overall|correctness|discoverability|effectiveness|efficiency|56%|86%|94%' \
skills/cuopt-install/benchmark/evals.json skills/cuopt-install/evals/evals.json || true
printf '%s\n' '--- repository diff metadata ---'
git status --short
git diff --stat -- skills/cuopt-install/BENCHMARK.md skills/cuopt-install/skill-card.mdRepository: NVIDIA/cuopt
Length of output: 3886
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
import re
from pathlib import Path
pattern = re.compile(
r"^\|\s*(?P<measure>[^|]+?)\s*\|\s*"
r"(?P<claude_base>\d+)%\s*→\s*(?P<claude_skill>\d+)%\s*"
r"\((?P<claude_uplift>[+-]?\d+|±0)\s+points\)\s*\|\s*"
r"(?P<codex_base>\d+)%\s*→\s*(?P<codex_skill>\d+)%\s*"
r"\((?P<codex_uplift>[+-]?\d+|±0)\s+points\)\s*\|$"
)
for filename in ("skills/cuopt-install/BENCHMARK.md", "skills/cuopt-install/skill-card.md"):
print(filename)
mismatches = []
for line_number, line in enumerate(Path(filename).read_text().splitlines(), 1):
match = pattern.match(line)
if not match:
continue
for agent in ("claude", "codex"):
base = int(match[f"{agent}_base"])
skill = int(match[f"{agent}_skill"])
actual = skill - base
shown = match[f"{agent}_uplift"]
shown_value = 0 if shown == "±0" else int(shown)
if actual != shown_value:
mismatches.append((line_number, match["measure"].strip(), agent, actual, shown))
print("mismatches:", mismatches)
PYRepository: NVIDIA/cuopt
Length of output: 583
Correct the inconsistent uplift values.
BENCHMARK.md defines uplift as skill score - baseline score. Correct the four mismatches in both benchmark tables:
- Claude Code correctness:
+8 - Claude Code effectiveness:
+17 - Codex discoverability:
+47 - Codex efficiency:
+64
If unrounded scores are used, document the calculation and rounding method.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skills/cuopt-install/BENCHMARK.md` around lines 37 - 42, Update the uplift
values in both benchmark tables in BENCHMARK.md to match the defined
skill-score-minus-baseline calculation: Claude Code correctness +8, Claude Code
effectiveness +17, Codex discoverability +47, and Codex efficiency +64. If these
values derive from unrounded scores, document the calculation and rounding
method alongside the tables.
Comment on lines
+69
to
+74
| | Overall | 56% → 90% (+34 points) | 63% → 91% (+28 points) | | ||
| | Security | 93% → 86% (-7 points) | 100% → 100% (±0 points) | | ||
| | Correctness | 86% → 94% (+9 points) | 89% → 100% (+11 points) | | ||
| | Discoverability | 27% → 96% (+69 points) | 47% → 94% (+46 points) | | ||
| | Effectiveness | 64% → 81% (+18 points) | 76% → 93% (+17 points) | | ||
| | Efficiency | 12% → 93% (+81 points) | 3% → 67% (+63 points) | |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
file="skills/cuopt-install/skill-card.md"
printf '%s\n' '--- target lines ---'
nl -ba "$file" | sed -n '35,85p'
printf '%s\n' '--- relevant evaluation and score references ---'
rg -n -i 'overall|security|correctness|discoverability|effectiveness|efficiency|unrounded|rounded|evaluation|2026-08-12|Claude Code|Codex' "$file"
printf '%s\n' '--- repository diff summary and file diff ---'
git diff --stat -- "$file"
git diff -- "$file"Repository: NVIDIA/cuopt
Length of output: 2082
Recompute the displayed uplift deltas.
The displayed integer values conflict with these deltas:
- Claude Code Correctness: +8, not +9.
- Claude Code Effectiveness: +17, not +18.
- Codex Discoverability: +47, not +46.
- Codex Efficiency: +64, not +63.
If the deltas use unrounded source values, state this in the table. Otherwise, update the deltas.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skills/cuopt-install/skill-card.md` around lines 69 - 74, Recompute the
uplift deltas in the comparison table, especially Claude Code Correctness and
Effectiveness plus Codex Discoverability and Efficiency, so each displayed
integer matches the shown endpoints. If the calculations use unrounded source
values, explicitly state that in the table; otherwise update the displayed
deltas to the correct rounded values.
Collaborator
Author
|
/ok to test 1da997d |
Iroy30
approved these changes
Aug 13, 2026
Collaborator
Author
|
/merge |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trivial blank-line addition after SKILL.md frontmatter to trigger NVSkills CI — fetch updated skill card and benchmark for
cuopt-install.