[gretadolcetti]

Pull Request

Issue

Fixes #1715

Description

This pull request extends the Qiskit code validation example to support additional validation steps beyond QKT (flake8-qiskit-migration) rules. It introduces optional validation using a problem's own test suite and the LintQ static analyzer, and updates both the documentation and code to reflect these capabilities. The changes provide a more robust validation and repair loop for Qiskit code generation.

Key changes:

New validation helpers

• Implemented validate_correctness() to run candidate code against a benchmark problem's own test suite, supporting both Qiskit HumanEval and QuantumKatas datasets.
• Implemented validate_lintq() to analyze code using the LintQ CodeQL queries, parsing SARIF output and reporting findings.

Integration into main example

• Updated qiskit_code_validation.py to import and use the new validation helpers, allowing optional LintQ validation via a use_lintq flag.

Attribution

• AI coding assistants used

---

Adding a new component, requirement, sampling strategy, or tool?

If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.

• Component
• Requirement
• Sampling Strategy
• Tool

[gretadolcetti]

@ajbozarth

[planetf1]

Thanks @gretadolcetti, it's great to see the sample extended to show more of the checks you can run with mellea. Quantum is a great showcase for it.

For the core repo, though, I wonder if we should stick to mainstream, actively maintained dependencies. LintQ is a nice idea, but it's a research artefact from a single paper (FSE 2024), with no commits since February 2025. Its model of Qiskit predates Qiskit 2: two of its ten rules look for APIs that have since been removed (c_if, iden), and it doesn't cover newer constructs such as if_test or prepare_state. CodeQL's licence also restricts automated use to research and open-source codebases. Updating a version pin wouldn't address any of this, since it comes from LintQ being a research prototype. That's fine for a blog post or a benchmark run, but harder to recommend as a general pattern in core.

On the mellea side, the LintQ check plugs in through the same req + simple_validate + extra_requirements pattern the example already uses. That makes it another interesting worked example of how far you can take a validator, which is exactly what a blog post does well. Since this is closely tied to generative-computing/mellea-website#80, could the LintQ specifics live with the blog instead, in a branch or gist the post links to? The test-suite validator looks like the part with lasting value for the core example. Running a problem's own tests inside the repair loop is a good fit there, and I'd be glad to see that go in on its own.

I've noted two issues with validate_correctness on generative-computing/mellea-website#80 (the check lookup, and sys.exit() hanging m.instruct()), which would be worth fixing before it goes in.

[gretadolcetti]

Thanks for the very valuable feedback, @planetf1.

Let me try to explain why we believe this could be a valuable addition to Mellea, even with the concerns you rightly raised.

First, it's true that LintQ hasn't been updated recently, but Qiskit is still a young domain, and well-established tools don't exist yet (https://dl.acm.org/doi/epdf/10.1145/3840563.3843709), moreover 80% of the rules LintQ checks for are still valid for Qiskit v2, and there is still plenty of Qiskit v1 code still around. Interest in this area is growing, and we expect it to keep growing over the next few years.

We also strongly believe in promoting a broader linting culture within the quantum community. This example also serves as a possible implementation of a paper we just had accepted at a NeurIPS workshop (https://openreview.net/forum?id=Q6uE2LmrH1), which could bring more visibility to the topic and consequently to Mellea itself. This extended example is a good candidate because, like the pipeline we propose in the paper, it is modular: other analyzers and unit tests can be plugged in with very little effort.

We expect new analyzers to be developed, and with this example Mellea will already be connected to that space. We hope it could become one of the tools that helps move linting culture in quantum computing forward.

Regarding the blog post we are discussing in generative-computing/mellea-website#80, we believe that having a pointer to a concrete example, built on an existing one, would not only allow users to play around in a very convinient and easy way, but also allow them to discover the other qiskit related examples.

Finally, I think this example shows what for me personally is the real value of Mellea: how easy it is to add, adapt, and modify requirements, even for non-experts. So, to conclude we believe it was a good idea to develop on an already existing example so that each part (the existing and the new) could benefit from each other, adding functionalities in a few lines (again, one of Mellea's best features) rather than adding a completely new thing from scratch.

[planetf1]

thanks @gretadolcetti - having another think over this .. will get back to you. Appreciate the quick turnaround.

[ajbozarth]

Sorry for the delay, I finally found time to do a deep dive into this (and the blog pr). The tl;dr is I don't think we actually want this PR in mellea like I originally thought when we talked previously.

This falls into two reasons:

1. LintQ: There are two key issues with adding LintQ to the example:
   1. LintQ requires CodeQL which has a license whose use in mellea falls into a distinct grey area
   2. LintQl requires a separate user install outside a pipit lib and would make the example not unable with a single uv run command like it is currently. (personally this was the reason I feel we should not include the LintQ additions)
      2.validate_correctness() is included but not used in the example. This is the weaker reason as it is documented in the README and the code is good. My reasoning that we shouldn't add it actually traces back to a decision I made when I added support for extra_requirements, when I wrote a similar function but decided not to include it because it was only useful when running against code that existing in the dataset it validates against.

I am open to still including validate_correctness() and the doc updates related to it if I can be convinced that the helper still makes sense to include by itself, in which case feel free to remove the lintq code and I can re-review what's left.

I'll be posting a PR review on the blog pr soon referring this review

[gretadolcetti]

Hi @ajbozarth, thanks for sharing your thoughts. Let me try to address both points.

1. LintQ

(i) Licensing. The changes are confined to docs/examples/instruct_validate_repair/qiskit_code_validation/: Mellea doesn't depend on, bundle or redistribute CodeQL, and it isn't in pyproject.toml or the script dependencies. Users who opt in install it themselves and accept GitHub's terms directly, which explicitly allow academic research and demonstration, i.e. what the example does. LintQ itself is MIT-licensed. This is the same approach the PR takes for QuantumKatas' CC BY-NC-SA license: document the terms and leave the choice to the user. I'm happy to add a pointer to the CodeQL terms in the LintQ setup section.

(ii) Setup. The default path is still a single uv run with QKT validation only; LintQ is opt-in via use_lintq.

2. validate_correctness It isn't tied to the benchmark datasets: it only needs prompt, test and entry_point, so users can pass their own tests.

To sum up, this PR is meant as an optional extension of the existing example. Users who want to try LintQ can install the external tools and accept their terms. Even for users who don't, it leaves room for other quantum analyzers and shows how easily they can be plugged into Mellea to improve the quantum code it generates. I hope this addresses your concerns, and thanks again for the discussion.

[ajbozarth]

Assuming we move forward with this, here's a first review pass; I'll do a deeper review tomorrow.

Some feedback from Claude — comments inline. One blocker: the diff isn't ruff format-clean, so pre-commit ruff-format will reject it; the rest is polish.

[gretadolcetti]

Hi @ajbozarth, I should have solved the first review pass' concerns.

[ajbozarth]

So I've been reviewing this in spurts over the last week, and I think I've partially circled back to my original push back.

Though the LintQ work is interesting, it requires a third party non-python install, thus making trying the example difficult. In addition it doesn't really show anything novel about using the IVR loop that your other addition validate_correctness() doesn't also show, but with less user overhead.

In general the mellea examples focus should be on how mellea can help solve problems and this introduces a very heavy LintQ addition to demonstrate a behavior that your already demonstrating with the validate_correctness req in the same work.

So my feedback is to slim down this PR to just the validate_correctness() code and documentation.

[gretadolcetti]

@ajbozarth, I removed all the LintQ features from the PR in gretadolcetti@c4d02d3.
Thanks for your time and consideration!

[ajbozarth]

Some feedback from Claude. LintQ removal looks good. One note here, the rest inline.

The PR title still says "Add support for LintQ" — since that's the squash-merge commit title and LintQ is gone, please update it to describe validate_correctness. (Body's a bit stale too but that's just cleanup.)

[ajbozarth]

Since validate_correctness is deliberately not wired into the example, this snippet is the only thing justifying the helper's inclusion — so it has to get a reader from the downloaded dataset to a call, and it doesn't yet. Setup downloads a .jsonl, but nothing turns it into the problem dict, and m/strategy are undefined. One-line fix, plus a pointer:

import json

# One dataset row = one problem dict (reuse m and strategy from qiskit_code_validation.py's setup)
problem = json.loads(next(open("data/qiskit_humaneval_hard.jsonl")))

code, success, attempts = generate_validated_qiskit_code(
    m, problem["prompt"], strategy, extra_requirements=[passes_tests(problem)]
)

And state that the snippet is a drop-in into the example's existing setup — as written it reads as a description of the helper, not instructions for using it.

[ajbozarth]

Optional trim: the "QKT rules" row here just restates what "How It Works" and the validators list already cover, so the table is really one new row (Test suite) plus the repair-reason framing. A sentence would carry it without the table scaffolding — keeps this section lighter against the rest of the README.

[ajbozarth]

Optional: __import__ is still here, so __import__("os").system(...) works — this is defense-in-depth, not isolation. Fine to keep given the "not a sandbox" warnings; drop it or say so in the comment.