[mscasso-scanoss]

What

Integrates github.com/scanoss/snippets-classifier to score each snippet match on whether its reported OSS and local line ranges correspond — proposing wrong candidate assignments (false positives) for a human to confirm.

The classifier itself is offline and embedded, but scoring a match needs three inputs: the local file, the matched OSS file, and the paired line ranges. The inventory carries the ranges and the OSS file's hash; the local file is on disk; the OSS file is fetched by hash. That fetch is the only network cost, so distinct OSS files are downloaded once and reused.

It only proposes (threshold 0.7) — it removes nothing.

Changes

• pkg/scanoss/filecontents.go — Contents.File(ctx, md5, start, end) → GET /v3/file-contents/{md5} (whole file or a line range).
• pkg/snippetmatch/ — Annotator: reads each local file, fetches every distinct OSS file once (cache by hash, bounded concurrency), classifies in parallel, sets FileEvidence.SnippetClassification. Non-fatal: a match it cannot score (no local file / no OSS content) is left unannotated and logged.
• pkg/sbom — SnippetClassification {verdict, probability, model_version} on FileEvidence, rendered in raw output.
• pkg/scanpipeline — opt-in ClassifySnippets stage after enrichment; new classify progress layer.
• cmd/scan.go — --classify-snippets flag (path scan only, raw format only; warns if used with another format).

Usage

scanoss-cli scan ./src --classify-snippets

Each snippet match in the raw output gains:

"snippet_classification": { "verdict": "false_positive", "probability": 0.91, "model_version": "lr-…" }

Dependency

snippets-classifier is a public, MIT licensed module (v0.2.0, model lr-d51f2e100211). The build needs no GOPRIVATE or token.

Testing

• go build ./..., go vet ./..., gofmt clean; full go test ./... and -race on the concurrent packages all pass.
• New unit tests: snippetmatch (real vs false-positive verdicts with the classifier's own fixtures, dedup-by-hash, non-fatal skips, non-snippet evidence ignored) and Contents.File (whole-file, line range, 404).
• Rebased on main (2026-10-26); build, gofmt and the tests of cmd, sbom, scanoss, scanpipeline and snippetmatch pass again.
• Not exercised end-to-end here (needs a live API key + a scan producing snippet matches).

🤖 Generated with Claude Code

[matiasdaloia]

Ran this branch (46abd2c) end to end against api.scanoss.com and validated the model on code outside the training corpus. Summary below.

End to end

• Works. file_hash comes back as a 16-hex CRC64 and /v3/file-contents/{crc64} answers 200, so the code is fine. The "md5" wording in filecontents.go is stale.
• Every snippet match was scored, none skipped.

Validation on application repos

8 public repos, none in the corpus: go-shiori/shiori, gotify/server, Klipper3d/klipper, louislam/uptime-kuma, macrozheng/mall, mealie-recipes/mealie, sherlock-project/sherlock, TeamNewPipe/NewPipe. They produced 876 snippet matches, 70 of them at P >= 0.7.

I reviewed a blind, stratified sample of 109 against the README criteria: all 39 in 0.1-0.7, 40 of the 70 at >= 0.7, and 30 of the 767 below 0.1. The figures are reweighted to the 876 matches. 13 cases came out unsure, mostly generated OpenAPI clients.

threshold	FP precision	FP recall	real matches flagged
0.5	0.68	0.96	~22
0.7	0.71	0.90	~18
0.9	0.79	0.83	~11
0.95	0.81	0.75	~9

Recall holds up. Precision is well below the corpus figure (0.95 at 0.7): about 3 in 10 flagged matches are real.

Where it goes wrong

1. Ranges made mostly of "trivial" lines score as FP even when they correspond line for line. Comments, import and export lists, __all__, Javadoc.

   • gotify/server gotify-server.env.example: 331/343 lines identical, P = 1.00.
   • mealie mealie/schema/*/__init__.py and shiori clients/ts/src/models/index.ts: identical export lists, P = 1.00.
   • TheAlgorithms/Java AbsoluteValue.java: identical Javadoc block with an int to long signature change, P = 0.99.

   It looks like normalisation leaves almost no informative lines, and a small or low-diversity range pushes towards FP. A fallback that also counts trivial lines when the informative count is near zero might fix most of these.

2. An older version of the same file with insertions. mealie new_meal.py, NewPipe Destination.kt, shiori system.go: the shared lines correspond in order, with some lines added locally in between, and they score FP at 0.83 to 0.95.

What this means for the rollout

• 0.7 is fine for proposing a false positive for a human to confirm.
• Dismissing without a human is not safe at any threshold until (1) is fixed. Even at 0.95, about 1 in 5 would be a real match.

On top of this PR

I'll open a stacked PR on this branch, so this one stays as it is. It adds:

• --classify-threshold;
• filtering, with a count of what was removed;
• CycloneDX output, and SPDX if it is simple;
• a "classified N of M" summary on stderr;
• docs.