Skip to content

Image series ingestion - #138

Open
miriyalamani wants to merge 23 commits into
mainfrom
image-series-ingestion
Open

miriyalamani wants to merge 23 commits into
mainfrom
image-series-ingestion

Conversation

@miriyalamani

@miriyalamani miriyalamani commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Purpose and change

This PR adds the Image Matching plugin (image-matching) and implements Ingestion as its first task (shown in the RescueBox UI as tool Image Ingestion). Image Match (local hash search and UMass essay compare) is described in the README but not implemented in this PR.

  • Ingestion hashes a folder of images into a named archive in shared PostgreSQL (files, archives, archive_files). Each distinct image is stored once by SHA-256; re-ingest only computes hashes that are still missing.
  • Ingest can optionally compute MD5, PDQ, semantic and private embeddings via image-similarity, and a redacted essay from Ollama qwen3-vl:4b-instruct with Presidio (en_core_web_lg), limited by Max essay words (default 60) and 350 characters. Ollama requests use bounded num_ctx / num_predict to avoid excessive context allocation.
  • Essays remain in the local database only. Upload through the Safe Match intermediary is planned on branch 137 and is not included here.
  • Archives can be deleted with optional hash retention or renamed; labels are unique. Supported formats are JPEG, PNG, BMP, and WebP.
  • Also adds chatbot entry 11. Image Ingestion, demo series under src-tauri/demo/image-matching/ from image-series-dataset (Wikimedia Commons), shared table definitions in rb.api.database (ESSAY_COLUMN_CHARS), a CLIPSeg download command in the image-similarity README, and Presidio/spaCy bundling in backend.spec for the desktop build.

How to try it

Full walkthrough: src/image-matching/README.md.

  • Setup: Run poetry install, then ./run_backend_server. MD5 and PDQ need no extra models. Semantic needs SigLIP2/CLIPSeg ONNX under image-similarity (download commands in that README). Essay needs Ollama and the Presidio spaCy model (installed with image-matching).
  • UI: Run ./run_ui_server, open http://localhost:8080/chatbot, choose Menu → 11. Image Ingestion → Open.
  • Quick check: Ingest src-tauri/demo/image-matching/series_0013_Alexandria_Ocasio-Cortez_2019 with Semantic and Essay set to No. Expect Archive 'series_0013_Alexandria_Ocasio-Cortez_2019': 3 files. Run the same ingest again and expect Label already in use.
  • API: Swagger at http://127.0.0.1:8000/docs under the image_matching tag (/image_matching/ingest_images, delete_archive, rename_archive).
  • Smoke test: With the backend running, ./src/image-matching/smoke_test.sh should end with PASS: ingest, duplicate label, rename, delete. For essays (Ollama + Docker for psql), use ESSAY=1 ./src/image-matching/smoke_test.sh.
  • Cleanup: The smoke test deletes its own archive. Stop Postgres with cd startup && docker compose down when you are done.

Validation and review

  • At commit a8ff568, poetry run pytest src/image-matching/tests reported 22 passed (local).
  • ruff check src/image-matching and black --check src/image-matching were clean (local).
  • The full repository test suite and a Windows PyInstaller build were not run in this refresh.
  • Manual checks (author): default smoke test, UI walkthrough in the plugin README, essays on demo images in about 2–3.5 seconds each with bounded Ollama context, and hashing across JPEG, PNG, BMP, and WebP.
  • Review follow-ups addressed: configurable Max essay words; ESSAY_COLUMN_CHARS for files.essay; per-file ingest commits; delete-with-retain and rename edge-case tests; removal of temporary essay logging (essays are not written to the backend log); backend.spec bundling; demo licensing notes; explicit CLIPSeg curl in image-similarity README.
  • Independent agent review: Not re-run for this description update.

Documentation and figures

  • Root README.md adds a sample analysis line and mentions image hash archives in the storage section.
  • src/image-matching/README.md documents tasks, requirements, demo images, smoke test (ESSAY=1), UI scenarios, unit tests, and a Mermaid diagram for ingestion vs future Safe Match flow.
  • image_ingestion/app-info.md and smoke_test.sh support in-app copy and automation.

Security and privacy audit

  • Affected sections: image_matching/hashing/essay.py, privacy.py, image_ingestion/tasks.py, archive_store.py, rb/api/database.py (archive_files.path, files.essay).
  • The essay path uses a restrictive prompt and Presidio on PERSON, EMAIL_ADDRESS, PHONE_NUMBER, and LOCATION; redaction remains best-effort. Redacted essays are stored in Postgres only and are not emitted in backend logs in this PR.
  • The database stores local file paths in archive_files.path.
  • Deleting an archive without retaining hashes may remove files rows that no other archive still references.
  • This PR does not send data to UMass services.

Compatibility and limitations

  • New tables are created when the backend starts. Rollback after reverting the code requires dropping archive_files, archives, and files (and any dependent data you rely on).
  • Ingest commits each file’s hashes before creating the archive row, so a crash mid-folder may leave files rows without a new archive until ingest is retried.
  • Essay hashing depends on en_core_web_lg and pulling qwen3-vl:4b-instruct (about 3 GB); the Windows PyInstaller build was not verified in this PR.
  • Essay upload will need Safe Match client fields that ingestion does not collect yet; pyproject.toml and poetry.lock may overlap with branch 137.
  • CLI parsers split on | and ,, so labels or descriptions containing those characters cannot be passed reliably through the CLI.

One row per unique file in files (keyed by sha256) holds md5, PDQ,
semantic and private embeddings, and the essay text. archives holds
investigation labels and archive_files links archives to files.
Add Shared essay, privacy and hash modules for safe matching, importable by
plugins without depending on other plugins. Pins presidio-analyzer,
presidio-anonymizer and the spaCy model so nothing is
downloaded at runtime.
redact_text replaces names, emails, phone numbers and places etc. The spaCy model loads on first use.
create_essay describes an image with Ollama (gemma3:4b) in at most 55
words, without quoting visible text or naming people or places.
create_redacted_essay chains it with PII redaction
- Cap redacted essays at 350 chars (~75 CLIP tokens) on a word boundary;
  files.essay stays VARCHAR(600) as headroom
- Remove detected PII outright (Presidio redact) and collapse whitespace
  instead of inserting <PERSON>-style tags
- Log each redacted essay in the backend during ingestion; the UI result
  still shows only counts
…e CRUD.

dd a second-step task picker for Image Ingestion (ingest, delete, rename), fix delete
archive foreign-key ordering, return a clear "Archive not found" response instead of 500,
shorten the essay hash UI copy, and switch essay generation to qwen3-vl:4b-instruct.
@miriyalamani
miriyalamani requested a review from nb950 November 12, 2026 19:30
@miriyalamani
miriyalamani marked this pull request as ready for review November 12, 2026 19:30
@miriyalamani
miriyalamani requested a review from gbiss November 12, 2026 19:48
… ingest persistence and essay column naming after review.
- Essay: generate one plain paragraph (people, objects, scene) instead
  of JSON facts; keep whole sentences within 60 words / 350 chars and
  cut an overlong first sentence so the essay is never empty. JSON took
  22-44 s per image and dropped scene details; a paragraph takes ~2.5 s.
- Ingest only JPEG, PNG, BMP and WebP; skip GIF and TIFF (Ollama rejects
  TIFF for the essay). Scan the folder in image_ingestion instead of
  importing image_similarity's private _collect_image_paths.
- Allow renaming an archive to its own label.
- backend.spec: bundle Presidio and spaCy en_core_web_lg for the
  desktop build.
- Tests: archive delete with and without retained hashes (in-memory
  SQLite), same-label rename, supported-format filter, paragraph essays.
- Docs: restructure the plugin README (tasks, requirements, demo image
  source, smoke test, UI walkthrough, unit tests); update app-info
  format list and essay description.
@gbiss

gbiss commented Nov 13, 2026

Copy link
Copy Markdown

an essay: a paragraph of up to 60 words

If it's not already, can we please make this maximum configurable?

Comment thread pyproject.toml Outdated
fusepy = "^3.0.1"

rb-lib = { path = "src/rb-lib", develop = true }
rb-safe-matching-common = { path = "src/rb-safe-matching-common", develop = true }

@nb950 nb950 Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why add new rb-safe-matching-common modules to rescuebox , its needed only for image_ingestion plugin ?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, it doesn't strictly need to be there. Image-ingestion already declares it, so Poetry installs it either way. I added it to match how rb-lib is listed, but I'm happy to drop it from the root to keep that list minimal.

@miriyalamani

Copy link
Copy Markdown
Collaborator Author

an essay: a paragraph of up to 60 words

If it's not already, can we please make this maximum configurable?

Not yet. I can add a max essay words field on Ingest form with default 60.

@gbiss

gbiss commented Nov 16, 2026

Copy link
Copy Markdown

The following command froze my machine after thrashing for about 5-10m.

poetry run python -m rb.safe_matching_common "src-tauri/demo/image-ingestion/series_0022_Angela_Merkel_2017/Angela_Merkel_2017_105_Angela Merkel and Mauricio Macri 07.jpg"

Claude is telling me that the problem is that my Ollama settings were defaulting to a higher context than was necessary for the essay. On one hand, manually changing the default in Ollama did the trick for me. But it would be better to set the context length in the code since we know that we don't need too much. Something like the following.

ESSAY_NUM_CTX: Final[int] = 8192   # image is ≤ ~4.1k tokens + ~150 prompt + ≤ ~100 output

response = ollama.generate(
    model=model, prompt=ESSAY_PROMPT, images=[image_path],
    options={"num_ctx": ESSAY_NUM_CTX, "num_predict": 256},
)

…eration.

Route plugin and subtask Run clicks through _handle_select so Menu mode
select_tool() works without a callback. Pass num_ctx and num_predict to
ollama.generate for essays (PR review).
@gbiss

gbiss commented Nov 17, 2026

Copy link
Copy Markdown

With Semantic set to Yes, ingest fails with CLIPSeg ONNX model not found at …/image_similarity/onnx_models/clipseg-rd64-refined.onnx. The README's Requirements say only that Semantic needs "the image-similarity ONNX models", and the image-similarity README links the CLIPSeg repo without a download command. Perhaps you could add the command to the Requirements section?

curl -L -o src/image-similarity/image_similarity/onnx_models/clipseg-rd64-refined.onnx \
  https://huggingface.co/Xenova/clipseg-rd64-refined/resolve/main/onnx/model.onnx

- image-similarity README: add the curl command for
  clipseg-rd64-refined.onnx and note that the image-ingestion Semantic
  hash loads these models from the same onnx_models/ folder.
- image-ingestion README: link the Semantic requirement to the
  image-similarity installation section.
Rename the image-ingestion plugin to image-matching and fold the
rb-safe-matching-common package into it, so ingestion, hashing and the
upcoming query live in one plugin with one shared database

- Layout: image_matching/main.py registers the tasks; image_ingestion/
  holds ingest, delete and rename; hashing/ holds hashers, essay and
  PII redaction; archive_store.py stays shared for the future query/.
- Routes move from /image_ingestion/* to /image_matching/*; the plugin
  registry, chatbot menu, backend.spec, pyproject, pytest.ini and the
  demo images folder are updated to match.
- Stop writing redacted essays to the backend log and drop the
  rb-safe-matching-common __main__ demo. ESSAY=1 smoke_test.sh now
  prints essays from the database.
- README and app-info describe ingestion as implemented and query and
  upload through the Safe Match intermediary as not yet implemented.
- Name the plugin Image Ingestion in the chatbot menu, plugin metadata
  and plugin registry.
- Move app-info.md into image_ingestion/ and load it from there in
  main.py and backend.spec.
- Restructure app-info.md and the README as Image Matching, then its
  Ingestion and Query tasks, then Ingestion's subtasks; mark each as
  implemented or not yet implemented.
- Describe what the plugin does instead of referring to the design
  document.
Add a "Max essay words" number input (default 60) to Ingest Images.
The value sets the essay prompt and word limit for newly generated
essays; the 350-character cap still applies. Stored essays are not
regenerated when the value changes.
Update the CLI parser, smoke test payloads, README, app-info, and
unit tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants