diff --git a/CHANGELOG.md b/CHANGELOG.md index 5247040..c575454 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,8 @@ Versioned entries describe their historical candidates/releases. Version 4.0.1 p ## [Unreleased] +- Qualify native RealityScan 2.2 input-state reports and Postshot 1.1.69 COLMAP text/binary exports on retained staircase projects. Accept zero-based unsigned camera/image IDs, scope appended logs before UTF-8 decoding, and apply the dark dialog theme to consumer reports. Preserve unknown coverage when a failed Postshot project cannot export camera poses; document the exact correspondence between five mask warnings and five absent Down views. + - Import RealityScan registration/input-state reports and Postshot COLMAP camera exports with optional mask-warning logs in Studio and CLI. Persist per-source evidence across extractions, report coverage per component, capture and direction, and distinguish explicit rejection, absence and unknown membership. - Preserve the documented dotted filename extension in the RealityScan report template and cancel active report imports when the dialog is dismissed with Escape. diff --git a/IMPROVEMENTS.md b/IMPROVEMENTS.md index 37236e3..eefd523 100644 --- a/IMPROVEMENTS.md +++ b/IMPROVEMENTS.md @@ -1,10 +1,10 @@ # Improvement roadmap -Updated 2026-11-01 after consolidating the October 28 trials, executing the paired 16/31-capture staircase pilot, repeating it with the merged capture-plan implementation and adding consumer-report import. This is the active roadmap for reliable, inspectable dataset preparation: prepare, diagnose, correct, export and verify in the target application. Dated reports retain their original scope; new evidence updates this roadmap rather than rewriting earlier results. +Updated 2026-11-02 after qualifying native consumer reports from the retained 16/31 staircase projects and correcting the importer. This is the active roadmap for reliable, inspectable dataset preparation: prepare, diagnose, correct, export and verify in the target application. Dated reports retain their original scope; new evidence updates this roadmap rather than rewriting earlier results. ## Implemented, exercised and distributed -The tested functional baseline is `dev` at `3bba6be` (PR #29). The public [v4.0.3 prerelease](https://github.com/nicolasdiolez/360Extractor/releases/tag/v4.0.3) predates projects, mask profiles, rolling storage checks and alignment diagnostics. These development features are not in that downloadable binary, even though the source version still reads 4.0.3. Version/build identification needs to distinguish the next candidate. The earlier repository inventory recorded `main` at v4.0.1 and GitHub's latest stable release as v3.3; this consumer revalidation did not re-audit or publish releases. +The native-report baseline is `dev` at `f780792` (PR #31); the retained extraction/reconstruction pair was produced with `3bba6be` (PR #29). The November 2 importer corrections are prepared on `codex/native-consumer-report-validation`, pending review and integration. The public [v4.0.3 prerelease](https://github.com/nicolasdiolez/360Extractor/releases/tag/v4.0.3) predates projects, mask profiles, rolling storage checks and alignment diagnostics. These development features are not in that downloadable binary, even though the source version still reads 4.0.3. Version/build identification needs to distinguish the next candidate. The earlier repository inventory recorded `main` at v4.0.1 and GitHub's latest stable release as v3.3; this consumer revalidation did not re-audit or publish releases. | Capability | Development sources | Real-world evidence / remaining limit | | :--- | :--- | :--- | @@ -21,7 +21,7 @@ The [consumer report](docs/implementation-2026-10-28/CONSUMER_VALIDATION.md) and The first follow-up implementation ([PR #28](https://github.com/nicolasdiolez/360Extractor/pull/28)), merged into `dev`, adds per-model directional coverage and missing views, plus saved per-video extraction ranges in Studio/CLI. See the [implementation checks and limits](docs/implementation-2026-11-01/SELECTION_VALIDATION.md). This does not change the 16-capture diagnostic limit or qualify another reconstruction engine. -1. **Make consumer results inspectable and repeatable.** The [new run with merged dev](docs/implementation-2026-11-01/DEV_REVALIDATION.md) verifies the 96 common JPEGs and 16 common masks by SHA-256 for each consumer profile. Postshot again changes from tracking failure to a reopened 5 kStep splat, now with 118/186 accepted views. RealityScan grows from 8 to 58 views in its largest component, covering 8/16 versus 31/31 physical captures, but still produces 12 fragments in each case. The [earlier pilot](docs/implementation-2026-11-01/SAMPLING_COMPARISON.md) produced different counts from identical image bytes; the cause is not isolated. The first consumer-result importer now records view identities, physical captures, directions and mask warnings; qualify its native exports across versions, then investigate the five unmatched Down masks on Postshot reopen with a separate masked/unmasked test. Run at least three repetitions per arm under a recorded cache/seed protocol before judging repeatability or timing. Keep current defaults and diagnostic limits; an explicit denser diagnostic needs its own resource qualification. No complete-scene or held-out quality claim is established. +1. **Make consumer results inspectable and repeatable.** The [new run with merged dev](docs/implementation-2026-11-01/DEV_REVALIDATION.md) verifies the 96 common JPEGs and 16 common masks by SHA-256 for each consumer profile. Postshot again changes from tracking failure to a reopened 5 kStep splat, now with 118/186 accepted views. RealityScan grows from 8 to 58 views in its largest component, covering 8/16 versus 31/31 physical captures, but still produces 12 fragments in each case. The [earlier pilot](docs/implementation-2026-11-01/SAMPLING_COMPARISON.md) produced different counts from identical image bytes; the cause is not isolated. The [native-report qualification](docs/implementation-2026-11-02/NATIVE_REPORT_VALIDATION.md) now verifies RealityScan 2.2 input states and Postshot 1.1.69 text/binary identities, including project save/reopen. It corrects zero-based Postshot IDs, log-session decoding and dialog contrast. The five mask warnings map exactly to the five absent Down views; the failed 16-capture project has no camera poses to export. Next isolate mask effects on the same selected views with a separate masked/unmasked test, and extend compatibility to other versions and native registration CSV. Run at least three repetitions per arm under a recorded cache/seed protocol before judging repeatability or timing. Keep current defaults and diagnostic limits; an explicit denser diagnostic needs its own resource qualification. No complete-scene or held-out quality claim is established. 2. **Complete inspectable extraction selection.** Saved inclusion ranges now restrict full extraction in `dev` (PR #28), preserve global cadence and combine with existing exact-frame exclusions. The editor provides boundary previews; diagnostics respect both extraction ranges and their separate test window. Controlled 16-to-31 densification preserving the original indices is implemented by [PR #29](https://github.com/nicolasdiolez/360Extractor/pull/29), with a frozen Studio/CLI plan and a reusable staircase configuration; see [validation and limits](docs/implementation-2026-11-01/DENSIFICATION_VALIDATION.md). Remaining: graphical timeline editing and an inspectable candidate list with selection/rejection reasons. Evaluate better window-based selection against fixed sampling on the same corpus; the current optical-flow filter demonstrated no benefit in the initial pilot. 3. **Add persistent manual mask correction.** Brush/polygon corrections and controllable margins should survive reopening and export through each destination profile. Measure missed occluders and false removal on annotated examples, including artwork, hands and poles. The consumer pilots verified association/mode, not the per-pixel effect of every mask. 4. **Complete project recovery and resource planning.** Add autosave/recovery, complete run history and verified resume using source/settings checks plus a validated output index. Retain fresh run folders as the current safe behavior. Extend planning to model caches and reconstruction workspaces, with measured RAM/VRAM and disk use. @@ -76,7 +76,8 @@ These implementations are not all qualified on real camera/GPU/software combinat - [x] Paired 16-versus-31-capture pilot at 90° on the beginning of the staircase; one run per case/consumer, with results and limits recorded. - [x] Repeat the staircase pair with merged `dev`, verify common image/mask hashes and retain native consumer projects. - [x] Initial durable consumer-result import in Studio/CLI ([PR #31](https://github.com/nicolasdiolez/360Extractor/pull/31)): RealityScan input states/registration, Postshot COLMAP camera identities, per-component capture/direction coverage and mapped/unmapped mask warnings. See [formats and limits](docs/CONSUMER_REPORTS.md) and [source validation and review](docs/implementation-2026-11-01/CONSUMER_IMPORT_VALIDATION.md). -- [ ] Qualify real exported reports across supported consumer versions and verify mask association/persistence per pixel. Postshot logs alone still cannot establish capture coverage. +- [x] Qualify native RealityScan 2.2 input-state reports and Postshot 1.1.69 COLMAP text/binary identities on retained projects, with exact direction/capture coverage and saved-project roundtrip. Corrections remain on the November 2 qualification branch pending integration. +- [ ] Extend native report qualification to registration CSV and other supported versions; verify mask effects/persistence per pixel. The five mapped warnings concern five absent Down views, and Postshot logs alone still cannot establish capture coverage. - [ ] Controlled repeatability qualification with at least three runs per arm; record cache state and seed availability. - [ ] Repeated consumer quality comparisons, mask-effect checks and complete-scene acceptance. - [ ] Explicit HDR/ICC/alpha handling. diff --git a/docs/CONSUMER_REPORTS.md b/docs/CONSUMER_REPORTS.md index 590355f..e6e5d94 100644 --- a/docs/CONSUMER_REPORTS.md +++ b/docs/CONSUMER_REPORTS.md @@ -2,6 +2,8 @@ Development sources provide **Project → RealityScan / Postshot Reports…** for one selected source. This reads existing reconstruction evidence; it does not launch either consumer or change extraction settings. The downloadable v4.0.3 preview does not include this feature. +[Native staircase qualification](implementation-2026-11-02/NATIVE_REPORT_VALIDATION.md) exercised the supplied template in RealityScan 2.2 and both COLMAP formats in Postshot 1.1.69. That follow-up corrects zero-based Postshot IDs and documents the failed-project case; it does not certify every consumer version or registration CSV. + Choose the completed extraction folder containing `manifest.json` and `images.jsonl`, then the camera/input report and optionally a consumer log. The folder does not need a `colmap/` subfolder and may use either destination mask profile. Select one component or Image Set to inspect its view states, physical-capture coverage, directions and mask warnings. Filter by state, filename, direction or frame; the view table pages in groups of 200. ## Supported evidence @@ -15,6 +17,8 @@ Choose the completed extraction folder containing `manifest.json` and `images.js Postshot's camera export is in the **Image Set actions**, distinct from **File → Export Cameras** (animated cameras). See the [Postshot Image Set guide](https://activation.jawset.com/docs/d/Postshot%2BUser%2BGuide/Interface/Scene%2BTree/Image%2BSet). Camera membership does not by itself establish which images participated in a particular training interval or whether their masks were applied correctly. +If Postshot reports **Image Set is empty** and refuses to export camera poses, import its log alone. Coverage remains unknown; partial solver counts are not camera identities. For RealityScan registration CSV, explicitly select **Internal/External camera parameters**. Naming a CLI output `.csv` does not select that format: `exportRegistration` uses current export settings unless a parameter file is supplied. + Use **Save RealityScan report template…**, or the CLI below, to obtain a plain-text report template. Export it from the project with RealityScan's `exportReport` command, for example: ```text @@ -28,7 +32,7 @@ The template iterates components, selects each one and reports its input names a - Capture indices, directions, timestamps and mask associations come from the **confirmed extraction index**, never from guessing a filename suffix. Custom filenames are supported. - Cross-platform paths match exact index-relative names first, then an unambiguous basename. Matching is case-sensitive. Foreign or ambiguous camera names, duplicates, nonfinite/truncated COLMAP records and incomplete exports fail the import. No inferred camera is silently accepted. - Coverage denominators are exported views/captures, not an assumed six views per capture, and not proof of all images submitted to the consumer. Physical-capture coverage distinguishes at least one retained view from all exported views retained. No union of components is represented as a coherent reconstruction. -- Logs use the **last UTC session header** when present; otherwise they use the entire supplied file. A session may contain several runs. Provide a run-specific excerpt when necessary. Warning-to-mask association is a filename correspondence, not proof of the warning's source run. Tracking counts and partial solver attempts are deliberately not used to infer view membership. +- Logs use the **last UTC session header** when present; otherwise they use the entire supplied file. Scope is selected before UTF-8 decoding, so older sessions with a different encoding do not block a valid current session. The selected scope must still be UTF-8. A session may contain several runs. Provide a run-specific excerpt when necessary. Warning-to-mask association is a filename correspondence, not proof of the warning's source run. Tracking counts and partial solver attempts are deliberately not used to infer view membership. - No supplied log means warnings are **unknown**. No warning in a supplied scope does not prove correct mask persistence or pixel effects. - Input-file SHA-256 digests, manifest/index digests, export run ID and source size/date are retained with the normalized facts. Names and size/date are not source-pixel verification. No media is copied or reconstructed during import. diff --git a/docs/implementation-2026-11-02/NATIVE_REPORT_VALIDATION.md b/docs/implementation-2026-11-02/NATIVE_REPORT_VALIDATION.md new file mode 100644 index 0000000..1c25cc9 --- /dev/null +++ b/docs/implementation-2026-11-02/NATIVE_REPORT_VALIDATION.md @@ -0,0 +1,48 @@ +# Native consumer-report qualification + +On 2026-11-02, reopen the retained staircase projects from the [merged-dev 16/31 comparison](../implementation-2026-11-01/DEV_REVALIDATION.md) on Windows/Shadow. Export fresh evidence with RealityScan **2.2.0.119430.RS** and Postshot **1.1.69**. No alignment, training or extraction was rerun, and the retained projects were not saved over. This qualifies report compatibility for these projects and versions, not a new reconstruction result. + +The importer baseline is `dev` **f7807929482b873126bd0a46c4eb2b743fc1c253** (PR #31). Corrections and this report are prepared on `codex/native-consumer-report-validation`; they are not part of a published binary. [Compact measurements](native-report-results.json) retain input/index/manifest digests, component-specific capture/direction coverage and version information without absolute private paths. + +## Native results + +| Evidence | Confirmed views | Retained views | Physical captures with a retained view | +| :--- | ---: | ---: | ---: | +| RealityScan, 16 captures, largest of 12 components | 96 | 8 | 8/16 | +| RealityScan, 31 captures, largest of 12 components | 186 | 58 | 31/31 | +| Postshot, 31 captures, Image Set exported as text and binary | 186 | 118 in both, identical identities | 31/31 | +| Postshot, 16 captures, reopened failed project | 96 | No export available | Unknown to the importer | + +RealityScan's native `exportReport` successfully executes the **exact product template**. Each of the 12 components reports every indexed input with native True/False alignment flags. The largest control component retains eight Left views. The dense component retains Left 31, Back 11, Up 7, Down 5, Front 4 and Right 0. The remaining 88/128 views are rejected **in that component**, not globally. Components remain separate. + +Postshot's dense Image Set retains Left 31, Down 26, Right 26, Back 13, Up 13 and Front 9. Neither dense result retains all six views of any physical capture. Text and binary camera exports identify exactly the same 118 images. The other 68 indexed views are absent from that exported set, with unknown cause. + +The failed 16-capture Postshot project reopens with **96 imported/selected images, 0 tracked cameras and 52 sparse points**. Its export action says **“There are no camera poses to export from this Image Set.”** The earlier two-view partial solver attempt is therefore not a saved two-camera model. No camera report was fabricated; importing the fresh log alone correctly leaves all 96 view memberships unknown. + +Both consumer profiles still have **96 common JPEGs and 16 common masks** between the 16/31 datasets, all SHA-256 identical in a fresh read-only check. The four confirmed index digests also match the preparation records. No original media was copied to the internal disk. + +## Masks and log scope + +The fresh dense Postshot session (`2026-11-02 09:26:09 UTC`) repeats five unmatched-mask warnings, for Down frames **5, 8, 11, 13 and 16**. All five map to the confirmed mask index. They are **exactly the five Down views missing from the exported Image Set**; the other 26 Down views are retained. This is correspondence evidence, not proof of why those views are absent or whether masks affected training pixels. Do not treat these warnings alone as five failed mask applications to retained cameras. + +The fresh control session (`2026-11-02 09:37:18 UTC`) has no warnings, but no camera export. That does not prove masking or tracking success. RealityScan was imported without a log, so its mask warnings remain unknown. + +The accumulated Windows Postshot log also contains non-UTF-8 bytes in an older session. Session excerpts were cut at the last native UTC header **before decoding**, retaining exact session bytes; only these small excerpts were transferred. The importer now applies that same order to accumulated logs. The selected session still must be valid UTF-8; unsupported current-session bytes fail explicitly. Whole-input SHA-256 remains attached to an import, even when only the last session is interpreted. This is not automatic encoding detection or run separation within one session. + +## Corrections and validation + +- **Native Postshot IDs:** both formats start image/camera IDs at zero. Baseline `f780792` rejects both real files with `Invalid COLMAP image ID or pose`. Accept zero and read binary IDs as unsigned 32-bit values, matching COLMAP's [identifier types and invalid sentinels](https://github.com/colmap/colmap/blob/main/src/colmap/util/types.h). Continue rejecting negative/out-of-range text IDs, the invalid sentinel, duplicate IDs, invalid poses and malformed records. Added regressions cover zero in each field and unsigned boundaries. +- **Old log encoding:** locate the last UTC session in bytes before UTF-8 decoding. A regression ensures ignored older bytes cannot block a valid current session, while invalid current/no-header logs still fail and the input digest covers all original bytes. +- **Report contrast:** include the consumer dialog in the existing dark dialog stylesheet. The native report table, headers, labels, buttons and warnings are readable with a light system palette as well. + +Five real imports completed through Studio's background import thread: both RealityScan reports, both dense Postshot formats and the control log. Saving and reopening the `.360proj` preserved every report. After reopening, filtering absent Down views displayed exactly five rows, each with one mapped warning. RealityScan's selected dense component displayed all 186 view states independently. These were macOS source/Qt offscreen checks with media preview disabled; they do not qualify Windows Studio or GPU inference. A real Postshot CLI import also succeeded with 186 confirmed views and five warnings. + +Checks: **521 tests and 20 subtests passed**, no skips, in 25.15 seconds; all **33 importer tests** passed again after explicitly preserving UTF-8 BOM handling; Ruff, whole-project mypy (56 files), critical-contract mypy (16 modules), release metadata and whitespace checks passed. The initial sandboxed suite could not invoke `ps` for its child-process test; the final complete run had that access and passed. No remote CI, installed-wheel or distributed-binary validation is claimed for this local branch. + +## Retained evidence and limits + +Private native files, normalized receipts, the reopened `.360proj` and screenshots are retained locally under ignored `plans/application360-native-reports-2026-11-02/`. The six-file transfer archive is **32,099 bytes**, SHA-256 `b812471bacf78927b7834da68ec1558bce445a36857154fee533dbb8f358d72f`, verified after the authorized private-Drive transfer. It contains no image, video or splat model. Public source changes contain compact measurements and synthetic regressions only. + +RealityScan **registration CSV** was not qualified here: `exportRegistration` without an explicit parameter file used the installation's current COLMAP export settings and created a COLMAP folder instead of CSV. The successful RealityScan qualification is the input-state template. For CSV, explicitly select **Internal/External camera parameters**, or supply the corresponding saved export settings, as described by the [RealityScan CLI documentation](https://rshelp.capturingreality.com/en-US/tutorials/commandline_1.htm). + +Next: isolate mask behavior with a recorded masked/unmasked comparison on the same selected views, then at least three controlled repetitions per arm for repeatability. Other consumer versions, native registration CSV, non-UTF-8 active sessions, reconstruction quality and Windows/CUDA distribution remain unqualified. Keep extraction defaults, diagnostic limits and release status unchanged. diff --git a/docs/implementation-2026-11-02/native-report-results.json b/docs/implementation-2026-11-02/native-report-results.json new file mode 100644 index 0000000..3740cca --- /dev/null +++ b/docs/implementation-2026-11-02/native-report-results.json @@ -0,0 +1,1258 @@ +{ + "baseline_commit": "f7807929482b873126bd0a46c4eb2b743fc1c253", + "versions": { + "RealityScan": "2.2.0.119430.RS", + "Postshot": "1.1.69" + }, + "transfer_sha256": "b812471bacf78927b7834da68ec1558bce445a36857154fee533dbb8f358d72f", + "datasets": { + "realityscan/control16": { + "run_id": "6dac30e43b4f43b088a481d689daf70e", + "images_written": 96, + "index_sha256": "4184b816b2e11bc265090b0fbb8aba769f1318a7e09475cdb952d3b6f6220dcd", + "manifest_sha256": "e0876fae1886ae5da131a3a8a0b2d07100c9b9fefe48c65ff2109ee890048d3f" + }, + "realityscan/trial31": { + "run_id": "0374a96ab42644a4826a3123bebf5046", + "images_written": 186, + "index_sha256": "0939c764ab2ad775b35690168b8edb4fb4afad8b78ca5982197a3f4790e3ce2b", + "manifest_sha256": "503797c30893d69c010982d0b1eaf8edb37b91356bc8ba2868a756864c943588" + }, + "postshot/control16": { + "run_id": "daab262713934ff3b38786dce3b027b0", + "images_written": 96, + "index_sha256": "5f0b407dc43ee4b65751cb9c775d6995d63e51a22ea8ad1501afa1c4275465da", + "manifest_sha256": "55559d86eeb392ad22095b69f48c77bf6b0d7eb6e7e3ecc77f5c9b798731f8fb" + }, + "postshot/trial31": { + "run_id": "0d479b29caca40c5a1c1b4c3fec062e5", + "images_written": 186, + "index_sha256": "32592fea892b0e7767f018250e7fff70e8e876394cb7c83484a896737af9f2dc", + "manifest_sha256": "859dc4b48e211f665a8413778eef182e5fea03fdf97c25105e0fe34052980d23" + } + }, + "imports": { + "realityscan-control16-txt": { + "inputs": [ + { + "name": "control16-input-states.txt", + "sha256": "f4efa880fb40a1c8e1a521da4087163276cc4138ec4f9cd73e80bb52a57d7fff", + "role": "cameras" + } + ], + "log_scope": null, + "models": [ + { + "label": "Component 0", + "totals": { + "rejected": 88, + "retained": 8 + }, + "capture_totals": { + "exported": 16, + "any_retained": 8, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 16 + }, + "Down": { + "rejected": 16 + }, + "Front": { + "rejected": 16 + }, + "Left": { + "retained": 8, + "rejected": 8 + }, + "Right": { + "rejected": 16 + }, + "Up": { + "rejected": 16 + } + }, + "retained_source_frames": [ + 0, + 5, + 11, + 16, + 22, + 27, + 33, + 38 + ] + }, + { + "label": "Component 1", + "totals": { + "rejected": 94, + "retained": 2 + }, + "capture_totals": { + "exported": 16, + "any_retained": 2, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 15, + "retained": 1 + }, + "Down": { + "rejected": 16 + }, + "Front": { + "rejected": 16 + }, + "Left": { + "rejected": 16 + }, + "Right": { + "rejected": 16 + }, + "Up": { + "rejected": 15, + "retained": 1 + } + }, + "retained_source_frames": [ + 5, + 11 + ] + }, + { + "label": "Component 2", + "totals": { + "rejected": 93, + "retained": 3 + }, + "capture_totals": { + "exported": 16, + "any_retained": 3, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 15, + "retained": 1 + }, + "Down": { + "rejected": 15, + "retained": 1 + }, + "Front": { + "rejected": 15, + "retained": 1 + }, + "Left": { + "rejected": 16 + }, + "Right": { + "rejected": 16 + }, + "Up": { + "rejected": 16 + } + }, + "retained_source_frames": [ + 16, + 22, + 27 + ] + }, + { + "label": "Component 3", + "totals": { + "rejected": 94, + "retained": 2 + }, + "capture_totals": { + "exported": 16, + "any_retained": 2, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 16 + }, + "Down": { + "rejected": 15, + "retained": 1 + }, + "Front": { + "rejected": 15, + "retained": 1 + }, + "Left": { + "rejected": 16 + }, + "Right": { + "rejected": 16 + }, + "Up": { + "rejected": 16 + } + }, + "retained_source_frames": [ + 16, + 22 + ] + }, + { + "label": "Component 4", + "totals": { + "rejected": 93, + "retained": 3 + }, + "capture_totals": { + "exported": 16, + "any_retained": 3, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 15, + "retained": 1 + }, + "Down": { + "rejected": 15, + "retained": 1 + }, + "Front": { + "rejected": 16 + }, + "Left": { + "rejected": 16 + }, + "Right": { + "rejected": 16 + }, + "Up": { + "rejected": 15, + "retained": 1 + } + }, + "retained_source_frames": [ + 16, + 22, + 27 + ] + }, + { + "label": "Component 5", + "totals": { + "rejected": 94, + "retained": 2 + }, + "capture_totals": { + "exported": 16, + "any_retained": 2, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 16 + }, + "Down": { + "rejected": 16 + }, + "Front": { + "rejected": 16 + }, + "Left": { + "rejected": 16 + }, + "Right": { + "rejected": 14, + "retained": 2 + }, + "Up": { + "rejected": 16 + } + }, + "retained_source_frames": [ + 22, + 27 + ] + }, + { + "label": "Component 6", + "totals": { + "rejected": 93, + "retained": 3 + }, + "capture_totals": { + "exported": 16, + "any_retained": 3, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 15, + "retained": 1 + }, + "Down": { + "rejected": 15, + "retained": 1 + }, + "Front": { + "rejected": 15, + "retained": 1 + }, + "Left": { + "rejected": 16 + }, + "Right": { + "rejected": 16 + }, + "Up": { + "rejected": 16 + } + }, + "retained_source_frames": [ + 44, + 49, + 55 + ] + }, + { + "label": "Component 7", + "totals": { + "rejected": 93, + "retained": 3 + }, + "capture_totals": { + "exported": 16, + "any_retained": 3, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 15, + "retained": 1 + }, + "Down": { + "rejected": 15, + "retained": 1 + }, + "Front": { + "rejected": 16 + }, + "Left": { + "rejected": 15, + "retained": 1 + }, + "Right": { + "rejected": 16 + }, + "Up": { + "rejected": 16 + } + }, + "retained_source_frames": [ + 44, + 49, + 55 + ] + }, + { + "label": "Component 8", + "totals": { + "rejected": 94, + "retained": 2 + }, + "capture_totals": { + "exported": 16, + "any_retained": 2, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 16 + }, + "Down": { + "rejected": 16 + }, + "Front": { + "rejected": 15, + "retained": 1 + }, + "Left": { + "rejected": 16 + }, + "Right": { + "rejected": 16 + }, + "Up": { + "rejected": 15, + "retained": 1 + } + }, + "retained_source_frames": [ + 49, + 55 + ] + }, + { + "label": "Component 9", + "totals": { + "rejected": 93, + "retained": 3 + }, + "capture_totals": { + "exported": 16, + "any_retained": 3, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 16 + }, + "Down": { + "rejected": 16 + }, + "Front": { + "rejected": 16 + }, + "Left": { + "rejected": 13, + "retained": 3 + }, + "Right": { + "rejected": 16 + }, + "Up": { + "rejected": 16 + } + }, + "retained_source_frames": [ + 60, + 66, + 71 + ] + }, + { + "label": "Component 10", + "totals": { + "rejected": 93, + "retained": 3 + }, + "capture_totals": { + "exported": 16, + "any_retained": 3, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 15, + "retained": 1 + }, + "Down": { + "rejected": 15, + "retained": 1 + }, + "Front": { + "rejected": 16 + }, + "Left": { + "rejected": 16 + }, + "Right": { + "rejected": 16 + }, + "Up": { + "rejected": 15, + "retained": 1 + } + }, + "retained_source_frames": [ + 71, + 77, + 82 + ] + }, + { + "label": "Component 11", + "totals": { + "rejected": 94, + "retained": 2 + }, + "capture_totals": { + "exported": 16, + "any_retained": 2, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 16 + }, + "Down": { + "rejected": 16 + }, + "Front": { + "rejected": 16 + }, + "Left": { + "rejected": 14, + "retained": 2 + }, + "Right": { + "rejected": 16 + }, + "Up": { + "rejected": 16 + } + }, + "retained_source_frames": [ + 77, + 82 + ] + } + ], + "mask_warning_frames": [], + "unknown_views_without_model": { + "unknown": 96 + } + }, + "realityscan-trial31-txt": { + "inputs": [ + { + "name": "trial31-input-states.txt", + "sha256": "e857c58a42a344fdff0929f22bdb302acf478b3aa2daa10919cea327577b4eee", + "role": "cameras" + } + ], + "log_scope": null, + "models": [ + { + "label": "Component 0", + "totals": { + "rejected": 184, + "retained": 2 + }, + "capture_totals": { + "exported": 31, + "any_retained": 2, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 31 + }, + "Down": { + "rejected": 31 + }, + "Front": { + "rejected": 31 + }, + "Left": { + "rejected": 31 + }, + "Right": { + "rejected": 31 + }, + "Up": { + "retained": 2, + "rejected": 29 + } + }, + "retained_source_frames": [ + 0, + 2 + ] + }, + { + "label": "Component 1", + "totals": { + "retained": 2, + "rejected": 184 + }, + "capture_totals": { + "exported": 31, + "any_retained": 2, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 31 + }, + "Down": { + "rejected": 31 + }, + "Front": { + "retained": 2, + "rejected": 29 + }, + "Left": { + "rejected": 31 + }, + "Right": { + "rejected": 31 + }, + "Up": { + "rejected": 31 + } + }, + "retained_source_frames": [ + 0, + 2 + ] + }, + { + "label": "Component 2", + "totals": { + "rejected": 184, + "retained": 2 + }, + "capture_totals": { + "exported": 31, + "any_retained": 2, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 31 + }, + "Down": { + "rejected": 31 + }, + "Front": { + "rejected": 31 + }, + "Left": { + "rejected": 31 + }, + "Right": { + "retained": 2, + "rejected": 29 + }, + "Up": { + "rejected": 31 + } + }, + "retained_source_frames": [ + 0, + 2 + ] + }, + { + "label": "Component 3", + "totals": { + "rejected": 184, + "retained": 2 + }, + "capture_totals": { + "exported": 31, + "any_retained": 2, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 31 + }, + "Down": { + "rejected": 31 + }, + "Front": { + "rejected": 31 + }, + "Left": { + "rejected": 31 + }, + "Right": { + "rejected": 29, + "retained": 2 + }, + "Up": { + "rejected": 31 + } + }, + "retained_source_frames": [ + 8, + 11 + ] + }, + { + "label": "Component 4", + "totals": { + "rejected": 184, + "retained": 2 + }, + "capture_totals": { + "exported": 31, + "any_retained": 2, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 30, + "retained": 1 + }, + "Down": { + "rejected": 30, + "retained": 1 + }, + "Front": { + "rejected": 31 + }, + "Left": { + "rejected": 31 + }, + "Right": { + "rejected": 31 + }, + "Up": { + "rejected": 31 + } + }, + "retained_source_frames": [ + 22, + 27 + ] + }, + { + "label": "Component 5", + "totals": { + "rejected": 184, + "retained": 2 + }, + "capture_totals": { + "exported": 31, + "any_retained": 2, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 31 + }, + "Down": { + "rejected": 30, + "retained": 1 + }, + "Front": { + "rejected": 30, + "retained": 1 + }, + "Left": { + "rejected": 31 + }, + "Right": { + "rejected": 31 + }, + "Up": { + "rejected": 31 + } + }, + "retained_source_frames": [ + 16, + 22 + ] + }, + { + "label": "Component 6", + "totals": { + "rejected": 184, + "retained": 2 + }, + "capture_totals": { + "exported": 31, + "any_retained": 2, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 30, + "retained": 1 + }, + "Down": { + "rejected": 31 + }, + "Front": { + "rejected": 31 + }, + "Left": { + "rejected": 31 + }, + "Right": { + "rejected": 31 + }, + "Up": { + "rejected": 30, + "retained": 1 + } + }, + "retained_source_frames": [ + 22, + 27 + ] + }, + { + "label": "Component 7", + "totals": { + "rejected": 184, + "retained": 2 + }, + "capture_totals": { + "exported": 31, + "any_retained": 2, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 31 + }, + "Down": { + "rejected": 31 + }, + "Front": { + "rejected": 30, + "retained": 1 + }, + "Left": { + "rejected": 31 + }, + "Right": { + "rejected": 31 + }, + "Up": { + "rejected": 30, + "retained": 1 + } + }, + "retained_source_frames": [ + 49, + 55 + ] + }, + { + "label": "Component 8", + "totals": { + "rejected": 184, + "retained": 2 + }, + "capture_totals": { + "exported": 31, + "any_retained": 2, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 31 + }, + "Down": { + "rejected": 30, + "retained": 1 + }, + "Front": { + "rejected": 30, + "retained": 1 + }, + "Left": { + "rejected": 31 + }, + "Right": { + "rejected": 31 + }, + "Up": { + "rejected": 31 + } + }, + "retained_source_frames": [ + 46, + 52 + ] + }, + { + "label": "Component 9", + "totals": { + "rejected": 165, + "retained": 21 + }, + "capture_totals": { + "exported": 31, + "any_retained": 21, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 31 + }, + "Down": { + "rejected": 31 + }, + "Front": { + "rejected": 31 + }, + "Left": { + "rejected": 31 + }, + "Right": { + "rejected": 10, + "retained": 21 + }, + "Up": { + "rejected": 31 + } + }, + "retained_source_frames": [ + 24, + 27, + 30, + 33, + 35, + 38, + 44, + 46, + 49, + 52, + 55, + 57, + 60, + 63, + 66, + 68, + 71, + 74, + 77, + 79, + 82 + ] + }, + { + "label": "Component 10", + "totals": { + "rejected": 128, + "retained": 58 + }, + "capture_totals": { + "exported": 31, + "any_retained": 31, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 20, + "retained": 11 + }, + "Down": { + "rejected": 26, + "retained": 5 + }, + "Front": { + "rejected": 27, + "retained": 4 + }, + "Left": { + "retained": 31 + }, + "Right": { + "rejected": 31 + }, + "Up": { + "rejected": 24, + "retained": 7 + } + }, + "retained_source_frames": [ + 0, + 2, + 5, + 8, + 11, + 13, + 16, + 19, + 22, + 24, + 27, + 30, + 33, + 35, + 38, + 41, + 44, + 46, + 49, + 52, + 55, + 57, + 60, + 63, + 66, + 68, + 71, + 74, + 77, + 79, + 82 + ] + }, + { + "label": "Component 11", + "totals": { + "rejected": 183, + "retained": 3 + }, + "capture_totals": { + "exported": 31, + "any_retained": 3, + "all_retained": 0 + }, + "directions": { + "Back": { + "rejected": 31 + }, + "Down": { + "rejected": 29, + "retained": 2 + }, + "Front": { + "rejected": 30, + "retained": 1 + }, + "Left": { + "rejected": 31 + }, + "Right": { + "rejected": 31 + }, + "Up": { + "rejected": 31 + } + }, + "retained_source_frames": [ + 77, + 79, + 82 + ] + } + ], + "mask_warning_frames": [], + "unknown_views_without_model": { + "unknown": 186 + } + }, + "postshot-trial31-txt": { + "inputs": [ + { + "name": "images.txt", + "sha256": "febe2db9a7d00b92db54ac212a165c18ff11a0da876151b390978ac11b2a2260", + "role": "cameras" + }, + { + "name": "postshot-trial31-session.log", + "sha256": "29eb04e7b02b20f74e4a66967c5328028caf370709169980c2179d427853c966", + "role": "log" + } + ], + "log_scope": "2026-11-02 09:26:09 UTC", + "models": [ + { + "label": "Exported Image Set", + "totals": { + "not_retained": 68, + "retained": 118 + }, + "capture_totals": { + "exported": 31, + "any_retained": 31, + "all_retained": 0 + }, + "directions": { + "Back": { + "not_retained": 18, + "retained": 13 + }, + "Down": { + "retained": 26, + "not_retained": 5 + }, + "Front": { + "not_retained": 22, + "retained": 9 + }, + "Left": { + "retained": 31 + }, + "Right": { + "not_retained": 5, + "retained": 26 + }, + "Up": { + "not_retained": 18, + "retained": 13 + } + }, + "retained_source_frames": [ + 0, + 2, + 5, + 8, + 11, + 13, + 16, + 19, + 22, + 24, + 27, + 30, + 33, + 35, + 38, + 41, + 44, + 46, + 49, + 52, + 55, + 57, + 60, + 63, + 66, + 68, + 71, + 74, + 77, + 79, + 82 + ] + } + ], + "mask_warning_frames": [ + 5, + 8, + 11, + 13, + 16 + ], + "unknown_views_without_model": { + "unknown": 186 + } + }, + "postshot-trial31-bin": { + "inputs": [ + { + "name": "images.bin", + "sha256": "88b114e65ee99f8db1651f118acdc33d56c9aac06adf6238fb3f853c4a312ec6", + "role": "cameras" + }, + { + "name": "postshot-trial31-session.log", + "sha256": "29eb04e7b02b20f74e4a66967c5328028caf370709169980c2179d427853c966", + "role": "log" + } + ], + "log_scope": "2026-11-02 09:26:09 UTC", + "models": [ + { + "label": "Exported Image Set", + "totals": { + "not_retained": 68, + "retained": 118 + }, + "capture_totals": { + "exported": 31, + "any_retained": 31, + "all_retained": 0 + }, + "directions": { + "Back": { + "not_retained": 18, + "retained": 13 + }, + "Down": { + "retained": 26, + "not_retained": 5 + }, + "Front": { + "not_retained": 22, + "retained": 9 + }, + "Left": { + "retained": 31 + }, + "Right": { + "not_retained": 5, + "retained": 26 + }, + "Up": { + "not_retained": 18, + "retained": 13 + } + }, + "retained_source_frames": [ + 0, + 2, + 5, + 8, + 11, + 13, + 16, + 19, + 22, + 24, + 27, + 30, + 33, + 35, + 38, + 41, + 44, + 46, + 49, + 52, + 55, + 57, + 60, + 63, + 66, + 68, + 71, + 74, + 77, + 79, + 82 + ] + } + ], + "mask_warning_frames": [ + 5, + 8, + 11, + 13, + 16 + ], + "unknown_views_without_model": { + "unknown": 186 + } + }, + "postshot-control16-log": { + "inputs": [ + { + "name": "postshot-control16-session.log", + "sha256": "ac484464185a647eea6ab9e99c3ec7d7893957302b956552f5236630cba7b259", + "role": "log" + } + ], + "log_scope": "2026-11-02 09:37:18 UTC", + "models": [], + "mask_warning_frames": [], + "unknown_views_without_model": { + "unknown": 96 + } + } + }, + "common_outputs": { + "realityscan": { + "images": 96, + "masks": 16 + }, + "postshot": { + "images": 96, + "masks": 16 + } + } +} diff --git a/src/extractor360/core/consumer_reports.py b/src/extractor360/core/consumer_reports.py index 7b61718..47eb3c2 100644 --- a/src/extractor360/core/consumer_reports.py +++ b/src/extractor360/core/consumer_reports.py @@ -22,6 +22,9 @@ MAX_VIEWS = 20000 MAX_MEMBERSHIPS = 100000 MAX_REPORTS = 10 +# COLMAP image_t/camera_t are uint32; only UINT32_MAX is the invalid sentinel. +# Native Postshot exports start both identifiers at zero. +INVALID_COLMAP_ID = 2**32 - 1 STATES = ('retained', 'rejected', 'not_retained', 'disabled', 'unknown') FORMAT = '360-extractor-consumer-report' RS_TEMPLATE = '''360EXTRACTOR_REALITYSCAN_V1 @@ -223,7 +226,9 @@ def _postshot_text(content, names, cancelled): if len(fields) != 10: raise ValueError('Invalid COLMAP image row; choose images.txt, not cameras.txt') identifier, camera = int(fields[0]), int(fields[8]) - if identifier <= 0 or identifier in ids or camera <= 0 or not all(math.isfinite(float(v)) for v in fields[1:8]): + if (not 0 <= identifier < INVALID_COLMAP_ID or identifier in ids + or not 0 <= camera < INVALID_COLMAP_ID + or not all(math.isfinite(float(v)) for v in fields[1:8])): raise ValueError('Invalid COLMAP image ID or pose') ids.add(identifier) _member(names, fields[9], states) @@ -258,8 +263,9 @@ def unpack(fmt): ids = set() for _ in range(count): _check(cancelled) - row = unpack('i7di') - if row[0] <= 0 or row[0] in ids or row[-1] <= 0 or not all(math.isfinite(v) for v in row[1:8]): + row = unpack('I7dI') + if (row[0] == INVALID_COLMAP_ID or row[0] in ids or row[-1] == INVALID_COLMAP_ID + or not all(math.isfinite(v) for v in row[1:8])): raise ValueError('Invalid COLMAP image ID or pose') ids.add(row[0]) name = bytearray() @@ -282,11 +288,14 @@ def unpack(fmt): def _log(content, views, cancelled): - text = _text(content) - headers = list(re.finditer(r'^\[0:00:00\.\d+\] (\d{4}-\d{2}-\d{2} [\d:]+ UTC)', text, re.M)) - scope = headers[-1].group(1) if headers else 'entire supplied file (no UTC session header)' + # Postshot appends sessions to one file. Older sessions may use a different + # encoding; only the selected session must satisfy the UTF-8 contract. + content = content.removeprefix(b'\xef\xbb\xbf') + headers = list(re.finditer(rb'^\[0:00:00\.\d+\] (\d{4}-\d{2}-\d{2} [\d:]+ UTC)', content, re.M)) + scope = headers[-1].group(1).decode('ascii') if headers else 'entire supplied file (no UTC session header)' if headers: - text = text[headers[-1].start():] + content = content[headers[-1].start():] + text = _text(content) masks = _Names(v['mask'] for v in views if v['mask']) mask_views = {v['mask']: v['image'] for v in views if v['mask']} warnings = [] diff --git a/src/extractor360/ui/styles.qss b/src/extractor360/ui/styles.qss index 769b40c..586b8ec 100644 --- a/src/extractor360/ui/styles.qss +++ b/src/extractor360/ui/styles.qss @@ -10,15 +10,17 @@ QMainWindow { color: #E6E6EA; } -QDialog#alignmentDiagnostic, QDialog#extractionRanges { +QDialog#alignmentDiagnostic, QDialog#extractionRanges, QDialog#consumerReports { background-color: #141417; } QDialog#alignmentDiagnostic QLabel, QDialog#alignmentDiagnostic QPushButton, +QDialog#consumerReports QLabel, QDialog#consumerReports QPushButton, QDialog#extractionRanges QLabel, QDialog#extractionRanges QPushButton, QDialog#extractionRanges QCheckBox { color: #E6E6EA; font-size: 12px; } QDialog#alignmentDiagnostic QTableWidget, QDialog#alignmentDiagnostic QPlainTextEdit, +QDialog#consumerReports QTableWidget, QDialog#consumerReports QPlainTextEdit, QDialog#extractionRanges QTableWidget { background-color: #1D1D23; color: #E6E6EA; @@ -27,23 +29,25 @@ QDialog#extractionRanges QTableWidget { selection-color: #FFFFFF; font-size: 12px; } -QDialog#alignmentDiagnostic QHeaderView, QDialog#extractionRanges QHeaderView { +QDialog#alignmentDiagnostic QHeaderView, QDialog#extractionRanges QHeaderView, QDialog#consumerReports QHeaderView { background-color: #1D1D23; } QDialog#alignmentDiagnostic QHeaderView::section, QDialog#alignmentDiagnostic QTableCornerButton::section, +QDialog#consumerReports QHeaderView::section, QDialog#consumerReports QTableCornerButton::section, QDialog#extractionRanges QHeaderView::section, QDialog#extractionRanges QTableCornerButton::section { background-color: #2A2A33; color: #E6E6EA; padding: 5px; border: 1px solid #3A3A46; } -QDialog#alignmentDiagnostic QPushButton, QDialog#extractionRanges QPushButton { +QDialog#alignmentDiagnostic QPushButton, QDialog#extractionRanges QPushButton, QDialog#consumerReports QPushButton { background-color: #2A2A33; border: 1px solid #4A4A56; padding: 6px; border-radius: 4px; } -QDialog#alignmentDiagnostic QPushButton:disabled, QDialog#extractionRanges QPushButton:disabled { +QDialog#alignmentDiagnostic QPushButton:disabled, QDialog#extractionRanges QPushButton:disabled, +QDialog#consumerReports QPushButton:disabled { color: #74747E; background-color: #1D1D23; } diff --git a/tests/test_consumer_reports.py b/tests/test_consumer_reports.py index c6b38fb..51da38a 100644 --- a/tests/test_consumer_reports.py +++ b/tests/test_consumer_reports.py @@ -122,6 +122,61 @@ def test_empty_registered_set_requires_declared_zero(dataset, tmp_path): assert coverage(report, 'image-set')['totals'] == {'not_retained': 6} +def test_log_ignores_encoding_of_older_sessions_but_not_current_scope(dataset, tmp_path): + log = tmp_path / 'Postshot.log' + old = b'[0:00:00.001] 2026-11-01 21:23:32 UTC\nWARNING: ancien chemin accentu\xe9\n' + current = (b'[0:00:00.001] 2026-11-02 09:26:09 UTC\n' + b"WARNING: no matching image found for image mask 'capture 5_Down.mask.png'\n") + for valid in (old + current, current, b'\xef\xbb\xbf' + current): + log.write_bytes(valid) + report = import_report(dataset, 'postshot', log_path=log) + assert report['log_scope'] == '2026-11-02 09:26:09 UTC' + assert len(report['warnings']) == 1 + assert report['warnings'][0]['image'] == 'capture 5_Down.jpg' + assert report['inputs'][0]['sha256'] == hashlib.sha256(valid).hexdigest() + assert coverage(report)['totals'] == {'unknown': 6} + for invalid in (old, current + b'WARNING: accentu\xe9\n', b'WARNING: accentu\xe9\n'): + log.write_bytes(invalid) + with pytest.raises(ValueError, match='UTF-8'): + import_report(dataset, 'postshot', log_path=log) + + +def postshot_id_model(path, identifier, camera, binary): + # Postshot 1.1.69 emits zero-based IDs and absolute Windows image names. + name = r'C:\qualification\images\capture 0_Left.jpg' + if binary: + path.write_bytes(struct.pack('