Skip to content

CHANGELOG.md

Latest commit

 

History

History
180 lines (155 loc) · 9.46 KB

CHANGELOG.md

File metadata and controls

180 lines (155 loc) · 9.46 KB
Raw

Changelog

All notable changes to this project are documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added

  • Image-mode output for the LeRobot v3 writer. LeRobotV3WriterConfig(video=False) PNG-encodes each camera frame inline into the data parquet (dtype: "image" features) instead of encoding MP4s — no videos/ directory and no per-camera videos/… pointer columns are written, and info.json's video_path / video_files_size_in_mb are null (the keys stay present, as v3 loaders require). Set it from a conversion config with writer_config: {video: false}. Needs Pillow (now part of the [lerobot] extra). Video mode stays the default and is unchanged. The LeRobot v3 reader decodes these inline PNG columns, so image-mode datasets round-trip through Forge.

  • Streaming reads for cloud datasets (LeRobot-v3, Zarr). forge inspect and forge ingest on s3://… / gs://… now read over the network with range requests instead of downloading the whole dataset.

    • Inspect fetches metadata only — a 464 MB cloud LeRobot-v3 dataset yields a full inspect from ~3 KB (its info.json), a ~150,000× reduction, in a fraction of a second.
    • Ingest (register + quality-score) reads only the proprio parquet columns (state/action/timestamp) via column/range reads — never the videos. Ingesting the same 464 MB dataset streams ~3 MB and produces identical episodes and quality scores. Content hashes are identical whether an episode is streamed or downloaded, so cross-path re-ingest is idempotent.
    • LeRobot-v3 gets a proprio-only streaming read_episodes; Zarr streams via its native fsspec store. forge inspect --deep, embeddings, and video quality still fetch frames; other formats/commands are unchanged (download-to-temp). New forge.io.DataSource helper; readers keep their local paths untouched.

  • Search- and selection-driven curation. forge curate now selects episodes three ways — a SQL --where predicate, an explicit --ids list, or a --from <selection.json> file — so semantic search and Forge Studio can drive curation, not just SQL. forge search --save sel.json writes the result ids (with provenance) and Studio's dedup tab exports a matching decisions file; forge curate --from sel.json applies them and records the source in labeled_by. Explicit ids are filtered to episodes that exist (no dangling labels) and can be intersected with --where.

  • Catalog dedup, curation + Forge Studio (Phase 3). Find near-duplicate episodes from the embeddings and curate a clean, labeled training set.

    forge catalog dedup -c ./forge-catalog --threshold 0.97
forge curate -c ./forge-catalog --where "overall_score > 6" \
    --dedup 0.97 --dedup-policy keep-higher-quality --label approved
forge studio -c ./forge-catalog -o studio.html
    • Two new tables (bumps catalog SCHEMA_VERSION 2 → 3, additive): dedup_edges (near-dup pairs as facts — similarity, not verdicts) and curation_labels (an append-log of approve/reject/hold decisions, latest-row-wins).
    • forge catalog dedup computes near-dup pairs (cosine over episode embeddings, max over shared cameras; idempotent). forge curate applies a WHERE filter + a dedup policy (keep-higher-quality / keep-longer / keep-first) and labels survivors approved, dedup losers rejected.
    • DuckDB views/macros: v_curation (latest label per episode) and v_dup_losers(threshold, policy).
    • forge studio generates a self-contained, themed HTML app (Overview · Corpus · Dedup review · Snapshot) from real catalog data and embedded video thumbnails — one shareable file, no server.
    • Reuses the Phase 2 vectors, the readers for thumbnails, and the catalog writer/commit machinery. The per-dataset forge dedup (perceptual-hash) is unchanged; catalog dedup lives under forge catalog dedup.
    • See forge/catalog/README.md.

  • Catalog embeddings + semantic search (Phase 2). Embed the episodes in a catalog and search them by natural language.

    pip install "forge-robotics[embed]"
forge embed --catalog ./forge-catalog
forge search "picks up the red cup" -c ./forge-catalog --top 10
forge search --like <episode_id> -c ./forge-catalog
    • New embeddings table (bumps catalog SCHEMA_VERSION 1 → 2, additive — v1 catalogs open unchanged). One row per (episode_id, model_id, level, camera): episode-level vision vectors per camera + an instruction (text) vector.
    • New forge/embed/ engine — an EmbeddingModel registry with a SigLIP (siglip-so400m, shared image–text space) implementation, so text queries match episode video. Device auto-selects CUDA → Apple MPS → CPU.
    • forge embed (backfill) and forge ingest --embed (opt-in stage); forge search (text and --like) and Catalog.search(...). Brute-force cosine in DuckDB — sub-second at lab scale.
    • Vectors are versioned per model (model_id = <name>@<ckpt-hash>, reproducible across machines); search enforces a single model_id (never mixes vector spaces).
    • Reuses existing internals — the readers behind forge inspect for frames, forge.io for cloud sources, [video] for decode, and the catalog writer/commit machinery. New deps behind the [embed] extra (torch, transformers); the base CLI never imports them.
    • See forge/embed/README.md.

  • The catalog (Phase 1) — an append-only, queryable registry of episodes. Turns Forge from a per-dataset tool into a system of record: a set of append-only Parquet tables (episodes, quality_scores) written with pyarrow and queried with embedded DuckDB, on a local directory or an s3:// / gs:// bucket.

    pip install "forge-robotics[catalog]"
forge catalog init ./forge-catalog
forge ingest ./my_dataset --catalog ./forge-catalog
forge query "SELECT task, count(*) FROM episodes GROUP BY task" -c ./forge-catalog
forge catalog stats --catalog ./forge-catalog
    • New module forge/catalog/ with a Catalog class as the single entry point (from forge.catalog import Catalog), plus forge/catalog/ingest.py.
    • New CLI commands: forge catalog init, forge catalog stats, forge ingest, forge query (SQL over episodes, quality_scores, and a v_latest_quality view; --format table|json|csv).
    • Ingestion reuses existing internals — the same format readers behind forge inspect for metadata and QualityAnalyzer.analyze_episode (the engine behind forge quality) for scoring. No metadata extraction or scoring logic was reimplemented, and no existing behavior changed.
    • Idempotent & crash-safe. Episodes are keyed by a content hash, so re-running an ingest over the same sources is a no-op. Each flush commits atomically via a manifest-last protocol; queries read only manifested part-files, so a crash never exposes a partial batch.
    • New deps behind the [catalog] extra: duckdb, xxhash (and an explicit pyarrow). The base CLI never imports them — they load only when a catalog command runs.
    • See forge/catalog/README.md and docs/forge_data_engine_design.md.

  • Cloud storage support (s3://, gs://). Every command that accepts a dataset path now also accepts Amazon S3 and Google Cloud Storage URIs, in addition to local paths and hf:// URLs. For example:

    forge inspect s3://my-bucket/datasets/run_0413
forge convert gs://lab-data/rosbags ./out --format lerobot-v3
forge quality s3://my-bucket/datasets/droid --report report.html
    • Backed by fsspec (now a core dependency), with s3fs / gcsfs as optional extras: pip install "forge-robotics[s3]" or [gcs]. Passing an s3:// URI without s3fs installed fails with the exact pip install command to run.
    • All filesystem access is routed through a single utility, forge.io.paths. Remote datasets are downloaded to a temporary directory on first access and cleaned up automatically at process exit, so every format (including video, HDF5, and rosbag, which need random file access) behaves identically to local paths.
    • Authentication uses each provider's default credential chain (AWS env vars / profiles / IAM roles; GCP Application Default Credentials). Forge never handles credentials itself.

Fixed

  • writer_config now reaches parallel conversion workers. ConversionConfig dropped writer_config when serialized, so with --workers > 1 every writer-specific option (including the new video: false) was silently ignored by the worker processes.

Changed

  • fsspec is now a core dependency.

Known limitations

  • Writing outputs directly to cloud URIs (e.g. forge convert … s3://bucket/out) is not supported yet; commands fail fast with a clear message. Write to a local directory and upload it afterwards.
  • Remote datasets are downloaded in full before processing. True range-read streaming (reading parquet/zarr remotely without a full download) is a planned follow-up; the fsspec plumbing in forge.io.paths is in place for it.