feat(file_storage): configurable bucket folder prefix + any S3 provider

docs/superpowers/specs/2026-08-06-s3-provider-prefix-design.md

@@ -0,0 +1,135 @@
+# Configurable S3 provider + bucket folder prefix
+
+**Date:** 2026-08-06
+**Module:** `modules/file_storage`
+
+## Problem
+
+Two gaps in the `file_storage` module:
+
+1. **Provider portability.** The S3 backend exposes `s3_endpoint_url`, so
+   MinIO/R2 partly work, but it never constructs a `botocore.config.Config`.
+   That makes path-style addressing unreachable, which is the single most
+   common breakage on MinIO, Ceph, and any endpoint that is an IP or
+   `localhost`. `s3_region` is also required by the settings validator, so
+   region-less providers (R2 wants `auto`) fail to boot. Presigned URLs are
+   minted against the internal endpoint, so a browser cannot open a download
+   when the app reaches storage over a private network.
+
+2. **No folder prefix.** `service._generate_key()` emits `YYYY/MM/DD/<uuid><ext>`
+   at the bucket root. There is no way to confine objects to a folder, and no
+   way for an operator to configure one.
+
+## Requirements
+
+- Any S3-compatible provider must be usable (AWS, MinIO, R2, Wasabi, B2,
+  DigitalOcean Spaces, self-hosted gateways).
+- All objects live under a configurable folder.
+- The folder is configured **at runtime through the admin Settings UI**, not
+  hardcoded and not env-only.
+
+## Design
+
+### Key prefix
+
+New DB-backed setting in a **General** group, so it renders in the existing
+per-module Settings UI and picks up hot reload via
+`settings.reload.apply_changes_and_reload`:
+
+```python
+key_prefix: str = Field(default="", json_schema_extra={"group": "General"})
+```
+
+Normalized and validated by a `field_validator` on `FileStorageSettings`, so a
+bad value fails at config time rather than at first upload:
+
+| Input | Stored |
+|---|---|
+| `media` | `media/` |
+| `/media/` | `media/` |
+| `a//b` | `a/b/` |
+| `` (blank) | `` (bucket root) |
+| `../escape`, `a/../b`, `C:\x` | rejected with `ValueError` |
+
+**Scope: generic, all backends.** One `key_prefix` applies to every backend
+rather than an S3-only field. The filesystem backend nests it under
+`fs_root_path` (`./uploads/media/2026/...`); S3 prepends it to the object key.
+
+**Application point: baked into the stored key.** The prefix is applied once,
+at upload, inside `_generate_key`:
+
+```python
+def _generate_key(filename: str, prefix: str = "") -> str:
+    return f"{prefix}{datetime.now(UTC):%Y/%m/%d}/{uuid.uuid4().hex}{Path(filename).suffix}"
+```
+
+The full prefixed key is persisted in `StoredFile.key`. Backends receive
+already-qualified keys and never prepend anything themselves.
+
+Consequence, and the reason for this choice: editing the prefix later steers
+only new uploads. Every previously stored file keeps resolving, because its row
+still carries the key it was written under. The transparent-backend alternative
+(store bare keys, prepend on every operation) would orphan every existing
+object the moment an operator edits the prefix.
+
+A blank default means existing installations observe no behavior change.
+
+### S3 provider compatibility
+
+New fields in the existing **S3** settings group, all optional and inert when
+blank:
+
+| Field | Purpose |
+|---|---|
+| `s3_addressing_style` | `auto` \| `path` \| `virtual`. Requires building a real `botocore.config.Config`, which `_build` does not do today. Unblocks MinIO/Ceph. |
+| `s3_public_endpoint_url` | Presign against this host instead of the internal one, so browser download redirects work across a private network boundary. |
+| `s3_signature_version` | Blank = botocore default. For gateways needing an explicit `s3v4`/`s3`. |
+| `s3_verify_ssl` | Allow self-signed internal gateways. |
+
+`s3_region` drops out of the hard validator and defaults to `us-east-1`, so R2
+and region-less providers boot. `s3_bucket` remains required when the S3
+backend is selected.
+
+`S3Backend` holds a second set of client kwargs used *only* by
+`presigned_get_url`, identical to the main set except for `endpoint_url`. When
+no public endpoint is configured the two are the same object and behavior is
+unchanged. Signing against the public host is correct precisely because that is
+the host the browser will contact.
+
+`botocore` is imported lazily alongside `aioboto3`, matching the existing
+pattern, so the module still loads without the `[s3]` extra installed.
+
+### Error handling
+
+Unchanged in shape. Config errors surface as `ConfigurationError` at boot
+(fail-fast on a misconfigured prod deploy); per-operation failures stay
+`StorageBackendError` / `StorageNotFoundError`. Prefix validation raises
+`ValueError` from the pydantic validator, which the settings UI surfaces as a
+field error.
+
+### Testing
+
+- Prefix normalization table, including every rejection case.
+- Prefixed round-trip (put → get → delete) through both backends.
+- `Config` carries path-style addressing when configured.
+- Presign uses the public endpoint when set, the internal one when not.
+- S3 backend builds with no region configured.
+- Regression that guards the core guarantee: a pre-existing prefix-free key
+  still downloads after a prefix is configured.
+
+## Out of scope
+
+- **Filesystem shard repair.** `FilesystemBackend._resolve` shards on `key[:2]`,
+  which is `20` for every date-based key ever written, so all objects already
+  land in one directory. A prefix makes it `me` rather than `20` — equally
+  degenerate, not worse. Fixing it properly would relocate existing files on
+  disk. This spec corrects the misleading docstring and leaves behavior alone.
+- Provider presets and a "test connection" button in the settings UI.
+- Multipart upload for very large files (already deferred to v2).
+
+## Docs
+
+`modules/file_storage/README.md` is stale: it documents `SM_FILE_STORAGE_*` env
+vars as the configuration path (settings are DB-backed now) and calls the
+filesystem backend `local` rather than `filesystem`. Corrected alongside the
+new fields.

modules/file_storage/README.md

@@ -17,7 +17,30 @@ pip install "simple_module_file_storage[s3]"
 - `POST /api/file-storage/upload` multipart upload endpoint.
 - `GET /api/file-storage/files` (paged list), `GET /api/file-storage/files/{file_id}` (metadata), `GET /api/file-storage/files/{file_id}/download` (signed-URL redirect or stream), `DELETE /api/file-storage/files/{file_id}`.
 - Pluggable backend selected by the `backend` setting (`filesystem` default | `s3`); third-party backends can register under any id.
-- Settings (`backend`, `s3_bucket`, `s3_region`, `s3_endpoint_url` for R2/MinIO/etc., `s3_access_key_id`, `s3_secret_access_key`, `s3_presign_ttl_seconds`) are configured from the DB settings store via the admin UI — they are not read from `SM_FILE_STORAGE_*` environment variables at runtime.
+- Works with any S3-compatible provider — AWS, MinIO, Cloudflare R2, Wasabi, Backblaze B2, DigitalOcean Spaces, self-hosted gateways.
+- All settings are configured from the DB settings store via the admin UI — they are **not** read from `SM_FILE_STORAGE_*` environment variables at runtime. Changing them rebuilds the backend in place; no restart needed.
+
+### Settings
+
+| Setting | Group | Notes |
+|---|---|---|
+| `backend` | General | `filesystem` (default) or `s3` |
+| `key_prefix` | General | Folder every object is stored under, e.g. `media/`. Blank = bucket root. Applies to all backends. |
+| `fs_root_path` | Filesystem | Local storage root |
+| `s3_bucket` | S3 | Required when `backend=s3` |
+| `s3_region` | S3 | Defaults to `us-east-1`; region-less providers (R2) accept `auto` |
+| `s3_access_key_id` / `s3_secret_access_key` | S3 | Blank falls back to the ambient AWS credential chain |
+| `s3_endpoint_url` | S3 | Custom endpoint for MinIO / R2. Blank uses the AWS default |
+| `s3_public_endpoint_url` | S3 | Host used to *sign* download URLs when it differs from the endpoint the app connects to |
+| `s3_addressing_style` | S3 | `auto` / `path` / `virtual`. Use `path` for MinIO, Ceph, and IP/localhost endpoints |
+| `s3_signature_version` | S3 | e.g. `s3v4`. Blank uses the default |
+| `s3_verify_ssl` | S3 | Set false only for internal gateways with self-signed certificates |
+| `s3_presign_ttl_seconds` | S3 | Lifetime of download redirect URLs |
+| `max_file_size_bytes` / `allowed_content_types` | Limits | Upload validation |
+
+**Folder prefix.** `key_prefix` is baked into each object's key at upload time and stored on the row, so changing it later affects only new uploads — every existing file keeps resolving. `media` and `/media/` both normalise to `media/`; `..` segments are rejected.
+
+**Presigning across a network boundary.** If the app reaches storage privately (`http://minio:9000`) but browsers need a public host, set `s3_public_endpoint_url` to `https://files.example.com`. Download URLs are signed for that host, since an S3 signature is bound to the host in the URL.
 
 ## Usage
 
@@ -38,7 +61,17 @@ async def attach_receipt(
     return {"file_id": record.id}
 ```
 
-Backend and S3 credentials are configured from the admin UI at **/settings/modules → FileStorage** (DB-backed settings); switch `backend` to `s3` and fill in `s3_bucket`, `s3_region`, `s3_endpoint_url` (for MinIO/R2), `s3_access_key_id`, and `s3_secret_access_key`.
+Backend and S3 credentials are configured from the admin UI at **/settings/modules → FileStorage** (DB-backed settings); switch `backend` to `s3` and fill in `s3_bucket`, `s3_access_key_id`, and `s3_secret_access_key`. Set `key_prefix` to keep every object inside one folder.
+
+Provider examples:
+
+| Provider | `s3_endpoint_url` | `s3_region` | `s3_addressing_style` |
+|---|---|---|---|
+| AWS S3 | *(blank)* | your region | `auto` |
+| MinIO | `http://minio:9000` | `us-east-1` | `path` |
+| Cloudflare R2 | `https://<account>.r2.cloudflarestorage.com` | `auto` | `auto` |
+| DigitalOcean Spaces | `https://<region>.digitaloceanspaces.com` | your region | `auto` |
+| Backblaze B2 | `https://s3.<region>.backblazeb2.com` | your region | `auto` |
 
 ## Depends on
 

modules/file_storage/file_storage/backends/filesystem.py

@@ -20,10 +20,14 @@
 class FilesystemBackend:
     """Stores objects on the local filesystem under ``root``.
 
-    Keys are sharded by their first two characters to keep any single
-    directory from accumulating millions of entries (which slows ``readdir``
-    on most filesystems). A key like ``2026/04/19/abc123.png`` is written to
-    ``<root>/20/2026/04/19/abc123.png``.
+    Keys are prefixed with their first two characters, so a key like
+    ``2026/04/19/abc123.png`` is written to ``<root>/20/2026/04/19/abc123.png``.
+
+    Note this is *not* an effective shard: every key begins with either the
+    year (``20``) or the configured ``key_prefix``, so in practice all objects
+    land in a single top-level directory and are spread only by the date
+    segments below it. Genuine sharding would need to key off the uuid, which
+    would relocate every already-stored file — so the layout stays as-is.
     """
 
     backend_id = constants.BackendId.FILESYSTEM

modules/file_storage/file_storage/backends/s3.py

@@ -36,7 +36,14 @@ class S3Backend:
     backend_id = constants.BackendId.S3
     supports_presigned_url = True
 
-    def __init__(self, *, bucket: str, region: str, client_kwargs: dict[str, Any]) -> None:
+    def __init__(
+        self,
+        *,
+        bucket: str,
+        region: str,
+        client_kwargs: dict[str, Any],
+        presign_client_kwargs: dict[str, Any] | None = None,
+    ) -> None:
         try:
             import aioboto3
         except ImportError as exc:
@@ -49,10 +56,18 @@ def __init__(self, *, bucket: str, region: str, client_kwargs: dict[str, Any]) -
         self.bucket = bucket
         self.region = region
         self.client_kwargs = client_kwargs
+        # Presigning may need to sign against a different host than the one we
+        # connect to — the signature is bound to the host in the URL, so a URL
+        # signed for an internal endpoint is invalid at the public one. Falls
+        # back to the same kwargs when no public endpoint is configured.
+        self.presign_client_kwargs = presign_client_kwargs or client_kwargs
 
     def _client(self):
         return self._session.client("s3", **self.client_kwargs)
 
+    def _presign_client(self):
+        return self._session.client("s3", **self.presign_client_kwargs)
+
     async def put(
         self,
         key: str,
@@ -116,7 +131,7 @@ async def exists(self, key: str) -> bool:
 
     async def presigned_get_url(self, key: str, ttl_seconds: int) -> str:
         try:
-            async with self._client() as client:
+            async with self._presign_client() as client:
                 return await client.generate_presigned_url(
                     "get_object",
                     Params={"Bucket": self.bucket, "Key": key},
@@ -139,18 +154,57 @@ def _is_not_found(exc: Exception) -> bool:
     } and "404" in str(exc)
 
 
+def _build_botocore_config(settings: FileStorageSettings):
+    """Assemble a ``botocore.config.Config``, or ``None`` when all defaults.
+
+    Addressing style and signature version are reachable *only* through this
+    object — they are not client kwargs — which is why non-AWS providers could
+    not be configured before. Returning ``None`` for an all-default setup keeps
+    botocore's own negotiation intact rather than freezing it.
+    """
+    try:
+        from botocore.config import Config
+    except ImportError as exc:  # pragma: no cover - aioboto3 always brings botocore
+        raise ConfigurationError(
+            "S3 backend requires the 'botocore' package (installed with aioboto3)."
+        ) from exc
+
+    config_kwargs: dict[str, Any] = {}
+    if settings.s3_addressing_style != constants.AddressingStyle.AUTO:
+        config_kwargs["s3"] = {"addressing_style": settings.s3_addressing_style}
+    if settings.s3_signature_version:
+        config_kwargs["signature_version"] = settings.s3_signature_version
+    return Config(**config_kwargs) if config_kwargs else None
+
+
 @register_backend(constants.BackendId.S3)
 def _build(settings: FileStorageSettings) -> S3Backend:
-    if not settings.s3_bucket or not settings.s3_region:
-        raise ConfigurationError("S3 backend requires s3_bucket and s3_region.")
-    client_kwargs: dict[str, Any] = {"region_name": settings.s3_region}
+    if not settings.s3_bucket:
+        raise ConfigurationError("S3 backend requires s3_bucket.")
+
+    region = settings.s3_region or constants.DEFAULT_S3_REGION
+    client_kwargs: dict[str, Any] = {"region_name": region}
     if settings.s3_endpoint_url:
         client_kwargs["endpoint_url"] = settings.s3_endpoint_url
     if settings.s3_access_key_id and settings.s3_secret_access_key:
         client_kwargs["aws_access_key_id"] = settings.s3_access_key_id
         client_kwargs["aws_secret_access_key"] = settings.s3_secret_access_key
+    if not settings.s3_verify_ssl:
+        client_kwargs["verify"] = False
+    config = _build_botocore_config(settings)
+    if config is not None:
+        client_kwargs["config"] = config
+
+    presign_client_kwargs = client_kwargs
+    if settings.s3_public_endpoint_url:
+        presign_client_kwargs = {
+            **client_kwargs,
+            "endpoint_url": settings.s3_public_endpoint_url,
+        }
+
     return S3Backend(
         bucket=settings.s3_bucket,
-        region=settings.s3_region,
+        region=region,
         client_kwargs=client_kwargs,
+        presign_client_kwargs=presign_client_kwargs,
     )

modules/file_storage/file_storage/constants.py

@@ -58,6 +58,20 @@ class BackendId:
     S3: Final = "s3"
 
 
+class AddressingStyle:
+    """S3 URL addressing modes, mapped straight onto botocore's values.
+
+    ``VIRTUAL`` puts the bucket in the hostname (``bucket.s3.amazonaws.com``);
+    ``PATH`` keeps it in the path (``host/bucket/key``). Providers reached by
+    IP or ``localhost`` — MinIO, Ceph — can only do path-style, since a bucket
+    hostname would not resolve. ``AUTO`` defers to botocore's own heuristic.
+    """
+
+    AUTO: Final = "auto"
+    PATH: Final = "path"
+    VIRTUAL: Final = "virtual"
+
+
 class Permission:
     UPLOAD: Final = "file_storage.upload"
     DOWNLOAD: Final = "file_storage.download"
@@ -95,6 +109,12 @@ class I18nKey:
 # ── Defaults ─────────────────────────────────────────────────────────
 DEFAULT_BACKEND: Final = BackendId.FILESYSTEM
 DEFAULT_FS_ROOT: Final = "./uploads"
+DEFAULT_KEY_PREFIX: Final = ""  # blank = bucket root, preserving legacy layout
+# Providers that ignore regions (R2, most MinIO deployments) still need *some*
+# region string for SigV4 to compute a signature, so we default rather than
+# require. ``us-east-1`` is the conventional filler.
+DEFAULT_S3_REGION: Final = "us-east-1"
+DEFAULT_ADDRESSING_STYLE: Final = AddressingStyle.AUTO
 DEFAULT_MAX_FILE_SIZE_BYTES: Final = 100 * 1024 * 1024  # 100 MB
 DEFAULT_PRESIGN_TTL_SECONDS: Final = 300  # 5 minutes
 DEFAULT_CHUNK_SIZE: Final = 64 * 1024  # 64 KB