Make the local Swift store work out of the box

.github/workflows/dockerpublish.yml

@@ -17,9 +17,10 @@ jobs:
     steps:
       - uses: actions/checkout@v4
       - name: Build and Start Images
+        # Base file only: the tests mock Swift, so skip the local dev overlay.
         run: |
           cp template.env .env
-          docker compose up -d --build
+          docker compose -f docker-compose.yml up -d --build
       - name: Validate deploy overlay merge
         # CI builds and starts the base above; the toolkit additionally layers
         # docker-compose.deploy.yml. Validate that merge too so an overlay typo

README.md

@@ -60,133 +60,40 @@ You can view this report in your browser (located at `htmlcov/index.html`) by ru
 docker exec -ti externallinks-externallinks-1 coverage html
 ```
 
-## Swift object store - local setup (separate container)
-
-IMPORTANT: This setup should NOT be used in production! It is only suitable for local testing.
-
-Reference: https://github.com/CSCfi/docker-keystone-swift
-
-- Setup a separate single container for the Swift object store + Keystone auth
-  - In terminal, run:
-  ```bash
-  docker run --name externallinks-swift -d \
-    --env S6_LOGGING=0 \
-    -p 8080:8080 \
-    -p 5000:5000 \
-    -p 6000:6000 -p 6001:6001 -p 6002:6002 \
-    -v externallinks_swift:/srv/node \
-    ghcr.io/cscfi/docker-keystone-swift:latest
-  ```
-  - This will create a new single container from a built image from ghcr.io, and will create a local volume (`-v`) so you don't lose everything when restarting the container
-  - `S6_LOGGING` is to enable openstack logging to the console (better visualization in Docker)
-  - Known possible errors:
-    - If you have a service already using one of those ports, you can remap as you wish `-p 5001:5000 \`
-
-- Connect the network between this new container and the Django `externallinks` stack
-  - When you run `docker network ls`, it is expected you'd get something like
-    ```
-    NETWORK ID     NAME                            DRIVER    SCOPE
-    12e998998050   externallinks_default           bridge    local
-    ```
-  - `externallinks_default` is the network from the `externallinks` stack (the Django app), connect it to the new container
-    ```
-    docker network connect externallinks_default externallinks-swift
-    ```
-  - To make calls from Django to Swift (locally), you can now use the URL `http://externallinks-swift:{port}`
-    - The env var `OPENSTACK_AUTH_URL` can now be set to `http://externallinks-swift:5001/v3` (in the Django app)
-    - Also set the env var `SWIFT_CONTAINER_NAME` to the name of the Swift container you want to use, e.g. 'wikilink'
-      - You do not need to create this Swift container yourself as it will be created automatically
-- Setup the new container
-  - Access the running container
-    ```
-    docker exec -it externallinks-swift bash
-    ```
-  - Create the wikilink project
-    ```
-    openstack project create --domain default --description "Wikilink" wikilink
-    +-------------+----------------------------------+
-    | Field       | Value                            |
-    +-------------+----------------------------------+
-    | description | Wikilink                         |
-    | domain_id   | default                          |
-    | enabled     | True                             |
-    | id          | 544303a08953422b848a2393616adf63 |
-    | is_domain   | False                            |
-    | name        | wikilink                         |
-    | options     | {}                               |
-    | parent_id   | default                          |
-    | tags        | []                               |
-    +-------------+----------------------------------+
-    ```
-  - Create a new user (you can use one of the pre-configured users, skip this if you want)
-    ```
-    openstack user create --domain default --password djangopass djangouser
-    +---------------------+----------------------------------+
-    | Field               | Value                            |
-    +---------------------+----------------------------------+
-    | domain_id           | default                          |
-    | enabled             | True                             |
-    | id                  | ec41d7c9923e4b06821eca9fb40df2ab |
-    | name                | djangouser                       |
-    | options             | {}                               |
-    | password_expires_at | None                             |
-    +---------------------+----------------------------------+
-    ```
-  - Assign user to the project
-    ```
-    openstack role add --project wikilink --user djangouser admin
-    ```
-  - Update the environment variables in your current bash session (not .env) to start using the created user and project
-    ```
-    export OS_USERNAME=djangouser
-    export OS_PASSWORD=djangopass
-    export OS_PROJECT_NAME=wikilink
-    ```
-    - You can set these values as default in your Docker, otherwise you need to remember to set this whenever you want to bash into this container
-  - Create an application credential (copy the obtained `id` and `secret`)
-    ```
-    openstack application credential create --role admin django_admin_credential
-    +--------------+----------------------------------------------------------------------------------------+
-    | Field        | Value                                                                                  |
-    +--------------+----------------------------------------------------------------------------------------+
-    | description  | None                                                                                   |
-    | expires_at   | None                                                                                   |
-    | id           | f85b36fcf5754ddb9fefe676b4e08286                                                       |
-    | name         | django_admin_credential                                                                |
-    | project_id   | 544303a08953422b848a2393616adf63                                                       |
-    | roles        | admin                                                                                  |
-    | secret       | LjEcrcS2A70DFYOxJayPuB336CqMPvUBr-8RWEV7anW3U-99bKPUqTf5jfwEr9dU-rhQt-WXRLfftGC04w4fzA |
-    | system       | None                                                                                   |
-    | unrestricted | False                                                                                  |
-    | user_id      | ec41d7c9923e4b06821eca9fb40df2ab                                                       |
-    +--------------+----------------------------------------------------------------------------------------+
-    ```
-    - Copy the credential id value to the env var `SWIFT_APPLICATION_CREDENTIAL_ID` in the Django app
-    - Copy the secret value to the env var `SWIFT_APPLICATION_CREDENTIAL_SECRET` in the Django app
-  - One final step now is to set the local Swift endpoint to the network we connected previously
-    - When you execute the command `openstack endpoint list`, you'd see something like
-      ```
-      +----------------------------------+-----------+--------------+--------------+---------+-----------+--------------------------------------------+
-      | ID                               | Region    | Service Name | Service Type | Enabled | Interface | URL                                        |
-      +----------------------------------+-----------+--------------+--------------+---------+-----------+--------------------------------------------+
-      | a4b6a66d5b204afaab8afbe6d3521c8a | RegionOne | keystone     | identity     | True    | public    | http://localhost:5000/v3                   |
-      | a98f5748c124489291747fcfb7a1a84a | RegionOne | keystone     | identity     | True    | internal  | http://localhost:5000/v3                   |
-      | c9c933e896074f7880ff00b9aafec468 | RegionOne | keystone     | identity     | True    | admin     | http://localhost:5000/v3                   |
-      | 3b5bdc053046452b943ed2d021cf1d3a | RegionOne | swift        | object-store | True    | internal  | http://0.0.0.0:8080/v1/AUTH_%(project_id)s |
-      | b5f6dc8fde7d4cd59ce616edcbf47a58 | RegionOne | swift        | object-store | True    | admin     | http://0.0.0.0:8080/v1                     |
-      | 3af31d12aa1b40c2b24fbc9eeef57f6d | RegionOne | swift        | object-store | True    | public    | http://0.0.0.0:8080/v1/AUTH_%(project_id)s |
-      +----------------------------------+-----------+--------------+--------------+---------+-----------+--------------------------------------------+
-      ```
-    - Notice how the URL for the Swift service are pointing to localhost, they need to target the connection we created
-    - Delete the existing Swift endpoints (pay attention to the IDs)
-      ```
-      openstack endpoint delete 3b5bdc053046452b943ed2d021cf1d3a
-      openstack endpoint delete b5f6dc8fde7d4cd59ce616edcbf47a58
-      openstack endpoint delete 3af31d12aa1b40c2b24fbc9eeef57f6d
-      ```
-    - And finally, create the endpoints with the correct URL
-      ```
-      openstack endpoint create --region RegionOne object-store public http://externallinks-swift:8080/v1/AUTH_%\(project_id\)s
-      openstack endpoint create --region RegionOne object-store internal http://externallinks-swift:8080/v1/AUTH_%\(project_id\)s
-      openstack endpoint create --region RegionOne object-store admin http://externallinks-swift:8080/v1
-      ```
+## Swift object store (local)
+
+IMPORTANT: This setup is only for local testing, never production.
+
+A bare `docker compose up` also loads `docker-compose.override.yml`, which runs
+a local Keystone + Swift ([CSCfi/docker-keystone-swift](https://github.com/CSCfi/docker-keystone-swift))
+as the `swift` service. The one-shot `swift-init` service then runs
+`bin/swift_local_init.sh`, which creates the `wikilink` project, the
+`djangouser` user, and the application credential named in `.env`, and points
+the Swift endpoints at `http://swift:8080`. The defaults in `template.env`
+already match, so the archive commands work without further setup. Swift
+containers are created on first upload. An `.env` copied before this setup
+existed needs the Swift block from `template.env` copied over it.
+
+The web app and crons wait for `swift-init` to finish before starting.
+Objects and Keystone's database live in the `swift_objects` and
+`swift_keystone` volumes, so like the database they survive
+`docker compose down` and go away with `docker compose down -v`.
+
+The image is pinned because it never migrates Keystone's database on its
+own. After bumping both image tags in `docker-compose.override.yml`,
+migrate it once:
+
+```bash
+docker compose up -d swift
+docker compose exec swift su -s /bin/sh -c "keystone-manage db_sync" keystone
+docker compose restart swift
+```
+
+To look at what's stored:
+
+```bash
+docker compose exec -e OS_USERNAME=djangouser -e OS_PASSWORD=djangopass \
+  -e OS_PROJECT_NAME=wikilink swift swift list archive-linkevents
+```
+
+If `swift-init` fails, `docker compose logs swift swift-init` shows why.

bin/swift_local_init.sh

@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+# Prepares the local dev Keystone + Swift container (see
+# docker-compose.override.yml) for Django. Safe to rerun: it creates only
+# what's missing and recreates the application credential so its secret
+# always matches .env. Never point this at a real Keystone.
+set -euo pipefail
+
+project=wikilink
+username="${SWIFT_APPLICATION_CREDENTIAL_USERNAME:?}"
+password=djangopass
+credential="${SWIFT_APPLICATION_CREDENTIAL_NAME:?}"
+secret="${SWIFT_APPLICATION_CREDENTIAL_SECRET:?}"
+swift_url=http://swift:8080/v1
+
+openstack project show "$project" >/dev/null 2>&1 \
+    || openstack project create --domain default --description Wikilink "$project"
+openstack user show "$username" >/dev/null 2>&1 \
+    || openstack user create --domain default --password "$password" "$username"
+openstack role add --project "$project" --user "$username" admin
+
+# Application credentials belong to the user, so manage them as that user.
+as_user=(openstack --os-username "$username" --os-password "$password"
+    --os-project-name "$project")
+"${as_user[@]}" application credential delete "$credential" >/dev/null 2>&1 || true
+"${as_user[@]}" application credential create --role admin --secret "$secret" "$credential" >/dev/null
+
+# The image registers Swift at 0.0.0.0:8080, which other containers can't reach.
+openstack endpoint list --service object-store -f value -c ID -c Interface \
+    | while read -r id interface; do
+        if [ "$interface" = admin ]; then
+            url="$swift_url"
+        else
+            url="$swift_url/AUTH_%(project_id)s"
+        fi
+        openstack endpoint set --url "$url" "$id"
+    done
+
+echo "Local Swift ready: project $project, credential $credential for $username"

docker-compose.override.yml

@@ -0,0 +1,57 @@
+---
+# Local dev overlay, loaded automatically by a bare `docker compose` command.
+# Deployed instances pin COMPOSE_FILE to the base plus docker-compose.deploy.yml,
+# so nothing here reaches production.
+#
+# Runs a local Keystone + Swift for the archive commands to talk to, kept in
+# volumes like the database so archives survive `docker compose down`.
+# swift-init reruns on every `up` to make sure the project, user, credential,
+# and endpoints match .env.
+volumes:
+  swift_objects:
+  swift_keystone:
+
+services:
+  externallinks:
+    depends_on:
+      swift-init:
+        condition: service_completed_successfully
+  crons:
+    depends_on:
+      swift-init:
+        condition: service_completed_successfully
+  swift:
+    # Pinned because Keystone's database persists and the image never
+    # migrates it; see the README before bumping.
+    image: ghcr.io/cscfi/docker-keystone-swift:2026.9.0
+    healthcheck:
+      test:
+        [
+          "CMD-SHELL",
+          "curl -sf http://localhost:5000/v3 && curl -sf http://localhost:8080/healthcheck",
+        ]
+      interval: 5s
+      timeout: 5s
+      retries: 30
+    volumes:
+      - type: volume
+        source: swift_objects
+        target: /srv/node
+      - type: volume
+        source: swift_keystone
+        target: /var/lib/keystone
+  swift-init:
+    image: ghcr.io/cscfi/docker-keystone-swift:2026.9.0
+    entrypoint: ["/bin/bash", "/app/bin/swift_local_init.sh"]
+    env_file:
+      - ".env"
+    # Keystone's catalog lists itself at localhost:5000, so share swift's
+    # network for the openstack CLI to follow it.
+    network_mode: service:swift
+    depends_on:
+      swift:
+        condition: service_healthy
+    volumes:
+      - type: bind
+        source: ./bin/swift_local_init.sh
+        target: /app/bin/swift_local_init.sh

extlinks/common/swift.py

@@ -19,6 +19,9 @@ def swift_connection() -> swiftclient.Connection:
     Creates a swiftclient Connection configured using environment variables.
 
     This method works with v3 application credentials authentication only.
+    The credential is identified by its ID, or by its name plus the owning
+    username when no ID is set. The local dev Swift container uses the
+    latter, since Keystone generates IDs we can't know ahead of time.
 
     Returns
     -------
@@ -28,20 +31,27 @@ def swift_connection() -> swiftclient.Connection:
 
     auth_url = os.environ.get("OPENSTACK_AUTH_URL")
     credential_id = os.environ.get("SWIFT_APPLICATION_CREDENTIAL_ID")
+    credential_name = os.environ.get("SWIFT_APPLICATION_CREDENTIAL_NAME")
+    credential_username = os.environ.get("SWIFT_APPLICATION_CREDENTIAL_USERNAME")
     credential_secret = os.environ.get("SWIFT_APPLICATION_CREDENTIAL_SECRET")
 
-    if not auth_url or not credential_id or not credential_secret:
+    has_identifier = credential_id or (credential_name and credential_username)
+    if not auth_url or not has_identifier or not credential_secret:
         raise RuntimeError(
-            "The 'OPENSTACK_AUTH_URL', 'SWIFT_APPLICATION_CREDENTIAL_ID' and "
-            "'SWIFT_APPLICATION_CREDENTIAL_SECRET' environment variables must "
-            "be defined to use the Swift client"
+            "The 'OPENSTACK_AUTH_URL' and 'SWIFT_APPLICATION_CREDENTIAL_SECRET' "
+            "environment variables must be defined, along with either "
+            "'SWIFT_APPLICATION_CREDENTIAL_ID' or both "
+            "'SWIFT_APPLICATION_CREDENTIAL_NAME' and "
+            "'SWIFT_APPLICATION_CREDENTIAL_USERNAME', to use the Swift client"
         )
 
     return swiftclient.Connection(
         session=session.Session(
             auth=identity.ApplicationCredential(
                 auth_url=auth_url,
                 application_credential_id=credential_id,
+                application_credential_name=credential_name,
+                username=credential_username,
                 application_credential_secret=credential_secret,
                 user_domain_id="default",
             )

extlinks/common/tests.py

@@ -23,6 +23,13 @@
     "SWIFT_APPLICATION_CREDENTIAL_SECRET": "fakecredentialsecret",
 }
 
+SWIFT_TEST_CREDENTIALS_BY_NAME = {
+    "OPENSTACK_AUTH_URL": "fakeauthurl",
+    "SWIFT_APPLICATION_CREDENTIAL_NAME": "fakecredentialname",
+    "SWIFT_APPLICATION_CREDENTIAL_USERNAME": "fakeusername",
+    "SWIFT_APPLICATION_CREDENTIAL_SECRET": "fakecredentialsecret",
+}
+
 
 class LinkSearchDataByTimeTest(TestCase):
     def setUp(self):
@@ -160,11 +167,29 @@ def test_swift_connection(self):
         conn = swift.swift_connection()
         self.assertIsInstance(conn, swiftclient.Connection)
 
+    @mock.patch.dict(os.environ, SWIFT_TEST_CREDENTIALS_BY_NAME, clear=True)
+    def test_swift_connection_by_name(self):
+        conn = swift.swift_connection()
+        self.assertIsInstance(conn, swiftclient.Connection)
+
     @mock.patch.dict(os.environ, {}, clear=True)
     def test_swift_connection_validation(self):
         with self.assertRaises(RuntimeError):
             swift.swift_connection()
 
+    @mock.patch.dict(
+        os.environ,
+        {
+            k: v
+            for k, v in SWIFT_TEST_CREDENTIALS_BY_NAME.items()
+            if k != "SWIFT_APPLICATION_CREDENTIAL_USERNAME"
+        },
+        clear=True,
+    )
+    def test_swift_connection_validation_name_without_username(self):
+        with self.assertRaises(RuntimeError):
+            swift.swift_connection()
+
 
 class SwiftUploadTest(TestCase):
     def setUp(self):

template.env

@@ -16,12 +16,16 @@ EXTERNALLINKS_TAG=latest
 # Change to the host backup path on real servers (/usr/local/backup on a
 # compose-deploy toolkit instance).
 HOST_BACKUP_DIR=./backup
-# Swift Object Store (for storing archives), `local` will just ignore Swift usage
-SWIFT_APPLICATION_CREDENTIAL_ID=local
-SWIFT_APPLICATION_CREDENTIAL_SECRET=local
+# Swift Object Store (for storing archives). The defaults match the local
+# Swift container from docker-compose.override.yml; the secret is local-only.
+# In production, set the credential ID and secret, and leave NAME and USERNAME
+# alone (an ID takes precedence over them).
+SWIFT_APPLICATION_CREDENTIAL_ID=
+SWIFT_APPLICATION_CREDENTIAL_NAME=django_admin_credential
+SWIFT_APPLICATION_CREDENTIAL_USERNAME=djangouser
+SWIFT_APPLICATION_CREDENTIAL_SECRET=local-dev-only-secret
 SWIFT_CONTAINER_NAME=archive-linkevents
 SWIFT_CONTAINER_AGGREGATES=archive-aggregates
-# See the readme file for setting up a local swift store
 # In production, it should use the known URL https://openstack.eqiad1.wikimediacloud.org:25000/v3
-OPENSTACK_AUTH_URL=http://externallinks-swift:5001/v3
+OPENSTACK_AUTH_URL=http://swift:5000/v3
 LINKEVENTS_ARCHIVE_OBJECT_STORAGE_ONLY=false