diff --git a/.github/workflows/dockerpublish.yml b/.github/workflows/dockerpublish.yml index fdc2f425..2d6270c7 100644 --- a/.github/workflows/dockerpublish.yml +++ b/.github/workflows/dockerpublish.yml @@ -17,9 +17,10 @@ jobs: steps: - uses: actions/checkout@v4 - name: Build and Start Images + # Base file only: the tests mock Swift, so skip the local dev overlay. run: | cp template.env .env - docker compose up -d --build + docker compose -f docker-compose.yml up -d --build - name: Validate deploy overlay merge # CI builds and starts the base above; the toolkit additionally layers # docker-compose.deploy.yml. Validate that merge too so an overlay typo diff --git a/README.md b/README.md index eb95673c..46339fb3 100644 --- a/README.md +++ b/README.md @@ -60,133 +60,40 @@ You can view this report in your browser (located at `htmlcov/index.html`) by ru docker exec -ti externallinks-externallinks-1 coverage html ``` -## Swift object store - local setup (separate container) - -IMPORTANT: This setup should NOT be used in production! It is only suitable for local testing. - -Reference: https://github.com/CSCfi/docker-keystone-swift - -- Setup a separate single container for the Swift object store + Keystone auth - - In terminal, run: - ```bash - docker run --name externallinks-swift -d \ - --env S6_LOGGING=0 \ - -p 8080:8080 \ - -p 5000:5000 \ - -p 6000:6000 -p 6001:6001 -p 6002:6002 \ - -v externallinks_swift:/srv/node \ - ghcr.io/cscfi/docker-keystone-swift:latest - ``` - - This will create a new single container from a built image from ghcr.io, and will create a local volume (`-v`) so you don't lose everything when restarting the container - - `S6_LOGGING` is to enable openstack logging to the console (better visualization in Docker) - - Known possible errors: - - If you have a service already using one of those ports, you can remap as you wish `-p 5001:5000 \` - -- Connect the network between this new container and the Django `externallinks` stack - - When you run `docker network ls`, it is expected you'd get something like - ``` - NETWORK ID NAME DRIVER SCOPE - 12e998998050 externallinks_default bridge local - ``` - - `externallinks_default` is the network from the `externallinks` stack (the Django app), connect it to the new container - ``` - docker network connect externallinks_default externallinks-swift - ``` - - To make calls from Django to Swift (locally), you can now use the URL `http://externallinks-swift:{port}` - - The env var `OPENSTACK_AUTH_URL` can now be set to `http://externallinks-swift:5001/v3` (in the Django app) - - Also set the env var `SWIFT_CONTAINER_NAME` to the name of the Swift container you want to use, e.g. 'wikilink' - - You do not need to create this Swift container yourself as it will be created automatically -- Setup the new container - - Access the running container - ``` - docker exec -it externallinks-swift bash - ``` - - Create the wikilink project - ``` - openstack project create --domain default --description "Wikilink" wikilink - +-------------+----------------------------------+ - | Field | Value | - +-------------+----------------------------------+ - | description | Wikilink | - | domain_id | default | - | enabled | True | - | id | 544303a08953422b848a2393616adf63 | - | is_domain | False | - | name | wikilink | - | options | {} | - | parent_id | default | - | tags | [] | - +-------------+----------------------------------+ - ``` - - Create a new user (you can use one of the pre-configured users, skip this if you want) - ``` - openstack user create --domain default --password djangopass djangouser - +---------------------+----------------------------------+ - | Field | Value | - +---------------------+----------------------------------+ - | domain_id | default | - | enabled | True | - | id | ec41d7c9923e4b06821eca9fb40df2ab | - | name | djangouser | - | options | {} | - | password_expires_at | None | - +---------------------+----------------------------------+ - ``` - - Assign user to the project - ``` - openstack role add --project wikilink --user djangouser admin - ``` - - Update the environment variables in your current bash session (not .env) to start using the created user and project - ``` - export OS_USERNAME=djangouser - export OS_PASSWORD=djangopass - export OS_PROJECT_NAME=wikilink - ``` - - You can set these values as default in your Docker, otherwise you need to remember to set this whenever you want to bash into this container - - Create an application credential (copy the obtained `id` and `secret`) - ``` - openstack application credential create --role admin django_admin_credential - +--------------+----------------------------------------------------------------------------------------+ - | Field | Value | - +--------------+----------------------------------------------------------------------------------------+ - | description | None | - | expires_at | None | - | id | f85b36fcf5754ddb9fefe676b4e08286 | - | name | django_admin_credential | - | project_id | 544303a08953422b848a2393616adf63 | - | roles | admin | - | secret | LjEcrcS2A70DFYOxJayPuB336CqMPvUBr-8RWEV7anW3U-99bKPUqTf5jfwEr9dU-rhQt-WXRLfftGC04w4fzA | - | system | None | - | unrestricted | False | - | user_id | ec41d7c9923e4b06821eca9fb40df2ab | - +--------------+----------------------------------------------------------------------------------------+ - ``` - - Copy the credential id value to the env var `SWIFT_APPLICATION_CREDENTIAL_ID` in the Django app - - Copy the secret value to the env var `SWIFT_APPLICATION_CREDENTIAL_SECRET` in the Django app - - One final step now is to set the local Swift endpoint to the network we connected previously - - When you execute the command `openstack endpoint list`, you'd see something like - ``` - +----------------------------------+-----------+--------------+--------------+---------+-----------+--------------------------------------------+ - | ID | Region | Service Name | Service Type | Enabled | Interface | URL | - +----------------------------------+-----------+--------------+--------------+---------+-----------+--------------------------------------------+ - | a4b6a66d5b204afaab8afbe6d3521c8a | RegionOne | keystone | identity | True | public | http://localhost:5000/v3 | - | a98f5748c124489291747fcfb7a1a84a | RegionOne | keystone | identity | True | internal | http://localhost:5000/v3 | - | c9c933e896074f7880ff00b9aafec468 | RegionOne | keystone | identity | True | admin | http://localhost:5000/v3 | - | 3b5bdc053046452b943ed2d021cf1d3a | RegionOne | swift | object-store | True | internal | http://0.0.0.0:8080/v1/AUTH_%(project_id)s | - | b5f6dc8fde7d4cd59ce616edcbf47a58 | RegionOne | swift | object-store | True | admin | http://0.0.0.0:8080/v1 | - | 3af31d12aa1b40c2b24fbc9eeef57f6d | RegionOne | swift | object-store | True | public | http://0.0.0.0:8080/v1/AUTH_%(project_id)s | - +----------------------------------+-----------+--------------+--------------+---------+-----------+--------------------------------------------+ - ``` - - Notice how the URL for the Swift service are pointing to localhost, they need to target the connection we created - - Delete the existing Swift endpoints (pay attention to the IDs) - ``` - openstack endpoint delete 3b5bdc053046452b943ed2d021cf1d3a - openstack endpoint delete b5f6dc8fde7d4cd59ce616edcbf47a58 - openstack endpoint delete 3af31d12aa1b40c2b24fbc9eeef57f6d - ``` - - And finally, create the endpoints with the correct URL - ``` - openstack endpoint create --region RegionOne object-store public http://externallinks-swift:8080/v1/AUTH_%\(project_id\)s - openstack endpoint create --region RegionOne object-store internal http://externallinks-swift:8080/v1/AUTH_%\(project_id\)s - openstack endpoint create --region RegionOne object-store admin http://externallinks-swift:8080/v1 - ``` +## Swift object store (local) + +IMPORTANT: This setup is only for local testing, never production. + +A bare `docker compose up` also loads `docker-compose.override.yml`, which runs +a local Keystone + Swift ([CSCfi/docker-keystone-swift](https://github.com/CSCfi/docker-keystone-swift)) +as the `swift` service. The one-shot `swift-init` service then runs +`bin/swift_local_init.sh`, which creates the `wikilink` project, the +`djangouser` user, and the application credential named in `.env`, and points +the Swift endpoints at `http://swift:8080`. The defaults in `template.env` +already match, so the archive commands work without further setup. Swift +containers are created on first upload. An `.env` copied before this setup +existed needs the Swift block from `template.env` copied over it. + +The web app and crons wait for `swift-init` to finish before starting. +Objects and Keystone's database live in the `swift_objects` and +`swift_keystone` volumes, so like the database they survive +`docker compose down` and go away with `docker compose down -v`. + +The image is pinned because it never migrates Keystone's database on its +own. After bumping both image tags in `docker-compose.override.yml`, +migrate it once: + +```bash +docker compose up -d swift +docker compose exec swift su -s /bin/sh -c "keystone-manage db_sync" keystone +docker compose restart swift +``` + +To look at what's stored: + +```bash +docker compose exec -e OS_USERNAME=djangouser -e OS_PASSWORD=djangopass \ + -e OS_PROJECT_NAME=wikilink swift swift list archive-linkevents +``` + +If `swift-init` fails, `docker compose logs swift swift-init` shows why. diff --git a/bin/swift_local_init.sh b/bin/swift_local_init.sh new file mode 100755 index 00000000..790990ff --- /dev/null +++ b/bin/swift_local_init.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# Prepares the local dev Keystone + Swift container (see +# docker-compose.override.yml) for Django. Safe to rerun: it creates only +# what's missing and recreates the application credential so its secret +# always matches .env. Never point this at a real Keystone. +set -euo pipefail + +project=wikilink +username="${SWIFT_APPLICATION_CREDENTIAL_USERNAME:?}" +password=djangopass +credential="${SWIFT_APPLICATION_CREDENTIAL_NAME:?}" +secret="${SWIFT_APPLICATION_CREDENTIAL_SECRET:?}" +swift_url=http://swift:8080/v1 + +openstack project show "$project" >/dev/null 2>&1 \ + || openstack project create --domain default --description Wikilink "$project" +openstack user show "$username" >/dev/null 2>&1 \ + || openstack user create --domain default --password "$password" "$username" +openstack role add --project "$project" --user "$username" admin + +# Application credentials belong to the user, so manage them as that user. +as_user=(openstack --os-username "$username" --os-password "$password" + --os-project-name "$project") +"${as_user[@]}" application credential delete "$credential" >/dev/null 2>&1 || true +"${as_user[@]}" application credential create --role admin --secret "$secret" "$credential" >/dev/null + +# The image registers Swift at 0.0.0.0:8080, which other containers can't reach. +openstack endpoint list --service object-store -f value -c ID -c Interface \ + | while read -r id interface; do + if [ "$interface" = admin ]; then + url="$swift_url" + else + url="$swift_url/AUTH_%(project_id)s" + fi + openstack endpoint set --url "$url" "$id" + done + +echo "Local Swift ready: project $project, credential $credential for $username" diff --git a/docker-compose.override.yml b/docker-compose.override.yml new file mode 100644 index 00000000..d0654e25 --- /dev/null +++ b/docker-compose.override.yml @@ -0,0 +1,57 @@ +--- +# Local dev overlay, loaded automatically by a bare `docker compose` command. +# Deployed instances pin COMPOSE_FILE to the base plus docker-compose.deploy.yml, +# so nothing here reaches production. +# +# Runs a local Keystone + Swift for the archive commands to talk to, kept in +# volumes like the database so archives survive `docker compose down`. +# swift-init reruns on every `up` to make sure the project, user, credential, +# and endpoints match .env. +volumes: + swift_objects: + swift_keystone: + +services: + externallinks: + depends_on: + swift-init: + condition: service_completed_successfully + crons: + depends_on: + swift-init: + condition: service_completed_successfully + swift: + # Pinned because Keystone's database persists and the image never + # migrates it; see the README before bumping. + image: ghcr.io/cscfi/docker-keystone-swift:2026.9.0 + healthcheck: + test: + [ + "CMD-SHELL", + "curl -sf http://localhost:5000/v3 && curl -sf http://localhost:8080/healthcheck", + ] + interval: 5s + timeout: 5s + retries: 30 + volumes: + - type: volume + source: swift_objects + target: /srv/node + - type: volume + source: swift_keystone + target: /var/lib/keystone + swift-init: + image: ghcr.io/cscfi/docker-keystone-swift:2026.9.0 + entrypoint: ["/bin/bash", "/app/bin/swift_local_init.sh"] + env_file: + - ".env" + # Keystone's catalog lists itself at localhost:5000, so share swift's + # network for the openstack CLI to follow it. + network_mode: service:swift + depends_on: + swift: + condition: service_healthy + volumes: + - type: bind + source: ./bin/swift_local_init.sh + target: /app/bin/swift_local_init.sh diff --git a/extlinks/common/swift.py b/extlinks/common/swift.py index 1be309e7..9b96da39 100644 --- a/extlinks/common/swift.py +++ b/extlinks/common/swift.py @@ -19,6 +19,9 @@ def swift_connection() -> swiftclient.Connection: Creates a swiftclient Connection configured using environment variables. This method works with v3 application credentials authentication only. + The credential is identified by its ID, or by its name plus the owning + username when no ID is set. The local dev Swift container uses the + latter, since Keystone generates IDs we can't know ahead of time. Returns ------- @@ -28,13 +31,18 @@ def swift_connection() -> swiftclient.Connection: auth_url = os.environ.get("OPENSTACK_AUTH_URL") credential_id = os.environ.get("SWIFT_APPLICATION_CREDENTIAL_ID") + credential_name = os.environ.get("SWIFT_APPLICATION_CREDENTIAL_NAME") + credential_username = os.environ.get("SWIFT_APPLICATION_CREDENTIAL_USERNAME") credential_secret = os.environ.get("SWIFT_APPLICATION_CREDENTIAL_SECRET") - if not auth_url or not credential_id or not credential_secret: + has_identifier = credential_id or (credential_name and credential_username) + if not auth_url or not has_identifier or not credential_secret: raise RuntimeError( - "The 'OPENSTACK_AUTH_URL', 'SWIFT_APPLICATION_CREDENTIAL_ID' and " - "'SWIFT_APPLICATION_CREDENTIAL_SECRET' environment variables must " - "be defined to use the Swift client" + "The 'OPENSTACK_AUTH_URL' and 'SWIFT_APPLICATION_CREDENTIAL_SECRET' " + "environment variables must be defined, along with either " + "'SWIFT_APPLICATION_CREDENTIAL_ID' or both " + "'SWIFT_APPLICATION_CREDENTIAL_NAME' and " + "'SWIFT_APPLICATION_CREDENTIAL_USERNAME', to use the Swift client" ) return swiftclient.Connection( @@ -42,6 +50,8 @@ def swift_connection() -> swiftclient.Connection: auth=identity.ApplicationCredential( auth_url=auth_url, application_credential_id=credential_id, + application_credential_name=credential_name, + username=credential_username, application_credential_secret=credential_secret, user_domain_id="default", ) diff --git a/extlinks/common/tests.py b/extlinks/common/tests.py index a5b4d833..cbdf130f 100644 --- a/extlinks/common/tests.py +++ b/extlinks/common/tests.py @@ -23,6 +23,13 @@ "SWIFT_APPLICATION_CREDENTIAL_SECRET": "fakecredentialsecret", } +SWIFT_TEST_CREDENTIALS_BY_NAME = { + "OPENSTACK_AUTH_URL": "fakeauthurl", + "SWIFT_APPLICATION_CREDENTIAL_NAME": "fakecredentialname", + "SWIFT_APPLICATION_CREDENTIAL_USERNAME": "fakeusername", + "SWIFT_APPLICATION_CREDENTIAL_SECRET": "fakecredentialsecret", +} + class LinkSearchDataByTimeTest(TestCase): def setUp(self): @@ -160,11 +167,29 @@ def test_swift_connection(self): conn = swift.swift_connection() self.assertIsInstance(conn, swiftclient.Connection) + @mock.patch.dict(os.environ, SWIFT_TEST_CREDENTIALS_BY_NAME, clear=True) + def test_swift_connection_by_name(self): + conn = swift.swift_connection() + self.assertIsInstance(conn, swiftclient.Connection) + @mock.patch.dict(os.environ, {}, clear=True) def test_swift_connection_validation(self): with self.assertRaises(RuntimeError): swift.swift_connection() + @mock.patch.dict( + os.environ, + { + k: v + for k, v in SWIFT_TEST_CREDENTIALS_BY_NAME.items() + if k != "SWIFT_APPLICATION_CREDENTIAL_USERNAME" + }, + clear=True, + ) + def test_swift_connection_validation_name_without_username(self): + with self.assertRaises(RuntimeError): + swift.swift_connection() + class SwiftUploadTest(TestCase): def setUp(self): diff --git a/template.env b/template.env index 83043bea..41d15fa7 100644 --- a/template.env +++ b/template.env @@ -16,12 +16,16 @@ EXTERNALLINKS_TAG=latest # Change to the host backup path on real servers (/usr/local/backup on a # compose-deploy toolkit instance). HOST_BACKUP_DIR=./backup -# Swift Object Store (for storing archives), `local` will just ignore Swift usage -SWIFT_APPLICATION_CREDENTIAL_ID=local -SWIFT_APPLICATION_CREDENTIAL_SECRET=local +# Swift Object Store (for storing archives). The defaults match the local +# Swift container from docker-compose.override.yml; the secret is local-only. +# In production, set the credential ID and secret, and leave NAME and USERNAME +# alone (an ID takes precedence over them). +SWIFT_APPLICATION_CREDENTIAL_ID= +SWIFT_APPLICATION_CREDENTIAL_NAME=django_admin_credential +SWIFT_APPLICATION_CREDENTIAL_USERNAME=djangouser +SWIFT_APPLICATION_CREDENTIAL_SECRET=local-dev-only-secret SWIFT_CONTAINER_NAME=archive-linkevents SWIFT_CONTAINER_AGGREGATES=archive-aggregates -# See the readme file for setting up a local swift store # In production, it should use the known URL https://openstack.eqiad1.wikimediacloud.org:25000/v3 -OPENSTACK_AUTH_URL=http://externallinks-swift:5001/v3 +OPENSTACK_AUTH_URL=http://swift:5000/v3 LINKEVENTS_ARCHIVE_OBJECT_STORAGE_ONLY=false