diff --git a/.env.example b/.env.example index 46a5703a..397a93f5 100644 --- a/.env.example +++ b/.env.example @@ -109,6 +109,18 @@ OPENREPL_MONGODB_DB=openrepl # worker never uses this. OPENREPL_FIRESTORE_CREDENTIALS= +# Optional. The service account key (the same kind of key: JSON, its base64, or the +# path of the file) that the server signs in to the Firebase Realtime Database +# with, for the cleanup of shared sessions: every so many hours (Site settings, +# default 24) it removes the sessions that ended, ran out or were left behind, from +# openrepl, editor_values, chat-list and file-browser. The Firebase Admin SDK key of +# the project (Project settings, Service accounts, Generate new private key) works +# for both this and OPENREPL_FIRESTORE_CREDENTIALS, so the same key can be given to +# both; when this one is empty the server uses OPENREPL_FIRESTORE_CREDENTIALS. With +# neither the cleanup does not run (the admin Health page says so) and ended +# sessions are removed only by the pages that see them. A worker never uses this. +OPENREPL_FIREBASE_ADMIN= + # Optional. A long random secret for the server's own use. It signs the session # cookies (through a key derived from it) and encrypts the API keys an admin # saves in the dashboard (Site settings; a key saved there wins over the ones @@ -135,10 +147,12 @@ OPENREPL_HOST=localhost # Delete from /tmp what nobody has touched for this long (the Docker image sets 1h). #OPENREPL_TMP_MAX_AGE=1h -# The Firebase project the page signs in with. Leave it out to use the built-in -# (production) project. To test sign-in with a development project of your own, -# give its web app config (Firebase console > Project settings > Your apps) in -# any of these forms. apiKey, authDomain and projectId are required: +# The Firebase project the page signs in with and shares sessions through. There is +# no built-in project: without this the server says so in its log and in the console of +# the page, and sign-in and shared sessions do not work. Give the web app config +# (Firebase console > Project settings > Your apps) of your project, or of a +# development project of your own, in any of these forms. apiKey, authDomain and +# projectId are required: # # - JSON on one line (the single quotes are for gotty's own reading; leave them # out for `docker run --env-file`): @@ -147,8 +161,7 @@ OPENREPL_HOST=localhost # - the snippet the console shows, on one line, in quotes: # OPENREPL_FIREBASE_CONFIG='const firebaseConfig = { apiKey: "AIza...", authDomain: "my-dev.firebaseapp.com", projectId: "my-dev" };' # -# - base64 of either, which has no quotes. This is also the form of the -# server's built-in config. Make it with: base64 < firebase-dev.txt | tr -d '\n' +# - base64 of either, which has no quotes. Make it with: base64 < firebase-dev.txt | tr -d '\n' # OPENREPL_FIREBASE_CONFIG=eyJhcGlLZXkiOiJBSXphLi4uIiwiYXV0aERvbWFpbiI6Ii4uLiIsInByb2plY3RJZCI6Ii4uLiJ9 # # A wrong value stops the server at start-up and says which field is wrong. diff --git a/README.md b/README.md index 01c47e17..f1d0fcbd 100755 --- a/README.md +++ b/README.md @@ -646,6 +646,7 @@ The server-side settings come from environment variables, and gotty can load the | `OPENREPL_ADMIN_EMAILS` | Owner accounts, comma-separated. Owners are admins and the only ones who can add or remove other admins in the dashboard. With none set, nobody is an admin. | `user.email` | | `OPENREPL_OPENAI_API_KEY` | The OpenAI key, as it is (not base64). | `user.OpenaiAPIKey` (base64) | | `OPENREPL_MONGODB_URI` | Optional. Keeps the admin settings and every database (sessions, feedback, blog, snippets, practice) in MongoDB (for example an Atlas `mongodb+srv://` URI) instead of files under `/opt/gotty`, so they survive deploys that wipe the disk. The first start copies the existing files in, once. A secret. Gateway and standalone only; a worker keeps files. | | +| `OPENREPL_FIREBASE_ADMIN` | Optional. A Google service account key (the JSON, its base64, or a file path) that the server signs in to the Firebase Realtime Database with, for the cleanup of shared sessions (every `sessionCleanupHours`, daily by default; an admin setting). Without it the server falls back to the key of `OPENREPL_FIRESTORE_CREDENTIALS`, and without either the cleanup does not run (the Health page says so) and ended sessions are removed only by the pages that see them. | | | `OPENREPL_FIRESTORE_CREDENTIALS` | Optional. A Google service account key (the JSON, its base64, or a file path) for the Firebase project's Firestore. Used when `OPENREPL_MONGODB_URI` is not set and Firestore answers; otherwise the server uses files. Keep the security rules of `kv_*` and `settings` closed. | | | `OPENREPL_FIRESTORE_PROJECT` | The project, when the key does not name it (the emulator). | | | `OPENREPL_MONGODB_DB` | The MongoDB database for it. Default `openrepl`. | | @@ -659,7 +660,7 @@ The server-side settings come from environment variables, and gotty can load the | `OPENREPL_LANDLOCK` | The filesystem sandbox of the REPLs (LLD 14): `auto` (the default) applies it where the kernel has Landlock and says in the log and on the admin health page where it does not, `off` switches it off, `require` refuses to start a REPL without it. | | | `OPENREPL_NET_ISOLATION` | The private network of a workspace for REPLs that have no kernel network namespace of their own (LLD 16): `auto` (the default) turns it on where a start-up probe passes and says in the log and on the admin health page where it does not, `off` switches it off, `require` refuses to start a REPL without it. | | | `OPENREPL_TMP_MAX_AGE` | A duration such as `1h`: the server deletes from `/tmp` what nobody has touched for that long. The Docker image sets `1h`. Not set: never. | | -| `OPENREPL_FIREBASE_CONFIG` | The Firebase project the page signs in with, as JSON or base64 of JSON. Default: the built-in production project. See below. | | +| `OPENREPL_FIREBASE_CONFIG` | The Firebase project the page signs in with, as JSON or base64 of JSON. Required for sign-in, shared sessions and the shared editor: there is no built-in project, and a server without it says so in its log and in the console of the page. See below. | | | `OPENREPL_ENV` | `dev` or `production` (the default). Dev shows the values of these settings in the start-up log; production only says which are set. | | The environment wins over the file. A variable that is empty counts as not set, so a half-filled `.env` does not switch the file off. The file is looked for in `/opt/gotty/.gitconfig`, then `~/.gitconfig`, then `/etc/.gitconfig`, and only the first one that exists is read. @@ -726,7 +727,7 @@ Sign-in goes through Firebase, so to test it without touching the production pro - **The snippet exactly as the console shows it** (`const firebaseConfig = { apiKey: "...", ... };`, with or without the first line). Nothing in it is run; only the `key: "value"` pairs are read. - **JSON on one line:** `{"apiKey":"AIza...","authDomain":"my-openrepl-dev.firebaseapp.com","projectId":"my-openrepl-dev","databaseURL":"https://my-openrepl-dev-default-rtdb.firebaseio.com"}`. - - **The base64 of either**, which has no quotes in it. The server's own built-in config is kept in this form, base64 of the snippet, so the same text works here. To make it, paste the snippet into a file and run `base64 < firebase-dev.txt | tr -d '\n'`. + - **The base64 of either**, which has no quotes in it. To make it, paste the snippet into a file and run `base64 < firebase-dev.txt | tr -d '\n'`. ```bash # ~/.env diff --git a/docs/lld/01-server-and-routing.md b/docs/lld/01-server-and-routing.md index 510ddc4f..2c594fb9 100644 --- a/docs/lld/01-server-and-routing.md +++ b/docs/lld/01-server-and-routing.md @@ -98,7 +98,7 @@ All paths are relative to `pathPrefix`: `/`, or `//` with `--random-url` | `/practice/dsa-questions` | GET | `handlePracticeQuestions` | public | Renders `static/practice.html`. | | `/practice/progress` | GET, PUT (POST accepted) | `handlePracticeProgress` | session | A signed-in user's practice questions and done marks (LLD 05). GET returns the stored document, or `{"signedIn": false}` for guests. PUT merges the body into the stored copy and returns the result; guests get 401. Bodies up to 2 MB, 200 questions, 1,000 marks; 240 writes per user per 10 minutes. | | `/sitemap.xml` | GET | `handleSitemap` | public | The home page, practice list, docs, about, privacy and terms pages, and every language page. `robots.txt` points to it. | -| `/snippet` | GET, POST | `handleSnippet` | public, 30 new links per IP per 10 minutes | Share-code links. POST `{lang, code}` (code up to 64 KB) stores a snapshot and returns `{id, url}`; GET `?id=` returns `{lang, code, created}`. Errors: 400 unknown language or no code, 413 too large, 429 rate limit, 503 store unavailable. | +| `/snippet` | GET, POST | `handleSnippet` | public, 30 new links per IP per 10 minutes | Share-code links. POST `{lang, code, expires}` (code up to 64 KB; `expires` the last day the link works, `YYYY-MM-DD` in UTC, from today up to today plus the admin's most, `snippetMaxDays`, default 30; empty means a month, or the most when shorter) stores a snapshot with an end (23:59:59 UTC of that day) and returns `{id, url, expires}`; GET `?id=` returns `{lang, code, created, expires}`, or **410** `{expired: true}` with no code once the end has passed (`/s/` answers 410 with a page too). Errors: 400 unknown language, no code, or a day outside the range (with the reason), 413 too large, 429 rate limit, 503 store unavailable. A link made before links had an end is given one a month from the first start that has this (`stampOldSnippets`, `snippet_expiry.go`, once; the most when shorter), and every week the server removes the links whose end has passed (`removeExpiredSnippets`; 10 minutes after the start, then every 7 days; not on a worker); the Health page shows the last run ("Code link cleanup"). Settings: `snippetMaxDays` ("Longest a code link works"), 1 to 365, default 30. | | `/s/` | GET | `handleSnippetLink` | public | Redirects (302) to the language's page with `?s=`, or to `/?repl=&s=`; 404 page for an unknown id. | | `/login` | GET, POST | `handleLoginSession` | public | GET: current session status as JSON. POST: create a session from the Firebase sign-in result (LLD 05). | | `/logout` | POST | `handleLogoutSession` | session | Deletes the session and cookie. | @@ -112,7 +112,7 @@ All paths are relative to `pathPrefix`: `/`, or `//` with `--random-url` | `/ws_filebrowser` | GET, POST | `Server.handleFileBrowser` | cookie homedir | File tree, load, save, zip, workspace usage (`?q=usage`) and file ops (LLD 04). Despite the name, this is plain HTTP. | | `/upload_file` | POST (multipart) | `Server.handleFileUpload` | cookie homedir | Upload into the homedir (LLD 04). | | `/auth_token.js` | GET | `handleAuthToken` | public | `var gotty_auth_token = '<--credential>'`. | -| `/config.js` | GET | `handleConfig` | public | `gotty_term`, `firebaseconfig` (the built-in production project, or `OPENREPL_FIREBASE_CONFIG`; see section 5), `openai_access_token` (LLD 07). | +| `/config.js` | GET | `handleConfig` | public | `gotty_term`, `firebaseconfig` (the project `OPENREPL_FIREBASE_CONFIG` names; with none set it is `null` and the page console says so; see section 5), `openai_access_token` (LLD 07). | | `/settings.js` | GET | `handleSettingsJS` | public | `var site_settings = {"colorOfTheDay": }`, sent with `Cache-Control: no-store` (LLD 05, 06). | | `/admin` | GET | `handleAdminPage` behind `wrapAdmin` | admin | The admin dashboard, a static page (`resources/admin.html`) with a restrictive Content-Security-Policy. | | `/admin/...` | GET, POST | `adminAPI` | admin; changes need the `X-Requested-With: openrepl-admin` header | The dashboard's JSON API: settings, parameters, health, usage, audit log, log, feedback, shared code, users and, on a gateway, workers and sessions. Routes and rules in LLD 13. | @@ -149,10 +149,10 @@ Settings (`utils/config.go`). Each is read from the environment first and from t | Cloudflare Workers AI (optional) | `OPENREPL_CLOUDFLARE_API_TOKEN` and `OPENREPL_CLOUDFLARE_ACCOUNT_ID`, both needed (`utils.CloudflareEnabled`) | none | | reCAPTCHA secret key (optional; checks the contact form on the server) | `OPENREPL_RECAPTCHA_SECRET`, as it is (the secret of the site key on the home page) | none | | Allowed host | `OPENREPL_HOST` | `user.host` | -| Firebase web config | `OPENREPL_FIREBASE_CONFIG`: JSON, or base64 of JSON | built in (production project) | +| Firebase web config | `OPENREPL_FIREBASE_CONFIG`: JSON, or base64 of JSON | none: sign-in, shared sessions and the shared editor do not work until it is set | | Mode | `OPENREPL_ENV`: `dev`, `development` or `local`; anything else is production | | -`OPENREPL_FIREBASE_CONFIG` (`utils.FirebaseConfigFromEnv`) replaces the Firebase web config that `/config.js` gives the page, so that a development project can be used. `apiKey`, `authDomain` and `projectId` are required; only the fields of a web app config are kept, and each is checked against the shape it must have (a pattern per field), because the value ends up in a script every visitor runs. `utils.FirebaseConfigJS` writes the object with `encoding/json`. A wrong value is an error that names the field, not its value, and `Options.Validate` returns it, so the server does not start; there is no quiet fall back to the built-in project. The start-up log line shows `firebase: the built-in project`, or `custom project (from env)`, and in dev mode the config itself, or a reminder that the built-in project is the production one. +`OPENREPL_FIREBASE_CONFIG` (`utils.FirebaseConfigFromEnv`) is the Firebase web config that `/config.js` gives the page. There is no built-in project (the production one used to be): a server started without it serves `const firebaseconfig = null;` and a console error, logs `firebase: NOT SET`, and the admin health page fails the "Firebase" line. `apiKey`, `authDomain` and `projectId` are required; only the fields of a web app config are kept, and each is checked against the shape it must have (a pattern per field), because the value ends up in a script every visitor runs. `utils.FirebaseConfigJS` writes the object with `encoding/json`. A wrong value is an error that names the field, not its value, and `Options.Validate` returns it, so the server does not start; there is no quiet fall back to any project. The start-up log line shows `firebase: NOT SET (...)`, or `custom project (from env)`, and in dev mode the config itself. An empty variable counts as not set. `utils.GetGitConfig()` reads the first file that exists among `/opt/gotty/.gitconfig`, `~/.gitconfig` (the home directory is expanded with `os.UserHomeDir`; it was a literal `~/` before) and `/etc/.gitconfig`, and flattens it into `section.key` pairs. A file that does not exist is skipped silently. diff --git a/docs/lld/06-frontend.md b/docs/lld/06-frontend.md index 44d73ea8..39bc65bf 100644 --- a/docs/lld/06-frontend.md +++ b/docs/lld/06-frontend.md @@ -52,8 +52,10 @@ Since T20 the page script is split into feature files under `src/js/src/page/`, | `03-run-and-files.js` | `LoadOptionFromUrl`, `StartTour`, `CompileandRun`, `RunandDebug`, editor download and upload, `uploadFile` | | `04-repl-demo.js` | The "Using the REPL" demo, usage table and language switching (`actionOnchange`) | | `05-auth.js` | Sign-in (FirebaseUI, `/login`), account menu, `logout` | -| `06-editor.js` | Ace set-up, themes, fonts, modes, Ctrl+S, editor sharing over Firebase | +| `06-editor.js` | Ace set-up, themes, fonts, modes, Ctrl+S, editor sharing over Firebase. The editor's language list has **C** and **C++** as two entries (values `c` and `cpp`, like the picker's two REPLs, whose `data-editor` is the same value, so the list and the picker follow each other one to one); Ace colours both with its one mode `c_cpp` (`aceMode`), a file by its extension (`codeext2menuoption`, `.c`, `.cpp`) and the default name of a new file (`EXT` in `22-editor-tabs.js`) use the two values, and a language record of an older page that says `c_cpp` is read as the one the picker is on (`listValue`). Test: `e2e/tests/editor-c-cpp.spec.js`. | | `07-file-browser.js` | The Files panel (jstree), context menu and live file events | +| `30-owner-presence.js` | A shared session whose owner is not there: the owner's page writes `editor_values//owner` `{online, at}` (set on connect, `onDisconnect` writes offline, a heartbeat every 25 s); a collaborator's page watches it and, after 8 s offline or 150 s of silence, shows a popup (left / nobody hosting; an id with no content: ended, and a link to an ended session no longer re-creates it), a bar above the editor, no sharing of its own edits and no Run, and "Copy it to my own editor" (a code link, `POST /snippet`, in a new tab). When the owner is back the page loads again. A missing record (an owner page of an older version) shows nothing. Test: `e2e/tests/shared-owner-left.spec.js`. Designed in the "Shared session: owner left notice" canvas. | +| `31-share-session.js` | Share with a time. Opening the Share popover starts a session (`ShareSession.start`, the time chosen last, 1 hour at first): `{id, endsAt, mins}` is saved in the browser (`localStorage` `openrepl-share`) and written to `editor_values//share` `{endsAt, mins, state}`. `getExampleRef` (`01-session.js`) gives the saved id back after a reload (`savedShareId`), unless another tab holds it (`openrepl-share-tab`, a 5 s heartbeat cleared on `pagehide`), so the link stays the same until it ends. The popover has the time list (guests 30 minutes, 1 hour; signed-in 30 minutes, 1 hour, 4 hours, 24 hours, 7 days; both cut at the admin's "Longest a share link works" settings, `shareLinkGuestMinutes` / `shareLinkUserMinutes`, default 60 and 10080, served as `site_settings.shareMaxGuestMinutes` / `shareMaxUserMinutes`), a countdown, Add time (up to the longest) and End session (asked first). The clock is checked every 15 s with the Firebase server time offset; five minutes before the end the owner is told once. End or expiry (`end`): the record says `ended` or `expired` for the collaborators, the owner's presence is stopped (`OwnerAway.stopHosting`), and after 2.5 s `wipe` removes `editor_values/`, `file-browser/` and the terminal records, and the page loads again (a new link next time). A collaborator's page (`30-owner-presence.js`) shows "The owner ended this session" or "This link has expired" and wipes the same records 15 s later; the owner's page wipes the records of a link that ran out while it was closed when it loads. A link that is over shows no code (`beforeInit`). A link whose data is gone says "This shared session has ended". **A collaborator's page of a session that is over writes nothing** (`30-owner-presence.js`, `lockPage`): the editor is read-only, every Firebase write from the page (`firebase.database.Reference.prototype` set, update, push, remove, transaction) is dropped except the wipe, the terminal and the peer chat send nothing, and the requests to `/ws_filebrowser` and `/upload_file` are refused. **Not a lock**: nothing on a server checks the time before the data is read. What is left behind when no page is open is removed by the server's cleanup job (`server/session_cleanup.go`): every `sessionCleanupHours` (admin setting, default 24; started 5 minutes after the server, not on a worker) it signs in to the Realtime Database REST API as the service account of `OPENREPL_FIREBASE_ADMIN` (or the key of `OPENREPL_FIRESTORE_CREDENTIALS`; `persist.RTDBTokens` mints the token, scope `firebase.database`), lists the ids under `openrepl`, `editor_values`, `chat-list` and `file-browser`, and for each reads `editor_values//share` and `.../owner`: an end that has passed, or a state `ended` or `expired`, is removed from the four places at once; an id with no share record is removed when its owner page is not saying it is there (no `owner` record, or one that is offline or older than 10 minutes) and it has been seen so for a whole interval (the job remembers it between runs, so a live session of an older owner page or one that has not pressed Share is not taken from under it). The Health page shows the last run. Test: `e2e/tests/shared-link-time.spec.js`. Designed in the "Shared session: owner left notice, and a link with a time" canvas (boards 5 to 8). | | `08-genie-nudge.js` to `17-share-code.js` | One file per Phase 1 and 2 feature: Genie nudge, hero chips, workspace controls, terminal state, landing, phone layout, accessibility, empty Files note, language pages, share-code links | | `18-genie-review.js`, `lib-diff.mjs` | Review of what Genie wants to put in the editor (below). `lib-diff.mjs` is the pure diff code; node tests in `src/js/test` (`npm test`) | @@ -82,7 +84,7 @@ What the parts do, together: - **Hero and landing.** Language chips and cards (`pickLanguage`), the usage heading, the docker command copy and the request form. - **Genie nudge.** `setupGenieErrorNudge` watches terminal output for the first error. - **Language pages.** `langPageForRepl` and a `#optionlist` change handler move the address to the language's page with `history.replaceState` and update the title, hero and canonical link (variants without a page keep the address). -- **Share-code links.** "Create code link" in the Share popover POSTs the editor content to `/snippet` and shows the link. On a page opened with `?s=`, `applySharedSnippet` puts the saved code in the editor once the language's starter code has loaded, and says so in a notice. +- **Share-code links.** "Create code link" in the Share popover POSTs the editor content and a day (`#code-link-until`, a date input: `min` tomorrow, `max` today plus the admin's most from `site_settings.snippetMaxDays`, value a month from today or the most when shorter, all counted in UTC like the server, `setupCodeLinkDate`) to `/snippet` and shows the link and "Works until ...". A link that has expired answers 410, and the notice says so. Test: `e2e/tests/code-link-expiry.spec.js`. On a page opened with `?s=`, `applySharedSnippet` puts the saved code in the editor once the language's starter code has loaded, and says so in a notice. - **Accessibility helpers.** `syncTerminalTabsA11y` and arrow-key handling for the terminal tabs, the Esc-then-Tab way out of the editor and terminal (`moveFocusOutOf`), `describeCodeInputs`, and the Share popover's keyboard support. - **Empty Files panel.** `updateFilesEmptyState` (LLD 04). diff --git a/docs/lld/15-object-storage.md b/docs/lld/15-object-storage.md index 8ead46a6..1af03659 100644 --- a/docs/lld/15-object-storage.md +++ b/docs/lld/15-object-storage.md @@ -36,7 +36,7 @@ A client with the standard library only (like `persist/firestore.go`): AWS Signa - **Who:** the node that is the authority for a home: a single server, or the gateway (which holds a copy of every home, LLD 12). Never a worker (`homesnap.Default.Enable()` is not called in worker mode; workers have no keys). Only signed-in users' homes: `guest-*` are never kept. - **Registered by use:** `touchHome` (`server/identity.go`) and the gateway's `PrepareHome` call `Touch(home)` when a signed-in user's request resolves its home. The first time a home is touched in a process: if its folder is missing or empty it is restored (below); from then on it is followed. A home that nobody touches is not asked about again. - **Snapshot (`SnapshotHome`):** a job every 5 minutes snapshots the followed homes whose fingerprint changed and in which nothing was written for 30 seconds; at a clean stop every changed home, without waiting. A snapshot holds plain files and folders (empty ones too), never a link, a socket or anything named `.wsync-*`; a file over 50 MB, or one that changes while it is read, is left out and noted in the manifest; a home over 100 MB is not snapshotted (an error in the job's result). -- **Storage (changed 2026-11-13):** one object for each UTC day, `homes///-.tar.gz` (`daily.go`), overwritten by every snapshot of that day (metadata: `sha256`, `generation`, `created`), and `latest.json` (generation, key, SHA-256, writer). The newest 3 days are kept (`Keep`). Where the bucket keeps versions (Backblaze B2 does, R2 does not), the older versions of a day's file stay in the bucket until its lifecycle rule removes them, and the profile page lists them (`objstore.ListVersions`, `GetVersion`). Objects of the earlier scheme (`-