Repository navigation
Conversation
Proposes the architecture of offline content generation on the CMS and its serving by the LMS. ADR-001 and ADR-002 each gain a status line pointing to it.
|
Thanks for the pull request, @cmltaWt0! This repository is currently maintained by Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review. 🔘 Get product approvalIf you haven't already, check this list to see if your contribution needs to go through the product review process.
🔘 Provide contextTo help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:
🔘 Get a green buildIf one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green. DetailsWhere can I find more information?If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources: When can I expect my changes to be merged?Our goal is to get community contributions seen and reviewed as efficiently as possible. However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:
💡 As a result it may take up to several weeks or months to complete a review and merge your PR. |
|
|
||
| Generation is triggered by the existing ``SignalHandler.course_published`` signal in the CMS, the hook that | ||
| already fans out search indexing, course outlines and special-exam registration, when the course waffle flag | ||
| ``offline_content.enable_offline_content`` (annotated per OEP-17) is on. |
There was a problem hiding this comment.
I'm curious why this is a per-course operator flag rather than a site-wide operator switch with a course setting that authors control, similar to "Mobile Course Available."
There was a problem hiding this comment.
Good question and the wording is a bit misleading. offline_content.enable_offline_content is a course waffle flag so it is a site-wide operator switch that can also be overridden per organisation or per course. That matches ADR-001 ("enabled for the course or entire platform using waffle flags"). I'll reword decision 2 so it does not read as per-course only.
It is an operator switch because the operator carries the cost and the exposure (if we agree with unsigned behaviour): generation load on the CMS workers, storage, and archive URLs that are fetched without credentials.
For the author's side I'd rather reuse the control authors already have than add a second one. Announcement already follows "Mobile Course Available": offline_download is added only when the learner has mobile access to the course and download_courses lists only mobile-available courses. I'll amend the ADR so that generation follows the same course setting and a course that is not mobile-available gets no archives at all.
A separate author setting for offline only ("on mobile, but not downloadable") is a product decision I'd keep out of this scope. It can be added later as one more condition on the same gate without changing the design.
There was a problem hiding this comment.
Ah, got it... thanks for clarifying @cmltaWt0..! Agreed on pushing 'on mobile, not downloadable' separation til later. I think once we get to assessments, some course teams may want to block certain content from offline. But for now, I feel good. thanks
| 6. The mobile API reads the manifest | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| ``/api/mobile/v4/course_info/blocks/``, in dict form, adds ``offline_download {file_url, last_modified, |
There was a problem hiding this comment.
Will the apps be able to check for updates efficiently for full-course and single-block downloads without requesting the full course outline?
There was a problem hiding this comment.
Actually it is a gap. The per-block last_modified is only in the v4 blocks response, and download_courses returns a size per course but no version so an app would have to fetch the blocks of every downloaded course to learn whether anything changed.
I'll add a course-level version of the offline content to decision 6 - a value returned per course by download_courses and at the course level of the v4 blocks response. It changes whenever an archive of the course is added, removed or rebuilt with different content and computed from the manifest with no call to the storage. The apps make one download_courses request for all courses and fetch the blocks response only for the courses whose version moved. The per-block last_modified there tells them which archives to fetch.
That covers both cases: a full-course download and a single downloaded block are latest as long as the course version has not moved.
For a single block I would not add an endpoint - with the course version the blocks response is requested only when something in the course actually changed.
| * Installing the platform's npm dependencies builds the MathJax file: three more packages at build time (about | ||
| 100 MB with their dependencies), nothing at run time; the component source lists the TeX packages by name and | ||
| is the place to add one. | ||
| * A platform or block-package upgrade reaches archives only through the command, and then every learner |
There was a problem hiding this comment.
Given a key use case for offline is for users who often have limited data/connectivity/free storage, is there a way we can avoid having learners re-download when archives are rebuilt but there are no changes?
There was a problem hiding this comment.
Archives are built deterministically and each row stores a content hash.
When a re-render produces the same hash, the row keeps its file and its last_modified so the apps see
nothing new and download nothing. I'll state the rule in decision 5 and reword this consequence.
An important note/exception is the platform upgrade case. Every archive is self-contained and carries the page scripts so a platform upgrade that changes those scripts changes every archive and then everything is downloaded again. An upgrade that does not touch them or a re-run of the command for any other reason costs learners only the archives whose content changed.
The shared-resources archive of ADR-002 is what removes the upgrade case but it is deferred until the applications can consume a second download.
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| ``offline_view`` is a named XBlock view that a block class opts into, implemented in the block's own package: | ||
| ``xblocks-contrib`` for the extracted HTML and problem blocks, ``xblock-poll`` for the survey, |
There was a problem hiding this comment.
I did not realize this earlier, but it seems like PollBlock and SurveyBlock are both implemented within xblock-poll, and most of their code is shared I propose that we implement offline mode for both, what do you think? Implementing it for only one might actually be more work because some of the shared could would need to be split up.
| 7. The bridge contract is the one the applications ship, completed | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| The page posts offline submissions to ``window.AndroidBridge.postMessage(json)`` and | ||
| ``window.webkit.messageHandlers.IOSBridge.postMessage(json)``, and exposes a global | ||
| ``markProblemCompleted(json)`` that the applications call when a stored submission is replayed into the page. | ||
| This name supersedes ``markCompleted`` in ADR-001. The payload is ``{url, type, data, content_type}``: the | ||
| handler URL, the HTTP method, the request body as a string, and its content type | ||
| (``application/x-www-form-urlencoded`` when absent, ``application/json`` for a block's ``json_handler``). | ||
| Nothing else, in particular no header of the learner's, is sent to the device. The applications replay a | ||
| submission by sending ``data`` verbatim to ``url`` with method ``type`` and that content type, authenticate it | ||
| themselves, surface a rejected replay to the learner, and quote the stored JSON when they call | ||
| ``markProblemCompleted``. Replaying the generic payload is the applications' part of this scope; the interactive | ||
| survey of decision 8 ships with it. |
There was a problem hiding this comment.
I'd like to dig into this part of the design. I will say "SurveyBlock" as an example, but this applies to any XBlock type, like ProblemBlock in the future.
I want to have just one SurveyBlock JS+HTML+CSS implementation, which works both online and offline. That means SurveyBlock can't call the mobile bridge explicitly. Instead, when SurveyBlock needs to talk to the backend, it would call the JS runtime object. The runtime object must decide whether to call the backend (online mode) or to call the bridge for later sync (offline mode). In other words: calling the bridge is an implementation detail of the runtime.
On the backend, I want offline_view to be a suitable base for student_view. Basically: offline_view should be stateless; it should render purely based on its content and settings; it should not need to know the request, a user, the time of day, etc. student_view should be rewritten to call offline_view, then call student_view_data (a json handler with student state), and then "hydrate" the offline view with the student data.
What do think of this direction @cmltaWt0 , and does it match what you had in mind? This would require rewriting making some changes to the XBlock JS runtime and rewriting. SurveyBlock's student_view. In case it helps explain, I had Claude look at xblock-poll and ProblemBlock and spec out the changes we'd need to make: https://github.com/raccoongang/edx-platform/pull/2683/files#diff-bd44597c93db908d92f5fe009934309b43ca13b2ce1f58d638dca94d2846c747
There was a problem hiding this comment.
Yes to both.
- Blocks and the bridge. As submitted, decision 7 has the page post stored HTTP requests to the bridge and decision 8 gives the problem a separate read-only rendering. Decision 11 already had state beside the archive and replay through the handler dispatch, but only for the next scope. Yours goes further and is better: the block's script calls only the runtime and the bridge is the archive runtime's business.
offline_viewas the base. Agreed (do we need to choose a better name of it's a content base?). The survey and poll get theirstudent_viewrebuilt on it in this scope. The problem keeps itsstudent_viewuntil the next scope as your document says. Drag and drop is nearly there: itsstudent_viewis already a static template, the author's settings as init data and state fetched by the script so two values that depend on the clock and the learner move to the state. The HTML and course info blocks have no learner state or handlers.
Two things the split leaves open:
- State offline. Decision 7 has the archive runtime read the state the application downloaded but the state service is decision 11's. Without it a learner who voted on the web sees an empty form offline. I'd add a per-block state fetch at download in this scope; the per-course state service of decision 11 can follow.
- Rejected replays. The survey document leaves open who reports one. The survey and poll refuse with
success: falseinside a 200 so the application cannot recognise a refusal. I'd have the block's script show it on its next load withloadStatealso returning the results of the last replay.
I'll take #2683 into the branch and amend on top (it's greatly describe the mechanics): decision 5 as in my other reply, the record's details, drag and drop and what I found checking the two documents against the code.
This is a larger scope than the ADR as submitted: a rebuilt online view for the survey and poll, runtime methods in four runtimes, a replay endpoint and matching work in both applications, with reviews in several repositories. I don't expect all of it to make the Willow cut and I'll follow up separately on what lands when.
| *Storage.* Archives are written and read through ``STORAGES["offline_content"]`` when it is defined, else | ||
| through ``default_storage``. The storage must return URLs the applications can fetch without credentials for | ||
| as long as the row exists; for S3 that means unsigned URLs, on a custom domain or CDN. Paths derive from the | ||
| usage key and the content hash, so nobody can compute them without the content. |
There was a problem hiding this comment.
Why must the files be fetchable without credentials?
There was a problem hiding this comment.
They don't have to be. The requirement comes from the applications as they ship today, not from the design: both download an archive with a bare HTTP client that sends no credentials (Android's FileDownloader, iOS's AF.download), and both store file_url when they read the course structure and download later, so a pre-signed storage URL can have expired by then. Unguessable paths are what works with those applications.
Since the applications change in this scope anyway, I'd replace it. file_url becomes an LMS endpoint that checks the learner's access and redirects to a storage URL signed for a few minutes, or streams the file where the storage cannot sign. The bucket stays private, access is checked at download time and unenrolling revokes it. It needs work on both sides: the endpoint on the backend and both applications sending their token on downloads from the LMS. I'll amend decision 5.
* docs: add offline_view proposals for the survey and problem blocks Two proposals supporting ADR-003: offline_view as the basis for a block's other views, with the runtime interface that requires, worked through for the xblock-poll survey; and a read-only offline_view for the problem block that the interactive scope can extend. * docs: align ADR-003 with the offline_view proposals Link ADR-003 to the two proposals and make it consistent with them: * offline_view is the basis for a block's other views, rendered with user-scoped fields and handler_url raising, and declined with a dedicated exception. * Blocks reach state and handlers through the runtime object; the mobile bridge is internal to the archive runtime, and stored calls are records of handler name and body replayed through one endpoint. * The problem view is read-only through its script, not a separate template; poll joins the survey as interactive. * Block resources are the ones the block's other views declare, including local_resource_url resources.
Description
Adds
openedx/features/offline_content/docs/003-offline-content-generation-architecture.rst(status: Proposed)and a status line to ADR-001 and ADR-002 pointing to it. Documentation only, no code.
ADR-001 (#35011) decided what mobile offline content is: a user-agnostic
offline_viewthat XBlocks opt into,archives generated on publish, an
offline_downloadfield in the mobile API and a JavaScript bridge. ADR-003supersedes its implementation decisions, narrows ADR-002 (the shared-resources archive waits for an application
change) and proposes the architecture of the platform side:
openedx.features.offline_content, installed in the LMS and the CMS; the CMS generates, theLMS serves;
SignalHandler.course_publishedin the CMS, gated by the course waffle flagoffline_content.enable_offline_content; one delayed task per course reconciling a manifest; a managementcommand for back-fill and prune; no new signal;
offline_viewis the only view rendered, with no user, through the field-data binding Studio's preview uses;the block declines by raising
NotImplementedError(reason); nostudent_viewfallback;Fragmentplus a DOM pass: platform static files from the staticfiles finders, courseassets from the contentstore, CSS
url()rewritten, one MathJax 4 file built atnpm installfor archiveswhose content has math, no external fetches, zip-slip-safe member names, a content hash;
supports it;
offline_download {file_url, last_modified, file_size}on/api/mobile/v4/course_info/blocks/from onemanifest query;
GET /api/mobile/v1/download_courses/{username}withtotal_sizefor the downloads page;window.AndroidBridge.postMessage(json),webkit.messageHandlers.IOSBridge, globalmarkProblemCompleted(json)(supersedesmarkCompleted),payload
{url, type, data, content_type};course_infoblocks, announced as two course-levelfields on the v4 response;
learner's state travels beside the archive, interactions are recorded and replayed as the learner.
Rejected alternatives and the reasons are in the ADR. Impacted roles: Developer and Operator (design document);
no learner- or author-facing change in this PR.
Supporting information
feat: [FC-0078] Downloads page openedx-app-android#432 (downloads page)