Skip to content

docs: ADR-003 offline content generation architecture - #39352

Open
cmltaWt0 wants to merge 2 commits into
openedx:masterfrom
raccoongang:max/offline-content-adr-003
Open

cmltaWt0 wants to merge 2 commits into
openedx:masterfrom
raccoongang:max/offline-content-adr-003

Conversation

@cmltaWt0

@cmltaWt0 cmltaWt0 commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Description

Adds openedx/features/offline_content/docs/003-offline-content-generation-architecture.rst (status: Proposed)
and a status line to ADR-001 and ADR-002 pointing to it. Documentation only, no code.

ADR-001 (#35011) decided what mobile offline content is: a user-agnostic offline_view that XBlocks opt into,
archives generated on publish, an offline_download field in the mobile API and a JavaScript bridge. ADR-003
supersedes its implementation decisions, narrows ADR-002 (the shared-resources archive waits for an application
change) and proposes the architecture of the platform side:

  1. one application, openedx.features.offline_content, installed in the LMS and the CMS; the CMS generates, the
    LMS serves;
  2. trigger = SignalHandler.course_published in the CMS, gated by the course waffle flag
    offline_content.enable_offline_content; one delayed task per course reconciling a manifest; a management
    command for back-fill and prune; no new signal;
  3. offline_view is the only view rendered, with no user, through the field-data binding Studio's preview uses;
    the block declines by raising NotImplementedError(reason); no student_view fallback;
  4. packaging from the Fragment plus a DOM pass: platform static files from the staticfiles finders, course
    assets from the contentstore, CSS url() rewritten, one MathJax 4 file built at npm install for archives
    whose content has math, no external fetches, zip-slip-safe member names, a content hash;
  5. one manifest model and one settings-driven storage; unguessable archive URLs, signed where the storage
    supports it;
  6. offline_download {file_url, last_modified, file_size} on /api/mobile/v4/course_info/blocks/ from one
    manifest query; GET /api/mobile/v1/download_courses/{username} with total_size for the downloads page;
  7. the bridge contract the applications ship, completed: window.AndroidBridge.postMessage(json),
    webkit.messageHandlers.IOSBridge, global markProblemCompleted(json) (supersedes markCompleted),
    payload {url, type, data, content_type};
  8. problems and drag and drop read-only; the survey interactive through the bridge;
  9. the flag gates generation and announcement together; prune removes the archives of flagged-off courses;
  10. the course handouts and updates are archives of their course_info blocks, announced as two course-level
    fields on the v4 response;
  11. interactive blocks are the next scope's and build on these decisions: content stays learner-free, the
    learner's state travels beside the archive, interactions are recorded and replayed as the learner.

Rejected alternatives and the reasons are in the ADR. Impacted roles: Developer and Operator (design document);
no learner- or author-facing change in this PR.

Supporting information

Proposes the architecture of offline content generation on the CMS and
its serving by the LMS. ADR-001 and ADR-002 each gain a status line
pointing to it.
@openedx-webhooks openedx-webhooks added open-source-contribution PR author is not from Axim or 2U core contributor PR author is a Core Contributor (who may or may not have write access to this repo). labels Nov 5, 2026
@openedx-webhooks

openedx-webhooks commented Nov 5, 2026 •

Copy link
Copy Markdown

Thanks for the pull request, @cmltaWt0!

This repository is currently maintained by @openedx/wg-maintenance-openedx-platform-oncall.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

  • If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
    • This process (including the steps you'll need to take) is documented here.
  • If it doesn't, simply proceed with the next step.
🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

  • Dependencies

    This PR must be merged before / after / at the same time as ...

  • Blockers

    This PR is waiting for OEP-1234 to be accepted.

  • Timeline information

    This PR must be merged by XX date because ...

  • Partner information

    This is for a course on edx.org.

  • Supporting documentation
  • Relevant Open edX discussion forum threads
🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details
Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

  • The size and impact of the changes that it introduces
  • The need for product review
  • Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

@github-project-automation github-project-automation Bot moved this to Needs Triage in Contributions Nov 5, 2026
@cmltaWt0
cmltaWt0 requested a review from kdmccormick November 5, 2026 15:30
@cmltaWt0 cmltaWt0 self-assigned this Nov 5, 2026
@cmltaWt0
cmltaWt0 marked this pull request as ready for review November 5, 2026 15:33

Generation is triggered by the existing ``SignalHandler.course_published`` signal in the CMS, the hook that
already fans out search indexing, course outlines and special-exam registration, when the course waffle flag
``offline_content.enable_offline_content`` (annotated per OEP-17) is on.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious why this is a per-course operator flag rather than a site-wide operator switch with a course setting that authors control, similar to "Mobile Course Available."

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question and the wording is a bit misleading. offline_content.enable_offline_content is a course waffle flag so it is a site-wide operator switch that can also be overridden per organisation or per course. That matches ADR-001 ("enabled for the course or entire platform using waffle flags"). I'll reword decision 2 so it does not read as per-course only.

It is an operator switch because the operator carries the cost and the exposure (if we agree with unsigned behaviour): generation load on the CMS workers, storage, and archive URLs that are fetched without credentials.

For the author's side I'd rather reuse the control authors already have than add a second one. Announcement already follows "Mobile Course Available": offline_download is added only when the learner has mobile access to the course and download_courses lists only mobile-available courses. I'll amend the ADR so that generation follows the same course setting and a course that is not mobile-available gets no archives at all.

A separate author setting for offline only ("on mobile, but not downloadable") is a product decision I'd keep out of this scope. It can be added later as one more condition on the same gate without changing the design.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, got it... thanks for clarifying @cmltaWt0..! Agreed on pushing 'on mobile, not downloadable' separation til later. I think once we get to assessments, some course teams may want to block certain content from offline. But for now, I feel good. thanks

6. The mobile API reads the manifest
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``/api/mobile/v4/course_info/blocks/``, in dict form, adds ``offline_download {file_url, last_modified,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the apps be able to check for updates efficiently for full-course and single-block downloads without requesting the full course outline?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually it is a gap. The per-block last_modified is only in the v4 blocks response, and download_courses returns a size per course but no version so an app would have to fetch the blocks of every downloaded course to learn whether anything changed.

I'll add a course-level version of the offline content to decision 6 - a value returned per course by download_courses and at the course level of the v4 blocks response. It changes whenever an archive of the course is added, removed or rebuilt with different content and computed from the manifest with no call to the storage. The apps make one download_courses request for all courses and fetch the blocks response only for the courses whose version moved. The per-block last_modified there tells them which archives to fetch.
That covers both cases: a full-course download and a single downloaded block are latest as long as the course version has not moved.

For a single block I would not add an endpoint - with the course version the blocks response is requested only when something in the course actually changed.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, this makes sense to me! 👍

* Installing the platform's npm dependencies builds the MathJax file: three more packages at build time (about
100 MB with their dependencies), nothing at run time; the component source lists the TeX packages by name and
is the place to add one.
* A platform or block-package upgrade reaches archives only through the command, and then every learner

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given a key use case for offline is for users who often have limited data/connectivity/free storage, is there a way we can avoid having learners re-download when archives are rebuilt but there are no changes?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Archives are built deterministically and each row stores a content hash.
When a re-render produces the same hash, the row keeps its file and its last_modified so the apps see
nothing new and download nothing. I'll state the rule in decision 5 and reword this consequence.

An important note/exception is the platform upgrade case. Every archive is self-contained and carries the page scripts so a platform upgrade that changes those scripts changes every archive and then everything is downloaded again. An upgrade that does not touch them or a re-run of the command for any other reason costs learners only the archives whose content changed.

The shared-resources archive of ADR-002 is what removes the upgrade case but it is deferred until the applications can consume a second download.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``offline_view`` is a named XBlock view that a block class opts into, implemented in the block's own package:
``xblocks-contrib`` for the extracted HTML and problem blocks, ``xblock-poll`` for the survey,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not realize this earlier, but it seems like PollBlock and SurveyBlock are both implemented within xblock-poll, and most of their code is shared I propose that we implement offline mode for both, what do you think? Implementing it for only one might actually be more work because some of the shared could would need to be split up.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree.

Comment on lines +185 to +198
7. The bridge contract is the one the applications ship, completed
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The page posts offline submissions to ``window.AndroidBridge.postMessage(json)`` and
``window.webkit.messageHandlers.IOSBridge.postMessage(json)``, and exposes a global
``markProblemCompleted(json)`` that the applications call when a stored submission is replayed into the page.
This name supersedes ``markCompleted`` in ADR-001. The payload is ``{url, type, data, content_type}``: the
handler URL, the HTTP method, the request body as a string, and its content type
(``application/x-www-form-urlencoded`` when absent, ``application/json`` for a block's ``json_handler``).
Nothing else, in particular no header of the learner's, is sent to the device. The applications replay a
submission by sending ``data`` verbatim to ``url`` with method ``type`` and that content type, authenticate it
themselves, surface a rejected replay to the learner, and quote the stored JSON when they call
``markProblemCompleted``. Replaying the generic payload is the applications' part of this scope; the interactive
survey of decision 8 ships with it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to dig into this part of the design. I will say "SurveyBlock" as an example, but this applies to any XBlock type, like ProblemBlock in the future.

I want to have just one SurveyBlock JS+HTML+CSS implementation, which works both online and offline. That means SurveyBlock can't call the mobile bridge explicitly. Instead, when SurveyBlock needs to talk to the backend, it would call the JS runtime object. The runtime object must decide whether to call the backend (online mode) or to call the bridge for later sync (offline mode). In other words: calling the bridge is an implementation detail of the runtime.

On the backend, I want offline_view to be a suitable base for student_view. Basically: offline_view should be stateless; it should render purely based on its content and settings; it should not need to know the request, a user, the time of day, etc. student_view should be rewritten to call offline_view, then call student_view_data (a json handler with student state), and then "hydrate" the offline view with the student data.

What do think of this direction @cmltaWt0 , and does it match what you had in mind? This would require rewriting making some changes to the XBlock JS runtime and rewriting. SurveyBlock's student_view. In case it helps explain, I had Claude look at xblock-poll and ProblemBlock and spec out the changes we'd need to make: https://github.com/raccoongang/edx-platform/pull/2683/files#diff-bd44597c93db908d92f5fe009934309b43ca13b2ce1f58d638dca94d2846c747

@cmltaWt0 cmltaWt0 Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes to both.

  • Blocks and the bridge. As submitted, decision 7 has the page post stored HTTP requests to the bridge and decision 8 gives the problem a separate read-only rendering. Decision 11 already had state beside the archive and replay through the handler dispatch, but only for the next scope. Yours goes further and is better: the block's script calls only the runtime and the bridge is the archive runtime's business.
  • offline_view as the base. Agreed (do we need to choose a better name of it's a content base?). The survey and poll get their student_view rebuilt on it in this scope. The problem keeps its student_view until the next scope as your document says. Drag and drop is nearly there: its student_view is already a static template, the author's settings as init data and state fetched by the script so two values that depend on the clock and the learner move to the state. The HTML and course info blocks have no learner state or handlers.

Two things the split leaves open:

  • State offline. Decision 7 has the archive runtime read the state the application downloaded but the state service is decision 11's. Without it a learner who voted on the web sees an empty form offline. I'd add a per-block state fetch at download in this scope; the per-course state service of decision 11 can follow.
  • Rejected replays. The survey document leaves open who reports one. The survey and poll refuse with success: false inside a 200 so the application cannot recognise a refusal. I'd have the block's script show it on its next load with loadState also returning the results of the last replay.

I'll take #2683 into the branch and amend on top (it's greatly describe the mechanics): decision 5 as in my other reply, the record's details, drag and drop and what I found checking the two documents against the code.

This is a larger scope than the ADR as submitted: a rebuilt online view for the survey and poll, runtime methods in four runtimes, a replay endpoint and matching work in both applications, with reviews in several repositories. I don't expect all of it to make the Willow cut and I'll follow up separately on what lands when.

Comment on lines +153 to +156
*Storage.* Archives are written and read through ``STORAGES["offline_content"]`` when it is defined, else
through ``default_storage``. The storage must return URLs the applications can fetch without credentials for
as long as the row exists; for S3 that means unsigned URLs, on a custom domain or CDN. Paths derive from the
usage key and the content hash, so nobody can compute them without the content.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why must the files be fetchable without credentials?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They don't have to be. The requirement comes from the applications as they ship today, not from the design: both download an archive with a bare HTTP client that sends no credentials (Android's FileDownloader, iOS's AF.download), and both store file_url when they read the course structure and download later, so a pre-signed storage URL can have expired by then. Unguessable paths are what works with those applications.

Since the applications change in this scope anyway, I'd replace it. file_url becomes an LMS endpoint that checks the learner's access and redirects to a storage URL signed for a few minutes, or streams the file where the storage cannot sign. The bucket stays private, access is checked at download time and unenrolling revokes it. It needs work on both sides: the endpoint on the backend and both applications sending their token on downloads from the LMS. I'll amend decision 5.

@mphilbrick211 mphilbrick211 moved this from Needs Triage to In Eng Review in Contributions Nov 16, 2026
* docs: add offline_view proposals for the survey and problem blocks

Two proposals supporting ADR-003: offline_view as the basis for a
block's other views, with the runtime interface that requires, worked
through for the xblock-poll survey; and a read-only offline_view for
the problem block that the interactive scope can extend.

* docs: align ADR-003 with the offline_view proposals

Link ADR-003 to the two proposals and make it consistent with them:

* offline_view is the basis for a block's other views, rendered with
  user-scoped fields and handler_url raising, and declined with a
  dedicated exception.
* Blocks reach state and handlers through the runtime object; the
  mobile bridge is internal to the archive runtime, and stored calls
  are records of handler name and body replayed through one endpoint.
* The problem view is read-only through its script, not a separate
  template; poll joins the survey as interactive.
* Block resources are the ones the block's other views declare,
  including local_resource_url resources.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core contributor PR author is a Core Contributor (who may or may not have write access to this repo). open-source-contribution PR author is not from Axim or 2U

Projects

Status: In Eng Review

Development

Successfully merging this pull request may close these issues.

4 participants