Skip to content

Deduplicate exchanged content - #1355

Open
eilmiv wants to merge 7 commits into
masterfrom
deduplicate_exchanged_content
Open

Deduplicate exchanged content#1355
eilmiv wants to merge 7 commits into
masterfrom
deduplicate_exchanged_content

Conversation

@eilmiv

@eilmiv eilmiv commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Summary of changes

  • Store the source a resource was ingested from
  • Don't update resource from a different source unless metadata is better

Motivation and context

Closes #1356

Checklist

  • I have read and followed the CONTRIBUTING guide.
  • I confirm that I have the authority necessary to make this contribution on behalf of its copyright owner and agree to license it to the TeSS codebase under the BSD license.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses metadata “flip-flopping” during ingestion by recording which Source last scraped a Scrapable resource and introducing logic to skip updates from other sources unless the incoming metadata is deemed richer. This aligns with issue #1356’s goal of stabilizing metadata when multiple sources provide conflicting versions.

Changes:

  • Add last_scraped_by_id to events and materials, and expose it via Scrapable#last_scraped_by_source.
  • Update the ingestor to persist last_scraped_by_source and to skip writes when the current source is not responsible.
  • Add/adjust unit tests and fixtures to cover responsibility switching behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
lib/ingestors/ingestor.rb Sets last_scraped_by_source on ingest and introduces responsibility logic to avoid cross-source flip-flops.
app/models/concerns/scrapable.rb Adds the belongs_to :last_scraped_by_source association for scrapable models.
db/migrate/20260729120000_add_last_scraped_by_to_scrapable_models.rb Adds the new DB columns for tracking scrape source.
db/schema.rb Reflects the new last_scraped_by_id columns in events and materials.
test/unit/ingestors/ingestor_test.rb Adds tests for setting and switching scrape-source responsibility.
test/fixtures/events.yml Adds an event fixture used by the new responsibility tests.
test/unit/ingestors/scraper_test.rb Updates expected activity count impacted by the new ingestion behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread lib/ingestors/ingestor.rb
Comment thread lib/ingestors/ingestor.rb Outdated
@eilmiv
eilmiv marked this pull request as ready for review August 4, 2026 15:00
@eilmiv
eilmiv requested a review from fbacall August 4, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ensure metadata is not changed back and forth with two conflicting sources for a learning resource

2 participants