Skip to content

Keep AM when searching for dates after a time - #1400

Merged
AdrianAtZyte merged 4 commits into
scrapinghub:masterfrom
AdrianAtZyte:search-am
Oct 21, 2026
Merged

AdrianAtZyte merged 4 commits into
scrapinghub:masterfrom
AdrianAtZyte:search-am

Conversation

@AdrianAtZyte

@AdrianAtZyte AdrianAtZyte commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Fixes #681, fixes #1037, fixes #894, fixes #504.

@codspeed

codspeed Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 9 untouched benchmarks


Comparing AdrianAtZyte:search-am (456cf6f) with master (db7bb24)

Open in CodSpeed

@codecov

codecov Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.98%. Comparing base (db7bb24) to head (456cf6f).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1400   +/-   ##
=======================================
  Coverage   98.97%   98.98%           
=======================================
  Files         239      239           
  Lines        3627     3628    +1     
=======================================
+ Hits         3590     3591    +1     
  Misses         37       37           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@serhii73 serhii73 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this fixes all four linked issues (May 31, 8AM UTC, tomorrow 2am, 12:47 am, on Monday at 7am). Two things before merging:

1. The updated test depends on the day of the month. "June 5 am utc" gets its day from the relative-base chain, which starts at today ("May 2020" → today's day, …, "January UTC" → 2023-01-). So the expected datetime(2023, 6, 23, 5, 0) only holds on the 23rd, the day the PR was pushed. On the 30th, test_relative_base_setting_2_en fails with 2023-06-30 05:00. Suggestion:

datetime.datetime(2023, 6, min(datetime.datetime.now(tz=datetime.timezone.utc).day, 30), 5, 0, tzinfo=pytz.utc)

(min(..., 30) because June has no 31st. The search clamps to June 30 on the 31st.)

2. German "am" ("on the") after a 24-hour time or a year is now kept as AM, and the date is lost:

from datetime import datetime
from dateparser.search import search_dates
rb = {"RELATIVE_BASE": datetime(2020, 6, 15, 12)}
search_dates("Das Konzert findet um 20 Uhr am 5. Mai 2021 statt.", languages=["de"], settings=rb)
# master: [('um 20 Uhr am 5. Mai 2021', datetime(2021, 5, 5, 0, 0))]
# PR:     [('Mai 2021', datetime(2021, 5, 15, 0, 0))]
search_dates("Er wurde 1990 am 3. März in Berlin geboren", languages=["de"], settings=rb)
# master: [('1990 am 3. März in', datetime(1990, 3, 3, 0, 0))]
# PR:     [('März in', datetime(2020, 3, 15, 0, 0))]

Only keeping am after a valid 12-hour value seems to cover both. For example:

item = re.sub(r"(?<!\b(?:0?[1-9]|1[0-2])(?:[:.][0-5]\d)?\s*)\bam\b", "", item)

With that, the issue examples still work, the German cases match master, and um 10 Uhr am 5. Mai 2021 even improves (2021-05-05 10:00 instead of 2021-05-10).

@serhii73 serhii73 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review (claude code-review, at ecffb5e). No new correctness regression found. Keeping "am" after a 1-12 time is the intended change, and the German "am 5. Mai" cases in the new tests behave as asserted.

Non-blocking notes:

  • The rule is a syntactic guess: any "am" after a 1-12 number or h:mm is kept, so German <1-12> am <date> (e.g. "Seite 4 am 7. Juni") will still read the preposition as AM. Using the source language, or the translation step, would be more robust than looking at what precedes the token.
  • "June 5 am utc" now expects min(now.day, 30), so its expected value depends on the real clock. A fixed RELATIVE_BASE would make it deterministic. The "May 31" and "8am UTC" cases were also replaced, which loses the original coverage.
  • The test for "um 20 Uhr am 5. Mai 2021" expects midnight, so the 20:00 is dropped (the same as before this PR); the 10 Uhr case keeps its time. The "1990 am 3. März in" test also pins the stray trailing "in" in the span.
  • The regex is case-sensitive and runs per candidate in parse_item; precompiling it at module level would be cheaper.

@AdrianAtZyte
AdrianAtZyte merged commit 2f87862 into scrapinghub:master Oct 21, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants