Update aftonbladet - #953
Conversation
Extend parser to include version 1.1 for Aftonbladet articles. - Add new article metadata to `meta.info` - Enhance JSON structure with detailed body, sections, and headlines - Include support for multiple image versions with captions and descriptions
Ensure accurate selection of image byline by updating the XPath expression. - Change `@class='image-byline'` to `contains(@Class,'image-byline')` - Adjust type annotation from `re.Pattern[str]` to `Pattern[str]`
MaxDall
left a comment
MaxDall
left a comment
There was a problem hiding this comment.
REQUEST_CHANGES — V1_1 drops in-body <ul> lists, and image caption/credits are wrong on most images.
se.Aftonbladet: read 10 of 100 scanned (18 flagged, 8 in draw); layout coverage news / opinion / listicle / image-heavy, over-capture, image attributes, VALID_UNTIL and the V1→V1_1 bump all checked. 3 blockers + 2 nits inline.
Not inline:
- Live-chat and liveblog pages (
/a/Rr77qd/aftonbladet-direktplus twochatta-…pages, 3 of 100 scanned) extract an empty body. They aren't articles and the defaultonly_complete=Truecrawl drops them — no action needed, noted so the scan counts aren't read as a parser fault. - The 10 flagged articles outside the draw fall into three benign classes: 7× the site masthead
<dl class='_editorial_…'>on those body-less pages, 2× the "Fakta" side box (a widget appended after the running text, excluded by V1 as well), 1× an image-gallery<ul>whose text is caption + credits rather than body prose.
|
|
||
| class V1_1(V1): | ||
| _summary_selector = XPath("(//header)[2]/p") | ||
| _paragraph_selector = XPath("(//section[@class='article-body'])[1]/p") |
There was a problem hiding this comment.
Blocker — <ul> lists inside the body are dropped. The p-only selector leaves the body reading "Nyckelfunktioner:" followed by nothing; the list that belongs there — "Självrengörande dockningsstation som tömmer dammbehållaren, tvättar mopparna och fyller på vatten automatiskt…" — is missing (9 such <ul class='_list_16k26_3 … layout-component'>, all direct children of section.article-body, in this one article). Fix: extend _paragraph_selector with | (//section[@class='article-body'])[1]/ul/li. [1]
| _paragraph_selector = XPath("(//section[@class='article-body'])[1]/p") | ||
| _subheadline_selector = XPath("(//section[@class='article-body'])[1]/h2") | ||
|
|
||
| _caption_selector = XPath("./ancestor::figure//figcaption/span") |
There was a problem hiding this comment.
Blocker — gallery figures extract neither caption nor credits. Their caption sits directly in the element, with no <span> child for /span to match: <figcaption class='_srOnly_…'>Över 49 000 har tagit sig över gränsen från Marocko senaste dygnen. <cite>Foto: Antonio Sempere /AP/TT</cite></figcaption>. 37 of the 42 images across the draw come back caption=None, authors=[] (all 13 in [1], all 11 in the robot-vacuum guide). Fix: point _caption_selector at the figcaption element itself. [2]
| _subheadline_selector = XPath("(//section[@class='article-body'])[1]/h2") | ||
|
|
||
| _caption_selector = XPath("./ancestor::figure//figcaption/span") | ||
| _image_author_selector = re.compile(r"(?i)foto:\s*(?P<credits>.*)\s*$") |
There was a problem hiding this comment.
Blocker — the "Visa mer" toggle leaks into image credits. figcaption ends with <span class='_showMore_…' hidden>Visa mer</span>, which the caption selector picks up and (?P<credits>.*)\s*$ then swallows: every non-empty authors in the draw ends that way, e.g. ['Emma Da Silva/AP/TT Visa mer'] (also 4 more), and the fixture added here records it for all six images ("Robin Lorentz Allard Visa mer"). Fix: exclude span[contains(@class,'showMore')] in _caption_selector. [1]
| ) | ||
| _image_selector = XPath("//figure//img") | ||
|
|
||
| _paywall_selector = XPath("//main/vev") |
There was a problem hiding this comment.
Nit — _paywall_selector is inherited unchanged and I could not confirm it still marks the new layout. //main/vev matches nothing in any of the 10 crawled pages, but all 10 are free ("isAccessibleForFree": true), so free_access=True is correct for them and the selector stays untested against a Plus article. Worth verifying on one — the new layout carries isAccessibleForFree in its ld+json.
| ) | ||
|
|
||
| class V1_1(V1): | ||
| _summary_selector = XPath("(//header)[2]/p") |
There was a problem hiding this comment.
Nit — positional selectors where a class is available. (//header)[2] and (//section[@class='article-body'])[1] select by position, yet the elements they hit carry class='article-intro' and class='article-body'. If a page ever grows another <header>, the summary silently comes from the wrong element rather than extracting nothing. //header[@class='article-intro']/p is equivalent today and fails loudly.
… figcaption and re-record fixture
MaxDall
left a comment
MaxDall
left a comment
There was a problem hiding this comment.
@addie9800 Thanks a lot for addressing these layout changes ❤️
2 participants
No description provided.