feat(publisher): add V2 parser for TheGatewayPundit - #956
Conversation
…abilities Introduce version 2 parser for TheGatewayPundit to improve parsing precision. - Add support for parsing nested sections, summaries, and enhanced paragraph filtering - Update image extraction and metadata integration - Deprecate older sitemap and introduce updated sitemap handling - Validate article versioning with expiration date - Extend test suite with new sample articles and metadata
MaxDall
left a comment
MaxDall
left a comment
There was a problem hiding this comment.
REQUEST_CHANGES — V2's body selectors only reach direct children of entry-content, so on nearly half the pool part or all of the article text is lost.
us.TheGatewayPundit: read 10 of 100 scanned (84 flagged, 10 in draw); layouts, over-capture, image attributes checked. 4 blockers + 2 nits inline.
Not inline:
- Blocker — 24 of 100 pool articles came back with no
bodyat all, and fundus' default crawl drops every one. They are not all sponsored posts — e.g.us-launches-second-consecutive-night-iran-attacks-powerful,russia-charges-telegram-founder-pavel-durov-aiding-terrorism,breaking-dr-fauci-invokes-5th-amendment-will-not. The two that landed in the cached draw are explained by theh3comment below; the rest I did not diagnose (they need a second crawl). Worth re-checking the whole class once the selector fixes land.
V1's VALID_UNTIL / the V2 major bump / the added fixture all look right, and the site is fully free so free_access is not needed.
| namespaces={"re": "http://exslt.org/regular-expressions"}, | ||
| ) | ||
| _paragraph_selector = XPath( | ||
| f"(//div[@class='entry-content'] | //div[@class='entry-content']/blockquote[not(@class='twitter-tweet')]) " |
There was a problem hiding this comment.
Blocker — paragraphs nested below entry-content are dropped, because both steps require a direct child. On [1] the last 7 of 15 paragraphs are lost, ending with "…History shows that you can vote your way into socialism, but you can seldom vote your way out of it." — Ad Inserter wraps the article tail in div.ai-viewport-*/div.code-block/div.ai-dynamic, so //div[@class='entry-content']/p and /blockquote never see it. 45 of 100 pool articles show the same <p>-dropped signature, up to 3,762 uncaptured chars. Fix: make both steps descendant (//div[@class='entry-content']//p, //blockquote[not(@class='twitter-tweet')]).
|
|
||
| _summary_selector = XPath( | ||
| f"//article//p[not(text())]/strong[text() and not(re:test(text(), '{_related}'))] |" | ||
| f"//div[@class='entry-content']/h3", |
There was a problem hiding this comment.
Blocker — entry-content/h3 routes an entire article into the summary, leaving zero paragraphs and an empty body. Posts that render every line as <h3> extract as summary-only: "…Treating cancer is big business and one that makes the big pharmaceutical companies…" lands in summary, _paragraph_selector matches nothing, and body comes back empty (also on [2]). Fix: pass h3 to subheadline_selector rather than folding it into summary_selector. [1]
There was a problem hiding this comment.
Unfortunately, I don't really see a solution to this. :/ And since the articles in the example are the less frequent ones, I would stick to this version.
| doc=self.precomputed.doc, | ||
| paragraph_selector=self._paragraph_selector, | ||
| image_selector=XPath("//div[@class='entry-content']//img"), | ||
| author_selector=XPath("./ancestor::figure//figcaption"), |
There was a problem hiding this comment.
Blocker — image authors is the caption verbatim, not a credit. author_selector picks the same <figcaption> the default caption_selector reads, so both come back identical: on [1] caption="Schumer derailed by heckler during anti-corruption speech" and authors=["Schumer derailed by heckler during anti-corruption speech"] (also "Screenshot" on [2], and the committed V2 fixture). TGP's figcaption holds no credit element — fix: drop author_selector and keep the default. V1 line 54 has the same bug, now that the layout emits captions.
| sitemap_filter=inverse(regex_filter("post-sitemap")), | ||
| reverse=True, | ||
| "https://www.thegatewaypundit.com/sitemap.xml", | ||
| sitemap_filter=inverse(regex_filter("sitemap-pt_post")), |
There was a problem hiding this comment.
Blocker — the new sitemap yields URLs from other domains. 5 of 100 crawled URLs were joehoft.com (3) and gatewayhispanic.com (2) — separate publications parsed as The Gateway Pundit; the Gateway Hispanic pages have a different layout and return neither title nor body, e.g. [1]. Fix: tighten sitemap_filter to the TGP host, or add a url_filter restricting to www.thegatewaypundit.com.
| "https://www.thegatewaypundit.com/sitemap_index.xml", | ||
| sitemap_filter=inverse(regex_filter("post-sitemap")), | ||
| reverse=True, | ||
| "https://www.thegatewaypundit.com/sitemap.xml", |
There was a problem hiding this comment.
Nit — the NewsMap source was removed without a replacement. NewsMap("https://www.thegatewaypundit.com/news-sitemap.xml") is gone, leaving the sitemap as the only source, so recency-oriented crawls lose their news map. Intentional (news sitemap retired alongside the layout change), or fallout from the sitemap URL swap? Crawl ordering itself looks fine without reverse=True — 95 of 100 drawn URLs were from the current month.
|
|
||
| class V2(BaseParser): | ||
| _related = ( | ||
| r"(?i)^(Click|This article appeared originally|(read )?more:|watch:|more from .{1,20}:|this video is)\s*" |
There was a problem hiding this comment.
Nit — _related misses lead-ins whose source name is a link. re:test(text(), …) only sees a node's first text child, so <p>More from <a>Fox</a>:</p> is tested as "More from ", never matches more from .{1,20}:, and the stub line leaks into the body on [1]. Fix: test the element's full string value (normalize-space(.)) instead of text().
MaxDall
left a comment
MaxDall
left a comment
There was a problem hiding this comment.
@addie9800 Thanks so much for adapting to the lastest layout changes 🚀
2 participants
Introduce version 2 parser for TheGatewayPundit to improve parsing precision.