Claude/spac ipo ingest issues p8x1n2 - #288
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens SPAC ingestion and identity normalization by (1) fixing company-name normalization edge cases that caused canonical splits/collisions, (2) improving S-1 segmentation/classification so post–de-SPAC registrations don’t incorrectly mint SPAC rows, and (3) adding operator tooling to repair identity tiers without full re-extraction.
Changes:
- Add EDGAR jurisdiction-suffix stripping and make company-ending stripping safe against regex metacharacters (prevents canonical collisions/splits).
- Extend SPAC candidate scanning/classification with an “as-filed header SIC 6770” signal sourced from parsed S-1 classifications.
- Improve S-1 segmentation reliability (swallowed-section truncation + line-scan fallback) and record a dedicated
CONVERTER_NO_STRUCTUREdiagnostic; addsec resolve --renormalizeandsec canonical suggest-aliases.
Reviewed changes
Copilot reviewed 31 out of 31 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/util/dataCleaningUtils.ts | Adds stripEdgarJurisdictionSuffix to remove trailing EDGAR /DE-style disambiguators before normalization. |
| src/task/spac/spacCandidateScan.ts | Adds filed_sic_6770 signal via s1_classification and keeps SQL vs repo implementations aligned. |
| src/task/spac/spacCandidateScan.sqlite.test.ts | Ensures the sqlite-vs-repository parity test registers the new classification repo dependency. |
| src/task/spac/IdentifySpacsTask.sqlite.test.ts | Updates pruning test setup to include the S-1 classification repository. |
| src/task/spac/classifySpacCandidate.ts | Incorporates signal_filed_sic_6770 into candidate gating and confidence grading. |
| src/task/spac/classifySpacCandidate.test.ts | Adds coverage for the new as-filed header SIC signal (including null/unknown semantics). |
| src/task/resolve/ResolveObservationsTask.ts | Adds optional renormalize pass to recompute derived identity columns before resolving. |
| src/task/resolve/renormalize.test.ts | Tests --renormalize behavior for both company and person observations (including idempotence). |
| src/task/canonical/SuggestAliasesTask.ts | Adds task to suggest aliases by scanning EDGAR entity name history under one CIK. |
| src/task/canonical/suggestAliases.ts | Implements edit-distance-based alias suggestion logic keyed by canonical identity inputs. |
| src/task/canonical/suggestAliases.test.ts | Tests alias suggestion behavior and edit-distance cutoff logic. |
| src/storage/spac/SpacCandidateSchema.ts | Adds persisted signal_filed_sic_6770 field to SPAC candidate records. |
| src/storage/observation/PersonObservationRepo.ts | Adds updateNormalizedParts to update only derived person identity columns. |
| src/storage/observation/CompanyObservationRepo.ts | Adds updateNormalizedName to update only derived company identity columns. |
| src/storage/dead-letter/ExtractionDeadLetterSchema.ts | Adds CONVERTER_NO_STRUCTURE dead-letter reason code. |
| src/storage/company/CompanyNormalization.ts | Fixes ending stripping to treat literals safely; strips EDGAR jurisdiction suffix early in normalization. |
| src/storage/company/CompanyNormalization.test.ts | Adds regression tests for literal-safe endings and jurisdiction suffix stripping. |
| src/storage/company/CompanyFamilyName.ts | Applies jurisdiction-suffix stripping to family-key derivation to avoid split families. |
| src/storage/company/CompanyFamilyName.test.ts | Adds tests to ensure /Cayman-style suffixes don’t split sponsor families. |
| src/storage/classification/S1ClassificationSchema.ts | Expands classifier_source description to include sgml-header-rejected. |
| src/sec/html/parseEdgarHtml.golden.test.ts | Adjusts expected sections/floors after segmenter improvements (incl. RELATED_PARTY expectations). |
| src/sec/forms/registration-statements/s1/headerSicDowngrade.test.ts | Adds focused tests for downgrading stale 6770 header SIC based on summary content. |
| src/sec/forms/registration-statements/s1/DocumentTreeSegmenter.ts | Adds swallowed-section truncation, general nesting fallback (excluding restating containers), and line-scan fallback. |
| src/sec/forms/registration-statements/s1/DocumentTreeSegmenter.test.ts | Adds test coverage for swallowed sections, line-scan fallback, and generalized nesting logic. |
| src/sec/forms/registration-statements/s1/DocumentSegmenter.ts | Extends heading patterns (foreign private issuer variants, “Certain Transactions”, offering variants). |
| src/sec/forms/registration-statements/Form_S_1.storage.ts | Records converter-structure failures, integrates new segmenter API, and adds header-SIC downgrade logic. |
| src/resolver/EntityObserver.ts | Extracts normalizePersonNameParts to share normalization logic with batch re-normalization. |
| src/eval/goldenS1Labels.ts | Adds/updates golden related-party labels for filings headed “CERTAIN TRANSACTIONS”. |
| src/cli/groups/resolve.ts | Adds --renormalize flag wiring for batch resolve CLI. |
| src/cli/groups/canonical.ts | Adds sec canonical suggest-aliases command with text/TSV output modes. |
| CLAUDE.md | Documents new tooling/behavior (renormalize, alias suggestion, segmentation + SIC downgrade details). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+189
to
+193
| const literal = stripLiteralSuffix(normalized); | ||
| if (literal !== null && literal !== normalized) { | ||
| suffix = normalized.slice(literal.length).trim() || null; | ||
| normalized = literal; | ||
| original = normalized; |
…ations
Company identity keys: `COMPANY_ENDINGS_TO_STRIP` carried the literal
`[related person is an entity]` into `new RegExp("\\b" + ending + "\\b$")`,
where its brackets are a character class. It deleted the final single-letter
word of any name drawn from {r,e,l,a,t,d,p,s,o,n,i,y}, so `Churchill Capital
Corp I` normalized to `Churchill Capital` and `Reinvent Technology Partners Y`
collided with `Reinvent Technology Partners` — two distinct SPACs, one
canonical company. The same list backs `hasCompanyEnding`, the
person-vs-company discriminator on Forms D/C/1-A/1-Z/3/4/5/144, which read
`Klein Michael S` as a company; the class also contained a literal space, so
`hasCompanyAnywhere` was true for every multi-word string. Literal suffixes are
now matched as text and word-shaped endings are escaped.
EDGAR's state-of-incorporation marker (`/DE`, `/CI`, `/Cayman`) is stripped by
a shared helper before either normalizer tokenizes. It split one sponsor's
vehicles across two families (`churchill-capital` vs
`churchill-capital-corp-cayman`) and blocked the legal-form strip behind it, so
`Blue Acquisition Corp/Cayman` minted a second canonical company.
Segmenter: a chosen section now stops where it has swallowed another chosen
section's body, unless the containment is one prospectuses really have. A
mis-leveled all-caps heading nests the rest of the document, which rendered
committed summaries of 966k and 1,008k characters and pushed one filing's risk
factors past MAX_RISK_FACTORS_CHARS so the disclosure was never extracted.
Heading vocabulary: `CERTAIN TRANSACTIONS` (the older SPAC spelling of Item
404, fatal in 3 of 62 sampled filings) and the Form 20-F spellings an F-1 uses
for management, ownership and the offering summary.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N6p43D17X9u7RE9hMyufwT
… minting a SPAC Line-scan fallback: when the tree walk resolves fewer than two target sections the segmenter scans the rendered text with the same heading patterns, slicing each hit to the next. Bridgetown Holdings' 3.2MB prospectus is typeset inside 295 tables, so the converter emits 4 heading nodes and the filing extracted nothing at all despite 97% of its text being present; it now recovers all ten target sections. The threshold is deliberately tiny — a line scan has no structural evidence and cannot tell a table-of-contents entry from the heading it points at — and no other filing in the committed corpus or a 62-filing sample falls below it. Recovery is still recorded: a new CONVERTER_NO_STRUCTURE reason code, under its own `converter` section name rather than the filing-level "" key that ProcessAccessionDocFormTask resolves after a successful store. Eight SECTION_NOT_FOUND entries were indistinguishable from a legitimately incorporation-by-reference S-1. Header-SIC downgrade: the SIC on a post-de-SPAC registration statement is stale — the surviving company keeps the shell's CIK and EDGAR keeps coding it 6770 — so a 6770 header now has to agree with the prospectus summary before it mints a known-SPAC row and gates the 8-K / merger-proxy / Form 25-15 tier. The gate reads the summary rather than the whole document, which a de-SPAC prospectus fills with its own SPAC history, and only a substantial summary can demote: silence is evidence only where there was room to speak. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N6p43D17X9u7RE9hMyufwT
…IC signal ResolveObservationsTask gains --renormalize: recompute each observation's derived identity columns from the name as filed, then resolve. Both tiers already persist the raw parts beside the derived ones, so the recompute is local — and it calls the same helpers the extraction path writes with (normalizePersonNameParts, normalizeCompanyName) so a second implementation cannot drift and re-key half the tier to a generation nothing else produces. Without it a normalizer change reached the database only by re-extracting every observing filing and re-paying its AI cost, which is why several sat unfixed. `sec canonical suggest-aliases` finds filers EDGAR has carried under two spellings of one name. Five SIC-6770 registrants were filed under a conformed name misspelling `Acquisition`; EDGAR later corrected the entity name, but the accession keeps the typo, so each issuer normalizes two ways and mints two canonical companies and two families. No normalizer can close that — the two are different words — but the evidence that they are one entity is that EDGAR filed both under one CIK. Pairs whose difference is only a series marker are excluded via the family key: `Corp I` is one edit from `Corp II` and they are different companies. spac_candidate gains signal_filed_sic_6770, read from s1_classification.sic where processFormS1 already writes the header SIC it parsed. It is the only signal a completed de-SPAC cannot erase — it recodes and renames, so the other three vanish together — and it grades `high`, since a registration filed under a 6770 header is a blank-check IPO by construction. Null means no registration has been parsed yet, which is not the same as false. Also in the segmenter: the line-scan fallback needs the document to render at least 50k characters before its lack of structure counts as a converter failure rather than as a short filing, and it slices each hit to the next hit of a DIFFERENT target — a typeset prospectus repeats its section name as a page header, which chopped Bridgetown's risk factors into a 5k fragment of a 177k section. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N6p43D17X9u7RE9hMyufwT
…401 heading Found by probing 30 further SPAC registrations against the fixed pipeline. `Lucent, Inc.` (CIK 1778343), a Montana shell registering a $30,000 offering, states outright that "the proposed business activities described herein classify the Company as a 'blank check' company" — and the header-SIC downgrade demoted it anyway, deleting its spac row. `looksLikeBlankCheck` needs two distinct signals by default, and that phrase is the only one such a filing has: a shell this size has no trust account, no founder shares and no sponsor, so it never reaches the SPAC-IPO vocabulary the signal list is built from. The two callers ask the same question with opposite error costs — as the AI pre-filter a false negative skips a model call, here it drops the whole 8-K / merger-proxy / Form 25-15 tier — so the demotion now requires a summary carrying NO blank-check signal at all. Measured over the committed corpus, all 20 labelled SPAC summaries carry at least two and every non-SPAC filed under a 6770 header (Ionetix, Moolec, Zhong Yuan) carries zero, so demoting only at zero separates every observed case. Also: the Item 401 heading a smaller reporting company uses adds a "promoters and control persons" clause, which the management patterns did not match — `Allure Worldwide` yielded no people because of it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N6p43D17X9u7RE9hMyufwT
The header-SIC downgrade judged every filing afresh on its prose, with no regard for whether the CIK was already a known SPAC. But a CIK that once registered as a blank check stays a SPAC CIK for good: the shell keeps its CIK through the combination and renames, which is precisely what the spac row's three eras (spac_* / post_merger_* / current_*) exist to model, and what recordDeSpacLinkage writes into them. A post-combination registration statement therefore reads like the operating company it now is — the expected shape, not evidence the vehicle was never a SPAC — so the content gate detached such a filing from the very lifecycle row it belongs to. The gate now consults the spac row first and only judges prose for a CIK nothing knows about yet, where the question is whether to MINT a row on the strength of a stale header rather than whether an established vehicle counts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N6p43D17X9u7RE9hMyufwT
Found in an unbiased systematic sample of 30 further SPAC registrations. `Mammon Omicron Acquisition Corp` marks `The Offering` as a bolded paragraph rather than a structural heading, so the tree walk saw no section for it and the offering-terms extractor got nothing — while 90k characters of unit terms sat inside the prospectus summary, which is exactly where LEGITIMATE_CONTAINMENTS already expects a summary to carry them. That is the same shape NESTED_SECTION_FALLBACKS already handles for the Item 402 disclosure nested in MANAGEMENT, so it is the same one-entry fix. It fires only when the tree walk found no section for the target, so a filing whose offering has a real heading is untouched — and no committed fixture changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N6p43D17X9u7RE9hMyufwT
Third instance of one shape: a filer bolds a section label instead of giving it a structural heading, so the tree walk sees no section and the extractor gets nothing. `TCG Growth Opportunities Corp.` prints `Principal stockholders` directly after the management roster that way, losing its whole ownership table. Generalizing the fallback — every missing target against every resolved container — was measured and rejected: it adds a section to 6 of the 45 committed fixtures with wrong slices, including a 208k "The Sponsor" carved out of a summary and a 135k "Management" for a filing whose roster is documented as bolded paragraphs with no section at all. findNestedSection slices to the next known heading line, which is the right boundary only for pairs where the block really runs that far. So this is one more measured pair: the committed corpus gains nothing from it, and TCG recovers a 10,817-char ownership section. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N6p43D17X9u7RE9hMyufwT
The three hand-declared (target, container) pairs were shipped on a measurement that generalizing added 6 wrong sections to the committed corpus. Those 6 were not scattered: every one came out of a `Prospectus Summary`, whose job is to restate the prospectus by name, so each bolded label in it is a cross-reference rather than a block. `RESTATING_CONTAINERS` names that container, the fallback is general over every other resolved section — tightest enclosing body first — and the declared list keeps only the summary's own offering table, consulted after the general containers so a real body section always wins. Zero additions across all 42 committed fixtures and no golden test moves, so the generalization is free. On a real filing outside the corpus (CIK 2042460) it recovers a 20,238-char `The Sponsor` block out of `Management`, a pair no ranking of observed section order would have named. A slice-size guard was measured as the alternative and does not separate the cases: five bad summary slices run 68-96% of their container while the trusted compensation-inside-Management recovery runs 7-81% across 18 fixtures, 14 of them above 68%, and the sixth bad slice is 14%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N6p43D17X9u7RE9hMyufwT
Each one lost a whole section on a real filing: - `UNDERWRITING—CONFLICTS OF INTEREST` (TPG Pace Beneficial Finance, 1819399). The FINRA Rule 5121 qualifier was matched only in its parenthesized spelling, so an em-dash clause dropped the underwriter list. Now either punctuation. - `DIRECTORS AND MANAGEMENT` (Fusion Fuel Green, 1819794). The Form 20-F Item 6 pattern alternated " senior" / " and executive", so a bare conjunction did not reach `management` and the F-1 yielded no roster. - `PRINCIPAL STOCKHOLDERS3` (BurTech Acquisition, 1871638), the anchor's superscript fused onto the heading. `matchTarget` retries once with a trailing 1-3 digit marker removed, and only when the heading did not match as printed, so it cannot change what an unambiguous heading resolves to. Adds `sectionHeadingPatterns.test.ts`, which pins every heading variant these rounds have added against the CIK it came from, plus the body prose whole-line anchoring must keep rejecting. Nothing pinned this surface before, and it is the one the rounds keep touching. Not fixed, and recorded rather than patched: `Maywood Acquisition Corp.` (2028355) typesets its entire offering block inside a bordered table, so the converter fuses the heading onto the first sentence of the body and `THE OFFERING` is a line of its own nowhere in the document. Across 91 filings (42 committed fixtures, 49 real submissions) that is the only target recoverable solely from a glued heading, and the rule to reach it would be a caps-run heuristic rather than a pattern. It lands on the dead-letter worklist as SECTION_NOT_FOUND, which is the honest outcome. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N6p43D17X9u7RE9hMyufwT
`suffix` and `original` are assigned in both stripping branches and never read — the function returns `normalized` alone. They predate the literal / pattern split and survived it unchanged. Behavior is identical; the loop now says only what it does. Reported by Copilot on #288. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N6p43D17X9u7RE9hMyufwT
sroussey
force-pushed
the
claude/spac-ipo-ingest-issues-p8x1n2
branch
from
August 15, 2026 04:41
4415f47 to
bb90329
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.