feat(import): deterministic sidebar discovery for Fern sites - #41
feat(import): deterministic sidebar discovery for Fern sites#41minhthanhdang wants to merge 7 commits into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🔗 Linked repositories identifiedCodeRabbit considers these linked repositories for cross-repo context during reviews:
🚧 Files skipped from review as they are similar to previous changes (2)
WalkthroughThe importer probes Fern sites between Mintlify and Archbee detection. It parses Fern Next.js RSC payloads, merges selected tab trees, preserves hierarchy and metadata, excludes hidden or unsupported nodes, and removes duplicate pages. Fern navigation becomes the scrape result when available. Orphan matching now traverses nested pages. The test helper exposes Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
src/commands/import.js (1)
1523-1533: 🩺 Stability & Availability | 🟠 Major | 💤 Low valueIsolate Fern tab failures and resolve tab URLs with
new URL. A malformedpointsTovalue or a rejected tab fetch currently runs inside the function-level failure path, so one tab can discard the complete navigation tree. Resolve each tab URL withnew URL(tab.pointsTo, \${origin}/`), and wrap each tab's fetch and collection in its owntry/catch` so later tabs and entry categories survive independently.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/commands/import.js` around lines 1523 - 1533, Update the tab URL construction in the extractFernTabs loop to resolve tab.pointsTo with the URL API against origin instead of concatenating strings. Preserve the existing normalizePath comparison and fetch flow, ensuring leading-slash, absolute, relative, and empty tab targets resolve correctly. Apply the same fix in `@src/commands/import.js` around lines 1523 - 1533.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/commands/import.js`:
- Around line 1661-1685: Limit Fern page nesting to five levels before
categories are uploaded. Update addFernCategories, using the page structures
returned by fernPageFromNode and processed by dedupeFernPages, to flatten or cap
descendants deeper than five levels beneath each category while preserving valid
shallower hierarchy.
---
Nitpick comments:
In `@src/commands/import.js`:
- Around line 1523-1533: Update the tab URL construction in the extractFernTabs
loop to resolve tab.pointsTo with the URL API against origin instead of
concatenating strings. Preserve the existing normalizePath comparison and fetch
flow, ensuring leading-slash, absolute, relative, and empty tab targets resolve
correctly.
Apply the same fix in `@src/commands/import.js` around lines 1523 - 1533.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 208ff203-9706-4a18-b768-24ab68a8f1fd
📒 Files selected for processing (2)
src/commands/import.jssrc/commands/import.test.js
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
readmeio/ai(manual)readmeio/gitto(manual)readmeio/markdown(manual)readmeio/readme(manual)
…ent page" This reverts commit df7189b.
…, tab-title fallback for untitled roots
2 participants
🧰 Changes
Fern sites (vapi, cohere, elevenlabs, zep, hume, ...) imported with scattered structure: the Firecrawl sidebar scrape fails acceptance on them, so the pipeline clustered llms.txt pages by URL path into invented categories. On vapi that meant 21 made-up categories instead of the site's real 12-section sidebar.
Every Fern page embeds the complete nav tree in its SSR payload, so we now read the sidebar from there instead of guessing.
🧬 QA & Testing
Fern sites result
https://claude.ai/code/artifact/5386a2a1-f0d1-47df-8523-0c45d7d97e3a?via=auto_preview