A small, practical C# sample that crawls a modern JavaScript-rendered page with Puppeteer Sharp, then parses and extracts structured data with AngleSharp.
The demo targets the public .NET organization repositories page on GitHub and prints the extracted repository list as JSON.
Many real-world pages are no longer static HTML. This repository shows a maintainable pattern for:
- Rendering dynamic pages in a headless Chromium browser (
PuppeteerSharp) - Parsing the resulting DOM with a standards-based HTML parser (
AngleSharp) - Mapping DOM nodes into typed models and serializing them with
System.Text.Json
It is intended as a learning / reference project for .NET developers building crawlers, scrapers, or HTML extraction tools.
- .NET 10 SDK
- Network access on first run (downloads a Chromium revision used by Puppeteer Sharp)
dotnet restore
dotnet run --project src/CrawlerSamples.ConsoleAppOr open CrawlerSamples.sln in Visual Studio / VS Code / Rider and run the console app.
dotnet publish src/CrawlerSamples.ConsoleApp -c Release -r win-x64 --self-contained true -o build/win/net10.0/x64You can also use the included publish profiles under src/CrawlerSamples.ConsoleApp/Properties/PublishProfiles/.
The app prints a JSON array of repository models, for example:
[
{
"url": "/dotnet/runtime",
"visibility": "Public",
"title": "runtime",
"description": ".NET is a cross-platform runtime...",
"language": "C#",
"license": "MIT"
}
]Screenshot:
CrawlerSamples.sln
src/CrawlerSamples.ConsoleApp/
Program.cs # Crawl + parse flow
RepoModel.cs # Extracted model + JSON source generation
- On first run, Puppeteer Sharp downloads a Chromium package into a local cache. This can take a while depending on your network.
- If the Chromium download fails, the process exits with an exception.
- GitHub's UI markup can change. If selectors stop matching, update them in
CreateModelWithAngleSharp. - The sample waits for a key press only in interactive terminals; redirected / CI environments exit immediately after printing results.
See CONTRIBUTING.md. Please open issues and pull requests — CI builds on every PR.
See SECURITY.md.