Skip to content

Speed up reading packages and fix some bugs - #37

Closed
w1tcherrr wants to merge 10 commits into
ValveResourceFormat:masterfrom
w1tcherrr:perf
Closed

w1tcherrr wants to merge 10 commits into
ValveResourceFormat:masterfrom
w1tcherrr:perf

Conversation

@w1tcherrr

@w1tcherrr w1tcherrr commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Changes

  • Archive files are opened and memory mapped once per package. GetMemoryMappedStreamIfPossible returns a lightweight stream over that mapping. ReadEntry reads with RandomAccess from the cached handles.
  • Both are now thread-safe.
  • The directory tree is parsed from a single buffer. Sorting is unchanged.
  • Verification reads into reused buffers.

Benchmark

Benchmark Unit Before After
Open CS2 pak01_dir.vpk ms 50.6 27.9
Open Dota 2 pak01_dir.vpk ms 210.6 101.7
Mapped stream + read, entries up to 4 KiB us 36.70 1.13
Mapped stream + read, entries of 16-256 KiB us 44.10 3.33
Mapped stream + read, entries of 4 MiB or more us 3137 321
ReadEntry, entries up to 4 KiB us 37.4 3.7
Mapped reads on 16 threads, entries up to 4 KiB reads/s 25,300 7,890,000
VerifyChunkHashes, CS2 archives 0-5 ms 1358 626
VRF loading 7,397 CS2 resources, 1 thread ms 544 320
VRF loading 7,397 CS2 resources, 16 threads ms 283 203

Fixes

  • Concurrent ReadEntry calls could return wrong data.
  • Entries cut off by the end of an archive were returned zero-filled. They now throw EndOfStreamException.
  • Calling Read again kept the previous package's signature and checksums, and IsDirVPK never reset in SetFileName.
  • AddFile on an optimized package appended unsorted, so FindEntry missed the file.
  • Read(Stream) and Write(Stream) broke when the package did not start at position 0, and writing into a longer stream hashed the leftover bytes.
  • Malformed signature and archive MD5 section sizes now throw InvalidDataException.

Behaviour changes

  • GetMemoryMappedStreamIfPossible returns an UnmanagedMemoryStream instead of MemoryMappedViewStream in 64-bit processes. Returned streams can only be read while the package is open, they throw ObjectDisposedException once it is disposed or read again.
  • Archive files stay open and mapped until the package is disposed, and chunk files use FileShare.ReadWrite.
  • ReadEntry, GetMemoryMappedStreamIfPossible and the Verify* methods throw ObjectDisposedException after Dispose.

I ran this PR over hundreds of different game paks and they read identical to before.

@xPaw xPaw closed this in 6d635de Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant