Skip to content

Fix hardware decoder selection and software fallback - #1110

Open
rajeshgk wants to merge 1 commit into
OpenShot:developfrom
rajeshgk:fix/hw-decoder-selection
Open

rajeshgk wants to merge 1 commit into
OpenShot:developfrom
rajeshgk:fix/hw-decoder-selection

Conversation

@rajeshgk

Copy link
Copy Markdown
Contributor

Summary

This pull request improves FFmpegReader hardware decoder selection, initialization, and fallback behavior.

Changes

  • Keep hardware decoder state specific to each FFmpegReader instance rather than sharing mutable global decoder state.
  • Preserve the selected hardware decoder throughout reader initialization and pixel-format negotiation.
  • Improve platform-specific hardware backend selection for Linux, Windows, and macOS.
  • Correct CUDA adapter selection to use a GPU index rather than a DRM render-node path.
  • Fall back to software decoding when hardware device initialization fails.
  • Improve hardware device context cleanup during initialization, retries, and reader closure.
  • Add a deterministic regression test that forces hardware device initialization failure and verifies successful software decoding.

Testing

Environment: Linux, hardware acceleration enabled.

  • Full C++ build: passed.
  • Python and Ruby SWIG bindings: built successfully.
  • FFmpegReader test suite: 266 assertions across 30 test cases passed.
  • Deterministic CUDA initialization failure regression: passed.
  • git diff --check: passed.

Expected behavior

When a selected hardware decoder cannot initialize, FFmpegReader should retry with software decoding rather than fail to open an otherwise decodable media file.

The decoder selection is maintained per reader to avoid unintended interference between readers.

Platform validation

Linux build and regression tests have passed. Native Windows and macOS builds have not yet been performed.

Keep hardware decoder state per FFmpegReader instance.

Preserve the selected backend during pixel format negotiation.

Fall back to software decoding when hardware device initialization fails.

Improve hardware device context cleanup and retry handling.

Add deterministic CUDA initialization failure regression coverage.
@rajeshgk

Copy link
Copy Markdown
Contributor Author

Hi @jonoomph,

I would appreciate your review of this pull request, which addresses the CUDA hardware-decoder selection issue reported in #1017.

The changes preserve explicit hardware-decoder selection, eliminate shared mutable decoder state, correct CUDA device indexing, and introduce a software-decoding fallback when hardware initialization fails.

Validation completed:

  • Full C++ build passed.
  • Python and Ruby SWIG bindings built successfully.
  • FFmpegReader tests passed: 266 assertions across 30 test cases.
  • Deterministic hardware-initialization failure regression passed.
  • Codacy static code analysis passed.

The PR is currently mergeable without conflicts. Native Windows/macOS and CUDA-capable hardware testing remain areas for additional validation.

Your feedback on the decoder initialization, fallback behavior, and cross-platform implications would be particularly valuable.

Thank you for your time and consideration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant