Repository navigation
AV1 support for HW decoding #1690
Description
Overview
#1685 implemented HW support for CUDA devices. This support works for H264, VP9 but not AV1 (on supported devices)
Expected behavior
Decoding to be done on GPU.
Actual behavior
Decoding is done in software (even though allow_software_fallback is False)
Traceback:
n/a
Investigation
sample media: av1.webm
import av
import time
file = 'av1.webm'
hwaccel = av.codec.hwaccel.HWAccel(device_type='cuda', allow_software_fallback=False)
container = av.open(file, hwaccel=hwaccel)
start_time = time.time()
frame_count = 0
for packet in container.demux(video=0):
for _ in packet.decode():
frame_count += 1
hw_time = time.time() - start_time
hw_fps = frame_count / hw_time
container.close()
print(f"Decoded with cuda in {hw_time:.2f}s ({hw_fps:.2f} fps).")Sanity Check:
FFmpeg:
$ ffmpeg -c:v av1_cuvid -i av1.webm -f null -
...
frame= 300 fps=0.0 q=-0.0 Lsize=N/A time=00:00:10.00 bitrate=N/A speed=17.3x
# 30 FPS * 17.3x realtime ~ 500FPS decode speed
Py Sample:
$ python test.py
Decoded with cuda in 3.18s (94.25 fps).
Reproduction
see above
Versions
- OS: ubuntu 24.04
- PyAV runtime:
Built from master, post PR Implemented hardware decoding #1685
PyAV v14.0.1
library configuration: --disable-static --enable-shared --libdir=/tmp/vendor/lib --prefix=/tmp/vendor --disable-alsa --disable-doc --disable-libtheora --disable-libfreetype --disable-libfontconfig --disable-libbluray --disable-libopenjpeg --disable-mediafoundation --enable-gmp --enable-gnutls --enable-libaom --enable-libdav1d --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-libspeex --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libxml2 --enable-lzma --enable-zlib --enable-version3 --enable-libx264 --disable-libopenh264 --enable-libx265 --enable-gpl
library license: GPL version 3 or later
libavcodec 61. 19.100
libavdevice 61. 3.100
libavfilter 10. 4.100
libavformat 61. 7.100
libavutil 59. 39.100
libswresample 5. 3.100
libswscale 8. 3.100
- I am/tried using the binary wheels
- I compiled from source
Research
I have done the following:
- Checked the PyAV documentation
- Searched on Google
- Searched on Stack Overflow
- Looked through old GitHub issues
Additional context
Tests done on an RTX4090, reproduced also on an L4 in GCP
Activity
matthewlai commentedon Dec 23, 2024
A few questions:
nvidia-smiwhile the decode is happening.ffmpeg -hwaccel cuda -i av1.webm -f null -legraphista commentedon Dec 23, 2024
I re-compiled ffmpeg without libaom and the above code stopped working, whereas the ffmpeg command that uses HW decoding continued to work
Full Logs
Tested on an GCP L4.
av1.mkv and av1.webm are the exact same file, github wouldn't allow me to upload an mkv.
Yes, compiled ffmpeg with CUDA support & shared libraries, installed on the system (sudo make install && sudo ldconfig) before (re-)installing pyav from master
matthewlai commentedon Dec 23, 2024
If you run
python -m av --hwconfigsdo you get a hwconfig for av1?Can you do
print(container.streams.video[0].codec_context.is_hwaccel)?The way allow_software_fallback works right now is a bit misleading. It controls the case where the hardware decoder can be found and opened, but something goes wrong trying to set it up to decode the actual video (eg. resolution too high, subsampling mode not supported, etc). In that case if allow_software_fallback is set, we allow the decoder to do software decoding instead.
In the case that the hardware decoder is never there to begin with, we silently fall back to software decoding, regardless of the allow_software_fallback setting. This is necessary because in many files you have multiple streams with different encodings, and there will be at least one that can't be hardware decoded (eg MJPEG). We don't want to fail in those cases. Maybe we can raise an exception if no stream can be accelerated.
legraphista commentedon Dec 23, 2024
strangely, the paths diverge here.
On the GCP L4, I cannot decode the stream.
But on my PC where I have libaom compiled and configured (and can decode the file using the weird software unintended fallback, verified with nvtop), I get the following:
both systems have AV1 HW support