[INVESTIGATION] Proxy clips not generating with the default profiles

Help
1

I had this issue where I could only get proxy clips to generate using software encoding. VAAPI encoding jobs would just hang. I found a solution, but am trying to figure out what’s wrong. I don’t really understand why this solution works or what the root cause is.

Here is my hardware and vaapi info:

$ vainfo
Trying display: wayland
vainfo: VA-API version: 1.22 (libva 2.22.0)
vainfo: Driver version: Mesa Gallium driver 25.1.4-arch1.2 for AMD Radeon RX 7700 XT (radeonsi, navi32, LLVM 20.1.6, DRM 3.61, 6.12.35-1-lts)
vainfo: Supported profile and entrypoints
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSlice
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileAV1Profile0            : VAEntrypointEncSlice
      VAProfileNone                   : VAEntrypointVideoProc

Here is the profile

-init_hw_device vaapi=vaapi0: -filter_hw_device vaapi0 -i -vf scale=%width:-2,format=nv12,hwupload -vsync 1 -c:v h264_vaapi -g 1 -bf 0 -qp 26 -c:a aac -ab 128k

Here is the ffmpeg command that kdenlive wants to run for the vaapi-x264 profile. It hangs at the end there. You can see a more detailed explanation in the output below but I am testing on a 30fps clip generated by kdenlive that is 4k 10bit.

$ ffmpeg -v verbose -init_hw_device vaapi=vaapi0: -filter_hw_device vaapi0 -i IMG_6126-30fps.mp4 -vf "scale=640:-2,format=nv12,hwupload" -vsync 1 -c:v h264_vaapi -g 1 -bf 0 -qp 26 -c:a aac -ab 128k test.mp4
ffmpeg version n7.1.1 Copyright (c) 2000-2025 the FFmpeg developers
  built with gcc 15.1.1 (GCC) 20250425
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-frei0r --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libdvdnav --enable-libdvdread --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgsm --enable-libharfbuzz --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libplacebo --enable-libpulse --enable-librav1e --enable-librsvg --enable-librubberband --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-vapoursynth --enable-version3 --enable-vulkan
  libavutil      59. 39.100 / 59. 39.100
  libavcodec     61. 19.101 / 61. 19.101
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample   5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
[AVHWDeviceContext @ 0x56db96a11540] Trying to use DRM render node for device 0.
[AVHWDeviceContext @ 0x56db96a11540] libva: VA-API version 1.22.0
[AVHWDeviceContext @ 0x56db96a11540] libva: Trying to open /usr/lib/dri/radeonsi_drv_video.so
[AVHWDeviceContext @ 0x56db96a11540] libva: Found init function __vaDriverInit_1_22
[AVHWDeviceContext @ 0x56db96a11540] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x56db96a11540] Initialised VAAPI connection: version 1.22
[AVHWDeviceContext @ 0x56db96a11540] VAAPI driver: Mesa Gallium driver 25.1.4-arch1.2 for AMD Radeon RX 7700 XT (radeonsi, navi32, LLVM 20.1.6, DRM 3.61, 6.12.35-1-lts).
[AVHWDeviceContext @ 0x56db96a11540] Driver not found in known nonstandard list, using standard behaviour.
-vsync is deprecated. Use -fps_mode
Passing a number to -vsync is deprecated, use a string argument as described in the manual.
[h264 @ 0x56db96c2ef80] Reinit context to 3840x2160, pix_fmt: yuv420p
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'IMG_6126-30fps.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf61.7.100
  Duration: 00:01:11.84, start: 0.000000, bitrate: 122826 kb/s
  Stream #0:0[0x1](und): Video: h264 (High), 1 reference frame (avc1 / 0x31637661), yuv420p(tv, bt2020nc/bt2020/arib-std-b67, progressive, left), 3840x2160, 122558 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
      Metadata:
        handler_name    : Core Media Video
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.19.101 h264_vaapi
      Side data:
        Ambient Viewing Environment, ambient_illuminance=314.000000, ambient_light_x=0.312700, ambient_light_y=0.329000
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 265 kb/s (default)
      Metadata:
        handler_name    : Core Media Audio
        vendor_id       : [0][0][0][0]
[out#0/mp4 @ 0x56db96aca940] No explicit maps, mapping streams automatically...
[vost#0:0/h264_vaapi @ 0x56db96c71680] Created video stream from input stream 0:0
[Parsed_scale_0 @ 0x56db96cc4740] w:640 h:-2 flags:'' interl:0
[aost#0:1/aac @ 0x56db97a27180] Created audio stream from input stream 0:1
File 'test.mp4' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_vaapi))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
[vost#0:0/h264_vaapi @ 0x56db96c71680] Starting thread...
[aost#0:1/aac @ 0x56db97a27180] Starting thread...
[vf#0:0 @ 0x56db96ac7980] Starting thread...
[af#0:1 @ 0x56db97ab2600] Starting thread...
[vist#0:0/h264 @ 0x56db96ab8300] [dec:h264 @ 0x56db96caf8c0] Starting thread...
[aist#0:1/aac @ 0x56db96ab7800] [dec:aac @ 0x56db97ab3b40] Starting thread...
[in#0/mov,mp4,m4a,3gp,3g2,mj2 @ 0x56db96c2d9c0] Starting thread...
Press [q] to stop, [?] for help
[h264 @ 0x56db96cb4dc0] Reinit context to 3840x2160, pixfmt: yuv420p
[graph-1_in_0:1 @ 0x79da2c004380] tb:1/44100 samplefmt:fltp samplerate:44100 chlayout:stereo
[Parsed_scale_0 @ 0x79da28004400] w:640 h:-2 flags:'' interl:0
[graph -1 input from stream 0:0 @ 0x79da28012700] w:3840 h:2160 pixfmt:yuv420p tb:1/30000 fr:30000/1001 sar:0/1 csp:bt2020nc range:tv
[Parsed_scale_0 @ 0x79da28004400] w:3840 h:2160 fmt:yuv420p csp:bt2020nc range:tv sar:0/1 -> w:640 h:360 fmt:nv12 csp:bt2020nc range:tv sar:0/1 flags:0x00000004
[Parsed_scale_0 @ 0x79da28004400] [framesync @ 0x79da28004528] Selected 1/30000 time base
[Parsed_scale_0 @ 0x79da28004400] [framesync @ 0x79da28004528] Sync level 1
[graph -1 input from stream 0:0 @ 0x79da28012700] video frame properties congruent with link at pts_time: 0
[h264_vaapi @ 0x56db96c30140] Using input frames context (format vaapi) with h264_vaapi encoder.
[h264_vaapi @ 0x56db96c30140] Input surface format is nv12.
[h264_vaapi @ 0x56db96c30140] Using VAAPI profile VAProfileH264High (7).
[h264_vaapi @ 0x56db96c30140] Using VAAPI entrypoint VAEntrypointEncSlice (6).
[h264_vaapi @ 0x56db96c30140] Using VAAPI render target format YUV420 (0x1).
[h264_vaapi @ 0x56db96c30140] RC mode: CQP.
[h264_vaapi @ 0x56db96c30140] Block Level bitrate control: OFF.
[h264_vaapi @ 0x56db96c30140] RC quality: 26.
[h264_vaapi @ 0x56db96c30140] RC framerate: 30000/1001 (29.97 fps).
[h264_vaapi @ 0x56db96c30140] Using intra frames only.
[h264_vaapi @ 0x56db96c30140] All wanted packed headers available (wanted 0xd, found 0x1f).
[h264_vaapi @ 0x56db96c30140] Using level 3.
Output #0, mp4, to 'test.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf61.7.100
  Stream #0:0(und): Video: h264 (High), 1 reference frame (avc1 / 0x31637661), vaapi(tv, bt2020nc/bt2020/arib-std-b67, progressive, left), 640x360, q=2-31, 29.97 fps, 30k tbn (default)
      Metadata:
        handler_name    : Core Media Video
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.19.101 h264_vaapi
      Side data:
        Ambient Viewing Environment, ambient_illuminance=314.000000, ambient_light_x=0.312700, ambient_light_y=0.329000
  Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, delay 1024, 128 kb/s (default)
      Metadata:
        handler_name    : Core Media Audio
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.19.101 aac
[out#0/mp4 @ 0x56db96aca940] Starting thread...
frame=    0 fps=0.0 q=0.0 size=       0KiB time=N/A bitrate=N/A speed=N/A

If I simply remove the -g 1 -bf 0 flags resulting in the following flags it runs properly:

-init_hw_device vaapi=vaapi0: -filter_hw_device vaapi0 -i -vf scale=%width:-2,format=nv12,hwupload -vsync 1 -c:v h264_vaapi -qp 26 -c:a aac -ab 128k
2

The hardware encoders usually only support a subset of what it is in theory possible to encode - so I’d first be checking what your hardware (and the driver you have for it) supposedly supports for configuring I and B frame rates.

You’re trading flexibility and best quality for speed when using them, which is fine for rendering low res/quality proxies, but your choice of format is still likely to be limited.