Having recently done a thorough set of tests of my new Nvidia RTX 4080 graphics card across a range of NLEs and intensive graphics applications, I wanted to delve deeper into the new 8th generation Nvidia Encoder (NVENC) that comes with the 4000 series which accelerates the encoding of AV1, H.265 (HVEC) and H.264 (AVC). I wanted to find out how to make the most of the hardware, especially the dual encoders – Nvidia having included two encoder chips this time around.

Some background on presets

It is often said, “Good, Fast, Cheap – Pick Two” and it fits post production pretty well. Encoding has a similar relationship between quality, speed and file size.

For example if you fix the file size by fixing the bitrate, as is common, you can have higher quality by taking longer to make the encode. If you fix the quality (using say the constant quality slider in Handbrake), you can get the file size a bit smaller by taking longer with the encode. The way you go about this is by choosing a different preset – seen here for the x264 encoder in Handbrake for example.

A preset is defined as follows:

A preset is a collection of options that will provide a certain encoding speed to compression ratio. A slower preset will provide better compression (compression is quality per filesize). This means that, for example, if you target a certain file size or constant bit rate, you will achieve better quality with a slower preset. Similarly, for constant quality encoding, you will simply save bitrate by choosing a slower preset (source).

The real question though is whether taking that extra time is worth it to you. Some people will just choose the slowest preset knowing they want the highest quality and are prepared to wait for it. Personally I would always like to know if it’s worth doing something slower – what if you’re waiting an extra 10 minutes every day for a 0.01% improvement that no-one will ever see? Well it turns out it’s exactly like that – not only for software encoders like x264, but also for the new RTX 4000 series, which I’ll get to later.

A while back I came across some great research from Jan Ozer called “Choosing an x264 preset” which clearly shows that you gain almost nothing by using anything slower than the “faster” preset in terms of quality, yet cost yourself a good bit of time if you do.

Presets for Nvidia’s NVENC in Davinci Resolve

The Nvidia hardware encoder (NVENC) also has presets – officially called P1 (fastest) through P7 (slowest) which according to Nvidia “determine for example GOP structure, B frames, look-ahead encoding, and so on” – stuff most of us don’t have time to understand, but it’s the same deal – spending a bit more time to get a bit more quality or a smaller file.

And it turns out the export time is massively different across these presets. I ran some tests on the RTX 4080 exporting a 5 minute DNxHR UHD 25fps clip to all three output options (H.264, H.265 and AV1) across all 7 presets. For H.265 and AV1, the fastest preset was around 6 times faster than the slowest one, going from around 50 FPS to around 300 FPS. To put that into some context, if you have a 30 minute timeline, the “Very Slow” preset will take over 15 minutes and the “Very Fast” preset will take 2.5 minutes. That kind of time adds up if you’re doing this every day. To get this export speed, the input codec does have to be ProRes or DNxHR (or equivalent mezzanine codec) with effects already cached/rendered – otherwise the encoder is not fed fast enough.

The dual encoders

There’s also a reason why the encoding speed jumps up sharply for the two fastest presets – it is here that the dual encoders come into play. Apparently under certain conditions, the new cards can split a video into two halves and pass each half to each encoder and then recombine them. In order to use the dual encoders on the new RTX 4000 series you need the following to be true:

• Resolution is UHD or above
• Preset is “Faster (P2)” or “Very Fast (P1)”
• Tuning is set as “High Quality” (this is the default anyway)
• Export codec is AV1 or H.265

I was also only able to achieve this in Davinci Resolve (not Premiere Pro or Shutter Encoder for example). It’s interesting that the H.265 encode is now overall faster than H.264 – I think it is time to start using it if you’re not already.

But what about quality?

Of course the big question is what is the tradeoff in quality for these speeds? While you can and should use your eyes to see if an export is at the quality you require, for more subtle differences it’s best to use something like PSNR or VMAF to assess the visual quality.

VMAF is an “Emmy-winning perceptual video quality assessment algorithm developed by Netflix”. And you can use it for free within FFmpeg – I followed this guide (again by Jan Ozer). The really good thing about VMAF is that it’s a score out of 100 which makes it quite easy to understand and according to Netflix a score of over 85 is good.

While there is a small drop-off in quality across the presets, it really is very small – less than half a VMAF point & not visible to the naked eye. Really with any score over 90 I find it hard to see any artifacting at all. A much bigger difference is seen by using H.265 instead of H.264 (and AV1 is even better). Of course the actual numbers will depend on the source material, but I did a few other tests and got similar results.

While speed of encode doesn’t affect the quality that much, the bitrate/filesize does. You can see here a 1080p test file encoded to H.265 at bitrates from 10Mbps down to 1Mbps – the quality varies little across the presets, but a lot across the bitrates. To my eye I was very happy with the encodes above 6Mbps.

Given most people are not short of drive space or internet speed, upping the bitrate is definitely the way to push the quality up, whilst keeping the benefit of getting the encode done quickly with the fastest presets.

Premiere Pro

In Premiere Pro there are more limited options, but you still can speed up your exports by some judicious tweaking of settings. Presets are not available in H.264 and AV1 is not available at all yet (aside from using 3rd party plugin Voukoder), but if you choose HVEC (H.265) as the format and you dial down in the Video settings, you see five options listed as “Quality” – why there are five and not seven I have no idea (my guess is trying to keep it simpler by leaving off the outer two). You also need to make sure you have chosen one pass hardware encoding which uses the Nvidia NVENC encoder.

Again there are substantial time savings to be made here, though sadly none of them seemed to access the dual encoders. But I saw about a 67% speed increase going from the default “Good” to the “Lower” setting – and this time the drop off in quality was completely negligible (only 0.1 VMAF points across all the presets). And getting to 167 FPS is pretty impressive even if not quite the 300 FPS of Davinci Resolve.

Shutter Encoder

Shutter Encoder is a popular free encoding tool so I thought I’d give it a look too. It’s very useful and better than Handbrake in lots of ways (e.g. keeps timecode intact). It’s a front end for FFmpeg and by default it’s using the CPU encoders like x264. This is great if you have an older graphics card or you are after the best possible quality. You can force it to use NVENC if you want to for much faster speeds – in my tests getting a 10x speed increase by changing from the default settings to NVENC “veryfast”. It performed similarly to Premiere with 183 FPS and again didn’t seem to be able to use the dual encoders.

I did find the software encoders gave 1 or 2 more VMAF points for the same material in general, so again it depends if you are trying to squeeze out the last bit of quality and don’t mind spending the time doing that.

Conclusions

The Nvidia hardware encoder (NVENC) has been of a very high quality for many generations – close enough to the quality of software encoders to make it worthwhile and much much faster. In this RTX 4000 series it is even better and the inclusion of dual encoders is quite exciting. Being able to access them in Davinci Resolve is fantastic, bringing encoding speeds up to an impressive 300 FPS by using the fastest presets. And those speeds involve very little drop off in quality, plus you could make up for that anyway by bumping up the bitrate a little, to no real disadvantage.

I also think it’s time to ditch H.264 for H.265. Hardware on the receiving end is now easily able to play it back and it’s now quicker to make as well as better quality. H.264 is coming on 20 years old now and H.265 can now be seen as the “safe” choice against AV1 if that makes you feel better.