Upscaling or “uprezzing” is something most editors do every day without thinking much about it. Drop in a 720p clip into your HD 1080p timeline and scale it up to fill the screen and move on. The issue is that simply scaling the clip looks soft. It might fail QC on a broadcast show or it might draw the ire of your client (even if they were the one who provided it!). If you have an eye for quality, you will not be happy to include lower resolution clips in your edit.
The problem is that it is common to have a mix of clip resolutions in a project, especially when working at 4K. Maybe you have some archive footage of a lower resolution or maybe one camera was set to the wrong resolution one day. Maybe you’re working at HD and have heard that there are advantages in uploading 4K to YouTube and want to uprez the whole thing. The question is – can it be done at a higher quality than simply zooming in?
All the NLEs default to using a simple zoom when scaling and for good reason – it is not processor intensive and so doesn’t get in the way of your editing. But when it comes to delivering your project, what are the options?
There have been algorithms around for a long time for upscaling that get a bit more quality out of a shot than a simple zoom. There is the After Effects “Detail-preserving Upscale” effect for example and there are hardware solutions like Grass Valley’s Alchemist (albeit primarily for frame rate conversion). There are open source solutions like using FFMPEG which have some high quality filters built in.
Enhance, Enhance
Hollywood has faked the “zoom in and enhance” effect for years, which became a meme for those in the know.
But in the last few years, A.I. has come into this race with the claim of genuinely clawing back missing resolution. Enhanced upscaling or superscaling is now in Davinci Resolve, Adobe Camera Raw, Topaz Video Enhance AI & various online sites like Pixop, as well as a whole world of open source options (if you’re really clever and have a lot of time on your hands!)
I recently edited a documentary on A.I. and it was a quick learning curve trying to get up to speed on the topic and not a little scary to see how far reaching it is and how quickly it is entering every sphere of life (with Facebook being a well known example). It is a rapidly emerging field in many industries, not least post production. Adobe has their pretty far reaching Sensei program for example and Nvidia are making advances all the time, building upscaling into games with DLSS and their Shield media player. Ways of using it are popping up all the time, like this A.I. driven lip sync.
On YouTube people like Denis Shiryaev have large followings, restoring old footage using a variety of tools, not only upscaling. His video “A Trip Through New York City in 1911” has 18M views.
A.I. for upscaling
When we talk about A.I. there are broadly two types of artificial intelligence. The first, called General A.I. (or AGI), is real intelligence like a human has – think of a robot that can really think for itself. The second is narrow A.I., a.k.a machine learning or pattern recognition or neural networks. It is this that is being leveraged to find those missing pixels.
Upscaling or superscaling images and video clips to re-create missing pixels is a task well suited to machine learning. It is a simple idea, though clearly pretty complicated to implement (see this blog if you don’t believe me!) – to feed a computer algorithm many pairs of images or clips – low-res and high res versions of the same thing, so it can learn what a low res image “should” look like when upscaled (and bear in mind that video is just a series of images).
As Eric Chan from Adobe says in a recent blog on the new Enhance feature in Adobe Camera Raw: “The idea is to train a computer using a large set of example photos. Specifically, we used millions of pairs of low-resolution and high-resolution image patches so that the computer can figure out how to upsize low-resolution images.” The algorithm then learns how to fill in the missing information in low res imagery given to it based on what it has learned and so adding realistic detail that was never there.
Testing
I decided to run two sample tests, pitting various options against each other. Of course, ideally you would test these on different types of footage and I’d encourage you to do that if you have time. I am judging these by eye and of course it’s my subjective opinion. The differences are subtle – but to my eye significant. I am looking for the processed file to be as close to the original as possible – minimal artifacts, as sharp as possible without looking digitally sharpened. This is where video and photography do diverge – screenshots from films should never look as sharp as a photo.
Test 1 – 2x upscale from 1080p to 4K UHD
I have taken an ungraded 10 second clip from a recent shoot, shot at 4K UHD on the Blackmagic Pocket 4K. I then exported it at 1080p and upscaled that clip back to 4K UHD with the various contenders and am looking at one frame from that clip. I am using Adobe Premiere to collate the clips.
And here is a tight 100% crop comparing the original to a simple 200% zoom:
And here is the leaderboard in order of quality:
- Adobe Camera Raw
- Topaz Video Enhance AI
- Pixop
- Topaz Gigapixel AI
- Alchemist
- Davinci Resolve Super Scale
- After Effects Detail-preserving Upscale
1. Adobe Camera Raw
For me, Adobe’s brand new Enhance feature in Adobe Camera Raw came out on top in the first test, barely distinguible from the original, but by far the worst in terms of workflow as it’s really designed for still images. It is a get-your-assistant-to-do-it workflow if there ever was one. For what it’s worth, it involves exporting the clip as an TIFF image sequence first (ACR can’t enhance PNGs), then getting those images into ACR via Bridge or Lightroom and then selecting all of them in the filmstrip, holding down option/alt (to suppress the dialog), right clicking the image & choosing Enhance (as detailed right at the bottom of the blog) which creates a superscaled DNG sequence. Then batch converting that into a TIFF or PNG sequence and batch re-naming it so it has numbers at the end, all so it can be imported properly into Premiere. Very few people are going to have that kind of time, but the technology is exciting.
2. Topaz Video Enhance AI
Coming in second is Topaz Video Enhance AI. In this 2x test I found the Artemis High Quality to work best. I found that when feeding it with a video clip, it messed with the colour in the clip (a non-starter), whereas it was fine with an image sequence. It is also very, very hardware intensive and slow. It has the option of mixing in a subtle amount of film grain which works well.
3. Pixop
Third was Pixop – to my eye a slight step down in quality from Topaz. However it is a pay per use model and the processing is done on the cloud – both of which will suit some people better.
4. Topaz Gigapixel AI
I’ve put Topaz’s Gigapixel AI (which is designed for stills) in fourth simply as I find it too sharp and I couldn’t see a way of turning that down in the software, though that may change in future versions. I have seen some people use this to good effect with video.
5. Alchemist
In fifth is Alchemist which does a pretty decent job of upscaling given that isn’t something Grass Valley shows off about much, compared to de-interlacing and frame rate conversion which is what it is known for. I was using the software version of Alchemist called “Alchemist File” (I didn’t know you could run it on your own computer before this test!)
It does have a built in engine called Quasar which says it’s all about upscaling, but in my test it was identical to the standard engine. The software is designed (and priced!) to run by a tape op in a machine room – it’s something you’d have at a post house and pay them to use for you.
6. DaVinci Resolve Super Scale
In sixth place I have Davinci Resolve’s Super Scale feature (found via clip attributes on an individual clip) which I tested on version 16 and 17 and it was identical on both. I am disappointed in the quality Resolve is giving, especially as BMD claim machine learning for this effect. For me I can see artifacting around the eyes and it doesn’t look very real. Still it’s better than a simple zoom and the big advantage is a much easier workflow if you are finishing in Resolve. And a good colourist will do other things to cover up the artifacts.
7. After Effects Detail-preserving Upscale
In seventh place I have After Effect’s Detail-preserving Upscale effect which if you are finishing in Premiere has a similarly easy workflow, but again doesn’t compete on quality, though is worth using rather than a simple zoom.
Test 2 – 3x upscale from 720p to 4K UHD
This test asked more of the software competitors, with a 3x zoom required to upscale from 720p to 4K UHD. Results were similar, but it did shuffle the leaderboard a bit, with Topaz moving to the top
The main thing here was that Adobe Camera Raw doesn’t let you do that 2x superscale twice. Of course you can work around this and I tried that, but the second file, although twice the pixels, was identical in quality.
Leaderboard
- Topaz Video Enhance AI
- Topaz Gigapixel AI
- Adobe Camera Raw
- Alchemist
- Pixop
- Davinci Resolve SuperScale
- After Effects Detail-preserving Upscale
1. Topaz Video Enhance AI
Topaz Video Enhance AI was very impressive here with a very useable image even at 300%. I found that this time the Gaia preset worked better & generally I find you do need to audition the presets for any given clip to see which works best (unfortunately this is very slow).
2. Topaz Gigapixel AI
Even though it’s designed for stills, I found Topaz’s other software was the second best option (and if they bring in the option of turning down the sharpening controls it may compete for the top spot).
3. Adobe Camera Raw
The main thing here was that Adobe Camera Raw doesn’t let you do that 2x superscale twice. Of course you can work around this and I tried that, but the second file, although twice the pixels, was identical in quality.
4. Alchemist
There is a noticeable drop in sharpness here and you may question the number four spot. The reason I have it here is that although softer, it at least doesn’t have any distracting artifacting,
5. Pixop
Although it’s pretty sharp, I found with Pixop’s offering the artifacting around the eyes and on the skin was distracting.
6. Davinci Resolve SuperScale
In a similar vein, Davinci Resolve’s artifacting was even worse and I found it hard to decide between it and After Effects.
7. After Effects Detail-preserving Upscale
After Effects comes out quite a bit softer similar to Pixop, but with slightly worse artifacts.
Summary
In summary, it’s a case of no pain, no gain. By far the easiest to work with are Davinci Resolve if you are finishing in it and After Effects if finishing in Premiere. Adobe Camera Raw is probably too tough a workflow for most to work with. Some machine rooms will have Alchemist in hardware or software and that’s a great option. But for me the dedicated AI tools from Topaz and the online providers like Pixop are the ones to watch.
AI already gives remarkable results and one of the big things about this is that it should get better every year. These companies will be training their algorithms every day and the exciting as well as scary thing, is that after a while they will be clever enough to train themselves.