Post Production

Advanced keying with After Effects: Part 4

FrankenAlpha. Igor, fetch me the matte


The point of using a chromakey plugin is to generate a matte / alpha channel based on the colour of a background. In Part 4 of this series on advanced chromakey in After Effects, I demonstrate the benefits of stitching together a matte from multiple different renders. This directly follows on from part 3, where I demonstrate setting up a bunch of different keys.

When I started work on this series, I was worried that I wouldn’t have enough material to do an entire tutorial just on combining different renders.  After all, that’s what After Effects does – it’s what everyone already knows how to do!  Bring in a few videos, stack them together in the timeline and then mask out different areas.  Simples! But is something so basic worthy of its own article?  Anyway I sat down and started recording and didn’t finish until 2 hours later.  So there must be something to it… The resulting tutorial is so long that I’ve split it into two halves, part 1 is on combining multiple mattes and part 2 is on colour correction.  I admire anyone who can sit through both parts in one sitting without drifting off, but there’s a lot of information in there.

As with the earlier parts, the videos cover all of the main information but there’s always a few things I like to emphasize in writing.

With such a broad series like this one I don’t see the value in taking a single clip and working through it to produce a final “perfect” result.  Every piece of footage is different.  Instead, this video serves as a sort of overall reminder of the tools you have available to you.  Because individual cases can vary so much, I’m a big fan of trying several different approaches and seeing what works – basic trial and error.  So in part 3 I demonstrated how I like to pre-render out a bunch of compositions that have different settings, and now in Part 4 I’m simply looking at them and deciding what works best.  As a reminder of the tools and techniques we have available in After Effects, here’s a list of things to try.

Basic masking and garbage matting:

  • Simple 2D masks
  • Masks with variable feather
  • Using the “track masks” feature in AE
  • Camera tracking and masking with 3D solids
  • Mocha

In the earlier parts of this series I’ve mentioned various 3rd party products that I believe are worth the price.  For noise reduction, I recommend Neat Video or Red Giant’s Magic Bullet denoiser.  For keying you have two different Primatte versions, and Composite Brush.  When it comes to advanced masking and roto, then upgrading to Mocha Pro is another example where a 3rd party plugin can pay for itself with a single job. While I haven’t used the latest version of Mocha Pro, the magnetic spline tool looks like an impressive time saver. Just to clarify – After Effects comes with a basic version of Mocha already included, which I demonstrate above.  Mocha Pro is a separate product that includes more powerful features, including the magnetic spline tool, that doesn’t come with After Effects by default.

When it comes to processing mattes and cleaning them up, most of the useful plugins are simply in the After Effects “Matte” menu, but there are a couple of others which can be overlooked.

Matte processing:

  • Matte choker
  • Refine Hard matte
  • Refine soft matte
  • Levels (use the alpha channel)
  • Curves (ditto, use the alpha channel)
  • CC Vector Blur (set to perpendicular)

As the matte choker plugin is so useful, it’s worth taking the time to understand what it’s doing and how it works.  This brilliant tutorial is devoted to explaining the matte choker and if you’ve made it this far through the series I recommend you watch it too.  While Adobe have provided us with the newer Refine Hard Matte and Refine Soft Matte plugins, I’m still learning exactly when they work the best.  These newer plugins rely on information in the colour channels to help process the alpha channel, so if you want to use them with pre-renders (as I heavily advocate in this series…) then you’ll want to render out with straight alpha channels.

Using CC Vector Blur isn’t a technique that made the final video, but is a useful technique to know if you have especially noisy mattes, or blocky / compressed source material.  Using Vector Blur to smooth out mattes is something I learned from After Effects Guru Brian Maffit, and it’s proved useful in a number of cases.  The trick is to set Vector Blur to “perpendicular”, which blurs out the matte along the edge.  I’d recommend using this on luma-mattes, so you can see what it’s doing, although you can set the input of the plugin to “alpha”.

Colour me white and call me Matte

Another point I make in the video that I’ll emphasize here, is the need to colour correct footage that has a separate matte applied.  When you use an advanced plugin like Keylight to pull a key, the software is colour correcting and de-spilling the footage at the same time as it’s extracting an alpha channel.  When we construct a matte separately and apply it to our footage, we’re missing out on that colour correction so we need to do it manually. That’s the topic of the second video:

As I demonstrate in the video, the newer “Advanced Spill Suppressor” plugin is a big step forwards from the earlier (now obsolete) spill suppressor.  While I show how Keylight is another option, it’s not something I’ve used myself – I just wanted to demonstrate that it’s possible.  The results you get out of Keylight will depend on the settings you have in the plugin, so you can still scrub the colour values and despill values as I demonstrated in Part 3.  The car example in the video above wasn’t great because Keylight was keying out the green spill on the dashboard and showing the black background underneath, but if I had tweaked the settings I might have come up with a better result. However when your footage keys more easily, as with the face example, Keylight does a really great job at colour correcting.

The Advanced Spill Suppressor plugin allows you to adjust the strength of the effect. For other plugins that don’t have this option, you can control the “Effect Opacity”, under the “Compositing Options”

Another thing I forgot to demonstrate in the videos is that you don’t need to use a spill suppressor plugin at 100% strength. While the Advanced Spill Suppressor plugin has a strength setting, if you’re using other plugins (eg. Keylight) then you can always open the “Compositing Options” and adjust the “Effect Opacity”.

Similarly, if you want to experiment with the DIY spill suppression method I demonstrate, the channel mixer layer can also have the opacity adjusted to suit your taste.  If you have the “darken colour” layer set to 100% then the results can be very strong, and dialing back the layer to 75% opacity or less can make the result look less processed.

Colour Correcting semi-transparent areas of the image is done with the “Remove Colour Matting” plugin, which is something I demonstrate in-depth in my earlier tutorial on alpha channels.  Because keying and alpha channels are closely related, I suggest you take the time out to watch the full video here.

Four down, 1 to go

So Part 4 has ended up being much longer than I anticipated, but if you’ve missed the other parts then you can watch them here: part 1, part 2, part 3.

In the final part of this series I’ll be looking at a few compositing tips, but until then you can check out all my other After Effects articles at the ProVideo Coalition.


Support ProVideo Coalition
Shop with Filmtools Logo

Share Our Article

Chris was born a geek, and was lucky enough to own a Commodore 64 at a time when the number of students at his primary school who owned a computer could be counted on one…

Leave a Reply

Notify of