With the launch of Visual Search in Final Cut Pro 12, the ability to find a clip simply based on what’s in the shot has finally become more mainstream. FCP wasn’t first, though: Premiere, Peakto and Jumper all include a visual search of some kind that video editors can make use of.
These tools aren’t all equivalent. FCP and Premiere’s tools are self-contained, while Jumper and Peakto can work with media from any app. Jumper has tighter integration with NLEs, while Peakto comes from a stills background, and can also manage all your photo libraries. Once you’ve found a clip, FCP has probably the best suite of tools (keywords, favorites and smart collections) for organizing those finds, but you can make a folder full of subclips in any of these tools in a pinch.
Here, let’s focus on the visual search side of things, because the differences between these tools aren’t obvious. Each person will have their own needs when it comes to searching for content, and may not discover any issues after a quick search or two.
To equalize the differences, I’ve given all these tools exactly the same dataset, including demo files from Final Cut Pro for iPad, old and new demo files from Premiere, a semi-random selection of clips from online stock site Pixabay, some of my own clips, and several finished productions.
Before we get into the details, let’s take a quick tour through how visual searching works in all these apps.
Analysis and integration
Final Cut Pro makes the analysis process pretty easy. You can analyze for visual search on import, or right-click any files and perform this analysis later. It’s not instant, and it is an intensive process — you may hear the fans on a MacBook Pro spin up. When complete, you can search in the Browser, typing your search terms in the top field while it’s set to “All” or just “Visual”. More complex searches can be stored using the floating Search panel (click the icon to the right of the search bar) and you can also save your searches as Smart Collections, allowing you to repeat these searches at any time.
Premiere can also analyze on import, but you’ll need to check the right box in the Media Analysis & Transcription pane in preferences first. You’ll then need to open the dedicated Search panel and search through “Everything” or “Visuals” in the top left. If you find a few clips and want to collect them, the best option is to right-click and choose New Sequence from Selection. Annoyingly, you can only display search results in thumbnail view, you can’t see the end of even moderately long clip names, and HDR clips aren’t displayed properly.
Jumper can integrate with your NLE, (FCP, Premiere, Resolve and Avid MC are all supported) but you can also use it as a standalone app. Media needs to be added to Jumper’s database manually, but once you find a clip in Jumper, it can select the original clip in your NLE, or add all the found clips to a timeline for you to cull further. Importantly, Jumper includes several different models, allowing you to trade speed for accuracy. I’ve gone with Most Accurate for these tests, but analysis with this slower model did take noticeably longer than a scan with the default Fast. Also helpful: Jumper can do face detection, and also find shots that are similar to existing shots.
Peakto exists as a standalone app, though integration with Premiere is available with the Pro version. You can add a Premiere or FCP project (or a folder full of files) to your catalog, to analyze that media and show it within the general photo/video grid. Peakto’s main focus isn’t on visual search, after all; it’s an app designed to let you access all your photo and video libraries at the same time. Still, analysis doesn’t take too long, and you can then search using the filter bar at the top.
Search terms
I’ll be testing a list of search terms to see how well each of these tools does at finding things I know should be findable, and how often they produce false positives, showing clips that don’t match what I searched for. Searching can be a bit of a dark art, but we’ll try to keep it as simple as possible, because that’s what most users will do.
Still, if you want to find a drone shot, do you search for “drone” or the actual content of the shot? Similarly, if you search for “bicycle” you might find shots with bicycles in them, or shots filmed from a bicycle. What will you find if you search for a color, or a trademarked name? Can you reliably find clapperboards?
As others have found, sometimes you’ll need to try a variant search term: dive vs diving, run vs running, jump vs jumping, or add additional qualifying words to your searches. Sometimes, adding a space at the end of your search term can make a difference. Sometimes you’ll find a clip with your search term in text, or even spoken out loud. AI models are weird and unpredictable, so expect imperfection. For an example of this in FCP, where “two women” didn’t find the same clips as “two women hugging” did, check out Robin Kurz’s video here.
I suspect most video editors will use a feature like this to search for nouns, so this list of search terms is noun-heavy. I’ve also included several locations, verbs and adjectives and camera movement descriptions and shot types to see how well these tools can help an editor find a shot. Some of these nouns have multiple potential meanings, which can lead to confusion. And will spelling make a difference? We’ll find out.
False positives and Relevance
Importantly, all these apps will present false positives to some degree, and you’ll need to use your own brain to filter those out. Finding the balance between too few results and too many false positives is tricky, but if good results are shown first, it won’t slow you down.
The different apps deal with relevance in different ways, too:
- Final Cut Pro shows High Relevance clips first, then Low Relevance clips. While the sorting is helpful, you can activate Group By Relevance to see the split more clearly in Filmstrip view, or show the Relevance column in List view.
- Premiere sorts by relevance, but doesn’t show how relevant each result is.
- Jumper shows the most relevant results first, but it doesn’t put a limit on the number of clips it finds — which does result in more false positives. If you want, you can keep loading in more results of decreasing confidence, but if you don’t see a useful clip in the first page of results, it’s probably not there.
- Peakto sorts by relevance, with a fixed number of results like FCP and Premiere. Unfortunately it doesn’t select parts of clips, but shows you the entire clip in a skimmable form, with a blue bar at the top where the search term was found. This makes it a little harder to scan all your results at a glance.
AI is weird
Some oddities did appear.
- I searched for a breadmaker (the kitchen appliance, using one and two word variations) and only Jumper found it successfully. The other apps found a toaster.
- Most of the time, Jumper surfaced the right shots quickly, but now and again it found too many false positives first. When searching for “banana” all apps found the one obvious clip of a woman eating a banana, but lots of non-banana shots too. The same thing happened with “snow” — the shots were there, but buried. If this happens to you, adding more words (for example to “woman eating banana”) to the search is the best way to focus your results.
- Because visual search is mostly about objects in shot, none of these apps could reliably find terms to do with how the shot was recorded: wide, close up, timelapse, gimbal, and so on. Searching for “drone” was only slightly more reliable, and I stopped searching this category when it was clear it wouldn’t succeed.
- A search for “colorful” should definitely have found one specific clip, but FCP couldn’t find it, even when spelt with a “u”.
- Searching for terms which could be interpreted in many different ways, like “cool”, produces unhelpful results that vary wildly between apps.
- Searching for “laugh” vs “laughing” produced quite different results in each app.
Results
Importantly, not all apps return results in the same way, so it can be a little hard to assess them. To keep things simple, I’m judging on just two factors:
- did it find some useful shots?
- did it find all the shots it should have?
With those criteria in mind, here’s how I awarded points:
- 2 = it found all the shots, mostly near the top of the search results.
- 1 = it found some, but not all of the shots
- 0 = it failed to find any shots
Here’s how the apps did, with handy color coding:
Obviously, this scale is arbitrary. You may prefer to assign more weight to a good result or even subtract points for incomplete or missing results, so if you want to play with the numbers, here they are in Numbers and Excel formats. I’d also recommend running your own tests.
Conclusions
Overall, as you can see from the chart above, Jumper gave me the most consistently good results. Premiere came second, but has a few UI niggles that make its results harder to use. Final Cut Pro and Peakto can definitely find clips on most of the searches I tried, but they fell short more than I’d like.
On the other hand, FCP’s skim-friendly interface, with its large preview, is probably the best way to assess the quality of your results. And because both FCP and Premiere can scan your clips on import, it’s easy to integrate their built-in features to your existing workflow. Peakto’s strengths are mostly in stills management, but has some workgroup-friendly features too.
What should you buy, if this is a useful feature for you? Casual users searching for something that’s relatively easy to describe could probably be happy with any of these tools — but they won’t know what they’re missing. For the best results, or if you need more advanced search options, I’d recommend Jumper.