MulticoreWare shows LipSync to autodetect a/v sync via deep learning + GPUs

MulticoreWare and Telestream are partnering to make LipSync available to enterprise customers.

MulticoreWare’s LipSync technology uses deep neural networks to autodetect audio/video sync errors by “watching” and “listening” to videos. According to MulticoreWare, NVIDIA GPU-accelerated models find and match instances of human faces and human speech in up to 2-3x realtime, enabling highly scalable quality control for file-based or streaming content. LipSync technology is being demonstrated at NAB2017 by MulticoreWare, developers of the x265 HEVC video encoder.

Typical causes of audio-video misalignment include transmission and transcode errors, incorrect video cuts, or incorrect framerate conversions. With an ever-increasing amount of video content, sources, transmissions, and transcodes, sync errors can occur more frequently. MulticoreWare developed LipSync to automatically detect synchronization errors to ensure content integrity at scales where manual verification is intractable or expensive.

According to its creator, LipSync combines the latest deep learning neural network techniques with statistical analysis to test videos without relying on digital fingerprinting or watermarking. Audio/video synchronization detection is performed by analyzing moving lips and faces and listening for human speech patterns, similar to how a human viewer would watch a video. Unlike a human viewer, LipSync can process file-based content at 2-3x real-time or analyze multiple video streams in real-time using NVIDIA GPU-accelerated servers.

“We are the first-to-market with a machine learning-based solution,” says Arun Ramanathan, VP and GM for Machine Learning at MulticoreWare. “This was made possible by combining our expertise in video processing, GPU computing and deep learning.”

NVIDIA GPUs enabled the development of LipSync into a realtime solution. “LipSync is an impressive example of how deep learning, accelerated by NVIDIA GPUs, solves major challenges in creating and distributing video content,” said Will Ramey, director of Developer Marketing at NVIDIA. “This innovative application addresses a pervasive problem for the entire industry.”

Telestream and MulticoreWare are partnering to make LipSync available to enterprise customers. Shawn Carnahan, CTO of Telestream said that, “Identifying audio-video sync errors has long been a challenge in our industry and Telestream is excited to offer an automated solution using deep learning technologies. Telestream is working closely with MulticoreWare to integrate LipSync into our products.”

Telestream recently expanded its video quality-control portfolio with the acquisition of VidCheck and IneoQuest.

MulticoreWare is currently demonstrating LipSync technology for new partners and licensees. Video quality control providers, broadcasters, and content distributors can integrate LipSync into their existing software or pipelines, or use it as a standalone product on-site or in the cloud. On-demand usage is supported on Amazon Web Services (AWS), Google Cloud Platform, and other GPU-accelerated cloud services running Windows or Linux. Licensing models include perpetual on-premise installations, integration licenses, and per-usage pricing.

For more information visit lipsync.multicorewareinc.com or MulticoreWare’s booth SU14002 at NAB 2017.

Upcoming articles, reviews, radio shows, books and seminars/webinars

Stand by for upcoming articles, reviews, and books. Sign up to my free mailing list by clicking here. Most of my current books are at books.AllanTepper.com, and my personal website is AllanTepper.com.

Si deseas suscribirte a mi lista en castellano, visita aquí. Si prefieres, puedes suscribirte a ambas listas (castellano e inglés).


Listen to his CapicúaFM show at CapicúaFM.com in iTunes or Stitcher.

FTC disclosure

No manufacturer is specifically paying Allan Tépper or TecnoTur LLC to write this article or the mentioned books. Some of the other manufacturers listed above have contracted Tépper and/or TecnoTur LLC to carry out consulting and/or translations/localizations/transcreations. Many of the manufacturers listed above have sent Allan Tépper review units. So far, none of the manufacturers listed above is/are sponsors of the TecnoTur programs, although they are welcome to do so, and some are, may be (or may have been) sponsors of ProVideo Coalition magazine. Some links to third parties listed in this article and/or on this web page may indirectly benefit TecnoTur LLC via affiliate programs. Allan Tépper’s opinions are his own.

Copyright and use of this article

The articles contained in the TecnoTur channel in ProVideo Coalition magazine are copyright Allan Tépper/TecnoTur LLC, except where otherwise attributed. Unauthorized use is prohibited without prior approval, except for short quotes which link back to this page, which are encouraged!


Was This Post Helpful:

0 votes, 0 avg. rating

Share:

Allan Tépper

Born in Connecticut, United States, Allan Tépper is a bilingual consultant, multi-title author, tech journalist, translator, and language activist who has been working with professional video since the eighties. Since 1994, Tépper has been consulting both end-users and manufacturers through his Florida company. Via TecnoTur, Tépper has been giving video tech seminars in several South Florida’s universities and training centers, and in a half dozen Latin American countries, in their native language. Tépper has been a frequent radio/TV guest on several South Florida, Guatemalan, and Venezuelan radio and TV stations. As a certified ATA (American Translators Association) translator, Tépper has translated and localized dozens of advertisements, catalogs, software, and technical manuals for the Spanish and Latin American markets. He has also written many contracted white papers for tech manufacturers. Over the past 18 years, Tépper’s articles have been published or quoted in more than a dozen magazines, newspapers, and electronic media in Latin America. Since 2008, Allan Tépper’s articles have been published frequently –in English– in ProVideo Coalition magazine, and since 2014, he is is the director of CapicúaFM.com. His website is AllanTépper.com.

Leave a Reply

Be the First to Comment!

Notify of
avatar
wpDiscuz
PVC Logo
PVC App Demo

Now Available in the App Store

Get access to our latest and featured articles from your favorite authors, wherever you are, directly on your phone!

  • Clean and Easy to Read
  • Blazing Fast Notifications for New Articles
  • Free Tuesday Webinar Access
Apple App Store
Google App Store
Get your FREE Search Bins Project File!
Want Us to Remind You?
Enter your email and subscribe!
Get Access!
Please enter your email address to view our free live webinar!
*you will be added to the ProVideoCoalition.com newsletter and if you’re already signed up, it won’t send you duplicate e-mails