Welcome to Lowpass, a newsletter about the future of entertainment and the next big hardware platforms, including smart TVs, ambient computing and AR / VR. This week: Netflix is looking to patent AI-automated video cropping.

Subscribe now to get the next issue delivered to your inbox for free.

Netflix has invented an AI Quibi

Remember Quibi? Jeffrey Katzenberg’s failed billion-dollar short-form video startup was, among other things, trying to woo viewers by giving them the option to watch videos with either vertical or horizontal screen orientation. To enable this feature, which Quibi called “Turnstyle,” the company’s editors made two distinct cuts for each and every of its clips.

Turnstyle clearly wasn’t enough to save Quibi (and asking people to pay for the resulting clips when YouTube and TikTok are free may not have been the best idea either), but the company was trying to address a real problem: Audiences have gotten used to vertical video on their phones, and Hollywood’s 16:9 content isn’t rendering particularly well on these devices.

Netflix has been exploring the use of AI to solve this issue. The company recently applied for a patent for automated video cropping, which aims to use algorithms to turn footage with horizontal formatting into mobile-friendly vertical videos.

• The goal of an automated video cropping solution is to not just simply cut off arbitrary parts of a video, but instead retain the elements that “the viewer is most likely to be interested in seeing,” as the patent application notes.

• That’s relatively easy if there’s just one person in a shot, but it gets harder when there are multiple people, animals or inanimate objects competing for attention.

• Netflix has been exploring the use of self-learning neural networks to determine the relative importance of an object in a scene. “Over time, these self-learning neural networks may improve the algorithms used for identifying objects in a video scene, even if those objects change positions or change orientations or change costumes or change their look in some other manner,” the patent application explains.

• Some of the criteria that can help algorithms determine the relative importance of an object include how often they appear in a scene or film, whether the objects move or not, and whether the filmmaker kept them in focus or not.

• Object recognition may also help, as it can determine whether an object is important in the context of the footage. “The neural networks may determine the semantic identity of the object (e.g., dog, tree, automobile, beach, etc.) or may simply identify the object as being a distinct object within the video scene and as potentially being the same object in other video scenes that occur earlier or later in the film or tv show,” the application notes.

• “Face recognition or object recognition algorithms may be used to identify particular cast members (e.g., A-list movie stars) or particular objects (e.g., the “Millennium Falcon” from “Star Wars”) in the video scene,” it adds. “These cast members or objects may be assigned a higher or lower importance based on who or what they are.”

Netflix’s algorithms aren’t going to crop “Star Wars” any time soon, and not just because it’s a Disney franchise. In the past, Netflix has faced a lot of backlash for anything that could be considered interfering with a film maker’s artistic vision.

• Judd Apatow got extremely upset when news surfaced in 2019 that Netflix was testing a feature to let viewers change the playback speed of individual videos. The company backpedaled at the time, only to launch the feature a year later.

• More recently, Netflix faced headwinds for experimenting with AI-generated art for a Japanese anime short film.

• It’s also worth noting that companies patent all kinds of things all the time, and only a subset of those inventions ever turn into actual products. Netflix has been exploring automated cropping for some time, and first filed for a related patent in 2019. And yet, four years later, watching Netflix shows in vertical orientation still doesn’t magically give you access to AI-generated crops.

Netflix did start cropping some videos manually in 2018. Instead of attempting to optimize its entire catalog for vertical viewing, the company focused on key scenes to generate promotional content for mobile devices.

• Back then, Netflix began using its own video editors to create 30-second previews for a more TikTok-like preview experience on mobile.

• Executives revealed at the time that 20% of all Netflix viewing was happening on mobile devices; 50% of all subscribers were accessing Netflix on mobile every month.

• While Netflix may not be using AI to crop feature-length films any time soon, one could speculate that these types of promo clips are a natural fit for AI cropping.

• The aforementioned patent application even suggests that human editors and AI could be working in concert, with algorithms improving over time as they learn from humans.

• Combine that with Netflix’s love for A/B testing, and one could envision a future in which human editors cut one promotional vertical video, with AI stepping in to generate additional versions that then all compete for views and swipes on the service.

AI is poised to become an important tool for Hollywood in the coming years, and automated cropping of promotional trailers is just one more example for this. Given the sheer amount of streaming content these days, and the desire of services like Netflix to personalize each and every viewer’s experience, it’s only a question of time until we’ll see AI-generated assets surface in streaming apps.

Netflix did not immediately respond to a request for comment.

What else

Netflix won big at this year’s Academy Awards. The company took home six Oscars — more than any other streaming service.

Meta is done with NFTs, for now. Creators won’t be able to share digital collectibles on Instagram and Facebook anymore going forward, according to its fintech head Stephane Kasriel.

Q&A with Spotify co-president Gustav Söderström. The music service’s CPO and CTO talks about its new mobile design, recent layoffs and future plans (“something is coming at some point").

Sony wants to sell more than 5 million PSVR2s. Sony CFO Hiroki Totoki told investors that the new device has “a good chance” to outsell the original PSVR headset.

Cord cutting is gaining steam in Europe. Western European pay TV revenues are expected to decline by 18% over the next five years.

Twitch’s CEO is stepping down. Emmet Shear is leaving after leading the company for over a decade.

Google Glass is dead. Google is discontinuing the enterprise edition of Glass, with support for shipped devices ceasing in September.

That’s it

Maybe it was all the puns on “The Last of Us,” or maybe the food-related jokes on “Extraordinary Attorney Woo.” Whatever it may have been, I really felt like ending this week’s newsletter with a topical joke … but I couldn’t think of one. AI to the rescue! Asked to write a Dad joke about VR, Notion’s new AI feature came up with this:

“Why did the virtual reality chicken cross the road?

To get to the other side of the simulation!”

Have a great weekend everyone, see you next week.