Welcome to Lowpass! This week: Samsung doesn’t want AR devices to replace your TV, and Sonos cancels its TV streamer.

How Samsung envisions AR wearables and smart TVs to work together

AR and VR enthusiasts have long envisioned a future in which glasses and headsets make other screens obsolete, starting with the biggest screen in your house: “We don't need a physical TV,” said Mark Zuckerberg in an interview about the future of AR back in 2017. “We can buy a $1 app TV and put it on the wall and watch it."

Samsung predictably doesn't share that vision. The world’s leading TV maker instead wants AR glasses and TVs to work together. In a newly-published patent application, Samsung details how AR wearables could be used in conjunction with your TV: As a fancy remote control, capable of casting, switching to media playback mode whenever you sit down in front of the TV, and even collecting viewing suggestions while out and about, Shazam-style. 

It’s a fascinating look at a future that may be just years away:

AR glasses as remote controls. Samsung’s patent application, which was published last month, focuses on AR glasses equipped with both outward-facing cameras to capture their owner’s field of view, and inward-facing eye tracking to figure out what their owner is looking at. Here’s how Samsung envisions to make use of those sensors:

• With their outward-facing camera, the glasses are capable of identifying whether there is a TV in the same room.

• Once a TV gets identified, the glasses can highlight it by displaying a glowing frame around it. Focus your eyes on it, and the glasses establish a connection to the smart TV, and check whether the two devices share accounts.

• After this kind of authentication, the glasses open a dedicated interface to interact with the TV, which could include a mirror image of whatever is streaming on the TV at the time.

• The glasses could also display media discovery elements, and for instance help people pick the next movie to watch on the TV.

• The AR glasses may also have media transfer capabilities. Upon looking at a TV that’s playing streaming video, a button labeled “Continue playing in app” would appear. Pressing it would pause playback on TV, and transfer the stream to the glasses.

Making AR living-room-aware. With AR glasses makers gearing up to release a first crop of consumer devices in the coming years, one major challenge will be how to integrate these devices with existing screens. Samsung’s patent application spends considerable time exploring potential solutions:

• By being able to recognize TV screens, AR glasses can help with multitasking in the living room: The glasses may, for instance, avoid displaying AR elements on top of the TV screen so that wearers can still watch TV while interacting with their friends via an AR chat app.

• The glasses could also keep track of where in your home you frequently stream video content based on a combination of GPS, SLAM and / or image recognition. After an initial learning period, they may automatically open up a menu to suggest new things to stream whenever you sit down on your living room couch.

• This is an interesting way to think about the way AR wearables will work even beyond this specific scenario: Instead of forcing you to scroll through long lists of apps, they’ll likely be much more context-relevant, and offer you access to specific features based on your location, objects in your field of view, and past usage patterns.

AR as a Shazam for TV.  The patent application doesn’t just stop at couch potato use cases. It also envisions people wearing AR glasses while out and about, where they might stumble across a TV screen or large outdoor display showing a snippet of a movie or TV show. 

• The glasses would then be able to identify the content in a Shazam-like fashion with the help of the outward-facing camera, and allow the wearer to watch it even after changing location.

Samsung is getting ready to re-enter the world of AR/VR with its new Android XR headset this year. The company is reportedly also working on AR glasses. At the same time, Samsung clearly wants to continue making TVs. 

While patent applications don’t necessarily translate into future products, Samsung’s stab at this does offer an interesting glimpse at how the company envisions tomorrow’s multi-screen world to work – and challenges entertainment companies to figure out how they want to utilize these technologies for their own benefit.

Why Sonos killed its TV streamer

Yesterday afternoon, The Verge’s resident Sonos scoop hound Chris Welch dropped a bombshell: Sonos has cancelled the planned release of an upcoming TV streaming product, with interim CEO Tom Conrad telling staff during an all-hands that “a push into video from Sonos is off the table ‘for now.’”

That’s a massive 180 for the company, which began showing the device to partners over a year ago, and had been testing it with select customers for some months. Pinewood, as the streaming box was internally called, was supposed to offer many of the same apps as competing products from Amazon, Google and Roku, but integrate more directly with Sonos home theater hardware, and also offer some HDMI switching capabilities.

The efforts were powered by an Android-based smart TV operating system developed by The Trade Desk, as I was first to report last year. Now, The Trade Desk apparently lost its flagship launch partner – and Sonos may have bought itself another lease on life.

What else

Scopely buys Pokemon Go for $3.5 billion. Niantic’s AR efforts are being spun out into a new company dubbed Niantic Spatial, which will receive $250 million in funding if the deal goes through.

Wonder acquires Tastemade for $90 million. The food delivery app also owns Grubhub and Blue Apron; I recently wrote about Tastemade here.

Apple considers building its own Meta Ray-Bans. Following the commercial flop of the Vision Pro, Apple is looking at a range of new form factors for AR and AI wearables, reports Mark Gurman.

Sphere Entertainment eyes baby Spheres. The company behind the Las Vegas Sphere is looking to build smaller venues with up to 5000 seats.

Meta will unveil a passthrough API for Quest headsets at GDC. A GDC session listing reveals that some major publishers are already testing the API, which will give developers access to the headset’s camera feed.

Lego wants to make its own video games. After seeing huge success in Fortnite, Lego is now forming an in-house game development unit.

Mike Verdu is leaving Netflix. Verdu had initially led Netflix’s gaming efforts, but more recently focused on an AI project.

That’s it

Ever wanted to build your own VR headset? A new surplus auction of equipment formerly used by Meta’s Reality Lab can help with that: The company is getting rid of a bunch of stuff associated with work on older-generation headsets, including a custom-built Oculus Final Optical Tester, a machine used to build components of the Meta Quest Pro, a proximity sensor testing machine and other testing and manufacturing equipment. Also included in the auction: Boxes of lenses and headset displays, some Oculus Rift headsets and a bunch of messy boxes that may or may not include 3D-printed prototypes … the auction will start later this month.

Thanks for reading, have a great weekend!

And many thanks to Kochava for sponsoring this issue of Lowpass.