Welcome to Lowpass! This week: Why mixed reality is about people and their furniture, and why Humane is doomed. Also: Paradise!

The future of mixed reality, according to Jesse Schell 

With both Meta and Apple selling mixed reality headsets to consumers now, many industry insiders are wondering: What is going to be the killer app for mixed reality? Will it be productivity, as favored by Apple? Or will it be gaming, as Meta is betting?

Jesse Schell believes it can be both – but he also thinks that we will see something totally new emerge. “I think a new genre is coming,” Schell recently told his audience at GDC in San Francisco. “I call this genre adaptive in-home storytelling.”

Schell has thought a lot about this subject, and in fact worked on it longer than most. In the 1990s, he developed some of the industry’s first VR rides for Disney, where he later also contributed to “Toontown Online.” He’s also Distinguished Professor at Carnegie Mellon University’s Entertainment Technology Center, and the author of “The Art of Game Design.” And as the founder and CEO of Schell Games, he has been leading a team that has worked on many popular VR games, including the I Expect You to Die franchise and Among Us VR.

Still, when mixed reality first emerged, Schell was skeptical about its potential for gaming. One reason: Mixed reality doesn’t offer the same kind of immersiveness and escapism as VR. “I don’t want every game to be in my dingy living room,” Schell said. “No, I want to go to amazing fantasy worlds.”

But what if that living room, dingy or not, became part of the plot?  “You’re talking about level design,” Schell told his GDC audience. “Guess what: Your house is the level. The characters have to figure out how to do their story in your house.”

Schell went on to spitball a scenario about someone ringing your doorbell after you put on your headset. You open the door and discover that it’s an AI-driven virtual character, who proceeds to enter your house, seek out the kitchen, unload virtual groceries on your real-life countertop, use your real oven to cook some virtual food, and then proceed to tell you about a quest that involves something hiding in your attic that the two of you will have to find together. “This is gonna be a thing, and it’s totally possible,” Schell insisted.

The team at Schell Games has also been experimenting with mixed reality multiplayer gaming, and Schell showed a video of one of these experiments at GDC: A virtual game of Hot Potato, but instead of beaming participants into each other’s living room, everyone gets to play in their own space, and objects are being tossed from space to space, with furniture doubling as obstacles.  “This combination of social and physicality, it's not like other media that we’ve played with before,” Schell said, adding: “Mixed reality is about people, and it’s about furniture.”

Schell also predicted that mixed reality will be huge with kids. “This is a medium children are going to love,” he said. “They’re so imaginative, and they love physical play. This brings those two things together.” 

But what about the grown-ups? In addition to having adventures with virtual, AI-driven characters who invite themselves over for virtual cook-outs, us old people will ultimately also use mixed reality for work, acknowledged Schell. The prime use case of the Vision Pro – a spatial computer that lets you work with multiple virtual screens at the same time – does have some merits, but won’t be fully embraced until we have more lifelike avatars for teleconferencing, he argued.

“Headset-based social telepresence could actually be better than Zoom,” Schell said. “That’s gonna be the killer app for adults.”

Schell’s entire GDC talk was published by Meta on YouTube this week, alongside a series of other sessions about mixed reality development, monetizing VR and the story behind Asgard’s Wrath 2, among other topics.

Too hot to handle

It’s been a harrowing few days for Humane, the startup behind the $700 AI Pin: Reviewing the AI Pin, The Verge called it “an infuriating product” that “just doesn’t work.” Engaged deemed the AI Pin “slow, finicky and barely even smart.” Bloomberg found that the device is “never going to work and not worth buying.” And Marques Brownlee called the AI Pin the worst product he had ever reviewed, adding: “Nobody should buy this device right now.”

My take is that Humane’s wounds are largely self-inflicted. If the company had released the AI Pin as an early access product to a small group of beta testers, then one might have been able to excuse its many flaws as necessary steps towards a future AI wearable. Instead, Humane styled itself as the next Apple, proclaimed that it was going to replace the smart phone, and maneuvered itself into a corner that it may not be able to escape from.

Sure, the company can improve the software of the AI Pin over time, and has already promised to add such incredibly complex things as timers and directions as part of an upcoming software update. But some of the most egregious flaws of the device are rooted in its hardware.

• Thermal and power issues appear to be widespread. Every review noted that the AI Pin frequently got too hot, forcing it to suspend operations until it cooled down. Brownlee suggested in his review that this was due to the company’s decision to charge the AI Pin wirelessly through clothing, which also seems to contribute to battery life issues. Sometimes, thermal issues can be addressed with software updates, but it’s hard to see if that can apply to charging-related heat.

• The laser projector just doesn’t seem to work. Reviewers universally panned the laser projector for low legibility, especially outdoors. My hunch is that the projector is a bit like the outward-facing Eyesight display on the Vision Pro: A feature each company committed to early on, forcing it to see it through despite obvious flaws. The key difference is that Vision Pro users don’t really need Eyesight, whereas the laser projector is essential to the AI Pin.

• Many AI Pin design decisions are questionable at best. As I suspected, the AI Pin doesn’t work with long hair, or certain types of clothing. But the company built it to be worn on your left upper chest, which is exactly where your typical seat belt sits, as Brownlee points out in his review video. How this obvious design flaw wasn’t detected before launch is beyond me.

Hardware issues like these are impossible to fix, unless you’re willing to start over from scratch – and I have my doubts that Humane is going to have the will, or the capital, to do so.

What else

This startup wants doctors to use music in the ICU. My latest for Fast Company is a look at MediMusic, which dispenses music as medicine. 

Take-Two Interactive is laying off 5% of its staff. The company is also cancelling the development of multiple games.

Prime Video now has more than 200 million monthly viewers. Amazon CEO Andy Jassy announced the milestone three months after the company began showing ads to most of its subscribers.

Roku got hacked, again. A new security breach affected more than half a million users.

Truth Social announces shiny new thing. Trump’s social media company has long said that it wanted to eventually get into the streaming business. Now, just as its stock is cratering, it’s reminding everyone of those plans.

YouTube VR gains 8K playback on Meta Quest 3. The resolution update was overdue to keep YouTube relevant in the immersive video space.

TikTok is launching a photo app. The app, dubbed Notes, is being tested in Australia and Canada for now.

YouTube now accounts for close to 10% of all U.S. TV viewing. The Google-owned service is more popular among U.S. TV viewers than Hulu, Prime Video, Disney+ and Max combined, according to Nielsen.

That’s it

I just finished a few TV shows, so I asked my longtime friend and former colleague Liz Shannon Miller for a viewing recommendation … and she suggested to watch the movie “Paradise,” which is now streaming on Tubi. Here’s why, in her own words:

Ella (Patricia Allison, one of the many breakout stars of Sex Education) lives a rough and tumble life in the small town of Paradise, but it's not until her father the sheriff (Bashir Salahuddin) gets killed under mysterious circumstances that she realizes just how rotten Paradise might be. I'm so glad I gave this admittedly violent but swiftly paced little film a shot, as a great cast (love Tate Donovan, and Tia Carrere has an eyepatch!), a rich color palette, and a compact runtime (83 minutes!) help the movie fly by. Watching this, I was primarily reminded of the novels of Carl Hiaasen — a little bloody, but very human and funny.

You can read more recommendations like this one in Liz’s Stream On newsletter, which gets published by Consequence every Friday. Check out the latest issue here, or simply join me as a happy subscriber.

Thanks for reading, have a great weekend!

Photo by Semeon Hrozian on Unsplash