A San Francisco-based gaming platform has developed a stellar immersive gaming partnership with Sky Live

Last Edited: August 3, 2023

Demond Cureton

As the gaming industry eclipses the movie and filmmaking sector, gaming firms are leveraging their technologies to create next-level immersive experiences.

Companies such as Meta, holoride, Improbable, Steam, Sony, and many others have begun tapping their serious gaming technologies to achieve this. With their innovative solutions, more and more companies and their customer bases have entered emerging technologies via mixed reality (MR) platforms.

Commonly used for creating real-time 3D (RT3D) content, motion capturing (mocap) has rapidly become a mainstay in the extended reality (XR) market, employed on the latest headsets.

Using their forward-facing cameras, hand and eye tracking, and 3-6 degrees of flight (DoF), the world’s most popular headsets have employed motion-capturing tools to boost user immersion.

A veteran firm in the computer vision and mocap gaming industry has shared its success stories with a simpler novel approach.

XR Today has the pleasure of interviewing David Lee, Co-Founder and Chief Executive, Nex Team Inc. In our discussion, we explore the rise of immersive motion-capturing technologies for the gaming industry.

We also examine his company’s Nex Playground mocap gaming solutions, its Unity-based motion developer kit (MDK), and its partnership with Sky Live.

David Lee: Our journey began in 2017 after we wanted to create motion capture using phone cameras. This took place around the release of Apple’s A12 Bionic chip. Apple had also released CoreML, which basically simplified how machine learning models ran on mobile devices.

Fast forward to today, and that level of processing power is in an increasing number of system-on-a-chip (SoC) processors.

Flagship SoCs now have those capabilities and leverage them for basic mocap and motion tracking on devices. Nex effectively uses these in their setup boxes, where the camera frame, captured on the device, is sent to the neural processing unit (NPU). This is where machine learning (ML) analysis extracts body pose, and we can capture a user’s motion.

This is later passed to the game engine to draw frames responding to it. It’s how we can actually interpret user movements on the screen, allowing gamers to interact with content.

We’ve been working with this for quite a long time and have built the set of technologies into our motion development kit (MDK). We have two kits developed on the Unity gaming engine and feature motion tracking via a single RGB camera.

This deploys motion capture capability across operating systems like Android, which runs on Nex Playground devices. We also feature hardware acceleration, which leverages the SoC’s underlying capabilities, to boost ML inference for pose models. It also contains lots of tools for building motion games more easily.

We use the MDK ourselves and share our strategy with external studios building video games for Nex Playground. Some of the technologies are built in-house, and others come from partnerships as well.

David Lee: Sky Live is an interactive camera for our Sky Glass interactive TV cameras. The TVs began shipping last year, and our interactive cameras launched last June.

These feature advanced SoC chips, processing 30 frames per second to extract motion. This is the core technology that enables people to build motion-based experiences.

Overall, the device has four main use cases: video conferencing, recorded call watching, connected show watching, and lastly, motion gaming and fitness tracking for feedback on workouts.

Nex is basically a major motion gaming partner of Sky, and of the 27 experiences that ship with the launch, Nex is responsible for 17 of those experiences.

This works with a partner beyond streaming by bringing interactive experiences to people’s living rooms. It will also help us expand how we distribute games to households and consumers in the UK market.

David Lee: Remember that mocap gaming is not new and has been around for a long time. You can count the Nintendo Wii, Microsoft Kinect, and the Nintendo Switch also continues what we’ve started.

Unlike the gaming systems before us, [we don’t] require either headsets, sensors on the body, or more complex camera systems. We instead want to build a future where TV and setup boxes, or simpler hardware like the Nex Playground, will have the capacity to let people play motion games, ready for mass adoption.

One could say we envision that the iPhone moment for the TV setup box ecosystem is coming, where use cases for them would expand beyond streaming content unilaterally, but with the camera being highly interactive.

There is already a growing level of interest in these experiences, and different manufacturers are entering the market with their own technologies. We are just one of the entrants in the market with a camera-enabled device that can go anywhere.

We’re excited to finally release our trailer for @starrigame on Steam! Get ready for new game scenes, outfits, characters, and songs coming your way with our Steam release this Fall! ?? Add us to your wishlist now! https://t.co/FjcHYAF6jl　#starri #movetomusic #steam pic.twitter.com/ZRkCGjufGI

— Nex (@nexmotiongames) July 21, 2023

We’ve shipped our games across mobile over the last few years after releasing our first Steam game. However, we note the best gaming experiences are in the living room, in communal spaces gathering the family to have fun.

Our company aims to turn screen time into active playtime, transforming living rooms into playgrounds.

David Lee: Again, the MDK is based on Unity, where developers can download the Unity plugin. Here, they can access the entire feature set of MDK tools.

We work with developers with experience with the Unity gaming engine, providing them with ease of use for our MDK.

Those interested in building games for Nex Playground can contact us directly and sign a non-disclosure agreement (NDA), where we later share the MDK with them.

Some create their first game within a few months, and we can quickly put it up to show other players at the Game Developers Conference (GDC).

It depends on the capacity of the game developers and whether they can have something up and running quickly. However, although getting started is easy, the normal game developer cycle for different studios operates on different timelines. This all depends on the depth, compressibility, and other factors related to the game and its developers.

Some developers brought their VR experiences and found it quite easy to port them onto the Nex Playground. Others would innovate and imagine new forms of gameplay that were impossible before, surprising us. It’s been an incredible journey for Nex to work with external developers to start bringing games onto the ecosystem.

David Lee: Our MDK already supports deploying motion games across phones, tablets, Macs, PCs, and others. However, we understand that the TV is still the best experience, so we keep adding more support for them, including for Sky Live.

In addition to supporting Nex Playground, which is dedicated hardware, we’ve added it for Apple tvOS 17. This occurred at Apple’s Worldwide Developer Conference (WWDC) 2023, where the company added continuity camera support for tvOS.

We’re seeing that all of these camera-enabled platforms are a major target for mocap gaming. This will allow us to build an ecosystem of games for the end user. We’re building services to bundle or integrate on devices like Sky Live, the Nex Playground, and others.

That way, we can keep improving our MDK’s capabilities, support more platforms, and enhance our kit. This will make it easier to build games and encourage developers to bring them onto the platform to build immersive experiences.

We also want to bring intellectual property (IP) on board, validating the platform and scaling it up as per our general strategy. Along with our distribution, development, IP, and other partners allow us to collectively port those really compelling experiences for consumers.