RISC-V Summit 2018 Keynote: Facebook, Silicon and Augmented Reality

Facebook is invested in their augmented reality-for-all platform, Spark AR, as evidenced by their recent event schedule to discuss it more in-depth. Robert Shearer Director of Silicon Architecture and Modeling at the Menlo Park-headquartered Facebook led a keynote at the inaugural RISC-V open source silicon-oriented conference in — of course — Silicon Valley.

Before discussing what was coming up next, Shearer first waxed wistful about the old days of the Commodore 64 era computing whose core experience was about enabling its users with the superpower of creating new and immersive experiences. Since then, PCs and other devices have matured into different form factors and screens. Pivoting to a familiar descriptor of augmented reality, as a delivery vehicle for “superpowers,” Shearer emphasized, “We’re on the cusp of a complete reinvention of a computer interface.”

Going back to the Commodore 64 era, Shearer discussed how he had a new superpower to create new and immersive worlds. PCs have come a long way since then, and with them about as many types of screens. Introducing augmented reality (AR), Shearer aimed for a high disruptive bar: “We’re on the cusp of a complete reinvention of a computer interface. We believe the next generation is augmented reality” Like the retro computers of old, augmented reality-style superpowers, through wearables, might include increasing human cognition or memory, using glasses to recognize someone and remind you of their name, and selectively muting background noise at a bar or background noise at a bar or restaurant. It’s worth noting that correlating augmented reality with superpowers has long been a key part of the branding of the AR-oriented Augmented World Expo, so the idea is not new although the capabilities of AR certainly has.

With more conventional modes of interaction, though, Shearer cautioned, “a little bit of the connection (between people and experiences) dies. However, with the help of hopefully stylish AR glasses, it might blur the line between a computer interface most people may be comfortable using.

Virtual reality, Shearer believes, will be the “super immersive environment,” while perfecting an augmented reality experience is the long-term objective that should be reached. however, to get there it might take as long as 15 years from now. What it will take to reach that point is improving the fundamental capabilities of the silicon baked into augmented reality.

How many of the one billion virtual reality users Facebook CEO and Founder Mark Zuckerberg quipped about as his vision for the technology at the Oculus Connect 5 conference will happen to be augmented reality users is certainly an open question.

Challenges of AR Silicon

Humans are used to living in a rich audio and visual; world, with natural day-to-day haptics, a package deal without latency. That’s how we define reality, Robert Shearer asserted. To get to a point where augmented reality can approximate it will require more than current-gen off-the-shelf silicon.

“Our appetite (for improved augmented reality experiences) is growing so much faster, so we have to build the silicon to give people the world they’re asking for,” Shearer said. To achieve that blurring of the line in reality between hardware, software, and human perception of reality, it would require a 100x factor of performance improvement from the silicon. In Shearer’s experience, it would require going back to the basics to re-architect what silicon can do.

“We have to understand the thermals, the plastics, the hinge. We have to get deeply engaged with the software team … and the users,” Shearer urged the silicon design and engineer-heavy audience.

Including optics, the discussion mentioned that the new optics, which has “the optics guys … extremely excited,” also means redesigning the silicon to run the displays and manage the optics. The latency requirements mean having to consider distributed computing in the designs. Distributed computing would division of different computation tasks across many different systems, to process the data more effectively to make high-quality, low-latency augmented reality possible.

Audio, too, serves an important role in creating immersive augmented reality. “Audio is such an incredibly important part of our immersive experience” If the audio is out of sync, creating latency, it can damage the immersion  To reach peak audio immersion in AR, as with other aspects, massive compute will be required to get there.

Interaction in AR, complete with responsive touch and ditching anything “click”-related is hard to do, but is another necessary hurdle to surmount to reach optimal AR interaction. User interface, Shearer added, requires far more than “slapping 2D (content) on a wall”. Computer vision, which allows AR to “see” the world around the device will likely still need cameras to recreate visual perception and recreate 3D models.

What talk about AR would be complete without including artificial intelligence (AI)? This keynote was no exception. Mapping space, processing audio and streamlining processes with both software and hardware systems can be accomplished with AI. As many articles, such as this one posted on Forbes, have noted, AI technology is evolving quickly with a lot of room to grow. AR and AI are technologies that are likely to evolve and mature together.

In the end, though, returning to the theme woven through the keynote, Shearer reminded the audience that this potential requires a re-thinking of silicon system design, creating a tightly integrated system from every aspect.

“We need your help. We need dozens of customized cores,” the Facebook veteran proclaimed. Getting FPGA (field-programmable gate array) silicon on developers desks, early developer environments, in the hands of AR developers early on, to help developers utilize every ounce of the power every new advancement in hardware and software, will help too.

Robert Shearer finished his entreaty where it began: with a big picture call to action. He portrayed AR as not a toy, not a fantasy, the next generation of computer interaction. “Let’s redefine the world,” Robert Shearer proclaimed. That should naturally include virtual worlds. The notion of ever-improving superpowers is a natural fit for games — presumably in augmented reality — and will likely lead the way in demonstrating the capabilities of what next-gen augmented reality in the next 15 years could be capable of.