Spatial AR: beyond the goggles

When you mention Augmented Reality, most people will think of goggle based systems like the Microsoft Hololens and the Magic Leap, or increasingly, phone or tablet based apps that overlay digital content on a camera feed.

However, the next frontier in Spatial AR requires thinking about more than what a user puts on their head, or holds in their hand. Spatial AR involves overlaying persistent, accurate digital information on real world environments. Since these applications go beyond the scope of a controlled environment, it will be necessary to work with alternative technology strategies, understanding of physical reality, developing useful augmentation content, and most importantly, understanding the user’s context and needs.

The good news is, that with a proper approach, many of the underpinnings of a successful AR application can be re-used across multiple technologies.

A good AR implementation partner should understand integration with enterprise APIs, 3D mapping and visualization and immersive UX, in addition to the actual AR technologies. Britelite Immersive has extensive in-house expertise in all of these areas.

What you need to succeed in Spatial AR

When setting out to craft a Spatial AR experience, it is tempting to just focus on the “sizzle”, the attention-getting technology that a user encounters directly. However, to make it really memorable and deliver true business value, there are some key factors that need to be taken into account: technology, reality capture, augmentation, and user sensing. I’ll cover these in greater depth below.

Expertise in technology delivery

By their very nature, many of the current technologies used to deliver AR experiences are individual. Only the person wearing the goggles or holding the tablet can see the augmented experience. Well known examples of these technologies include:

• - Goggles (Hololens, Magic Leap)
• - Phones and tablets (ARKit, ARCore, WebAR)

Many of the tools used to develop both for goggles and screen-based AR are the same. Systems like Unity, Unreal Engine and XCode are able to manage 3D assets and augmentation layers in ways that allow for apps to be deployed on multiple different systems.

In addition, web-based technologies such as WebAR are quickly getting to the point where complete AR experiences can be built using familiar tools like Javascript, node.js and an entire ecosystem of tools that are already able to connect to enterprise systems.

At Britelite, we are enthusiastic users of all the user-facing technologies, and we’ve integrated them with back-end tools to deploy large-scale, reliable applications. Many of our experiences involve seamlessly connecting powerful front-end immersive technologies to robust back-end systems. AR is just one of the ways that we deliver memorable experiences.

Creative Technologists at Britelite are AR enthusiasts

A big step forward in Spatial AR is creating a multi-user infrastructure, so that multiple people using these systems can experience a shared world. Both ARCore and ARKit have recently announced features that enable such shared experiences (WorldMaps and Cloud Anchors).

These technologies for sharing an AR experience require a robust back-end app to manage the shared world.

In addition, there are other systems (in addition to goggle and tablet based AR) that allow multiple users to participate in an AR experience simultaneously:

• - Projection mapping: A physical model or space can be augmented by accurately projection mapping relevant information onto a surface. For this Britelite project, key features of a building were highlighted on a model in sync with an informational presentation.

• - Object sensing tables: tables that can sense objects placed on them, and adapt their presentation, are a form of Augmented Reality, and one that lends itself readily to sharing. Britelite is deeply familiar with the use of object sensing tables as a form of collaborative AR. We have used it to build many applications, including a detailed robotic surgery simulator.

[caption id="attachment_3703" align="alignnone" width="512"]

Robotic surgery simulation on an object detection table[/caption]

A good implementation partner for a Spatial AR experience should be familiar with multiple different ways of delivering an immersive interactive experience, and architect them such that most of the development work can support multiple different user-facing technologies.

Mapping and understanding physical spaces

In order to augment reality, it is important to capture it in detail in the first place. Headset and phone based AR systems rely on various techniques to generate a spatial map of their location, but this can be limited by device and user constraints.

At Britelite we are deeply familiar with the tools and techniques required to model physical reality in a digital world.

• - Matterport scans: Britelite used this extremely detailed indoor scanning technique to generate realistic 3D models of spaces in a new hospital that would not be accessible to visitors, and applied an information overlay. By using a touchscreen or a tablet, visitors were able to access detailed information about all the equipment in the room.

• - Drone scans/3D photogrammetry is a powerful tool for building 3D models of buildings and outdoor sites - we have used these as input for AR and VR tools

• - Depth cameras: cameras that can sense depth are extremely useful in Augmented Reality, since they can understand the physical space in front of them, and use this to guide the placement of augmented content. In a project called Perfect Pint, Britelite used Intel Realsense cameras to map a user’s face and apply virtual facepaint in the colors of their preferred team.

[caption id="attachment_3706" align="alignnone" width="512"]

Guests could receive their AR selfie by entering their email address[/caption]

Building a truly useful augmentation layer

The augmentation layer is far more powerful when it really augments the user experience: to do so, it needs to be connected to real systems and data sources. A Spatial AR implementation partner should have the capability to interface with real world data sources and enterprise systems.

• - Tempur AR experience: For this Britelite project, when a user inspected a mattress with their phone, not only did they get relevant information about that item, the app was able to use retailer APIs to give accurate information about pricing and stock levels.

Britelite has executed multiple projects where we connected an interactive front-end to real enterprise systems via APIs. Uniquely for a digital creative agency, we have a solid in-house back-end development team that can deliver these projects.

Knowing the user

Augmented reality content needs to be relevant to the user, and so it is key to get accurate information about their situation. At the most basic level, an AR app needs to know:

• - Where they are: their position in the space where the experience takes place
• - What they are looking at: their “pose”, the direction they are looking at and their physical stance

However, to really create impact, beyond a tech demo, it is critical to know:

• - Who they are: information that identifies a user as an individual.
• - What is relevant to them: once we know who they are, we can present an augmentation layer that is customized for the.

Britelite has extensive experience with many different ways of sensing user interactions in addition to AR headsets and tablets.

• - Depth cameras: We have used Kinect and Intel Realsense to derive very accurate information about a user’s position, and used it to map very accurate interactive content. In this example, for a high-profile gala event, visitor motion triggered animations of plants and animals on a virtual “living wall”.

• - Wearable: Users can be encouraged to wear a Bluetooth enabled bracelet as part of an event. For this high-end gala, Britelite integrated smart bracelets with various different interactive installations. Guests were greeted by name when they interacted with the different activities, and had content tailored to their specific interests.

At Britelite, we have found that integrating multiple different sensing techniques specific to an installation can yield the best results. A wearable bracelet or an NFC reader can help identify a specific individual, and then a tablet or AR goggles can deliver information to them. This requires a well thought out back-end system to tie everything together.

Why AR is for your organization

Organizations like leading museums, and themed experiences are using spatial AR as a powerful way to create memorable experiences that are impossible any other way. The growing interested in spaces like Meow Wolf, Atelier Des Lumieres, and OneDome Global is partly due to the way that they integrate digital media with physical spaces to create compelling visitor experiences.

[caption id="attachment_3710" align="alignnone" width="512"]

Experiencing virtual artwork at OneDome’s Unreal Garden[/caption]

However, other organizations can benefit from the techniques that have been used in these spaces.

For example, lobby and reception areas can be turned into experiences that:

• - Create a memorable impression for clients, vendors and employees
• - Reinforce branding and values
• - Create a powerful impact in social media

By adding an augmentation layer to existing physical premises (or new ones) it is possible to dramatically enrich visitor experiences, in a way that can be updated and changed to keep it fresh without additional physical build.

For this project, Britelite used a combination of detailed scientific content, high-resolution custom LED panels, and multiple interactive sensors to transform the lobby of a well-known healthcare and pharmaceuticals company:

Beyond the demo - adding real value

When setting out to build an AR project, it is easy to focus solely on the tech that is used to deliver the experience. However, it is important to take into consideration all the other factors that deliver real value from the experience. Britelite has a unique combination of design, creative technology, front end and back end software development, all in house.

The Britelite process, with a focused discovery phase, identifies clearly what the source of value of an AR (or other immersive) experience is to be, and lines up all the various components that are needed to make it work.

We are able to develop the front-end application, and also build an integrated back-end software platform that accesses real data sources such as enterprise APIs.

By going beyond an attention-getting front-end, and connecting a spatial AR experience to real systems, your organization can go beyond the demo.