AV and the new realties

Magic Bench – a shared mixed reality experience.

The customer sits on a bench in front of a Large Format Display. On screen, the customer is joined by a virtual personal shopping avatar that reviews a selection of garments, helping the customer make a choice. Is this futuristic vision of the distant future? No, with the slightest of adjustments, it’s here and now.

The video showing Disney Research’s Magic Bench features a new spin on mixed-reality that lets multiple users interact with animated characters and other CGI elements together. Unlike typical augmented-reality or VR implementations, the prototype doesn’t use headsets or handheld devices: the users’ interaction is shown on a large format display and the content can be shared, taking the solution way beyond the headset-based technologies.

Users seated on the Magic Bench, developed by a Disney R&D team in Pittsburgh, see a mirrored image on a large display in front of them, creating a third-person point of view. The scene is reconstructed using a depth sensor, allowing participants to occupy the same actual digital 3D space as the computer-generated characters or objects, rather than superimposing one video feed onto another.

The goal of the Mouse House researchers was to create a ‘walk-up-and-play’ experience. The bench contains haptic actuators, which provide vibrations that let users feel the presence of the CGI characters and objects. The Magic Bench also identifies the location and the number of participants, and can infer where they’re looking. In addition, different formations of people seated create different types of experiences, assisted by sensors which trigger motion, vibration and other real-world elements that enhance the mixed reality experience.

To crack the problem of ‘depth shadows’, which occur in areas where the depth sensor has no corresponding line of sight with the camera, the Magic Bench system uses a modified algorithm to create a 2D backdrop. The 3D and 2D reconstructions are positioned in virtual space and populated with 3D characters and effects. The real-time rendering is a composite image capable of interacting with virtual physics, light and shadows.

The Magic Bench’s 3D reconstruction uses a combination of the depth and colour sensors on an off-the-shelf Microsoft Kinect motion-detection device. Here’s the technical explanation, for those interested in this level of detail: “We draw polygons using each point in the point cloud as a vertex, creating the appearance of a solid mesh. The mesh is then aligned to the RGB camera feed of the scene from the same Kinect. This alignment gives the mesh colour, and completes a 3D reconstructed video feed,” explained the spokesman for the Disney Research project team.

Commercial Reality

If the Disney example is still a little futuristic for you, how about some examples that are very much here and now. A construction industry consortium comprising Autodesk, Microsoft, Laing O’Rourke, AECOM and Doosan Babcock have been exploring the effective use of AR on the construction site. This drive in applying technology to some of the industry’s most fundamental challenges has consultants Soluis to win a competition to develop augmented reality (AR) software to be utilised on site at Crossrail’s sites in London.

In a recent video (https://www.youtube.com/watch?v=MOk_rgN3G3o), Martin McDonnel chairman of Soluis explains recent developments in AR and MR, and particularly its integration with games engine technology. While the Augmented Worker project employs headset technology in order to be deployed on site, McDonnel is a leading advocate of immersive environments that permit shared user experiences.

For example, Soluis Group and Graven Images have teamed up to design a new hotel for Radisson RED, Carlson Rezidor’s new upscale lifestyle select brand “inspired by the millennial lifestyle and fuelled by art, design, music and fashion”. Designers at Graven Images developed sketches and initial design ideas, which Soluis Group then turned into 3D visualisations.

Traditionally, it could take several hours for each frame to render, making the design process – especially small tweaks to the lighting and materials – drawn out and cumbersome. Using Unreal Engine’s real-time capabilities, Soluis Group was able to visualise scenes and all changes live – effectively building a high-end, fully interactive virtual environment where clients could experiment with different elements live. Unreal’s real-time capabilities were invaluable to the client, allowing them to effectively ‘walk through’ the hotel. using the Soluis Immersive Reality Portal running a number of projectors to create a visual experience that is wider than human field of view. (See the solution video at https://vimeo.com/220461080).

AR, MR or VR?

The terminology of the new realties – AR, MR and VR – is becoming increasingly confused, as McDonnel says: “There are just too many (R’s). The differences are important to an understanding of where the market sits for each flavour of technology. ‘Virtual Reality’ is defined as “the computer-generated simulation of a three-dimensional image or environment that can be interacted with in a seemingly real or physical way by a person using special electronic equipment, such as a helmet with a screen inside or gloves fitted with sensors.”

ABI Research says of the VR market the technology: “is often viewed through the lens of consumer-driven gaming, but in a recent B2B technology survey of 455 U.S.-based companies across nine vertical markets, we find that while only 4% of respondents have VR in operation, 85% are at least in the stages of early investigation. In fact, of the 13 technologies highlighted in the survey, VR fell roughly in the middle of the pack, ahead of other innovative technologies like AI and indoor location.”

Michael Inouye, Principal Analyst at ABI Research, adds: “Despite VR being a new technology, with some setbacks already evident, the survey yielded some surprisingly positive results for VR in enterprise and commercials spaces. The B2B market will take longer to develop than the consumer space, but its expansion, at least in the U.S., could occur at a faster rate than we had previously estimated.”

ABI’s findings show that the verticals with the most VR activity or interest, are healthcare, retail, automotive/transportation, and consumer packaged goods. Training, testing, and marketing are all early target functions for VR, along with vertical specific applications like treatments/therapies for anxiety conditions in medicine or virtual showrooms in retail.

Summing up the potential in the VR market, Sam Rosen, Managing Director and VP at ABI Research, concludes: “For any application that benefits from deeply immersive experiences, VR is often a natural fit.  We’re starting to see some early experimentation where VR will expand its horizons. The combination of a VR headset with a camera pass-through for merged reality experience in particular, will open it up to a much wider range of applications. We still expect the consumer segment of the VR market to hold the largest revenue share over the next five years, but eventually we anticipate the B2B opportunity will overtake the consumer space, especially if VR and related technologies do become the next compute platform.”

AR v MR?

‘Mixed Reality’ (MR), sometimes referred to as hybrid reality, “is the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real time.” A distinction is sometimes drawn between ‘mixed’ and ‘augmented’ reality technology, but differences are subtle and largely semantic, with AR defined as a technology that “superimposes a computer-generated image on a user’s view of the real world, thus providing a composite view.”

Rosen forecasts that there be different timelines for mainstream adoption of the various new realities: “We continue to view VR as a long-term market proposition, a technology that will really start to show its true transformative capabilities five plus years from now. In addition, while we currently see clear delineations between AR and VR today and into the near future. We expect these lines of demarcation to begin fading as the technologies increasingly reach maturity and saturation, perhaps then becoming better suited to the mixed and merged reality monikers that some use today.”

But impact on the AV sector is likely to be a good deal sooner than 4 years. Integrated Systems Events Managing Director Mike Blackman commented: “The use of Augmented and Virtual Reality products and solutions is already playing a significant role in the AV business for both our exhibitors and attendees. There is much to learn and much to get excited about. It’s vital that ISE is able to provide a platform for companies and individuals that are keen to understand the current state-of the-art; what’s available today and what’s coming in the future. VR Days has an unrivalled position to help us deliver this and one that we are proud to be involved with and help to develop.”

ISE has announced the start of an official cooperation with the VR Days Europe festival which takes place in Amsterdam on 25-27 October 2017. VR Days is a unique celebration of Virtual and Augmented content, creativity and production taking place in five different venues across the historic city.

Now in its third year, VR Days Europe 2017 – Where Realities Meet – will be comprised of a mix of events featuring a compelling range of keynotes, sessions, workshops and seminars. VR Days 2016 drew over 3,000 attendees. Integrated Systems Events will provide content, marketing and organisational support for VR Days Europe. In return, VR Days Europe will produce a new VR Days event at ISE 2018, specifically targeted towards ISE exhibitors and attendees.

VR Days Europe Festival Director Benjamin de Wit commented: “Our mission is to unite the VR and AR communities, to inspire creative minds and to drive business forward. I am delighted that we are working with ISE as we complement one another perfectly. Together we are confident that we can develop the event and the brand. A number of the vertical market sectors we cover are extremely relevant for the AV market-place. There is the potential for a great deal of synergy.”

VR Days Europe 2017 comprises five key components, hosted in five different locations: The VR Days Europe Conference, held at the prestigious DeLaMar Theatre, will showcase the leading VR and AR creative professionals in a dedicated one-day conference; Expert Tracks, hosted by leading personnel working on content, technology and research, will explore vertical market sectors such as Media, Arts, Entertainment, Healthcare, Education and Enterprise Solutions; the VRackathon will see VR experts tackle technical challenges set by companies and organisations; a CineMart will feature a jury of VR film and technology leaders who will debate projects looking for finance and distribution; the Church of VR will showcase the finest VR content. VR Days Europe will also feature an exhibition and the Halo Awards which will honour leading AR, VR, and MR projects.

Hardware implications

Whether AR, MR or VR, these new realities will have a disruptive impact on display technologies in all display categories from HMDs to LFDs. The sea-change here is the migration of new realties from one to one, solo experiences to shared experiences, with considerable AV / IT integration potential. ABI Research anticipates a growing demand for significantly higher quality displays, with almost two-thirds (66%) of VR HMDs expected to support 4K (Ultra HD) resolution in 2022. Displays with higher pixel density, wider field of view (FOV), and higher refresh rates are being developed to provide consumers with a more immersive experience.

The Augmented Worker – a construction industry consortium comprising Autodesk, Microsoft, Laing O’Rourke, AECOM and Doosan Babcock exploring the effective use of AR on the construction site.
Display resolution is one of the biggest challenges in head mounted displays (HMDs) as well as power consumption, size and weight of the VR HMD. Higher resolution displays are required to solve the ‘screen door effect’ caused by short distances between the user’s eyes and the display. Although the majority of VR HMDs available today support resolution of 2K or less, HMDs with higher resolutions are starting to enter the market.
“Tethered VR devices which are usually targeted at gaming applications, support higher resolutions displays compared to mobile or standalone segments. A number of tethered VR devices provide 2K resolution and some with 4K resolution displays have already hit the market,” commented Khin Sandi Lynn, industry analyst from ABI Research.

VR prototypes with even higher resolution have already been developed. In early 2017, Panasonic demonstrated VR HMD of 6400 x 1440 resolution, 200-degree FOV, by using 4 LCD display with 1600 x 1440 resolution each. Another VR HMD maker which showcased high resolution display was Pimax – its VR prototype supports 8K resolution with 200-degree FOV. Gaming and high-end entertainment requiring higher graphics are likely to drive the demand for high resolution VR displays.


The Soluis Immersive Reality Portal uses a number of projectors to create a visual experience that is wider than human field of view.

While improving the VR display resolution, headset makers are also working towards development of foveated rendering, which computes the highest quality image only at the center of the human visual field. “While our eyes can see full resolution only at the center of vision, foveated rendering tracks eye movement and enables the processor to render full resolution on display any area where the eyes are focusing,” Lynn explains. With efficient eye-tracking technology, foveated rendering sharpens the image at the focus point of the eyes, and reduce the resolution outside the focus point saving the graphic processing loads. Foveated rendering and eye tracking are likely to become important technologies in future VR HMDs for rendering high resolution images.

Future prospects

According to the latest figures from ABI Research, virtual reality (VR) technology will grow to an installed base of 256 million users worldwide and generating revenues of over US$ 60 billion in 2022. While the consumer market is expected to account for most of the market’s revenue, the commercial and enterprise space will expand its share reaching over 40% of the market by 2022, up from 26% in 2015.

“Content availability remains an issue, but healthy strides were made in the past year, particularly in Asia-Pacific and China with location based VR. Further trials and launches will accelerate the adoption of immersive content across the market and expand the use of VR as a tool for training, design, preparatory work and planning,” says Michael Inouye, Principal Analyst at ABI Research. “While the consumer space often garners the largest share of attention the market potential is much wider and we’ve already seen very promising interest in verticals like Retail, Healthcare, Automotive, Education, and Real Estate / Architecture / Engineering / Construction.”

VR experiences still range quite widely from more basic seated/standing mobile HMD configurations to dedicated location based VR installations and VR arcades that allow for 6DOF or room-scale experiences. The rise of wireless HMDs, standalone VR, and 6DOF mobile VR will help narrow the gaps between these different classes of experiences. Additionally, further technology advancements/introductions are on the horizon like eye tracking, foveated rendering, and increasingly higher resolution screens. The advances in shared experience VR, MR and VR will drive the next wave of developments in the AV market.

Related Posts