Virtual reality and augmented reality barely have their foot in the door to the mass market and their successor is already knocking. Something that has what it takes to outstrip everything that’s gone before it: “mixed reality” is one of the biggest transformations of all time. Kevin Prösel explains what that means for brands and creatives.

Just imagine a vision that has been around for more than 60 years in the minds of individual creatives: the idea of creating an artificial experience that appeals to all the observer’s senses and conveys to them the most tangible picture of another, enhanced reality. One of these visionaries was Morton Heilig, who in 1962 produced the first prototype of an “Experience Theatre” that played multisensory films in a mechanical way, adding vibrations, stereo sound and even wind and smells. For the price of 25 cents you could ride a motorbike around Brooklyn or bomb around the Californian dunes in a buggy. Morton was a cameraman and had developed a method to produce 3D films shot on a 35-mm movie camera, which he wanted to play inside his “Sensorama” machine as the cinema of the future.

Sensorama was way ahead of its time but a financial flop. Nobody back then could imagine how to possibly sell such a futuristic experience at a time when there weren’t even decent arcade games (Atari’s Pong only came along in 1972). Which was a shame as I would have loved to find out what an artificially replicated motorcycle ride through 1960s Brooklyn might have smelled like.

Today, more than six decades later, several parameters have changed, and we’re now calling such an experience “virtual reality”. It is currently the big thing in the entertainment industry and, according to current Statista forecasts, expected to generate a global turnover of more than 20 billion US dollars in the year 2020. If he had have come up with his invention a few decades later, Morton Heilig would be on a monetary par with Marc Zuckerberg and probably be the owner of a huge villa with pool in Beverly Hills by now.

Or at least that’s the theory. It seems to be a tragedy of history that the great visionaries are always ahead of their time. Ivan Sutherland, the spiritual father of augmented reality, a reality overlapped with virtual objects, might have gained worldwide recognition for the invention of the first graphic programme Sketchpad in 1963 – but not many people know that he also built the world’s first AR goggles in 1968. But these “goggles” can’t really be compared to today’s consumer headsets: they were more of a monstrous apparatus with two big cathode ray tubes, one for the left and one for the right eye, which projected a computer-generated wire-frame cube into the user’s field of vision. As the apparatus was too heavy for the user to carry on their own, it had to be affixed to the ceiling above them – hence the wonderful name “Sword of Damocles”.

Fortunately, today’s technology has nothing more in common with lethal Greek weapons.

Size and performance have both evolved to such an extent that a computing power unimaginable at the time can now fit into small, practical smartphones. The data headsets we are wearing now are lightweight, stand-alone mini-computers. At the same time, in addition to increasingly detailed depictions, the range of functions has also been extended to include previously unimagined possibilities: spatial computing, i.e. the opportunity to scan the space around the wearer of the glasses using sensors and then perfectly complement it with virtual presentations, is the crucial difference between mixed reality and augmented or virtual reality.

The world’s leading tech companies are also acknowledging this difference. As well as Microsoft, big names like Acer, HP, Lenovo and Samsung are also working on low-cost MR devices. In the shadow of the big players, disruptive start-ups like Meta and Magic Leap are hoping to establish themselves in this field too, promising an unprecedented expansion of reality with the aim of further stirring up the market. But the technology will really make an impact on the mass market when Apple integrates its MR/AR ambitions on a wider scale into the palette of new iPhones and iPads. Not an improbable scenario when you consider that Tim Cook has already bought a Munich start-up specialised in AR and the first insights into Apple’s new AR applications already looked very promising. His statements on the topic also suggest AR will be playing an important role for the company: “I regard AR as a big idea like the smartphone. I think AR is that big, it’s huge.” With the release of the ten-year anniversary edition of the iPhone, it is rumoured that Apple will be launching a portable MR device. The corresponding market pressure with presumed sales in the millions would suddenly elevate the technology to a level that would make it possible to offer MR in a manageable size for everyday use.

MR will come, perhaps even with more of a vengeance than VR and AR – but what are we going to do with it?

MR headsets merge virtual holograms with real-life, physical spaces. Using integrated gesture recognition, it is possible to even interact with the holographic objects with your bare hands, without having to use any other joysticks or controllers. Due to the fact that you’re still seeing the real space, unlike with VR you also have the undeniable advantage that the user won’t feel dizzy or disoriented and can enjoy the experience without being isolated. That means the field of creative possibilities goes way beyond the scenography possible up to now: interactive holograms will mean games integrated into the space, statistics, extended workspaces, insights into products, X-ray vision through buildings (finally!), holographic communication and telepresence, to name just a few possibilities. Entire industries can now think in applications that were impossible before, from the entertainment sector to the architecture and construction industry, the museum and cultural sector, the military and civil transport and traffic, down to education and research – and the sector that affects me and you the most: advertising! Or to be precise, digital advertising. A true revolution of communication that can lead to an enormous increase in product sales and, above all, means a never-seen-before interaction of customers with products and information.

Every space, every area – and yes, even every object – can be recognised by the MR technology and become a projection surface for holographic content. This content is no longer limited to two-dimensional, but can be experienced three dimensionally. You can look at it from all sides, virtually touch it, move it – and even speak to it. It is therefore breaking down the boundaries between the physical and the digital space – the mantra of clearly distinguishable media channels chanted by previous generations of advertisers is melting away like an ice cream on the last day of the summer holidays.

And because technically anything in the world could become a possible communication channel, we are moving beyond the limiting frameworks of concepts of “classic” desktop, tablet or smartphone devices with their restrictive screen estates.

New forms of presentation, completely integrated into the surroundings, can establish themselves and let every user experience the future as if they were a protagonist in a sci-fi Hollywood blockbuster.

But it will be the present and no longer a distant future in a far-off galaxy. For example, walking through a supermarket in the near future will be a whole new experience. The labels on products flap open and tell me about the quality or origin. I will be able to look inside packaging to find out everything I can do with the products. On promotional areas, three-dimensional holograms of celebrities will inform me about the best way to drink my coffee. And my wife will call me via holo-video to remind me to buy milk for our son.

But sadly, I’m no Luke Skywalker, and until that time comes, new schools of thought and completely new approaches of storytelling and production are needed – but also new models for managing and gauging campaigns. And we shouldn’t forget: with a scalable availability and growing acceptance of the MR technology, this also means a growing demand for suitable digital content and suitable applications. The expectations of millions of users will increase, and they will be demanding a greater presence with appropriate contents and formats from brands. And not just art for art’s sake, but with relevant concepts and stories.

Who is in and who is out? The race for the most innovative and best-positioned brand in mixed reality has begun.

If we ask ourselves the question of whether agencies and clients are already prepared, the clear answer is: no. And that’s a good thing. After all, despite all this optimistic and future-oriented technological savvy, we have to look at the situation realistically. We are currently experiencing the first generation of MR devices, which – although impressive and robust in their technological potential – are mere pioneers, opening the door for us into the next dimension of communication. And this is exactly where the big opportunity currently lies: it’s still not too late to jump on the bandwagon. The next device generation will be even smaller, even more powerful and, above all, offer an even more immersive experience. We will develop creative concepts that will even surprise us – because the technology offers unlimited opportunities. We have to seize it, use and test it in all settings and scenarios if we want to be actively involved in shaping this revolution of communication. Brands can gain initial experience and build up expertise in the field to make sure they are sitting firmly in the saddle when the breakthrough happens and become big holographic players. A step that is possible without any long-term investments and commitments. In exchange for that, agencies have to develop necessary process structures and expertise, because as I mentioned earlier, the old paradigms no longer apply. At the same time, it will be our key task to work out the added value for brands.

For example, the increased use of permanently portable MR headsets will lead to a higher demand for contents and all kinds of data. Every single thing and every place needs its own holographic content – preferably with the right brand message. The Internet of Things was yesterday, clever minds are already proclaiming the Internet on Things.

In such a usage scenario, an incredible amount of behavioural data would also be collected to enable additional insights into a quality of information that gets you even closer to the customer than all previous measures could ever have. Simply due to the fact that we are following the user’s view through their everyday life, scanning their surroundings and receiving virtual maps of places they are in, which – incidentally – will also open up a brand-new field of activity in terms of data protection. Apart from that, advertising messages can be inserted into these surroundings in a much more targeted way than was ever possible before. Especially as it will take place in completely new forms and formats that will be much less annoying for the user – because they will fit into the space, correspond to the user’s current situation and, where possible, even support it. Other scenarios outside the advertising sector enable applications in which it can be helpful to make invisible things visible – such as X-ray vision into complex technical machines or into the human body. Information located within the space can simplify a navigation or offer instructions for complicated machines (I need something like that for our washing machine at home…).

The true disruptive power of this technology lies in the realisation that, with mixed reality, we have the opportunity to break down the borders between space and time.

It no longer matters which continent the person I’m talking to is currently in: looking through MR glasses makes it feel as if they are standing right in front of me as a hologram. I can have a conversation with them in real time while virtually modelling the space that we’re standing in. I can even follow where their gaze is wandering within the space. Or I might be conversing with someone currently inside a space capsule – perhaps they don’t even have to be a real person anymore. With the possibility of holographic telepresence, I could also create holograms of fictional or deceased people and communicate with them. The performance by rapper Tupac Shakur, who was killed in 1996, as a hologram at the 2012 Coachella music festival in California was certainly a start. Which of course raises the question, which great personality from world history would you like to meet in this way?

Present, future and past are blending together, the world is becoming smaller – but at the same time, the level of content is increasing. Mixed reality is making it possible. We are making it possible. After all, at the end of the day we will be the ones who pave the way as visionaries. And who knows, maybe my son will one day remind me: “Mixed reality, holographic experiences?! That was cool, but so… 2017!”

Illustrations: Tatiana Sukarja