UX Design for Hololens & AR

Not too long ago I found myself standing in the middle of a restored mine, birds were chirping and wind was rustling the trees. The amazing thing about it, was that I was actually in the office of LOOOK; a Hololens development studio in Seattle, one of the first of its kind. I had a Hololens headset on and was demoing one of their current projects, and having issues with the voice control and hand gesture interactions (they take a bit to get used to…).  The Creative Director, John Howard, agreed to chat with me about the landscape of virtual and augmented reality, and what it was like to be working in such a new and fascinating medium. Mainly I was curious about how one might get involved in designing experiences for AR and VR, and what kinds of technical skills are required to break into the field.

The office was in a brick building in the middle of Capitol Hill, with high ceilings and lots of space. Most of the room was actually completely empty because of course that’s where they would test the products they were working on. John was very welcoming and immediately hooked me up with a coffee and a Hololens to check out some of their work. Looking around the room, it was suddenly full of colorful objects such as a spinning ballerina figure, a cake sitting on top of the fridge, and a floating ‘menu’ type object directly in front of me. I could walk up to the holograms and interact with some of them, it was all very cool and quirky.

One of the common criticisms of the Hololens is its narrow field of view. This was definitely true, the holograms were visible in a small window directly in front of my eyes, anywhere in my peripheral vision the holograms disappeared. John pointed out that it’s amazing how quickly we expect magic from new technologies. Maybe we should blame it on movies like Minority Report and Avatar, for making us feel entitled to flawless holographic computing. I do have to admit i was guilty of high expectations as well. After the demo we sat down and chatted about some of the hurdles facing mass consumer adoption of AR and VR.

John explained that he believed three emerging technologies are going to drastically change the world as we know it in the near future; mixed reality, the internet of things, and machine learning/machine vision. So much has already been accomplished in these areas, and we are close to a shift in our technological landscape that is on par with the coming of personal computers and the internet. A book he recommended which I have yet to read is “The Inevitable” by Kevin Kelley, about the 12 technological forces that will shape our future. Some of my favorite books on the subject are the best sellers by Michio Kaku, a futurist who has been studying underground technological advancements and making them accessible to the public. It’s pretty clear that UX design is about to change drastically as products we are designing evolve from 2D web apps and software to physical objects, 3D holograms, and virtual environments.

There will be a strong need for competent UX/UI designers to help create a framework and set of best practices for product development in these new areas. There are so many challenges in making these interfaces intuitive to use because 2D solutions no longer work. In a way, we have to revert back to how our species evolved; interacting in three dimensions, and smoothly assimilate our newfound technology back into the familiar world around us.

 So far there have already been many unexpected hurdles that emerged when creating experiences in mixed reality. A commonly cited issue is the phenomenon of occlusion; where virtual objects are blocked by objects in the real world. Extremely complex programming is required to counteract the vast array of variables that the software must react to such as shifts in light, weather, or environment. Another challenge is finding the right balance of user interaction with the computer generated objects and environments. Available options for user input are gaze, voice, hand gestures, a combinations of these, as well as hand held controllers such as those available for the Oculus Rift. There have been documented cases of arm fatigue for gesture interaction, and many usability concerns with gaze controlled interfaces. On the other hand, many augmented reality applications have ‘non-command user interfaces’ which don’t require input from the user but are constantly ‘listening’ for changes in the environment and responding to them automatically. Here’s a scenario that serves as an example of a non-command UI:


“consider an airplane mechanic who crawls around inside the guts of an aircraft for an inspection and needs to check for how long a certain part has been in service. With a traditional screen-based user interface, the mechanic would have to somehow “save” the part number (by remembering it, taking a picture of it on a smartphone, or writing it down on a piece of paper) and then access a phone or computer-based system to determine for how long that part has been in operation. But with an AR technology like HoloLens or Google Glass, the service record could be displayed right on top of the item, with little to no commands from the user.”


In these types of applications, it’s data visualization that presents the biggest challenge, or how to overlay the data onto the environment in a meaningful and easy to digest way.  John explained that in his opinion, the two main use cases for augmented reality are communicating ideas and concepts from one person to another, and exposing the hidden data present in the world around us at just the right time in context of the activities being performed.


As you can see, there is no shortage of where these technologies can take us, and simultaneously no limit to the types of usability issues we will be tasked with solving. So how can we prepare ourselves for this transition? There is very little material available online directly geared towards design for mixed or virtual reality. John’s advice was to look toward game design. Being used to thinking in 3 dimensions, game designers have a huge advantage in the VR/AR space. LOOOK studio uses Cinema4D to create image assets, others use Unity3D or Maya LT. He suggested finding a local community of people working on AR/VR applications, start learning about the challenges they are facing, and get used to thinking and talking about solutions. At RED Academy in Vancouver, they are offering a course called ‘Fundamentals of AR/VR and Mixed Reality’, which covers Unity3D basics, limitations of hardware currently available, and frameworks for designing in VR (among other things).  There really is no one stop shop to learn about this field as it is so new and best practices haven’t had time to emerge yet. The best we can do is reach out in our communities and start working on solutions that will contribute to the global knowledge base on the subject. I want to thank John again for the great conversation, insight, and advice!

Check Out John Howard speaking about UX/UI Design for Augmented Reality and spatial experiences: