Going hands-free and heads-up is good for productivity, but the real-world context that you get with HoloLens can improve collaboration in profound and unexpected ways.
By Mary Branscombe
In the year since Microsoft’s HoloLens became commercially available, developers have created 150 apps for the mixed reality headset that you can download from the Microsoft Store, including one that shows you which keys to strike as you play the piano. That's nifty, but the real strength of HoloLens is in commercial and industrial use.
Architects, designers, engineers and maintenance experts are finding that HoloLens improves collaboration, gives them new insights and helps them fix problems more effectively, because they can see things they never could before.
Here's how some companies have moved past the inspiration phase and are using HoloLens in production.
Faster fixes
Since last year, 24,000 ThyssenKrupp service technicians who fix elevators in customer buildings have been using HoloLens and Skype to collaborate with subject matter experts back in the call center. “The technical and the expert can highlight and annotate components in the equipment and that makes their call more efficient,” Thomas Felis, vice president of innovation at ThyssenKrupp Elevator Americas told CIO.com. “The expert can identify the parts and components that need to be replaced and the technician can work on the parts while they’re looking at the parts.”
Previously, the engineer had to have a phone in one hand for video conferencing with the expert and a tool in the other for fixing the problem, or they had to turn away and look at a laptop screen. Now they can use both hands for the job.
And unlike the augmented reality glasses ThyssenKrupp’s researchers had previously experimented with, HoloLens doesn’t need the technician to use markers so the system can keep track of what they’re looking at. “HoloLens is able to keep track of where you are in space,” says Felis. “With HoloLens, we have a complete standalone computer; you don’t need to be tethered to any other device like your phone [which you do need] to allow other augmented reality glasses to work, and having the entire feature set of a laptop in HoloLens allows us to put far more features into the application. And it allows you to act hands free.”
Feedback from the technicians is positive. “It's an aha! moment that's really cool; ‘I can work on the component and the arrow remains on it as I work on it, and I get immediate feedback from the subject matter expert.’ The system means they’re finishing jobs in a quarter of the time.”
ThyssenKrupp did need to hire developers with skills the company hasn’t needed before in order to build the HoloLens system. "When you go back to your engineers and say you’re going to hire game designers it's a surprise, but they understood that this kind of app development in 3D, using Unity, requires game developers and storytellers.”
The next step is building a HoloLens app that integrates with MAX, ThyssenKrupp’s predictive maintenance solution that runs on Azure. The company would also like to see HoloLens adapted for hazardous environments. Most troubleshooting calls are in machine rooms where HoloLens is safe to wear, but being able to wear the HoloLens and a hard hat at the same time would allow technicians to use HoloLens when they’re installing a new elevator. “We’d like a safety glass certification for the front shield too,” says Felis.
Faster, cheaper, clearer
The architects at Gensler are also taking HoloLens out into the real world, but they start using it in the office. “What HoloLens allow us to do is to take the digital assets we’re already creating as part of the design process and put them right into the environment and give us the opportunity of evaluating them in their context,” Alan Robles, experience designer at Gensler tells CIO.com.
Gensler uses Trimble’s SketchUp design software, which works on both PCs and HoloLens. “From an integration standpoint, it's been really easy to take what we’re doing and present it in a new way. We don’t want to be creating offshoot, dead-end workflows where assets are created specifically for an app but you can't recover the work you've done in it. With HoloLens, we’re able to visualize things we're already creating and then go back and work on them.”
“We can take our SketchUp models and put them into the environment. Sometimes they’re objects like a new sign or a new desk. Other times, we’re able to take a raw space where the ceilings are exposed and the floor is still concrete and load in the floor plan for the future space where you’ve got walls and rooms, and then we’re able to do the virtual walkthroughs with the context of the real space.”
On site, Gensler can show different ideas for layout. “Maybe we’ll have one with the conference rooms spread out and another with them clustered together and you can see how it feels to walk through that.”
In addition to providing better presentations, HoloLens also speeds up designing the details of a project and reduces the cost. Scale models are expensive and take time to produce — and if you make significant changes to a design, you need to wait for a new model to show the client. “Often when we develop a retail store, our process would involve foam core and card mockups of the store. Sometimes we’re creating those mockups right down to models of the products on the tables. With HoloLens we can eliminate a lot of the process around getting to the mockup, and sometimes we eliminate the mockup altogether. Or we can propose more iterations because they’re digital.” That also saves on waste, Robles points out. “There’s a great environmental case to be made, and it economizes on time. We’re able to propose more, get to a solution quicker and get to a resolution quicker.”
Although SketchUp lets you make changes to the model using HoloLens, you can also work on a PC or tablet to make changes. Unlike a virtual reality (VR) headset, you can see through the HoloLens. This means you don't have to take off the headset to work in the full CAD software, and the customer can keep their HoloLens on and see the changes live, in 3D, right on the model, but in a view that’s easy for them to understand.
“Clients gain a lot of reassurance,” says Robles. “There’s less question about what they’re buying when they understand it in a one to one manner. HoloLens has opened the door for a spectrum of clients to become part of the design process in a more practical and efficient way. Before we had HoloLens, when we were pitching an idea, we expected our clients to translate what they were seeing on the screen to the idea of what the physical experience of this design proposal would be. With HoloLens, we’re eliminating those barriers to understanding."
Gensler is also using HoloLens during construction to "identify places where the physical implementation has varied from the design intent.” That speeds up an expensive process called ‘clash detection.’ “Say a trade worker came in and installed a water pipe a foot over, and suppose that’s where the electrical wire was going to go.” SketchUp lets the site foreman see the plan against the wall, as if they had X-ray vision, and experiment with dragging elements around. They can also leave ‘air notes’ (voice recordings that you make on HoloLens) tagged to the exact location of the problem, so colleagues can hear the notes while they’re looking at the problem.
Putting mixed reality into the workflow
3D models aren’t new, but the real-world context that you get with HoloLens is vastly different from looking at models on screen or even in VR. And that context is as useful for customers as it is for designers and engineers.
“In the physical world, my understanding of scale and navigation is much better,” says Aviad Almagor, director of Trimble's Mixed-Reality Program. He calls mixed reality a transformative technology that is "changing the way our customers deal with data, changing the way they handle data, the way they communicate, consume and interact with data.”
“Architects and engineers are heavy users of 3D. They’re supposed to understand spatial relationships, and what we saw with them was that once they could put the 3D model in the physical world — whether that’s isolated or aligned with the real space — just how fast they could understand design changes and the impact they have on the environment."
But taking advantage of mixed reality at all these different stages of a project means it has to become part of the way people already work, rather than a separate and special thing, Almagor says, “It’s not enough to talk about visualization; what we’re dealing with now is how to bring that into the day to day workflow. If I’m a construction management engineer doing a design review, I start by visualizing the model, but I’m not just wandering around. I have a list of tasks associated with this design, and I can click on a task and be transferred into that location in the model and see what the issues are going to be.”
One unexpected benefit is how useful HoloLens is proving to be for remote collaboration, Almagor says. “It’s been embraced by customers because it allows them to save time and money doing remote sessions where they’re sharing graphic conference between remote teams.”
Mary Branscombe is a freelance journalist who has been covering technology for over two decades and has written about everything from programming languages, early versions of Windows and Office and the arrival of the web to consumer gadgets and home entertainment.