AR/VR: AR Mobile App for Equipment Maintenance

Customer

A European provider of workforce mobility solutions for manufacturing and industrial companies.

Need

The Customer turned to HQSoftware to develop a cross-platform mobile app that would allow for on-site maintenance of complex industrial equipment without the help of experts. As the tool was to create and play back interactive repair and service guides with augmented reality, the project team faced the challenge of ensuring image recognition with unmatched precision.

Business Value of AR in Media & Entertainment

How do modern businesses use AR to increase their revenues and outperform their industry competitors?

Download the whitepaper

Solution

Functionality

For those who use the app for the first time, the interactive, visualized tutorial comes as helpful, showing the basics of creating a guide.

To begin with, a user takes a photo of the equipment they want to create a repair or service guide for. The picture is then processed by Metaio SDK and turned into an image based on which the app will recognize the equipment during playback. That’s where HQSoftware’s decision to employ Metaio’s library pays off, as the tool offers in-built image recognition capabilities, allowing to save on cumbersome programming of custom markers from scratch.

Once the picture is taken, the user, pointing their tablet’s camera at the equipment, goes on to create a series of scenes with comprehensive, step-by-step explanations of all repair operations.

For each scene, the user can apply – that is, lay over the camera view on the screen – a wealth of animated, drag-and-drop 3D objects, such as a screwdriver, a nut or an arrow showing the direction, to better illustrate how to handle a specific action. All objects are scalable, moveable and can be placed at any angle. In addition to that, the user can attach text, images and videos.

When playing back the guide, all the user needs is to simply point their tablet at the equipment and follow the steps. Should the camera deviate so that image recognition becomes impossible, the app will notify the user to move it to the initial position. There is also a possibility to switch between scenes by using a QR code.

When the process is stuck or some elements of the equipment need replacement, the user can leave a maintenance request, and a relevant technical expert will get in touch to help solve the problem.

Technical details

At the technical level, a scene is a JavaScript object containing information on the attached 3D objects with the relevant coordinates and tilt angles. By collating it with the image initially generated from the photo of the equipment, Metaio SDK knows precisely where to place the 3D objects in relation to the real-world picture seen through the tablet’s camera when the guide is being played back. For quick and unparalleled access from the library, 3D objects are stored on an Amazon S3 server.
To ensure smooth synchronization of scenes and attached 3D objects, HQSoftware’s team implemented a proven combination of PouchDB and Apache CouchDB on the frontend and on the backend, respectively. In addition to their light weight, these databases allow storing data offline – for example, when the user is leaving a maintenance request – and sharing it when going online again.

Coupled with the open-source Polymer Project library, Metaio SDK was instrumental in creating iOS and Android native experiences while exploiting a single JavaScript code.

Outcome

HQSoftware developed a cross-platform mobile app to create and play back interactive repair and service guides enriched with augmented reality. Allowing for on-site maintenance of complex industrial equipment without training, the tool is already in use at multiple industrial companies.

Read more about our AR/VR development services.