Augmented Reality and Deep Learning towards the Management of Secondary Building Assets

The retrieval of as-is information for existing buildings is a prerequisite for effectively operating facilities, through the creation or updating of Building/Asset Information Models (BIM/AIM), or Digital Twins. At present, many studies focus on the capture of geometry for the modelling of primary components, overlooking the fact that many recurring actions need to be conducted on assets inside buildings. Furthermore, highly accurate survey techniques like laser scanning need long offsite processing for object recognition. Performing such process on site would dramatically impact efficiency and also prevent the need to revisit the site in the case of insufficient/incomplete data. In this paper, an Augmented Reality (AR) system is proposed enabling inventory, information retrieval and information update directly on-site. It would reduce post-processing work and avoid loss of information and unreliability of data. The system has a Head-Mounted Display (HMD) AR interface that lets the technician interact handsfree with the real world and digital information contained in the BIM/AIM. A trained Deep Learning Neural Network operates the automatic recognition of objects in the field of view of the user and their placement into the digital BIM. In this paper, two uses cases are described: one is the inventory of small assets inside buildings to populate a BIM/AIM, and the second is the retrieval of relevant information from the AIM to support maintenance operations. Partial development and feasibility tests of the first use case applied to fire extinguishers, have been carried out to assess the feasibility and value of this system.