Multi-camera real time 3d modeling Intelligent system for IKM

The Challenge

IKM approached us with an idea of an idea to make something that would be able to map humans and their environment captured from a camera to a virtual 3D representation.

We got excited, and thus began this journey of innovation, mad development, and norm breaking intelligent models and we were able to create HUMANS 3D

Solution

We created a python desktop application with a super intuitive and interactive UI. The architecture was designed into a fully scalable solution which could be extended to include data from other sensors like LIDAR, LASER Cameras, etc.

The outdoor model

We first analyzed camera feeds with our computer vision models to segment people, objects and to get a dense mapping of human poses. Then this camera feed was mapped to the 3d environment using our calibration app. The final result was people walking inside a Google Earth like 3D environment.
The indoor model

We used depth information from the camera to generate 3d maps of the building. Once a 3d map was obtained, we placed models of people from our deep learning module in the indoor environment.

All the camera feeds we used were gathered in real time and ran in real time too on an nVidia Tesla GPU.

Challenges

Like any other project, there were a few challenges but this project was novel. Presenting the challenge of nothing to learn from any history implementations. Everything we were doing, we were doing out of the churned-out thought processes and rigorous discussion sessions of our experts.

The first challenge presented itself when we converted people to 3D models. The movement was super jittery and not life-like.

The second issue came with the 3D environment. It had to run smoothly in Python and not many open-source 3D models were available that would load correctly in Python. We had to implement custom fixes and converters for those formats.

Then, it became very difficult to achieve real time performance with high polygon 3D graphics and deep learning models running in parallel.