Like any other project, there were a few challenges but this project was novel. Presenting the challenge of nothing to learn from any history implementations. Everything we were doing, we were doing out of the churned-out thought processes and rigorous discussion sessions of our experts.
The first challenge presented itself when we converted people to 3D models. The movement was super jittery and not life-like.
The second issue came with the 3D environment. It had to run smoothly in Python and not many open-source 3D models were available that would load correctly in Python. We had to implement custom fixes and converters for those formats.
Then, it became very difficult to achieve real time performance with high polygon 3D graphics and deep learning models running in parallel.