We needed to fetch real-time and updated data from a wide variety of sources and latency in getting live data and storing it into the database was a problem we could not tolerate. For run-time parallel processing and data fetching, we introduced a Redis server to act as a liaison between the two endpoints. It solved the speed problem like a charm.
The second head-scratcher presented itself in the form of multiple outliers. We noticed the variance in data distribution was not properly skewed. The inconsistency of labeled data among three classifiers cheap, super cheap, and hyper cheap, became a challenge because we needed a balanced and evenly distributed set of data. So, in order to generate unbiased and inaccurate insights about car prices, we came up with our own algorithm to deal with inconsistent records.
One major roadblock was to keep pace with the current changing trends. We had to maintain our pace with a rapidly evolving market that was struck by a global pandemic. External factors play a major role in evolving the model and enable us to change the old statistics and update the model with new data sets. We managed to tackle the segregation of the data sets for pre and post covid times. Model predictions were quite different after we maintained and updated it with new information.