Question 9
You are appointed as a Data Engineer in a company that has a legacy reporting application written in Java which suffers from performance problems. The reporting application plots dashboards with daily refresh of key business indicators to help management take the best decisions. The application reads data directly from the source database of MongoDB, aggregates using simple counts and shows them visually in a UI. The performance problem of this application comes because the source database is at times overloaded and therefore the dashboard is not able to retrieve answers fast enough making the end user wait for the result. Choose the best option that gives the best performance with minimal maintenance effort:
Since MongoDB is OLTP, it is not able to support business reporting. So, bringthe data into Hadoop end of day, and run OLAP queries on it from the same UI.
Convert the application from using plain Java to using Spark Streaming in Java
Extract raw data from MongoDB using Change Data Capture (CDC) once everyminute into Kafka, and then use Spark Streaming to compute the KPIs and then populate into a NoSQL DB like Redis for the UI to consume.
Query MongoDB every 1 minute for new data using a check on documentinserted timestamp, use Spark Streaming to compute the KPIs with the queried data, and then populate into a NoSQL DB like Redis for the UI to consume.
Convert application to using Python along with a NoSQL database for storingand retrieving the aggregated counts.