Question 6
You are appointed as a Data Engineer in a company that has a legacy reporting application written in Java which suffers from both performance and maintainability problems. The reporting application plots dashboards with near real-time refresh (once every minute) of key business indicators to help management take live decisions. The application reads data directly from the source database of MongoDB, aggregates using simple counts and shows them visually in a UI. The performance problem of this application comes because the source database is at times overloaded and therefore the dashboard is not able to refresh fast enough. The maintainability issue is because every time a new KPI summary is required, the source has to be changed to include it. Choose from the options below the option that best tackles both problems:
Since MongoDB is OLTP, it is not able to support business reporting. So, replace it with Hadoop which supports OLAP better
Extract raw data from MongoDB using Change Data Capture (CDC) once every minute into Kafka, and then use Spark Streaming to compute the KPIs and then populate into a NoSQL DB like Redis for the UI to consume.
Convert the application from using plain Java to using Spark Streaming in Java
Query MongoDB every 1 minute for new data using a check on document inserted timestamp, use Spark Streaming to compute the KPIs with the queried data, and then populate into a NoSQL DB like Redis for the UI to consume.