Question 2
You are given a Spark program that runs on a Google Dataproc cluster on a daily schedule starting execution at 7AM running typically for 2 hours, to produce as output the total amount spent on purchases made by every customer the previous day. The input data is coming into GCS every minute from a variety of sources as standalone files. Therefore, the business leader now feels that having to wait till 9AM the next day is no longer acceptable and instead ideally wants purchase information for each customer at least every 5 minutes during the day itself. What’s more, she wants to be able to change this time configuration later without involving you.
Which amongst the below represents the best option to achieve the above?
Change the schedule to run every 5 minutes, no other change required.
Change the code to leverage Spark Streaming with streaming window as “5minutes”, & let her manage the execution of the code on Google Dataproc
Change the code to leverage Spark Streaming with streaming window as “5minutes”, go from using Dataproc to Dataflow, & let her manage the execution of the code on Dataflow
Write a Cloud Function to move all incoming per-minute standalone files fromGCS to Pub/Sub, change the code to leverage Spark Streaming with streaming window as “5 mins”, convert from Dataproc to Dataflow, point source to Pub/Sub, & let her manage the execution of the code on Dataflow