Question 18
You are provided with a Spark program that picks out a list of suspicious transactions. Its logic is based on both the financial value of the transaction and the geographic location of the transaction. If the financial value is higher than a threshold and the geographic location is from a set of suspected locations (provided as a 1MB file), then the program deems the transaction as suspicious. The version of the Spark program given to you is written in such a way that it pulls all the transactions from the Workers to the Driver and then applies the logic. The fraud control team that runs this program is having to constantly bother you with new threshold values since they are not able to change the Spark code themselves due to lack of technical knowledge. Which 2 changes from the list below will get you the most benefit in performance while also meeting the needs of your stakeholders?
Use broadcast variables for the 1MB file
Hardcode threshold value as a filter condition in the Driver program so thatthe stakeholders can directly edit the Driver program without having to understand Spark
Reorder operations on the Driver such that geographic location is checkedfirst before filtering high value transactions
Hardcode threshold value as a filter condition in the Workers itself
Accept the threshold value as a parameter and filter using the threshold in theWorkers itself.