Question 2
A website that started 3 years ago now sees 1 Billion hits every month. The website owner wants to count average hits per customer across all months, where a customer is denoted by the IP address of the device from which the customer is accessing the website. The owner has at his disposal a minimal Hadoop cluster of 2 workers and 1 master each with 1GB of RAM. He would like to run this job every month going forward. Which of the following methods is the most likely to finish every time it is run in the months and years ahead yielding the right result?
Write a MapReduce program where the Map does nothing useful, Combinecomputes the aggregated hits per customer, Shuffle combines data based on IP address across workers, and the Reduce builds a hash table on each machine with hash key = IP address and hash value = running total and sum, with a final Map that emits the avg per customer.
Write a Spark program that forms a Dataframe as grouping by IP address withcount as aggregate, followed by a take into a list in the Spark driver which further computes the average of all the individual counts in the list
Write a Spark program that forms a Dataframe as grouping by IP address withcount as aggregate, followed by another stage that computes the avg on top of the Dataframe of the first stage
Write a MapReduce program where the Map does nothing useful, Combinecomputes the aggregated hits per customer, Shuffle combines data based on IP address across workers, and the Reduce sorts the data on sort key = IP address and then calculates the final avg per customer.
All will finish every time it is run without issues.