Question 5
A website sees 1 Billion hits every month. The website owner wants to count average hits per customer in the latest month, where a customer is denoted by the IP address of the device. The owner has at his disposal a Hadoop cluster of 5 workers and 2 masters each with 1GB of RAM. Which of the following methods is the most likely to finish fastest?
Write a Spark program that forms a Dataframe as grouping by IP address withcount as aggregate, followed by another stage that computes the avg on top of the Dataframe ofthe first stage
Write a MapReduce program where the Map does nothing useful, Combinecomputes the aggregated hits per customer, Shuffle combined data based on IP address acrossworkers, and in the Reduce, build hash table on each machine with hash key = IP address andhash value = counter, followed by another Reduce that finally computes the avg on top of all hashvalues.
Write a Spark program that forms a Dataframe as grouping by IP address withcount as aggregate, followed by a take into a list in the Spark driver which further computes theaverage of all the individual counts in the list
Write a MapReduce program where the Map does nothing useful, Shuffle databased on IP address across workers, and in the Reduce, build hash table on each machine withhash key = IP address and hash value = counter, followed by another Reduce that finally computesavg on top of all hash values.