Question 4
Consider the problem of sorting a 1 petabyte file of numbers stored in a Hadoop cluster of 10 machines where the size of each HDFS block is 100MB. Which of the below methods is the most likely to finish soonest in practice?
Use quicksort on each machine and then merge each machine's sorted data onthe master
Exploit duality of hashing vs sorting and write a hash function that assignseach line in the file to a distinct machine, and then sort within each machine
Write a general-purpose MapReduce / Spark program that picks the Nthhighest descending value from a set, invoke it iteratively from 1 to S (S=Number of lines in file),and create a new file of sorted data
Mergesort on each machine by limiting use of memory, and then between allpossible pairs of machines