Intro to Big Data End Term: 1 September 2024, Set QDB1 (May 2024 term)

It is a rule-of-thumb where one program does all the work required to “cook” the data fully for the sake of completeness

It is a rule-of-thumb applicable mainly to batch processing of data where fresh data is prioritized for processing ahead of older data

It is a rule-of-thumb applicable only in the world of stream processing where a program does the minimal processing required to produce a logical output which is then handed off via a message store to the next program that is also designed similarly, and so on, until the bigger problem is solved.

It is a principle for network packet routing, and not really applicable in big data.

Spark and Map

Serverless and Message Broker

Hadoop and Spark Streaming

MapReduce and Google Cloud Functions

Write a MapReduce program where the Map does nothing useful, Combine computes the aggregated hits per customer, Shuffle combined data based on IP address across workers, and in the Reduce, build hash table on each machine with hash key = IP address and hash value = counter, followed by another Reduce that finally computes the avg on top of all hash values.

Write a Spark program that forms a Dataframe as grouping by IP address with count as aggregate, followed by a take into a list in the Spark driver which further computes the average of all the individual counts in the list

Write a Spark program that forms a Dataframe as grouping by IP address with count as aggregate, followed by another stage that computes the avg on top of the Dataframe of the first stage

Write a MapReduce program where 2 pairs of Map Reduce are chained together: 1^(st) pair is where the Map does nothing useful, Shuffle data based on IP address across workers, and in the Reduce, build hash table on each machine with hash key = IP address and hash value = counter, while the 2^(nd) pair is another Map that does nothing useful followed by a Reduce that finally computes avg on top of all hash values.

All will finish in approximately the same time.