Introduction to Big Data Quiz 2: 6 August 2023 (May 2023 term)

1. A

   Big data is used to refer to the set of technologies built around Hadoop

2. B

   Big data refers to the cloud-native design principle

3. C

   Big data is about All data, Any Time, Any Method

4. D

   Big data is about volume, velocity, variety

1. A

   Spark and MapReduce

2. B

   Map and Reduce

3. C

   Hadoop and Spark

4. D

   MapReduce and Hadoop

Select all that apply.

1. A

   MapReduce forces barrier synchronization after every step, while Spark's execution is gated on Actions

2. B

   MapReduce leverages memory heavily, while Spark optimizes for disk-based computations

3. C

   MapReduce leverages disk heavily, while Spark optimizes for memory-based computations

4. D

   MapReduce executes linearly without parallelization, while Spark executes in parallel using Directed Acyclic Graphs

5. E

   MapReduce enables massively parallel computation, while Spark's driver program sequentially executes each worker

1. A

   A Spark program generates a set of tasks that are grouped into stages, each of which executes concurrently with the driver being responsible for collecting results from each stage for final output.

2. B

   A Spark action generates a job, with each job being broken down into multiple stages where a stage is a set of tasks that can be executed in parallel and stages themselves are interdependent, with each task being the unit of work mapped to an executor.

3. C

   A Spark program generates a job, with each job being broken down into multiple stages where a stage is a set of tasks that can be executed in parallel and stages themselves are interdependent, with each task being the unit of work mapped to an executor.

4. D

   A Spark action generates a task that is further broken down into multiple stages where a stage is a set of jobs that can be executed in parallel and stages themselves are interdependent, with each job being the unit of work mapped to an executor.

5. E

   A Spark program generates a task that is further broken down into multiple stages where a stage is a set of jobs that can be executed in parallel and stages themselves are interdependent, with each job being the unit of work mapped to an executor.

1. A

   Use quicksort on each machine and then merge each machine's sorted data on the master

2. B

   Exploit duality of hashing vs sorting and write a hash function that assigns each line in the file to a distinct machine, and then sort within each machine

3. C

   Write a general-purpose MapReduce / Spark program that picks the Nth highest descending value from a set, invoke it iteratively from 1 to S (S=Number of lines in file), and create a new file of sorted data

4. D

   Mergesort on each machine by limiting use of memory, and then between all possible pairs of machines

1. A

   Write a Spark program that forms a Dataframe as grouping by IP address with count as aggregate, followed by another stage that computes the avg on top of the Dataframe of the first stage

2. B

   Write a MapReduce program where the Map does nothing useful, Combine computes the aggregated hits per customer, Shuffle combined data based on IP address across workers, and in the Reduce, build hash table on each machine with hash key = IP address and hash value = counter, followed by another Reduce that finally computes the avg on top of all hash values.

3. C

   Write a Spark program that forms a Dataframe as grouping by IP address with count as aggregate, followed by a take into a list in the Spark driver which further computes the average of all the individual counts in the list

4. D

   Write a MapReduce program where the Map does nothing useful, Shuffle data based on IP address across workers, and in the Reduce, build hash table on each machine with hash key = IP address and hash value = counter, followed by another Reduce that finally computes avg on top of all hash values.

1. A

   Build the data pipeline using Spark, data storage on HDFS, and deploy both on Hadoop using IaaS

2. B

   Build the data pipeline using Python run on Serverless where the data is stored on cloud storage

3. C

   Build the data pipeline using Spark PaaS option on top of data stored on cloud storage

4. D

   Build the data pipeline using MapReduce, data storage on HDFS, and deploy both on Hadoop using IaaS

1. A

   This application has adopted cloud-native design

2. B

   This application cannot be called as cloud-native since it is not observable at all times

3. C

   This application cannot be called as cloud-native since it is not manageable easily

4. D

   This application cannot be called as cloud-native since it is not resilient

Select all that apply.

1. A

   No, because each use of the application by a prospective passenger for ticket search, date choosing, ticket seat blocking and ticket purchase is together a high complexity operation

2. B

   No, because each use of the application by a prospective passenger for ticket search, date choosing, ticket seat blocking and ticket purchase will take a long time

3. C

   No, because a failed step in the use of the application can result in duplicate payments

4. D

   No, because the number of ticket searches cannot be scaled by simply running the application in multiple machines

5. E

   No, because multiple prospective passengers have to use on a single machine (i.e. no embarrassingly parallel computation)

1. A

   Hive, because it executes SQL queries without users having to worry about memory and disk, much like Linux

2. B

   Spark, because it executes Python jobs without users having to worry about memory and disk, much like Linux

3. C

   Zookeeper, because it ensures the consistency of state for all machines in the cluster

4. D

   YARN, because it manages available resources of the cluster

1. A

   Set of lines one for each ID

2. B

   Set of words one for each occurrence against each ID

3. C

   Set of words one for each unique occurrence across IDs

4. D

   Set of lists where each line becomes a list of words

Select all that apply.

1. A

   Use broadcast variables

2. B

   Hardcode threshold value as a filter condition in the Driver program

3. C

   Filter for transactions in the Workers before bringing to the Driver

4. D

   Hardcode threshold value as a filter condition in the Workers itself

1. A

   Bucket name of the added file

2. B

   Name of the added file

3. C

   Time of creation of the file

4. D

   Event type of the triggered event

1. A

   All odd numbers are computed

2. B

   The program throws an error

3. C

   Nothing is computed

1. A

   Since data.bin is compliant with the RFC 4180, use PySpark’s read_csv() to read the data as is.

2. B

   Rename the file data.bin to data.csv to make it compliant with RFC 4180 and then use PySpark’s read_csv() to read the data

3. C

   The problem cannot be solved since the file cannot be converted to a valid format for reading consistently without additional information

4. D

   Write PySpark code to read all lines in data.bin, use string split on “,” as delimiter, and then collect all column names and corresponding values into a RDD for further processing

1sc = pyspark.SparkContext ()
2train_master = … #python dictionary of trains in India (about 1L trains in total)
3bookings_rdd = … #Spark RDD that is having all ticket bookings of format (train_no, date, seats, ticket_price)
4bookings_rdd.map( train_no => train_master.has_key(train_no) ).count ()

1. A

   i & 1

2. B

   i & 1 followed by 2

3. C

   i & 1 followed by 2 & 3

4. D

   ii & 2

5. E

   ii & 2 & 3

6. F

   ii & 3

1sc = pyspark.SparkContext ()
2words = sc.parallelize (
3   ['arude','rudy','helloworld','big data','rudy is rude','pyspark is not rude','rudeity','rude']
4)
5words_filter = words.filter(lambda x:'rude' in x)
6filtered = words_filter.collect()
7print(len(filtered))

1. A

   1

2. B

   5

3. C

   3

4. D

   4

1sc = pyspark.SparkContext ()
2data = [1, 2, 3, 4, 5]
3total = 0
4rdd = sc.parallelize (data)
5def increment (x):
6        global total
7        total += x
8rdd.foreach (increment)
9print (total)

1. A

2. B

3. C

4. D

1. A

2. B

3. C

4. D

Select all that apply.

1. A

   A MySQL table where all Creates are new rows and all Updates are in-place, and these can happen anytime during the day.

2. B

   A Linux machine’s system.log file into which all processes append their events

3. C

   Share transactions stored as facts & share prices stored as SCD Type II dimension tables in a PostgreSQL database

4. D

   A weather API that provides the temperature & rainfall readings for all its weather stations in India for the specific time instant when being queried

5. E

   Google drive account that stores all historical census data for India

Select all that apply.

1. A

   Snapshots of source systems can be created using an event capture tool like CDC and then replaying the events in sequence for that time period.

2. B

   A data lake is a collection of data to be provided as input for data science algorithms.

3. C

   Zookeeper is a library for ensuring all services used in big data are monitored, administered and cleaned at appropriate intervals.

4. D

   A single Spark cluster can have multiple “leader” master nodes.

5. E

   Spark is optimized for in-memory computation.

6. F

   Given the RDD underlying a Dataframe, you can recreate the same Dataframe provided you know the schema

Select all that apply.

1. A

   Google Cloud functions work on stateless paradigm - one function invocation doesn’t depend on the previous function invocations.

2. B

   Google cloud functions can be triggered by deletion of an object in GCS.

3. C

   Google Cloud Functions can automatically scale-up compute in case of high load.

4. D

   A single execution of a google cloud function is timed indefinitely till the operation finishes.