Introduction to Big Data Quiz 2: 3 December 2023 (September 2023 term)

1. A

   Big data is used to refer to the set of technologies built around Hadoop

2. B

   Big data refers to the cloud-native design principle

3. C

   Big data is about All data, Any Time, Any Method

4. D

   Big data is about volume, velocity, variety

1. A

   Spark and MapReduce

2. B

   Map and Reduce

3. C

   Hadoop and Spark

4. D

   MapReduce and Hadoop

1. A

   A Spark program generates a set of tasks that are grouped into stages, each of which executes concurrently with the driver being responsible for collecting results from each stage for final output.

2. B

   A Spark action generates a job, with each job being broken down into multiple stages where a stage is a set of tasks that can be executed in parallel and stages themselves are interdependent, with each task being the unit of work mapped to an executor.

3. C

   A Spark program generates a job, with each job being broken down into multiple stages where a stage is a set of tasks that can be executed in parallel and stages themselves are interdependent, with each task being the unit of work mapped to an executor.

4. D

   A Spark action generates a task that is further broken down into multiple stages where a stage is a set of jobs that can be executed in parallel and stages themselves are interdependent, with each job being the unit of work mapped to an executor.

5. E

   A Spark program generates a task that is further broken down into multiple stages where a stage is a set of jobs that can be executed in parallel and stages themselves are interdependent, with each job being the unit of work mapped to an executor.

1. A

   Use quicksort on each machine and then merge each machine's sorted data on the master

2. B

   Exploit duality of hashing vs sorting and write a hash function that assigns each line in the file to a distinct machine, and then sort within each machine

3. C

   Write a general-purpose MapReduce / Spark program that picks the Nth highest descending value from a set, invoke it iteratively from 1 to S (S=Number of lines in file), and create a new file of sorted data

4. D

   Mergesort on each machine by limiting use of memory, and then between all possible pairs of machines

1. A

   Write a Spark program that forms a Dataframe as grouping by IP address with count as aggregate, followed by another stage that computes the avg on top of the Dataframe of the first stage

2. B

   Write a MapReduce program where the Map does nothing useful, Combine computes the aggregated hits per customer, Shuffle combined data based on IP address across workers, and in the Reduce, build hash table on each machine with hash key = IP address and hash value = counter, followed by another Reduce that finally computes the avg on top of all hash values.

3. C

   Write a Spark program that forms a Dataframe as grouping by IP address with count as aggregate, followed by a take into a list in the Spark driver which further computes the average of all the individual counts in the list

4. D

   Write a MapReduce program where the Map does nothing useful, Shuffle data based on IP address across workers, and in the Reduce, build hash table on each machine with hash key = IP address and hash value = counter, followed by another Reduce that finally computes avg on top of all hash values.

1. A

   Build the data pipeline using Spark, data storage on HDFS, and deploy both on Hadoop using IaaS

2. B

   Build the data pipeline using Python run on Serverless where the data is stored on cloud storage

3. C

   Build the data pipeline using Spark PaaS option on top of data stored on cloud storage

4. D

   Build the data pipeline using MapReduce, data storage on HDFS, and deploy both on Hadoop using IaaS

1. A

   This application has adopted cloud-native design

2. B

   This application cannot be called as cloud-native since it is not observable at all times

3. C

   This application cannot be called as cloud-native since it is not manageable easily

4. D

   This application cannot be called as cloud-native since it is not resilient

1. A

   Hive, because it executes SQL queries without users having to worry about memory and disk, much like Linux

2. B

   Spark, because it executes Python jobs without users having to worry about memory and disk, much like Linux

3. C

   Zookeeper, because it ensures the consistency of state for all machines in the cluster

4. D

   YARN, because it manages available resources of the cluster

1. A

   Set of lines one for each ID

2. B

   Set of words one for each occurrence against each ID

3. C

   Set of words one for each unique occurrence across IDs

4. D

   Set of lists where each line becomes a list of words

1. A

   Bucket name of the added file

2. B

   Name of the added file

3. C

   Time of creation of the file

4. D

   Event type of the triggered event

1. A

   All odd numbers are computed

2. B

   The program throws an error

3. C

   Nothing is computed

1. A

   Since data.bin is compliant with the RFC 4180, use PySpark’s read_csv() to read the data as is.

2. B

   Rename the file data.bin to data.csv to make it compliant with RFC 4180 and then use PySpark’s read_csv() to read the data

3. C

   The problem cannot be solved since the file cannot be converted to a valid format for reading consistently without additional information

4. D

   Write PySpark code to read all lines in data.bin, use string split on “,” as delimiter, and then collect all column names and corresponding values into a RDD for further processing

1. A

   i & 1

2. B

   i & 1 followed by 2

3. C

   i & 1 followed by 2 & 3

4. D

   ii & 2

5. E

   ii & 2 & 3

6. F

   ii & 3

1. A

   1

2. B

   5

3. C

   3

4. D

   6

1. A

   [1, 2, 3, 4, 5]

2. B

   15

3. C

   0

4. D

   [0, 1, 2, 3, 4, 5]

1. A

2. B

3. C

4. D

1. A

   To persist the RDD in memory across operations.

2. B

   To delete the RDD from memory.

3. C

   To free up memory used by the RDD.

4. D

   To copy the RDD to a different cluster node.

5. E

   None of these

1. A

   It ensures data is immediately cleaned and transformed upon ingestion.

2. B

   It provides information about the structure, context, and validity of data.

3. C

   It dictates the storage format of all incoming data.

4. D

   It restricts data access to authorized users only.

5. E

   It defines the access control policies for the entire Data Lake

Select all that apply.

1. A

   MapReduce enforces synchronization barriers following each stage, whereas Spark's progress hinges on the occurrence of specific actions.

2. B

   MapReduce leverages memory heavily, while Spark optimizes for disk-based computations

3. C

   MapReduce leverages disk heavily, while Spark optimizes for memory-based computations

4. D

   MapReduce executes linearly without parallelization, while Spark executes in parallel using Directed Acyclic Graphs

5. E

   MapReduce enables massively parallel computation, while Spark's driver program sequentially executes each worker

Select all that apply.

1. A

   No, because each use of the application by a prospective passenger for ticket search, date choosing, ticket seat blocking and ticket purchase is together a high complexity operation

2. B

   No, because each use of the application by a prospective passenger for ticket search, date choosing, ticket seat blocking and ticket purchase will take a long time

3. C

   No, because a failed step in the use of the application can result in duplicate payments

4. D

   No, because the number of ticket searches cannot be scaled by simply running the application in multiple machines

5. E

   No, because multiple prospective passengers have to use the application on a single machine (i. e. No embarrassingly parallel computation)

Select all that apply.

1. A

   Use broadcast variables

2. B

   Hardcode threshold value as a filter condition in the Driver program

3. C

   Filter for transactions in the Workers before bringing to the Driver

4. D

   Hardcode threshold value as a filter condition in the Workers itself

Select all that apply.

1. A

   A MySQL table where all Creates are new rows and all Updates are in-place, and these can happen anytime during the day.

2. B

   A Linux machine’s system.log file into which all processes append their events

3. C

   Share transactions stored as facts & share prices stored as SCD Type II dimension tables in a PostgreSQL database

4. D

   A weather API that provides the temperature & rainfall readings for all its weather stations in India for the specific time instant when being queried

5. E

   Google drive account that stores all historical census data for India

Select all that apply.

1. A

   Snapshots of source systems can be created using an event capture tool like CDC and then replaying the events in sequence for that time period.

2. B

   A data lake is a collection of data to be provided as input for data science algorithms.

3. C

   Zookeeper is a library for ensuring all services used in big data are monitored, administered and cleaned at appropriate intervals.

4. D

   A single Spark cluster can have multiple “leader” master nodes.

5. E

   Spark is optimized for in-memory computation.

6. F

   Given the RDD underlying a Dataframe, you can recreate the same Dataframe provided you know the schema

Select all that apply.

1. A

   Google Cloud functions work on stateless paradigm - one function invocation doesn’t depend on the previous function invocations..

2. B

   Google cloud functions can be triggered by deletion of an object in GCS.

3. C

   Google Cloud Functions can automatically spin up more execution resources in case of high load.

4. D

   A single execution of a google cloud function is timed indefinitely till the operation finishes.

Select all that apply.

1. A

   Spark Driver Program: Coordinates and manages the application, running the main function and defining one or more RDDs.

2. B

   Hadoop Cluster Manager: Allocates resources and manages the distribution of tasks across the cluster. Examples include YARN, Mesos, and Standalone.

3. C

   Executor: Runs tasks on worker nodes and stores the data in memory or disk.

4. D

   Worker Node: Hosts executor processes, which perform data processing tasks.

5. E

   None of these

Select all that apply.

1. A

   YARN provides a centralized service for resource management across a cluster of machines

2. B

   YARN manages distributed computing frameworks whereas Spark is an example of a distributed computing framework

3. C

   Spark can run on YARN, the same way Hadoop Map Reduce can run on YARN

4. D

   YARN is a key feature of Spark