Introduction to Big Data Quiz 2: 2 April 2023, Set QPE3 (January 2023 term)

1. A

   Big data is used to refer to the set of technologies built around Hadoop

2. B

   Big data refers to the cloud-native design principle

3. C

   Big data is about All data, Any Time, Any Method

4. D

   Big data is about volume, velocity, variety

1. A

   Spark and MapReduce

2. B

   Map and Reduce

3. C

   Hadoop and Spark

4. D

   MapReduce and Hadoop

1. A

   A Spark program generates a set of tasks that are grouped into stages, each ofwhich executes concurrently with the driver being responsible for collecting results from eachstage for final output.

2. B

   A Spark action generates a job, with each job being broken down into multiplestages where a stage is a set of tasks that can be executed in parallel and stages themselves areinterdependent, with each task being the unit of work mapped to an executor.

3. C

   A Spark program generates a job, with each job being broken down intomultiple stages where a stage is a set of tasks that can be executed in parallel and stagesthemselves are interdependent, with each task being the unit of work mapped to an executor.

4. D

   A Spark action generates a task that is further broken down into multiplestages where a stage is a set of jobs that can be executed in parallel and stages themselves areinterdependent, with each job being the unit of work mapped to an executor.

5. E

   A Spark program generates a task that is further broken down into multiplestages where a stage is a set of jobs that can be executed in parallel and stages themselves areinterdependent, with each job being the unit of work mapped to an executor.

1. A

   Use quicksort on each machine and then merge each machine's sorted data onthe master

2. B

   Exploit duality of hashing vs sorting and write a hash function that assignseach line in the file to a distinct machine, and then sort within each machine

3. C

   Write a general-purpose MapReduce / Spark program that picks the Nthhighest descending value from a set, invoke it iteratively from 1 to S (S=Number of lines in file),and create a new file of sorted data

4. D

   Mergesort on each machine by limiting use of memory, and then between allpossible pairs of machines

1. A

   Write a Spark program that forms a Dataframe as grouping by IP address withcount as aggregate, followed by another stage that computes the avg on top of the Dataframe ofthe first stage

2. B

   Write a MapReduce program where the Map does nothing useful, Combinecomputes the aggregated hits per customer, Shuffle combined data based on IP address acrossworkers, and in the Reduce, build hash table on each machine with hash key = IP address andhash value = counter, followed by another Reduce that finally computes the avg on top of all hashvalues.

3. C

   Write a Spark program that forms a Dataframe as grouping by IP address withcount as aggregate, followed by a take into a list in the Spark driver which further computes theaverage of all the individual counts in the list

4. D

   Write a MapReduce program where the Map does nothing useful, Shuffle databased on IP address across workers, and in the Reduce, build hash table on each machine withhash key = IP address and hash value = counter, followed by another Reduce that finally computesavg on top of all hash values.

1. A

   Build the data pipeline using Spark, data storage on HDFS, and deploy both onHadoop using IaaS

2. B

   Build the data pipeline using Python run on Serverless where the data is storedon cloud storage

3. C

   Build the data pipeline using Spark PaaS option on top of data stored on cloudstorage

4. D

   Build the data pipeline using MapReduce, data storage on HDFS, and deployboth on Hadoop using IaaS

1. A

   This application has adopted cloud-native design

2. B

   This application cannot be called as cloud-native since it is not observable at alltimes

3. C

   This application cannot be called as cloud-native since it is not manageableeasily

4. D

   This application cannot be called as cloud-native since it is not resilient

1. A

   Hive, because it executes SQL queries without users having to worry aboutmemory and disk, much like Linux

2. B

   Spark, because it executes Python jobs without users having to worry aboutmemory and disk, much like Linux

3. C

   Zookeeper, because it ensures the consistency of state for all machines in thecluster

4. D

   YARN, because it manages available resources of the cluster

1. A

   Set of lines one for each ID

2. B

   Set of words one for each occurrence against each ID

3. C

   Set of words one for each unique occurrence across IDs

4. D

   Set of lists where each line becomes a list of words

1. A

   Bucket name of the added file

2. B

   Name of the added file

3. C

   Time of creation of the file

4. D

   Event type of the triggered event

1. A

   All odd numbers are computed

2. B

   The program throws an error

3. C

   Nothing is computed

1. A

   Since data.bin is compliant with the RFC 4180, use PySpark’s read_csv() to readthe data as is.

2. B

   Rename the file data.bin to data.csv to make it compliant with RFC 4180 andthen use PySpark’s read_csv() to read the data

3. C

   The problem cannot be solved since the file cannot be converted to a validformat for reading consistently without additional information

4. D

   Write PySpark code to read all lines in data.bin, use string split on “,” asdelimiter, and then collect all column names and corresponding values into a RDD for furtherprocessing

1. A

   i & 1

2. B

   i & 1 followed by 2

3. C

   i & 1 followed by 2 & 3

4. D

   ii & 2

5. E

   ii & 2 & 3

6. F

   ii & 3

1. A

   1

2. B

   5

3. C

   3

4. D

   4

Select all that apply.

1. A

   MapReduce forces barrier synchronization after every step, while Spark'sexecution is gated on Actions

2. B

   MapReduce leverages memory heavily, while Spark optimizes for disk-basedcomputations

3. C

   MapReduce leverages disk heavily, while Spark optimizes for memory-basedcomputations

4. D

   MapReduce executes linearly without parallelization, while Spark executes inparallel using Directed Acyclic Graphs

5. E

   MapReduce enables massively parallel computation, while Spark's driverprogram sequentially executes each worker

Select all that apply.

1. A

   No, because each use of the application by a prospective passenger for ticketsearch, date choosing, ticket seat blocking and ticket purchase is together a high complexityoperation

2. B

   No, because each use of the application by a prospective passenger for ticketsearch, date choosing, ticket seat blocking and ticket purchase will take a long time

3. C

   No, because a failed step in the use of the application can result in duplicatepayments

4. D

   No, because the number of ticket searches cannot be scaled by simply runningthe application in multiple machines

5. E

   No, because multiple prospective passengers have to use the application onthe single machine (i.e. no embarrassingly parallel computation)

Select all that apply.

1. A

   Use broadcast variables

2. B

   Hardcode threshold value as a filter condition in the Driver program

3. C

   Filter for transactions in the Workers before bringing to the Driver

4. D

   Hardcode threshold value as a filter condition in the Workers itself

Select all that apply.

1. A

   A MySQL table where all Creates are new rows and all Updates are in-place

2. B

   A Linux machine’s system.log file into which all processes append their events

3. C

   Share transactions stored as facts & share prices stored as SCD Type IIdimension tables in a PostgreSQL database

4. D

   A weather API that provides the temperature & rainfall readings for all itsweather stations in India at the point of querying

5. E

   Google drive account that stores all historical census data for India

Select all that apply.

1. A

   Snapshots of source systems can be created using an event capture tool likeCDC and then replaying the events in sequence for that time period.

2. B

   A data lake is a collection of data to be provided as input for data sciencealgorithms.

3. C

   Zookeeper is a library for ensuring all services used in big data are monitored,administered and cleaned at appropriate intervals.

4. D

   A single Spark cluster can have multiple “leader” master nodes.

5. E

   Spark is optimized for in-memory computation.

6. F

   Given the RDD underlying a Dataframe, you can recreate the same Dataframeprovided you know the schema

Select all that apply.

1. A

   Google Cloud functions work on stateless paradigm - one function invocationdoesn’t depend on the previous function invocations..

2. B

   Google cloud functions can be triggered by deletion of an object in GCS.

3. C

   Google Cloud Functions can automatically scale-up compute in case of highload.

4. D

   A single execution of a google cloud function is timed indefinitely till theoperation finishes.