Introduction to Big Data End Term: 28 April 2024 (January 2024 term)

1. A

   Big data is used to refer to the set of technologies built around Hadoop

2. B

   Big data refers to the cloud-native design principle

3. C

   Big data is about All data, Any Time, Any Method

4. D

   Big data is about volume, velocity, variety

1. A

   Spark and Map

2. B

   Serverless and Message Broker

3. C

   Hadoop and Spark

4. D

   MapReduce and Google Cloud Functions

1. A

   Write a MapReduce program where the Map does nothing useful, Combine computes the aggregated hits per customer, Shuffle combined data based on IP address across workers, and in the Reduce, build hash table on each machine with hash key = IP address and hash value = counter, followed by another Reduce that finally computes the avg on top of all hash values.

2. B

   Write a Spark program that forms a Dataframe as grouping by IP address with count as aggregate, followed by a take into a list in the Spark driver which further computes the average of all the individual counts in the list

3. C

   Write a Spark program that forms a Dataframe as grouping by IP address with count as aggregate, followed by another stage that computes the avg on top of the Dataframe of the first stage

4. D

   Write a MapReduce program where 2 pairs of Map Reduce are chained together: 1^(st) pair is where the Map does nothing useful, Shuffle data based on IP address across workers, and in the Reduce, build hash table on each machine with hash key = IP address and hash value = counter, while the 2^(nd) pair is another Map that does nothing useful followed by a Reduce that finally computes avg on top of all hash values.

5. E

   All will finish in approximately the same time.

1. A

   Build the data pipeline using VMs – one for Python & one for storing files

2. B

   Build the data pipeline using Python running on Google Cloud Functions where the data is stored on GCS

3. C

   Build the data pipeline using MapReduce on, data storage on HDFS, and deploy both on Dataproc

4. D

   Build the data pipeline using Dataflow on top of data stored on GCS

1. A

   This application has adopted cloud-native design

2. B

   This application cannot be called as cloud-native since it is not observable at all times

3. C

   This application cannot be called as cloud-native since it is not manageable easily

4. D

   This application cannot be called as cloud-native since it is not resilient

1. A

   Since data.bin is compliant with the RFC 4180, use PySpark’s read_csv() to read the data as is.

2. B

   Rename the file data.bin to data.csv to make it compliant with RFC 4180 and then use PySpark’s read_csv() to read the data

3. C

   The problem cannot be solved since the file cannot be converted to a valid format for reading consistently without additional information

4. D

   Write PySpark code to read all lines in data.bin, use string split on “,” as delimiter, and then collect all column names and corresponding values into a RDD for further processing

1. A

   i and 1

2. B

   i and 1 followed by 2

3. C

   i and 1 followed by 2 & 3

4. D

   ii and both 2 & 3

5. E

   ii and 2

6. F

   ii and 3

1. A

   Failure of a machine in the Kafka cluster will result in an Exception in the Spark pipeline which will then fail and halt.

2. B

   The Spark pipeline will not be able to start again from previously committed offset by restarting itself, resulting in at least-once processing semantics

3. C

   Irrespective of whatever machine fails, Spark will throw an error and halt.

4. D

   Data that is being processed will not be processed again, resulting in atmost- once semantics.

5. E

   The pipeline will be restarted automatically by Spark which is able to pick up the exact data from Kafka which was being processed at the time of error, resulting in exactly- once semantics.

1. A

   Add disks to each broker in the cluster, and disks are the cheapest computer component

2. B

   Increase memory in each of the brokers in the cluster. While cost of memory is more than cost of disks, it is still cheaper than adding brokers and helps to scale.

3. C

   Add new brokers to the cluster, even though this is more expensive than the other options this is the only foolproof way to scale.

4. D

   Create more topics and change input application to reroute data to all topics to be able to spread input data better. This is nearly the least expensive since only developer effort is required to change application.

5. E

   Double the number of partitions for this single topic to be able to spread input data better. This is the least expensive since only administrator effort is required without changing application.

1. A

   The business time for the final weekly sum operation needs to be changed to that of Monday 3AM India time instead of Sunday 3AM Middle East time.

2. B

   There is zero change needed since neither event time nor business time is changing whereas only the operational time is changing.

3. C

   Since time zone has changed as well as week definition too, the definition of business time has changed. So, the ETL has to be rewritten entirely.

4. D

   Nothing needs to change since daily sales is available at 1AM Middle East time which is anyway behind India time and so the numbers will be available before leadership comes in at 9AM.

5. E

   Event time has changed since the event of week ending has changed in definition, and so the ETL needs to be changed to consider the new event in the data.

1. A

   Since MongoDB is OLTP, it is not able to support business reporting. So, replace it with Hadoop which supports OLAP better

2. B

   Convert the application from using plain Java to using Spark Streaming in Java

3. C

   Extract raw data from MongoDB using Change Data Capture (CDC) once every minute into Kafka, and then use Spark Streaming to compute the KPIs and then populate into a NoSQL DB like Redis for the UI to consume.

4. D

   Query MongoDB every 1 minute for new data using a check on document inserted timestamp, use Spark Streaming to compute the KPIs with the queried data, and then populate into a NoSQL DB like Redis for the UI to consume.

5. E

   Convert application to using Python along with a NoSQL database for storing and retrieving the aggregated counts.

1. A

   Ingest all data into Pub/Sub, process using Google Cloud Dataflow, invoke the ML model, and then write back output from Cloud to the spider cam to relay to the bat.

2. B

   Compress the ML model to fit into the spider cam’s available resource, and write pipelines to execute the model in the spider cam itself

3. C

   Compress the ML model to fit into the spider cam’s available resource, and write pipelines to execute in the spider cam itself, with periodically data being sent to the cloud, retrain the model using Google Cloud ML and then redeploy the model to the spider cam.

4. D

   Compress the data in the spider cam every 5 seconds, write to Pub/Sub the compressed data, invoke the ML model and then write back output from Cloud to the spider cam to relay to the bat.

1. A

   A UDF is a scalar operation executing on 1 row at a time and producing output immediately, and therefore there is no intermediate output necessary. Whereas, a UDAF operates on multiple rows which will require multiple passes for the final output thereby requiring a buffer.

2. B

   A UDF is a scalar operation executing on 1 row at a time and producing output immediately, and therefore there is no intermediate output necessary. Whereas, a UDAF operates on multiple rows of unbounded size requiring a divide-and-conquer approach for computing aggregates, which uses the intermediate buffer to store partial values before finalizing the result aggregate.

3. C

   A UDF is a scalar operation executing on many rows at a time, grouped by a key and producing a single output, and therefore there is no intermediate output necessary. Whereas, a UDAF operates on multiple rows which will require multiple passes for the final output thereby requiring a buffer.

4. D

   A UDF is a scalar operation executing on many rows at a time, grouped by a key and producing a single output, and therefore there is no intermediate output necessary. Whereas, a UDAF operates on multiple rows of unbounded size requiring a divide-and-conquer approach for computing aggregates, which uses the intermediate buffer to store partial values before finalizing the result aggregate.

1. A

   Yes, since they check the syntax of the model function so that there are no errors

2. B

   Yes, they invoke PyTorch libraries that have already been setup for model scoring on GCP using APIs embedded within the function

3. C

   No, since the primary function of these lines of code is to eliminate repeated DL model loads as DL models are large in size

4. D

   Yes, they are Map-style UDFs that make it an embarrassingly parallel computation thus making the execution parallelized and fast.

5. E

   Yes, since the new model will potentially have large load times.

1. A

   Build a custom model that compresses the highest recall rate model just enough to be able to execute within the stipulated time, and measure recall.

2. B

   Use a different DL model from PyTorch that is already compressed to half the size.

3. C

   Remove complexity associated with Spark Streaming and convert the model execution pipeline into a single threaded Python application running on the same GPU machine.

4. D

   Change Spark machine to use CPUs and train a fresh pipeline to achieve objectives.

1. A

   No, the failure is not visible. The only visible effect for the external world would be in the form of a slowdown in runtime for completion of that mini-batch as Structured Streaming retries the mini-batch that failed thus taking twice as much time as normal.

2. B

   No, the failure is not visible since Structured Streaming uses transactions and idempotence to achieve exactly-once processing.

3. C

   No, the failure is not visible since Structured Streaming can process the same data in a retry resulting in the same outputs again.

4. D

   Yes, the failure is visible because the side effect of emitting timestamps in a batch will be visible as 2 consecutive Start timestamps without any end timestamp as Structured Streaming retries the failed batch.

5. E

   Yes, the failure is visible since the logs in the backend of Spark Structured Streaming are also logging the state of the machine.

1. A

   The read and write commands will remain the same, but the remaining code will need to be modified, as operations on streaming dataframes are not supported on static dataframes.

2. B

   The entire code will need to be modified as the APIs for stream and batch processing are completely different.

3. C

   The read and write commands need to be modified to specify that it’s a batch operation. Further, the specific logic of window functions will also need to be modified since there are no time windows anymore in batch processing.

4. D

   Only the read and write commands need to be modified to specify that it's a batch operation.

1. A

   Only (ii)

2. B

   Only (i)

3. C

   Both (i) and (ii)

4. D

   Neither (i) nor (ii)

1. A

   (i), (v), (iii), (iv), (ii)

2. B

   (i), (ii), (iii), (iv), (v)

3. C

   (ii), (iii), (i), (iv), (v)

4. D

   (v), (iii), (i), (iv), (ii)

5. E

   All are equally PaaS / IaaS

1clicks = (     # schema - adId: String, clickTime: Timestamp, ...
2  spark
3    .readStream
4    .format("kafka")
5    .option("subscribe", "clicks")
6    …
7    .load()
8)
9clicks
10  .join(pages, “pageId”)
11  .groupBy(“pageName”)
12  .count()

1. A

2. B

3. C

4. D

1. A

   Change the code to run every 5 minutes, no other change required

2. B

   Change the code to leverage Spark Streaming with streaming window as “5 minutes”, & let her manage the execution of the code on Dataproc

3. C

   Change the code to leverage Spark Streaming with streaming window as “5 minutes”, convert from Dataproc to Dataflow, & let her manage the execution of the code on Dataflow

4. D

   Write a Cloud Function to move all incoming per-minute standalone files from GCS to Pub/Sub, change the code to leverage Spark Streaming with streaming window as “5 mins”, convert from Dataproc to Dataflow, point source to Pub/Sub, & let her manage the execution of the code on Dataflow

1. A

   Route a copy of the ticket purchase to a Kafka topic, use Spark Structured Streaming to continuously read from this topic and update the aggregates by destinations, and emit using output mode “Update”.

2. B

   Route a copy of the ticket purchase to a Kafka topic, use Spark Structured Streaming to periodically read from this topic every 1 minute and update the aggregates by destinations, and emit all aggregates using the output mode “Complete”.

3. C

   Route a copy of the ticket purchase to a Kafka topic, use Spark Structured Streaming to periodically read from this topic every 1 minute and count the destinations in that batch, and emit only all aggregates in that batch using the output mode “Append”.

Select all that apply.

1. A

   MapReduce leverages memory heavily, while Spark optimizes for disk-based computations

2. B

   MapReduce forces barrier synchronization after every step, while Spark uses directed acyclic graphs to execute as many steps as possible in parallel

3. C

   MapReduce leverages disk heavily, while Spark optimizes for memory-based computations

4. D

   MapReduce enables massively parallel computation, while Spark's driver program sequentially executes each worker

5. E

   MapReduce is restricted in flexibility since only Map and Reduce are possible steps, while Spark has a variety of Actions possible making it highly flexible

Select all that apply.

1. A

   Use broadcast variables for the 1MB file

2. B

   Hardcode threshold value as a filter condition in the Driver program

3. C

   Reorder operations on the Driver such that geographic location is checked first before filtering high value transactions

4. D

   Hardcode threshold value as a filter condition in the Workers itself

Select all that apply.

1. A

   A MySQL table that is modelled as a Type II SCD where all Creates are new rows and all Updates are appended only, and these can happen anytime during the day.

2. B

   A Linux machine’s system.log file into which all processes append their events

3. C

   Share transactions stored as facts & share prices updated in-place in a PostgreSQL database

4. D

   A weather API that provides the temperature & rainfall readings for all its weather stations in India for the specific time instant when being queried

5. E

   Google drive account that stores all historical census data for India

Select all that apply.

1. A

   Snapshots of source systems can be created using an event capture tool like CDC and then replaying the events in sequence for that time period.

2. B

   A data lake is a collection of data to be provided as input for data science algorithms.

3. C

   Zookeeper is a library for ensuring that critical services used in big data are able to stay in sync with each other as to the health of those services.

4. D

   A single Spark cluster can have multiple “leader” master nodes.

5. E

   Spark is optimized for in-memory computation.

6. F

   Given the RDD underlying a Dataframe, you can recreate the same Dataframe provided you know the schema

Select all that apply.

1. A

   Batch operates on a set of data elements taken together while streaming can also operate on a set of data elements as determined by the window

2. B

   Batch operates on data that is static while streaming operates on data that is dynamically changing

3. C

   Batch processing can assume data as fully specified and complete while streaming cannot make that assumption

4. D

   Batch processing is typically high latency while streaming processing is necessary for real-time latencies

5. E

   Both Streaming & Batch processing can operate on massively large data sets

6. F

   Both Streaming and Batch processing in Spark can use the same syntax for programming the main functional logic of the application

Select all that apply.

1. A

   Use a message store that supports message replay so that no data is lost in processing.

2. B

   “Hot potato” principle is when the streaming application operates on the minimum amount of processing to produce valid output, which then becomes input for the next minimal processor, and so on, thus ensuring that no single process is doing too many things in one go.

3. C

   Hadoop is best suited for executing Streaming applications.

4. D

   Use checkpointing when faced with mission-critical workloads that require 100% accuracy.

5. E

   Handle state pollution by restarting the persistent store software periodically.

Select all that apply.

1. A

   Create all tables as external tables.

2. B

   Ensure cost based optimizer (CBO) is ON.

3. C

   Partition all tables on the same key on which the aggregate is happening.

4. D

   Run ANALYZE on all tables to ensure the right estimates are available to the optimizer.

5. E

   Cache the table in a step with actions ahead of the SQL statement that is the culprit.

Select all that apply.

1. A

   Each subscriber gets one third of all messages published into Topic 1.

2. B

   Publisher 1 can safely send data only for Subscriber 1’s consumption while restricting access to Subscribers 2 & 3.

3. C

   Subscriber 2 gets all messages published into Topic 1 by Publisher 1 but only half of Publisher 2’s messages.

4. D

   Each subscriber gets all messages published into Topic 1.