Intro to Big Data End Term: 24 December 2023 (September 2023 term)

1. A

   Failure of a machine in the Kafka cluster will result in an Exception in the Spark pipeline which will then fail and halt.

2. B

   The Spark pipeline will not be able to start again from previously committed offset by restarting itself, resulting in at least-once processing semantics

3. C

   Irrespective of whatever machine fails, Spark will throw an error and halt.

4. D

   The pipeline will be restarted automatically by Spark which is able to pick up the exact data from Kafka which was being processed at the time of error, resulting in exactly- once semantics.

5. E

   Data that is being processed will not be processed again, resulting in atmost- once semantics.

1. A

   Add disks to each broker in the cluster, and disks are the cheapest computer component

2. B

   Add new brokers to the cluster, even though this is more expensive than the other options this is the only foolproof way to scale.

3. C

   Increase memory in each of the brokers in the cluster. While cost of memory is more than cost of disks, it is still cheaper than adding brokers and helps to scale.

4. D

   Create more topics and change input application to reroute data to all topics to be able to spread input data better. This is nearly the least expensive since only developer effort is required to change application.

5. E

   Double the number of partitions for this single topic to be able to spread input data better. This is the least expensive since only administrator effort is required without changing application.

1. A

   The business time for the final weekly sum operation needs to be changed to that of Monday 3AM India time instead of Sunday 3AM Middle East time.

2. B

   Since time zone has changed as well as week definition too, the definition of business time has changed. So, the ETL has to be rewritten entirely.

3. C

   Nothing needs to change since daily sales is available at 1AM Middle East time which is anyway behind India time and so the numbers will be available before leadership comes in at 9AM.

4. D

   Event time has changed since the event of week ending has changed in definition, and so the ETL needs to be changed to consider the new event in the data.

5. E

   No change required since neither event time nor business time is changing whereas only the operational time is changing.

1. A

   Since MongoDB is OLTP, it is not able to support business reporting. So, replace it with Hadoop which supports OLAP better

2. B

   Convert the application from using plain Java to using Spark Streaming in Java

3. C

   Query MongoDB every 1 minute for new data using a check on document inserted timestamp, use Spark Streaming to compute the KPIs with the queried data, and then populate into a NoSQL DB like Redis for the UI to consume.

4. D

   Extract raw data from MongoDB using Change Data Capture (CDC) once every minute into Kafka, and then use Spark Streaming to compute the KPIs and then populate into a NoSQL DB like Redis for the UI to consume.

5. E

   Convert application to using Python along with a NoSQL database for storing and retrieving the aggregated counts.

1. A

   Ingest all data into Pub/Sub, process using Google Cloud Dataflow, invoke the ML model, and then write back output from Cloud to the spider cam to relay to the bat.

2. B

   Compress the ML model to fit into the spider cam’s available resource, and write pipelines to execute the model in the spider cam itself

3. C

   Compress the data in the spider cam every 5 seconds, write to Pub/Sub the compressed data, invoke the ML model and then write back output from Cloud to the spider cam to relay to the bat.

4. D

   Compress the ML model to fit into the spider cam’s available resource, and write pipelines to execute in the spider cam itself, with periodically data being sent to the cloud, retrain the model using Google Cloud ML and then redeploy the model to the spider cam.

1. A

   A UDF is a scalar operation executing on 1 row at a time and producing output immediately, and therefore there is no intermediate output necessary. Whereas, a UDAF operates on multiple rows which will require multiple passes for the final output thereby requiring a buffer.

2. B

   A UDF is a scalar operation executing on many rows at a time, grouped by a key and producing a single output, and therefore there is no intermediate output necessary. Whereas, a UDAF operates on multiple rows which will require multiple passes for the final output thereby requiring a buffer.

3. C

   A UDF is a scalar operation executing on 1 row at a time and producing output immediately, and therefore there is no intermediate output necessary. Whereas, a UDAF operates on multiple rows of unbounded size requiring a divide-and-conquer approach for computing aggregates, which uses the intermediate buffer to store partial values before finalizing the result aggregate.

4. D

   A UDF is a scalar operation executing on many rows at a time, grouped by a key and producing a single output, and therefore there is no intermediate output necessary. Whereas, a UDAF operates on multiple rows of unbounded size requiring a divide-and-conquer approach for computing aggregates, which uses the intermediate buffer to store partial values before finalizing the result aggregate.

1. A

   Yes, since they check the syntax of the model function so that there are no errors

2. B

   They invoke PyTorch libraries that have already been setup for model scoring on GCP using APIs embedded within the function

3. C

   No, since the primary function of these lines of code is to eliminate repeated DL model loads as DL models are large in size but with XGboost this isn’t required since model load time is negligible

4. D

   They are Map-style UDFs that make it an embarrassingly parallel computation thus making the execution parallelized and fast.

1. A

   Use a different DL model from PyTorch that is already compressed to half the size.

2. B

   Build a custom model that compresses the highest recall rate model just enough to be able to execute within the stipulated time, and measure recall.

3. C

   Remove complexity associated with Spark Streaming and convert the model execution pipeline into a single threaded Python application running on the same GPU machine.

4. D

   Change Spark machine to use CPUs and train a fresh pipeline to achieve objectives.

1. A

2. B

3. C

4. D

1. A

   No, the failure is not visible. The only visible effect for the external world would be in the form of a slowdown in runtime for completion of that batch as Structured Streaming retries the batch that failed thus taking twice as much time as normal.

2. B

   No, no failure is visible since Structured Streaming uses transactions and idempotence to achieve exactly-once processing.

3. C

   Yes, the failure is visible because the side effect of emitting timestamps in a batch will be visible as 2 consecutive Start timestamps without any end timestamp as Structured Streaming retries the failed batch.

4. D

   No, no failure is visible since Structured Streaming can process the same data in a retry resulting in the same outputs again.

1. A

   The read and write commands will remain the same, but the remaining code will need to be modified, as operations on streaming dataframes are not supported on static dataframes.

2. B

   Only the read and write commands need to be modified to specify that it's a batch operation.

3. C

   The entire code will need to be modified as the APIs for stream and batch processing are completely different.

4. D

   The read and write commands need to be modified to specify that it’s a batch operation. Further, the specific logic of window functions will also need to be modified since there are no time windows anymore in batch processing.

1. A

   (i), (v), (iii), (iv), (ii)

2. B

   (i), (ii), (iii), (iv), (v)

3. C

   (ii), (iii), (i), (iv), (v)

4. D

   (v), (iii), (i), (iv), (ii)

5. E

   All are equally PaaS / IaaS

1. A

   Route a copy of the ticket purchase to a Kafka topic, use Spark Structured Streaming to continuously read from this topic and update the aggregates by destinations, and emit using output mode “Update”.

2. B

   Route a copy of the ticket purchase to a Kafka topic, use Spark Structured Streaming to periodically read from this topic every 1 minute and update the aggregates by destinations, and emit all aggregates using the output mode “Complete”.

3. C

   Route a copy of the ticket purchase to a Kafka topic, use Spark Structured Streaming to periodically read from this topic every 1 minute and count the destinations in that batch, and emit only all aggregates in that batch using the output mode “Append”.

1. A

   Dynamic scaling adjusts the number of partitions dynamically based on data volume, optimizing resource usage.

2. B

   It helps in controlling the frequency of data persistence in the streaming application.

3. C

   Dynamic scaling is necessary for managing backpressure in the system.

4. D

   It optimizes the execution plan of Spark SQL queries for better performance.

1. A

   The watermark defines the maximum allowed lateness for events.

2. B

   It controls the frequency of checkpointing in the streaming application.

3. C

   The watermark is a timestamp indicating the current processing time.

4. D

   It specifies the size of the time window for aggregations.

1. A

   on_completion

2. B

   on_failure

3. C

   on_acknowledge

4. D

   on_success

1. A

   It specifies the Kafka topic to consume messages from.

2. B

   It uniquely identifies the consumer group to which the consumer belongs.

3. C

   It defines the deserialization format for consumer records.

4. D

   It sets the maximum number of records to be polled in each request.

1. A

   It uses the default partitioning strategy.

2. B

   It forces all records to be sent to partition 0.

3. C

   It dynamically selects the partition based on the key value.

4. D

   It partitions records based on a round-robin distribution.

1. A

   Modify the existing Spark Streaming code to run every 5 minutes and deploy it on the existing cluster.

2. B

   Rewrite the application using Spark Structured Streaming and allow the business stakeholders to configure the processing interval.

3. C

   Develop a Cloud Function to trigger the Spark Streaming job every 5 minutes and deploy it on a cloud platform.

4. D

   Convert the Spark Streaming application to use Apache Flink for better real- time processing capabilities.

1. A

   Each subscriber gets one third of all messages published into Topic 1.

2. B

   Publisher 1 can safely send data only for Subscriber 1’s consumption while restricting access to Subscribers 2 & 3.

3. C

   Subscriber 2 gets all messages published into Topic 1 by Publisher 1 but only half of Publisher 2’s messages.

4. D

   Each subscriber gets all messages published into Topic 1.

1clicks = (     # schema - adId: String, clickTime: Timestamp, ...
2  spark
3    .readStream
4    .format("kafka")
5    .option("subscribe", "clicks")
6    …
7    .load()
8)
9clicks
10  .join(pages, “pageId”)
11  .groupBy(“pageName”)
12  .count()

1. A

2. B

3. C

4. D

1. A

   Change the code to run every 5 minutes, no other change required

2. B

   Change the code to leverage Spark Streaming with streaming window as “5 minutes”, & let her manage the execution of the code on Dataproc

3. C

   Write a Cloud Function to move all incoming per-minute standalone files from GCS to Pub/Sub, change the code to leverage Spark Streaming with streaming window as “5 mins”, convert from Dataproc to Dataflow, point source to Pub/Sub, & let her manage the execution of the code on Dataflow

4. D

   Change the code to leverage Spark Streaming with streaming window as “5 minutes”, convert from Dataproc to Dataflow, & let her manage the execution of the code on Dataflow

Select all that apply.

1. A

   Use a message store that supports message replay so that no data is lost in processing.

2. B

   “Hot potato” principle is when the streaming application operates on data from cache (i.e. the hot area of memory) and therefore is able to produce very high throughput

3. C

   Hadoop is best suited for executing Streaming applications

4. D

   Use checkpointing when faced with mission-critical workloads that require 100% accuracy.

5. E

   Handle state pollution by restarting the persistent store software periodically.