Introduction to Big Data End Term: 22 December 2024, Set QDB4 (September 2024 term)

1. A

   The data being processed when the failure happened will have produced partial results that result in incorrect outputs.

2. B

   The pipeline will not be restarted automatically.

3. C

   The pipeline will be restarted automatically by Spark which is able to pick up the exact data from Kafka which was being processed at the time of error

4. D

   Spark will throw an error and halt

5. E

   Spark will gracefully shut down, reboot that machine which failed and start the Structured Streaming pipeline again to continue from where it left off

1. A

   No change to business time since that remains constant at 1AM local time, only the schedule for the final weekly sum operation needs to be changed to the operational time of Monday 3AM India time instead of Sunday 3AM Middle East time.

2. B

   Since time zone has changed as well as week definition too, the definition of business time has changed. So, the ETL has to be rewritten entirely.

3. C

   Nothing needs to change since daily sales is available at 1AM Middle East time which is anyway behind India time and so the numbers will be available before leadership comes in at 9AM.

4. D

   Event time has changed since the event of week ending has changed in definition, and so the ETL needs to be changed to consider the new event in the data.

5. E

   No change required.

1. A

   They check the syntax of the model function so that there are no errors

2. B

   They load the model into memory so that the loop iteration over each image does not need to do the same operation again and again given model load times are prohibitively expensive for DL models

3. C

   They invoke PyTorch libraries that have already been setup for model scoring on GCP using APIs embedded within the function

4. D

   They are Map-style UDFs that make it an embarrassingly parallel computation thus making the execution parallelized and fast.

1. A

   Ingest all data into Pub/Sub, process using Google Cloud Dataflow, invoke the ML model, and then write back output from Cloud to the spider cam to relay to the bat.

2. B

   Compress the ML model to fit into the spider cam’s available resource, and write pipelines to execute the model in the spider cam itself

3. C

   Compress the data in the spider cam every 5 seconds, write to Pub/Sub the compressed data, invoke the ML model and then write back output from Cloud to the spider cam to relay to the bat.

4. D

   Compress the ML model to fit into the spider cam’s available resource, and write pipelines to execute in the spider cam itself, with periodically data being sent to the cloud, retrain the model using Google Cloud ML and then redeploy the model to the spider cam.

1. A

   A UDF is a scalar operation executing on 1 row at a time and producing output immediately, and therefore there is no intermediate output necessary. Whereas, a UDAF operates on multiple rows which will require multiple passes for the final output thereby requiring a buffer.

2. B

   A UDF is a scalar operation executing on many rows at a time, grouped by a key and producing a single output, and therefore there is no intermediate output necessary. Whereas, a UDAF operates on multiple rows which will require multiple passes for the final output thereby requiring a buffer.

3. C

   A UDF is a scalar operation executing on 1 row at a time and producing output immediately, and therefore there is no intermediate output necessary. Whereas, a UDAF operates on multiple rows of unbounded size requiring a divide-and-conquer approach for computing aggregates, which uses the intermediate buffer to store partial values before finalizing the result aggregate.

4. D

   A UDF is a scalar operation executing on many rows at a time, grouped by a key and producing a single output, and therefore there is no intermediate output necessary. Whereas, a UDAF operates on multiple rows of unbounded size requiring a divide-and-conquer approach for computing aggregates, which uses the intermediate buffer to store partial values before finalizing the result aggregate.

1. A

   Since MongoDB is OLTP, it is not able to support business reporting. So, replace it with Hadoop which supports OLAP better

2. B

   Extract raw data from MongoDB using Change Data Capture (CDC) once every minute into Kafka, and then use Spark Streaming to compute the KPIs and then populate into a NoSQL DB like Redis for the UI to consume.

3. C

   Convert the application from using plain Java to using Spark Streaming in Java

4. D

   Query MongoDB every 1 minute for new data using a check on document inserted timestamp, use Spark Streaming to compute the KPIs with the queried data, and then populate into a NoSQL DB like Redis for the UI to consume.

1. A

   Use a different DL model that is faster but has half the recall rate.

2. B

   Build a custom model that compresses the highest recall rate model just enough to be able to execute within the stipulated time, and measure recall.

3. C

   Parallelize the DL model code using Divide-and-Conquer so that it runs faster.

4. D

   Change Spark machine to use GPUs

1. A

2. B

3. C

4. D

1. A

   Route a copy of the ticket purchase to a Kafka topic, use Spark Structured Streaming to continuously read from this topic and update the aggregates by destinations, and emit using output mode “Update”.

2. B

   Route a copy of the ticket purchase to a Kafka topic, use Spark Structured Streaming to periodically read from this topic every 1 minute and update the aggregates by destinations, and emit top aggregates using the output mode “Complete”.

3. C

   Route a copy of the ticket purchase to a Kafka topic, use Spark Structured Streaming to periodically read from this topic every 1 minute and count the destinations in that batch, and emit only top aggregates in that batch using the output mode “Append”.

1. A

   Yes, the failure is visible. The only visible effect for the external world would be in the form of a slowdown in runtime for completion of that batch as Structured Streaming retries the batch that failed thus taking twice as much time as normal.

2. B

   Yes, the failure is visible because the side effect of emitting timestamps in a batch will be visible as 2 consecutive Start timestamps without any end timestamp as Structured Streaming retries the failed batch.

3. C

   No, no failure is visible since Structured Streaming uses transactions and idempotence to achieve exactly-once processing.

4. D

   No, no failure is visible since Structured Streaming can process the same data in a retry resulting in the same outputs again.

1. A

   Only (i)

2. B

   Only (ii)

3. C

   Both (i) and (ii)

4. D

   Neither (i) nor (ii)

1. A

   The read and write commands will remain the same, but the remaining code will need to be modified, as operations on streaming data frames are not supported on static data frames.

2. B

   Only the read and write commands need to be modified to specify that it's a batch operation.

3. C

   The entire code will need to be modified as the APIs for stream and batch processing are completely different.

1. A

   (i), (v), (iii), (iv), (ii)

2. B

   (i), (ii), (iii), (iv), (v)

3. C

   (ii), (iii), (i), (iv), (v)

4. D

   (v), (iii), (i), (iv), (ii)

5. E

   All are equally PaaS / IaaS

Select all that apply.

1. A

   Increase memory in each of the brokers in the cluster

2. B

   Add disks to each broker in the cluster

3. C

   Add new brokers to the cluster

4. D

   Create more topics and change input application to reroute data to all topics to be able to spread input data better

5. E

   Double the number of partitions for this single topic to be able to spread input data better

Select all that apply.

1. A

   Create all tables as native Spark SQL tables (i.e. available as CatalogTables).

2. B

   Partition all tables on the same key on which the aggregate is happening.

3. C

   Ensure cost based optimizer (CBO) is ON.

4. D

   Run ANALYZE on all tables.

5. E

   Cache the table in a step with actions ahead of the SQL statement that is the culprit.

Select all that apply.

1. A

   Batch operates on a set of data elements taken together while streaming operates on data individually as it streams in

2. B

   Batch operates on data that is static while streaming operates on data that is dynamically changing

3. C

   Batch processing can assume data as fully specified and complete while streaming cannot make that assumption

4. D

   Batch processing is typically high latency while streaming processing is necessary for real-time latencies

5. E

   Batch processing can operate on massively larger data sets than streaming can

Select all that apply.

1. A

   Data needs to be retrievable from a persistent store that supports message replay. Replay refers to being able to fetch a specific set of data from that persistent store on- demand., where the data is chosen based on filters usually defined on sequence numbers or timestamps

2. B

   Data processing needs to be splittable across machines using divide-and- conquer, and composable into steps that execute very quickly

3. C

   Hadoop needs to be setup for its big data capabilities

4. D

   The application needs to have the cloud-native properties of being resilient, manageable and observable

5. E

   Application needs to be deployable on public cloud natively using PaaS components

Select all that apply.

1. A

   In pull delivery, your subscriber application initiates requests to the Pub/Sub server to retrieve messages.

2. B

   In push delivery, Pub/Sub initiates requests to your subscriber application to deliver messages.

3. C

   Pull subscription is preferred when efficiency and throughput of message processing is critical.

4. D

   The rate of delivery needs to be controlled by the subscriber client in push subscription.

Select all that apply.

1. A

   Each subscriber gets one third of all messages published into Topic 1.

2. B

   Each subscriber gets one third of all messages published into Topic 1 by Publisher 1

3. C

   Each subscriber gets all messages published into Topic 1.

1clicks = (     # schema - adId: String, clickTime: Timestamp, ...
2  spark
3    .readStream
4    .format("kafka")
5    .option("subscribe", "clicks")
6    …
7    .load()
8)
9clicks
10  .join(pages, “pageId”)
11  .groupBy(“pageName”)
12  .count()

Select all that apply.

1. A

2. B

3. C

4. D

Select all that apply.

1. A

   Change the code to assume hourly data instead of daily data, change scheduler to run every hour & let her manage the execution of the code on Dataproc

2. B

   Change the code to leverage Spark Streaming with streaming window as “1 hour”, & let her manage the execution of the code on Dataproc

3. C

   Change the code to leverage Spark Streaming with streaming window as “1 hour”, convert from Dataproc to Dataflow, & let her manage the execution of the code on Dataflow

4. D

   Write a Cloud Function to move all incoming per-minute standalone files from GCS to Pub/Sub, change the code to leverage Spark Streaming with streaming window as “1 hour”, convert from Dataproc to Dataflow, point source to Pub/Sub, & let her manage the execution of the code on Dataflow