Introduction to Big Data End Term: 13 April 2025, Set 1-6 (January 2025 term)

1. A

   Route a copy of every quick commerce item ordered to a Kafka topic, useSpark Structured Streaming to continuously read from this topic and update the current counts of supplies, and emit using output mode “Update”.

2. B

   Route a copy of every quick commerce item ordered to a Kafka topic, useSpark Structured Streaming to periodically read from this topic every 1 minute and update the current counts of supplies, and emit all aggregates using the output mode “Complete”.

3. C

   Route a copy of every quick commerce item ordered to a Kafka topic, useSpark Structured Streaming to periodically read from this topic every 1 minute and count the supplies ordered in that batch, and emit only all aggregates in that batch using the output mode “Append”.

1. A

   Change the schedule to run every 5 minutes, no other change required.

2. B

   Change the code to leverage Spark Streaming with streaming window as “5minutes”, & let her manage the execution of the code on Google Dataproc

3. C

   Change the code to leverage Spark Streaming with streaming window as “5minutes”, go from using Dataproc to Dataflow, & let her manage the execution of the code on Dataflow

4. D

   Write a Cloud Function to move all incoming per-minute standalone files fromGCS to Pub/Sub, change the code to leverage Spark Streaming with streaming window as “5 mins”, convert from Dataproc to Dataflow, point source to Pub/Sub, & let her manage the execution of the code on Dataflow

1. A

   (i), (v), (iii), (iv), (ii)

2. B

   (i), (ii), (iii), (iv), (v)

3. C

   (ii), (iii), (i), (iv), (v)

4. D

   (v), (iii), (i), (iv), (ii)

5. E

   All are equally PaaS / IaaS

1. A

   Since data.bin is compliant with the RFC 4180, use PySpark’s read_csv() to readthe data as is.

2. B

   Rename the file data.bin to data.csv to make it compliant with RFC 4180 andthen use PySpark’s read_csv() to read the data

3. C

   The problem cannot be solved since the file cannot be converted to a validformat for reading consistently without additional information

4. D

   Write PySpark code to read all lines in data.bin, use string split on “,” asdelimiter, and then collect all column names and corresponding values into a RDD for further processing

1. A

   The task is impossible since Spark 1.0 used RDDs which cannot be convertedto Dataframes without the correct data schema.

2. B

   The task is impossible since Spark core and syntax has changed so completelythat those data files it processed then can no longer be processed by current Spark versions.

3. C

   The task is possible with multiple trial-and-error schema experimentsmatching data files with the older code.

4. D

   The task is possible since Spark has backwards compability.

5. E

   None of these,since not enough information is available.

1. A

   Failure of a machine in the Kafka cluster will result in an Exception in the Sparkpipeline which will then fail and halt.

2. B

   The pipeline will be restarted automatically by Spark which is able to pick upthe exact data from Kafka which was being processed at the time of error, resulting in exactly-once semantics.

3. C

   The Spark pipeline will not be able to start again from previously committedoffset by restarting itself, resulting in at least-once processing semantics

4. D

   Irrespective of whatever machine fails, Spark will throw an error and halt.

5. E

   Data that is being processed will not be processed again, resulting in at most-once semantics.

6. F

   Spark will be able to pick up data from Kafka from exactly that offset whichfailed but may produce duplicate output on the target resulting in at least once semantics for the consumer of the output.

1. A

   Add disks to each broker in the cluster, and disks are the cheapest computercomponent

2. B

   Add new brokers to the cluster, even though this is more expensive than theother options this is the only foolproof way to scale.

3. C

   Create more topics and change input application to reroute data to all topicsto be able to spread input data better. This is nearly the least expensive since only developer effort is required to change application.

4. D

   Double the number of partitions for this single topic to be able to spread inputdata better. This is the least expensive since only administrator effort is required without changing application.

1. A

   The business time for the final weekly sum operation needs to be changed tothat of Monday 3AM India time instead of Sunday 3AM Middle East time.

2. B

   There is zero change needed since neither event time nor business time ischanging whereas only the operational time is changing.

3. C

   Since time zone has changed as well as week definition too, the definition ofbusiness time has changed. So, the ETL has to be rewritten entirely.

4. D

   Nothing needs to change since daily sales is available at 1AM Middle East timewhich is anyway behind India time and so the numbers will be available before leadership comes in at 9AM.

5. E

   Event time has changed since the event of week ending has changed indefinition, and so the ETL needs to be changed to consider the new event in the data.

1. A

   Since MongoDB is OLTP, it is not able to support business reporting. So, bringthe data into Hadoop end of day, and run OLAP queries on it from the same UI.

2. B

   Convert the application from using plain Java to using Spark Streaming in Java

3. C

   Extract raw data from MongoDB using Change Data Capture (CDC) once everyminute into Kafka, and then use Spark Streaming to compute the KPIs and then populate into a NoSQL DB like Redis for the UI to consume.

4. D

   Query MongoDB every 1 minute for new data using a check on documentinserted timestamp, use Spark Streaming to compute the KPIs with the queried data, and then populate into a NoSQL DB like Redis for the UI to consume.

5. E

   Convert application to using Python along with a NoSQL database for storingand retrieving the aggregated counts.

1. A

   Ingest all spider cam data including video feeds into Pub/Sub, process usingGoogle Cloud Dataflow where you invoke the vision model and the adjustment model, relaying this output to the spider cam continuously.

2. B

   Compress the vision model and adjustment model to fit into the spider cam’savailable resource, and write pipelines to execute the models in the spider cam itself

3. C

   Compress the vision model and adjustment model to fit into the spider cam’savailable resource, and write pipelines to execute in the spider cam itself, with periodically data being sent to the cloud for retraining the vision model using Google Cloud ML and then redeploy the model to the spider cam.

4. D

   Compress the data in the spider cam every 5 seconds, write to Pub/Sub thecompressed data, invoke the vision model and, if needed, the adjustment model, and then write back output from Cloud to the spider cam to relay to the fielder.

1. A

   Yes, since they check the syntax of the model function so that there are noerrors

2. B

   Yes, they invoke PyTorch libraries that have already been setup for modelscoring on GCP using APIs embedded within the function

3. C

   No, since the primary function of these lines of code is to eliminate repeatedDL model loads as DL models are large in size

4. D

   Yes, they are Map-style UDFs that make it an embarrassingly parallelcomputation thus making the execution parallelized and fast.

5. E

   Yes, since the new model will potentially have large load times.

1. A

   Build a custom model that compresses the highest recall rate model justenough to be able to execute within a single Spark worker and thus reduce cost of the cluster.

2. B

   Use a different DL model from PyTorch that has better efficiency at lesserrecall levels so that the cluster size can be minimized.

3. C

   Remove complexity associated with Spark Streaming, remove GPUs to savesignificant costs, and convert the model execution pipeline into a single threaded Python application using traditional ML running on a CPU-only machine.

4. D

   Change Spark machine to use CPUs and train a fresh pipeline to achieveobjectives.

1. A

   No, the failure is not visible to the consumers of NoSQL DB outputs, asStructured Streaming retries the mini-batch that failed thus taking maybe twice as much time as normal.

2. B

   No, the failure is not visible since Structured Streaming uses transactions andidempotence to achieve exactly-once processing.

3. C

   No, the failure is not visible since Structured Streaming can process the samedata in a retry resulting in the same outputs again.

4. D

   Yes, the failure is visible because the side effect of logging for debugging willbe visible as repeated entries when Structured Streaming retries the failed batch.

5. E

   Yes, the failure is visible since the logs in the backend of Spark StructuredStreaming are also logging the state of the machine.

1. A

   Code up a new batch processing job and process the 1 month of missed dataas a standalone job. Then, reactivate the structured streaming job once the latest date is caught up.

2. B

   Code up a new batch processing job and process the 1 month of missed dataas a standalone job. Then, reactivate the structured streaming job once the latest date is caught up. But copy the code over from the current structured streaming job where the read and write commands will remain the same, but the remaining code will need to be modified, as operations on streaming dataframes are not supported on static dataframes.

3. C

   Launch the structured streaming job using starting offset as that lastsuccessfully processed before the job went on hiatus, and launch in a streaming mode with a very high time frequency of repetition. This simulates a batch execution of the same logic so as to catch up for 1 month of data processing. Once done, relaunch the structured streaming using the earlier-used configuration parameters.

4. D

   Code up a new batch processing job and process the 1 month of missed dataas a standalone job. Then, reactivate the structured streaming job once the latest date is caught up.But copy the code over from the current structured streaming job where the read and write commands need to be modified to specify that it’s a batch operation. Further, the specific logic of window functions will also need to be modified since there are no time windows anymore in batch processing.

5. E

   Code up a new batch processing job and process the 1 month of missed dataas a standalone job. Then, reactivate the structured streaming job once the latest date is caught up. But copy the code over from the current structured streaming job where only the read and write commands need to be modified to specify that it's a batch operation.

Select all that apply.

1. A

   Each of the Consumers 1,2,3 and 4 gets all messages published into Topic.

2. B

   Publisher 1 can safely send data only for Consumer Group 1’s consumptionwhile preventing access to Consumer 3 & 4.

3. C

   Consumers 1 & 2 may not get every message published into Topic.

4. D

   Each Publisher is able to have the guarantee that its messages are reliablystored in Topic and made available to every consumer / consumer group that needs it.

5. E

   Consumer 3 gets all messages published into Topic.

6. F

   Consumer Group 1 gets all messages published by Publisher 1.

Select all that apply.

1. A

   Create all tables as external tables.

2. B

   Ensure cost based optimizer (CBO) is ON.

3. C

   Partition all tables on the same key on which the aggregate is happening.

4. D

   Run ANALYZE on all tables to ensure the right estimates are available to theoptimizer.

5. E

   Cache the table in a step with actions ahead of the SQL statement that is theculprit.

Select all that apply.

1. A

   Use a message store that supports message replay so that no data is lost inprocessing.

2. B

   In order to get “exactly once” data processing, it is sufficient that the messagestore supports transactional semantics of all-or-nothing operations.

3. C

   Hadoop is best suited for executing Streaming applications.

4. D

   Use checkpointing when faced with mission-critical workloads that require100% accuracy.

5. E

   Handle state pollution by restarting the persistent store software periodically.

Select all that apply.

1. A

   Batch operates on data that typically has a static schema but can adjust tochanges in schema, and the same syntax will also help in streaming mode

2. B

   Batch operates on a set of data elements taken together while streaming canalso operate on a set of data elements as determined by the window

3. C

   Batch processing can assume data as fully specified and complete whilestreaming cannot make that assumption

4. D

   Batch processing is typically high latency while streaming processing isnecessary for real-time latencies

5. E

   Batch can operate on massively large data sets, while streaming can alsooperate on massively large data but expressed as streams

Select all that apply.

1. A

   Subscribers to Google Pub/Sub can decide at what rate they want to consumedata without depending on or informing the publisher of the data.

2. B

   Snapshots of source systems can be created using an event capture tool likeCDC and then replaying the events in sequence for that time period.

3. C

   A shared-memory algorithm for a ML problem is one where the algorithmassumes that the memory available to it is shared with multiple other processes that run on the same machine.

4. D

   A data lake is a collection of data to be provided as input for data sciencealgorithms.

5. E

   Zookeeper is a library for ensuring that critical services used in big data areable to stay in sync with each other as to the health of those services.

6. F

   Distributed model training is not practical since reducing estimation errorsrequires constant exchange of enormous information across the machines in the cluster.

7. G

   Spark has support for declarative programming, functional programming aswell as procedural programming