Introduction to Big Data End Term: 30 April 2023 (January 2023 term)

Select all that apply.

1. A

   Batch operates on a set of data elements taken together while streamingoperates on data individually as it streams in

2. B

   Batch operates on data that is static while streaming operates on data that isdynamically changing

3. C

   Batch processing can assume data as fully specified and complete whilestreaming cannot make that assumption

4. D

   Batch processing is typically high latency while streaming processing isnecessary for real-time latencies

5. E

   Batch processing can operate on massively larger data sets than streamingcan

Select all that apply.

1. A

   Data needs to be retrievable from a persistent store that supports messagereplay. Replay refers to being able to fetch a specific set of data from that persistent store on-demand., where the data is chosen based on filters usually defined on sequence numbers ortimestamps

2. B

   Data processing needs to be splittable across machines using divide-and-conquer, and composable into steps that execute very quickly

3. C

   Hadoop needs to be setup for its big data capabilities

4. D

   The application needs to have the cloud-native properties of being resilient,manageable and observable

5. E

   Application needs to be deployable on public cloud natively using PaaScomponents

Select all that apply.

1. A

   Increase memory in each of the brokers in the cluster

2. B

   Add disks to each broker in the cluster

3. C

   Add new brokers to the cluster

4. D

   Create more topics and change input application to reroute data to all topics tobe able to spread input data better

5. E

   Double the number of partitions for this single topic to be able to spread inputdata better

Select all that apply.

1. A

   Create all tables as native Spark SQL tables (i.e. available as CatalogTables).

2. B

   Partition all tables on the same key on which the aggregate is happening.

3. C

   Ensure cost based optimizer (CBO) is ON.

4. D

   Run ANALYZE on all tables.

5. E

   Cache the table in a step with actions ahead of the SQL statement that is theculprit.

Select all that apply.

1. A

   In pull delivery, your subscriber application initiates requests to the Pub/Subserver to retrieve messages.

2. B

   In push delivery, Pub/Sub initiates requests to your subscriber application todeliver messages.

3. C

   Pull subscription is preferred when efficiency and throughput of messageprocessing is critical.

4. D

   The rate of delivery needs to be controlled by the subscriber client in pushsubscription.

1. A

   The data being processed when the failure happened will have producedpartial results that result in incorrect outputs.

2. B

   The pipeline will not be restarted automatically.

3. C

   The pipeline will be restarted automatically by Spark which is able to pick upthe exact data from Kafka which was being processed at the time of error

4. D

   Spark will throw an error and halt

5. E

   Spark will gracefully shut down, reboot that machine which failed and start theStructured Streaming pipeline again to continue from where it left off

1. A

   No change to business time since that remains constant at 1AM local time,only the schedule for the final weekly sum operation needs to be changed to the operational timeof Monday 3AM India time instead of Sunday 3AM Middle East time.

2. B

   Since time zone has changed as well as week definition too, the definition ofbusiness time has changed. So, the ETL has to be rewritten entirely.

3. C

   Nothing needs to change since daily sales is available at 1AM Middle East timewhich is anyway behind India time and so the numbers will be available before leadership comesin at 9AM.

4. D

   Event time has changed since the event of week ending has changed indefinition, and so the ETL needs to be changed to consider the new event in the data.

5. E

   No change required.

1. A

   Since MongoDB is OLTP, it is not able to support business reporting. So,replace it with Hadoop which supports OLAP better

2. B

   Extract raw data from MongoDB using Change Data Capture (CDC) once everyminute into Kafka, and then use Spark Streaming to compute the KPIs and then populate into aNoSQL DB like Redis for the UI to consume.

3. C

   Convert the application from using plain Java to using Spark Streaming in Java

4. D

   Query MongoDB every 1 minute for new data using a check on documentinserted timestamp, use Spark Streaming to compute the KPIs with the queried data, and thenpopulate into a NoSQL DB like Redis for the UI to consume.

1. A

   Ingest all data into Pub/Sub, process using Google Cloud Dataflow, invoke theML model, and then write back output from Cloud to the spider cam to relay to the bat.

2. B

   Compress the ML model to fit into the spider cam’s available resource, andwrite pipelines to execute the model in the spider cam itself

3. C

   Compress the data in the spider cam every 5 seconds, write to Pub/Sub thecompressed data, invoke the ML model and then write back output from Cloud to the spider camto relay to the bat.

4. D

   Compress the ML model to fit into the spider cam’s available resource, andwrite pipelines to execute in the spider cam itself, with periodically data being sent to the cloud,retrain the model using Google Cloud ML and then redeploy the model to the spider cam.

1. A

   A UDF is a scalar operation executing on 1 row at a time and producing outputimmediately, and therefore there is no intermediate output necessary. Whereas, a UDAF operateson multiple rows which will require multiple passes for the final output thereby requiring a buffer.

2. B

   A UDF is a scalar operation executing on many rows at a time, grouped by akey and producing a single output, and therefore there is no intermediate output necessary.Whereas, a UDAF operates on multiple rows which will require multiple passes for the final outputthereby requiring a buffer.

3. C

   A UDF is a scalar operation executing on 1 row at a time and producing outputimmediately, and therefore there is no intermediate output necessary. Whereas, a UDAF operateson multiple rows of unbounded size requiring a divide-and-conquer approach for computingaggregates, which uses the intermediate buffer to store partial values before finalizing the resultaggregate.

4. D

   A UDF is a scalar operation executing on many rows at a time, grouped by akey and producing a single output, and therefore there is no intermediate output necessary.Whereas, a UDAF operates on multiple rows of unbounded size requiring a divide-and-conquerapproach for computing aggregates, which uses the intermediate buffer to store partial valuesbefore finalizing the result aggregate.

1. A

   They check the syntax of the model function so that there are no errors

2. B

   They load the model into memory so that the loop iteration over each imagedoes not need to do the same operation again and again given model load times are prohibitivelyexpensive for DL models

3. C

   They invoke PyTorch libraries that have already been setup for model scoringon GCP using APIs embedded within the function

4. D

   They are Map-style UDFs that make it an embarrassingly parallel computationthus making the execution parallelized and fast.

1. A

   Use a different DL model that is faster but has half the recall rate.

2. B

   Build a custom model that compresses the highest recall rate model justenough to be able to execute within the stipulated time, and measure recall.

3. C

   Parallelize the DL model code using Divide-and-Conquer so that it runs faster.

4. D

   Change Spark machine to use GPUs

1. A

   Yes, the failure is visible. The only visible effect for the external world would bein the form of a slowdown in runtime for completion of that batch as Structured Streaming retriesthe batch that failed thus taking twice as much time as normal.

2. B

   Yes, the failure is visible because the side effect of emitting timestamps in abatch will be visible as 2 consecutive Start timestamps without any end timestamp as StructuredStreaming retries the failed batch.

3. C

   No, no failure is visible since Structured Streaming uses transactions andidempotence to achieve exactly-once processing.

4. D

   No, no failure is visible since Structured Streaming can process the same datain a retry resulting in the same outputs again.

1. A

   The read and write commands will remain the same, but the remaining codewill need to be modified, as operations on streaming dataframes are not supported on staticdataframes.

2. B

   Only the read and write commands need to be modified to specify that it's abatch operation.

3. C

   The entire code will need to be modified as the APIs for stream and batchprocessing are completely different.

1. A

   Only (i)

2. B

   Only (ii)

3. C

   Both (i) and (ii)

4. D

   Neither (i) nor (ii)

1. A

   (i), (v), (iii), (iv), (ii)

2. B

   (i), (ii), (iii), (iv), (v)

3. C

   (ii), (iii), (i), (iv), (v)

4. D

   (v), (iii), (i), (iv), (ii)

5. E

   All are equally PaaS / IaaS

1. A

   Route a copy of the ticket purchase to a Kafka topic, use Spark StructuredStreaming to continuously read from this topic and update the aggregates by destinations, andemit using output mode “Update”.

2. B

   Route a copy of the ticket purchase to a Kafka topic, use Spark StructuredStreaming to periodically read from this topic every 1 minute and update the aggregates bydestinations, and emit top aggregates using the output mode “Complete”.

3. C

   Route a copy of the ticket purchase to a Kafka topic, use Spark StructuredStreaming to periodically read from this topic every 1 minute and count the destinations in thatbatch, and emit only top aggregates in that batch using the output mode “Append”.

1. A

   Each subscriber gets one third of all messages published into Topic 1.

2. B

   Each subscriber gets one third of all messages published into Topic 1 byPublisher 1

3. C

   Each subscriber gets all messages published into Topic 1.

1. A

   Change the code to assume hourly data instead of daily data, changescheduler to run every hour & let her manage the execution of the code on Dataproc

2. B

   Change the code to leverage Spark Streaming with streaming window as “1hour”, & let her manage the execution of the code on Dataproc

3. C

   Change the code to leverage Spark Streaming with streaming window as “1hour”, convert from Dataproc to Dataflow, & let her manage the execution of the code onDataflow

4. D

   Write a Cloud Function to move all incoming per-minute standalone files fromGCS to Pub/Sub, change the code to leverage Spark Streaming with streaming window as “1 hour”,convert from Dataproc to Dataflow, point source to Pub/Sub, & let her manage the execution ofthe code on Dataflow