Introduction to Big Data · End Term · 13 Apr 2025 · January 2025 term · Set 1-2

Question 15: Let us say we are using structured streaming for continu…

Question 15

+2 marksOne correct option

Let us say we are using structured streaming for continuously reading data from Kafka and storing the results back into a Kafka topic using window function aggregates. Due to various reasons, the job did not run for 1 month. Now, we need to again continue the runs without compromising on the correctness of the results. What can you do that will take the least effort?

A
Code up a new batch processing job and process the 1 month of missed dataas a standalone job. Then, reactivate the structured streaming job once the latest date is caught up.
B
Code up a new batch processing job and process the 1 month of missed dataas a standalone job. Then, reactivate the structured streaming job once the latest date is caught up. But copy the code over from the current structured streaming job where the read and writecommands will remain the same, but the remaining code will need to be modified, as operations on streaming dataframes are not supported on static dataframes.
C
Launch the structured streaming job using starting offset as that lastsuccessfully processed before the job went on hiatus, and launch in a streaming mode with a very high time frequency of repetition. This simulates a batch execution of the same logic so as to catch up for 1 month of data processing. Once done, relaunch the structured streaming using the earlier-used configuration parameters.
D
Code up a new batch processing job and process the 1 month of missed dataas a standalone job. Then, reactivate the structured streaming job once the latest date is caught up. But copy the code over from the current structured streaming job where the read and write commands need to be modified to specify that it’s a batch operation. Further, the specific logic of window functions will also need to be modified since there are no time windows anymore in batch processing.
E
Code up a new batch processing job and process the 1 month of missed dataas a standalone job. Then, reactivate the structured streaming job once the latest date is caught up. But copy the code over from the current structured streaming job where only the read and write commands need to be modified to specify that it's a batch operation.

Show answer

Correct answer

C
Launch the structured streaming job using starting offset as that lastsuccessfully processed before the job went on hiatus, and launch in a streaming mode with a very high time frequency of repetition. This simulates a batch execution of the same logic so as to catch up for 1 month of data processing. Once done, relaunch the structured streaming using the earlier-used configuration parameters.

See the whole paper Take it as a mock test

Question 15 of 20 in the IIT Madras BS Introduction to Big Data (Intro to Big Data) End Term paper sat on 13 Apr 2025, in the January 2025 term (IIT M IMPROVEMENT FN EXAM QIM2 13 Apr). It carries 2 marks.

This question was also asked in

Intro to Big Data End Term — 13 Apr 2025, Set 1 · question 15

Question 15: Let us say we are using structured streaming for continu…

Question 15

This question was also asked in

More questions from this paper