Question 14
Let us say we are using structured streaming for continuously reading data from Kafka and storing the results back into a Kafka topic using window function aggregates. Due to various reasons, the job did not run for 1 month. Now, we need to again continue the runs without compromising on the correctness of the results. What can you do that will take the least effort?
Code up a new batch processing job and process the 1 month of missed dataas a standalone job. Then, reactivate the structured streaming job once the latest date is caught up.
Code up a new batch processing job and process the 1 month of missed dataas a standalone job. Then, reactivate the structured streaming job once the latest date is caught up. But copy the code over from the current structured streaming job where the read and write commands will remain the same, but the remaining code will need to be modified, as operations on streaming dataframes are not supported on static dataframes.
Launch the structured streaming job using starting offset as that lastsuccessfully processed before the job went on hiatus, and launch in a streaming mode with a very high time frequency of repetition. This simulates a batch execution of the same logic so as to catch up for 1 month of data processing. Once done, relaunch the structured streaming using the earlier-used configuration parameters.
Code up a new batch processing job and process the 1 month of missed dataas a standalone job. Then, reactivate the structured streaming job once the latest date is caught up.But copy the code over from the current structured streaming job where the read and write commands need to be modified to specify that it’s a batch operation. Further, the specific logic of window functions will also need to be modified since there are no time windows anymore in batch processing.
Code up a new batch processing job and process the 1 month of missed dataas a standalone job. Then, reactivate the structured streaming job once the latest date is caught up. But copy the code over from the current structured streaming job where only the read and write commands need to be modified to specify that it's a batch operation.