Question 14
Consider a Structured Streaming application running on Google Dataproc firing up every 10 seconds, consuming any number of records from Kafka available since last read, and emitting some computed answers to a NoSQL DB. Consider also that apart from the functional logic, the same application is also emitting into a file some log statements for debugging purposes meant for use by the developer of the application only in the event that something goes wrong but is otherwise not intelligible.
Assume there is a failure in one of the Dataproc machines that results in a failure of a specific run. For anybody consuming the NoSQL DB outputs, will they see any change in output as a result of the failure at all, or will the only visible impact of failure be of slower performance for the failed-and-retried run?
No, the failure is not visible to the consumers of NoSQL DB outputs, asStructured Streaming retries the mini-batch that failed thus taking maybe twice as much time as normal.
No, the failure is not visible since Structured Streaming uses transactions andidempotence to achieve exactly-once processing.
No, the failure is not visible since Structured Streaming can process the samedata in a retry resulting in the same outputs again.
Yes, the failure is visible because the side effect of logging for debugging willbe visible as repeated entries when Structured Streaming retries the failed batch.
Yes, the failure is visible since the logs in the backend of Spark StructuredStreaming are also logging the state of the machine.