Question 6
What happens when a Spark Structured Streaming pipeline operating with Kafka as source and console output as target is subject to a failure of a machine in either of the Kafka cluster or the Spark cluster?
Failure of a machine in the Kafka cluster will result in an Exception in the Sparkpipeline which will then fail and halt.
The pipeline will be restarted automatically by Spark which is able to pick upthe exact data from Kafka which was being processed at the time of error, resulting in exactly-once semantics.
The Spark pipeline will not be able to start again from previously committedoffset by restarting itself, resulting in at least-once processing semantics
Irrespective of whatever machine fails, Spark will throw an error and halt.
Data that is being processed will not be processed again, resulting in at most-once semantics.
Spark will be able to pick up data from Kafka from exactly that offset whichfailed but may produce duplicate output on the target resulting in at least once semantics for the consumer of the output.