Question 5
You join the data engineering team at a company that has good big data expertise already. Your first assignment is to convert Spark code written by previous engineers that used Spark 1.0 to use newer features and best practices. None of those previous engineers work in the company anymore, and nobody in the present team can tell you what data files available in the company data lake today correspond to which part of the processing done then. Which of the following most accurately captures your ability to do the task at hand?
The task is impossible since Spark 1.0 used RDDs which cannot be convertedto Dataframes without the correct data schema.
The task is impossible since Spark core and syntax has changed so completelythat those data files it processed then can no longer be processed by current Spark versions.
The task is possible with multiple trial-and-error schema experimentsmatching data files with the older code.
The task is possible since Spark has backwards compability.
None of these,since not enough information is available.