Question 12
Consider a file “data.bin” which is formatted as follows: every data record is in the form of pairs of values of the form “column name,column value”, with each pair in its own line. You are asked to write a data processing script using Python that scales with big data. Which of the following represents your approach?
Since data.bin is compliant with the RFC 4180, use PySpark’s read_csv() to read the data as is.
Rename the file data.bin to data.csv to make it compliant with RFC 4180 and then use PySpark’s read_csv() to read the data
The problem cannot be solved since the file cannot be converted to a valid format for reading consistently without additional information
Write PySpark code to read all lines in data.bin, use string split on “,” as delimiter, and then collect all column names and corresponding values into a RDD for further processing