Question 5
Consider a file “data.bin” which is formatted as follows: every data record has 10 key-value pairs of the format “key,value”, with each pair separated by a comma, where the “key” is the name of a field and the “value” is the value for that field in that record. Every data record occurs in its own line. You are asked to write a data processing script using Python that scales with big data. Which of the following represents your approach?
Since data.bin is compliant with the RFC 4180, use PySpark’s read_csv() to readthe data as is.
Rename the file data.bin to data.csv to make it compliant with RFC 4180 andthen use PySpark’s read_csv() to read the data
The problem cannot be solved since the file cannot be converted to a validformat for reading consistently without additional information
Write PySpark code to read all lines in data.bin, use string split on “,” asdelimiter, and then collect all column names and corresponding values into a RDD for further processing