The IIT Madras BS Introduction to Big Data (Intro to Big Data) End Term paper sat on 13 Apr 2025, in the January 2025 term, set 1-4: 20 questions for 50 marks in 180 minutes. Every question is below with its answer. Take it as a timed mock test to be marked, or read it through first.
Which of the following is a false characterization of "hot potato” principle?
It is a rule-of-thumb, not a principle, and therefore is not mandatory
It provides the same benefits when incorporated into the design of both batchas well as streaming data applications.
It emphasizes minimal work in each processing step for maximum scalabilityand optimal recoverability in the event of failures.
It is critically dependent on a message store as the via media for itsintermediate outputs.
Correct answer
It provides the same benefits when incorporated into the design of both batchas well as streaming data applications.
A website that started 3 years ago now sees 1 Billion hits every month. The website owner wants to count average hits per customer across all months, where a customer is denoted by the IP address of the device from which the customer is accessing the website. The owner has at his disposal a minimal Hadoop cluster of 2 workers and 1 master each with 1GB of RAM. He would like to run this job every month going forward. Which of the following methods is the most likely to finish every time it is run in the months and years ahead yielding the right result?
Write a MapReduce program where the Map does nothing useful, Combinecomputes the aggregated hits per customer, Shuffle combines data based on IP address across workers, and the Reduce builds a hash table on each machine with hash key = IP address and hash value = running total and sum, with a final Map that emits the avg per customer.
Write a Spark program that forms a Dataframe as grouping by IP address withcount as aggregate, followed by a take into a list in the Spark driver which further computes the average of all the individual counts in the list
Write a Spark program that forms a Dataframe as grouping by IP address withcount as aggregate, followed by another stage that computes the avg on top of the Dataframe of the first stage
Write a MapReduce program where the Map does nothing useful, Combinecomputes the aggregated hits per customer, Shuffle combines data based on IP address across workers, and the Reduce sorts the data on sort key = IP address and then calculates the final avg per customer.
All will finish every time it is run without issues.
Correct answer
Write a MapReduce program where the Map does nothing useful, Combinecomputes the aggregated hits per customer, Shuffle combines data based on IP address across workers, and the Reduce sorts the data on sort key = IP address and then calculates the final avg per customer.
An enterprise software designer wants to leverage the best of Google cloud to minimize the number of administrative overheads associated with her payment processing pipeline while also getting on-demand scalability without sacrificing flexibility. What option should she choose to best serve these needs?
Build the payment processer using VMs – one for Python for the logic, one forinvoking the external payment engine, and one for storing the results
Build the payment processor using Python running on Google Cloud Functionswhere the results are stored on GCS
Build the payment processor using MapReduce with input data and results arestored on HDFS, and deploy both on Dataproc
Build the payment processor using Dataflow on top of data stored on GCS
Correct answer
Build the payment processor using Python running on Google Cloud Functionswhere the results are stored on GCS
Consider an application that can scale from handling 1000 users to handling 100 million users by simply making copies of itself, has automation to detect task failures so that it can boot up a new task automatically, and logs operational metadata on a central logger backed by Kafka. Which of the following statements is true?
This application has adopted cloud-native design
This application cannot be called as cloud-native since it is not observable at alltimes
This application cannot be called as cloud-native since it is not manageableeasily
This application cannot be called as cloud-native since it is not resilient
Correct answer
This application has adopted cloud-native design
Consider a file “data.bin” which is formatted as follows: every data record has 10 key-value pairs of the format “key,value”, with each pair separated by a comma, where the “key” is the name of a field and the “value” is the value for that field in that record. Every data record occurs in its own line. You are asked to write a data processing script using Python that scales with big data. Which of the following represents your approach?
Since data.bin is compliant with the RFC 4180, use PySpark’s read_csv() to readthe data as is.
Rename the file data.bin to data.csv to make it compliant with RFC 4180 andthen use PySpark’s read_csv() to read the data
The problem cannot be solved since the file cannot be converted to a validformat for reading consistently without additional information
Write PySpark code to read all lines in data.bin, use string split on “,” asdelimiter, and then collect all column names and corresponding values into a RDD for further processing
Correct answer
Write PySpark code to read all lines in data.bin, use string split on “,” asdelimiter, and then collect all column names and corresponding values into a RDD for further processing
You join the data engineering team at a company that has good big data expertise already. Your first assignment is to convert Spark code written by previous engineers that used Spark 1.0 to use newer features and best practices. None of those previous engineers work in the company anymore, and nobody in the present team can tell you what data files available in the company data lake today correspond to which part of the processing done then. Which of the following most accurately captures your ability to do the task at hand?
The task is impossible since Spark 1.0 used RDDs which cannot be convertedto Dataframes without the correct data schema.
The task is impossible since Spark core and syntax has changed so completelythat those data files it processed then can no longer be processed by current Spark versions.
The task is possible with multiple trial-and-error schema experimentsmatching data files with the older code.
The task is possible since Spark has backwards compability.
None of these, since not enough information is available.
Correct answer
The task is possible with multiple trial-and-error schema experimentsmatching data files with the older code.
What happens when a Spark Structured Streaming pipeline operating with Kafka as source and console output as target is subject to a failure of a machine in either of the Kafka cluster or the Spark cluster?
Failure of a machine in the Kafka cluster will result in an Exception in the Sparkpipeline which will then fail and halt.
The pipeline will be restarted automatically by Spark which is able to pick upthe exact data from Kafka which was being processed at the time of error, resulting in exactly-once semantics.
The Spark pipeline will not be able to start again from previously committedoffset by restarting itself, resulting in at least-once processing semantics
Irrespective of whatever machine fails, Spark will throw an error and halt.
Data that is being processed will not be processed again, resulting in at most-once semantics.
Spark will be able to pick up data from Kafka from exactly that offset whichfailed but may produce duplicate output on the target resulting in at least once semantics for the consumer of the output.
Correct answer
Spark will be able to pick up data from Kafka from exactly that offset whichfailed but may produce duplicate output on the target resulting in at least once semantics for the consumer of the output.
A big data streaming application that uses Kafka as source is observed to be really lagging behind currently live data. The Kafka cluster has 2 broker nodes and this application is reading from 1 topic that has 10 partitions. On closer investigation, it was found that Kafka is not scaling to the velocity of input data coming in. What can you first try to do to scale Kafka further while incurring minimal overall costs?
Add disks to each broker in the cluster, and disks are the cheapest computercomponent
Add new brokers to the cluster, even though this is more expensive than theother options this is the only foolproof way to scale.
Create more topics and change input application to reroute data to all topicsto be able to spread input data better. This is nearly the least expensive since only developer effort is required to change application.
Double the number of partitions for this single topic to be able to spread inputdata better. This is the least expensive since only administrator effort is required without changing application.
Correct answer
Add new brokers to the cluster, even though this is more expensive than theother options this is the only foolproof way to scale.
A company with headquarters (HQ) in the Middle East operates on a Sunday-Thursday weekday schedule with Friday & Saturday as weekend days. It computes end of week revenue numbers using an ETL pipeline by first computing sales for each day at 1AM local time of the next day, and then summing up the weekly sales every Sunday early morning at 3AM local time. This number gets reported to leadership every Sunday morning 9AM local time. Therefore, the ETL pipeline is scheduled to run every Sunday morning at 8AM local time. As a result of management change, the company has decided to shutdown its business in the Middle East and relocate all of its operations and business HQ to India. Which of the following changes will need to be done to its ETL pipeline to ensure the correct output continues to be produced?
The business time for the final weekly sum operation needs to be changed tothat of Monday 3AM India time instead of Sunday 3AM Middle East time.
There is zero change needed since neither event time nor business time ischanging whereas only the operational time is changing.
Since time zone has changed as well as week definition too, the definition ofbusiness time has changed. So, the ETL has to be rewritten entirely.
Nothing needs to change since daily sales is available at 1AM Middle East timewhich is anyway behind India time and so the numbers will be available before leadership comes in at 9AM.
Event time has changed since the event of week ending has changed indefinition, and so the ETL needs to be changed to consider the new event in the data.
Correct answer
Since time zone has changed as well as week definition too, the definition ofbusiness time has changed. So, the ETL has to be rewritten entirely.
You are appointed as a Data Engineer in a company that has a legacy reporting application written in Java which suffers from performance problems. The reporting application plots dashboards with daily refresh of key business indicators to help management take the best decisions. The application reads data directly from the source database of MongoDB, aggregates using simple counts and shows them visually in a UI. The performance problem of this application comes because the source database is at times overloaded and therefore the dashboard is not able to retrieve answers fast enough making the end user wait for the result. Choose the best option that gives the best performance with minimal maintenance effort:
Since MongoDB is OLTP, it is not able to support business reporting. So, bringthe data into Hadoop end of day, and run OLAP queries on it from the same UI.
Convert the application from using plain Java to using Spark Streaming in Java
Extract raw data from MongoDB using Change Data Capture (CDC) once everyminute into Kafka, and then use Spark Streaming to compute the KPIs and then populate into a NoSQL DB like Redis for the UI to consume.
Query MongoDB every 1 minute for new data using a check on documentinserted timestamp, use Spark Streaming to compute the KPIs with the queried data, and then populate into a NoSQL DB like Redis for the UI to consume.
Convert application to using Python along with a NoSQL database for storingand retrieving the aggregated counts.
Correct answer
Since MongoDB is OLTP, it is not able to support business reporting. So, bringthe data into Hadoop end of day, and run OLAP queries on it from the same UI.
Fielder on the midwicket boundary is wearing a smart watch. Unlike other smart watches, this one is unique in that it helps him field. Whenever the ball is headed his direction, the watch vibrates to alert him. When there is a chance of a catch by this fielder, the watch continually whispers every second to go forward, back, left or right thus improving his chances of settling under the ball and taking the catch. The watch is connected to the spider cam. The spider cam is itself a powerful ARM-based computer which has connectivity to the Cloud, an all-seeing AI-powered superbeing, through the wire on which it hangs. Using this connectivity, it can send as much or as little data as required and also receive instructions from the cloud. Your task is to design the data pipeline that enables such feedback to the fielder for every ball that comes his way with as much accuracy as possible throughout the match. You are given two ML models: a vision model that given a frame from the spider cam computes an alert if the ball is headed to him, and an adjustment model that helps adjust catch positioning of the fielder which takes as input a continuous video feed. Which of the following options best satisfies the requirements?
Ingest all spider cam data including video feeds into Pub/Sub, process usingGoogle Cloud Dataflow where you invoke the vision model and the adjustment model, relaying this output to the spider cam continuously.
Compress the vision model and adjustment model to fit into the spider cam’savailable resource, and write pipelines to execute the models in the spider cam itself
Compress the vision model and adjustment model to fit into the spider cam’savailable resource, and write pipelines to execute in the spider cam itself, with periodically data being sent to the cloud for retraining the vision model using Google Cloud ML and then redeploy the model to the spider cam.
Compress the data in the spider cam every 5 seconds, write to Pub/Sub thecompressed data, invoke the vision model and, if needed, the adjustment model, and then write back output from Cloud to the spider cam to relay to the fielder.
Correct answer
Compress the vision model and adjustment model to fit into the spider cam’savailable resource, and write pipelines to execute the models in the spider cam itself
Yes, since they check the syntax of the model function so that there are noerrors
Yes, they invoke PyTorch libraries that have already been setup for modelscoring on GCP using APIs embedded within the function
No, since the primary function of these lines of code is to eliminate repeatedDL model loads as DL models are large in size
Yes, they are Map-style UDFs that make it an embarrassingly parallelcomputation thus making the execution parallelized and fast.
Yes, since the new model will potentially have large load times.
Correct answer
No, since the primary function of these lines of code is to eliminate repeatedDL model loads as DL models are large in size
You are given a Spark Streaming pipeline that invokes a pre-trained DL model for every image it receives as input and produces the classification result in quick time. The model with the best recall rate from the PyTorch library runs in under 3 seconds on an average, when executing on a GPU-powered Spark cluster. Your management has now instructed you to reduce the cost of AI projects significantly, and has given guidance that latency of execution is not a concern since the consumer for the model output is batch oriented, and also that they would be ok with a lesser recall rate provided the drop isn’t significant. What is the best option to explore first to meet the expectations?
Build a custom model that compresses the highest recall rate model justenough to be able to execute within a single Spark worker and thus reduce cost of the cluster.
Use a different DL model from PyTorch that has better efficiency at lesserrecall levels so that the cluster size can be minimized.
Remove complexity associated with Spark Streaming, remove GPUs to savesignificant costs, and convert the model execution pipeline into a single threaded Python application using traditional ML running on a CPU-only machine.
Change Spark machine to use CPUs and train a fresh pipeline to achieveobjectives.
Correct answer
Remove complexity associated with Spark Streaming, remove GPUs to savesignificant costs, and convert the model execution pipeline into a single threaded Python application using traditional ML running on a CPU-only machine.
Consider a Structured Streaming application running on Google Dataproc firing up every 10 seconds, consuming any number of records from Kafka available since last read, and emitting some computed answers to a NoSQL DB. Consider also that apart from the functional logic, the same application is also emitting into a file some log statements for debugging purposes meant for use by the developer of the application only in the event that something goes wrong but is otherwise not intelligible.
Assume there is a failure in one of the Dataproc machines that results in a failure of a specific run. For anybody consuming the NoSQL DB outputs, will they see any change in output as a result of the failure at all, or will the only visible impact of failure be of slower performance for the failed-and-retried run?
No, the failure is not visible to the consumers of NoSQL DB outputs, asStructured Streaming retries the mini-batch that failed thus taking maybe twice as much time as normal.
No, the failure is not visible since Structured Streaming uses transactions andidempotence to achieve exactly-once processing.
No, the failure is not visible since Structured Streaming can process the samedata in a retry resulting in the same outputs again.
Yes, the failure is visible because the side effect of logging for debugging willbe visible as repeated entries when Structured Streaming retries the failed batch.
Yes, the failure is visible since the logs in the backend of Spark StructuredStreaming are also logging the state of the machine.
Correct answer
No, the failure is not visible to the consumers of NoSQL DB outputs, asStructured Streaming retries the mini-batch that failed thus taking maybe twice as much time as normal.
Let us say we are using structured streaming for continuously reading data from Kafka and storing the results back into a Kafka topic using window function aggregates. Due to various reasons, the job did not run for 1 month. Now, we need to again continue the runs without compromising on the correctness of the results. What can you do that will take the least effort?
Code up a new batch processing job and process the 1 month of missed dataas a standalone job. Then, reactivate the structured streaming job once the latest date is caught up.
Code up a new batch processing job and process the 1 month of missed dataas a standalone job. Then, reactivate the structured streaming job once the latest date is caught up. But copy the code over from the current structured streaming job where the read and write commands will remain the same, but the remaining code will need to be modified, as operations on streaming dataframes are not supported on static dataframes.
Launch the structured streaming job using starting offset as that lastsuccessfully processed before the job went on hiatus, and launch in a streaming mode with a very high time frequency of repetition. This simulates a batch execution of the same logic so as to catch up for 1 month of data processing. Once done, relaunch the structured streaming using the earlier-used configuration parameters.
Code up a new batch processing job and process the 1 month of missed dataas a standalone job. Then, reactivate the structured streaming job once the latest date is caught up. But copy the code over from the current structured streaming job where the read and write commands need to be modified to specify that it’s a batch operation. Further, the specific logic of window functions will also need to be modified since there are no time windows anymore in batch processing.
Code up a new batch processing job and process the 1 month of missed dataas a standalone job. Then, reactivate the structured streaming job once the latest date is caught up. But copy the code over from the current structured streaming job where only the read and write commands need to be modified to specify that it's a batch operation.
Correct answer
Launch the structured streaming job using starting offset as that lastsuccessfully processed before the job went on hiatus, and launch in a streaming mode with a very high time frequency of repetition. This simulates a batch execution of the same logic so as to catch up for 1 month of data processing. Once done, relaunch the structured streaming using the earlier-used configuration parameters.
Which one of these is not an implementation of the divide-and-conquer data processing paradigm?
Spark Streaming
Hive
YARN
PySpark
Spark MLlib
Correct answer
YARN
What option(s) best describe the differences between MapReduce and Spark?
MapReduce leverages disk heavily, while Spark optimizes for memory-basedcomputations
MapReduce forces barrier synchronization after every step, while Spark usesdirected acyclic graphs to execute as many steps as possible in parallel
MapReduce is comprised of both Map and Reduce steps alternatingnecessarily, while Spark allows for any combination of actions and transformations.
MapReduce enables massively parallel computation, while Spark's driverprogram sequentially executes on each worker
MapReduce is restricted in flexibility since only Map and Reduce are possiblesteps, while Spark has a variety of Actions possible making it highly flexible
Correct answers
MapReduce leverages disk heavily, while Spark optimizes for memory-basedcomputations
MapReduce forces barrier synchronization after every step, while Spark usesdirected acyclic graphs to execute as many steps as possible in parallel
MapReduce is comprised of both Map and Reduce steps alternatingnecessarily, while Spark allows for any combination of actions and transformations.
You are provided with a Spark program that picks out a list of suspicious transactions. Its logic is based on both the financial value of the transaction and the geographic location of the transaction. If the financial value is higher than a threshold and the geographic location is from a set of suspected locations (provided as a 1MB file), then the program deems the transaction as suspicious. The version of the Spark program given to you is written in such a way that it pulls all the transactions from the Workers to the Driver and then applies the logic. The fraud control team that runs this program is having to constantly bother you with new threshold values since they are not able to change the Spark code themselves due to lack of technical knowledge. Which 2 changes from the list below will get you the most benefit in performance while also meeting the needs of your stakeholders?
Use broadcast variables for the 1MB file
Hardcode threshold value as a filter condition in the Driver program so thatthe stakeholders can directly edit the Driver program without having to understand Spark
Reorder operations on the Driver such that geographic location is checkedfirst before filtering high value transactions
Hardcode threshold value as a filter condition in the Workers itself
Accept the threshold value as a parameter and filter using the threshold in theWorkers itself.
Correct answers
Use broadcast variables for the 1MB file
Accept the threshold value as a parameter and filter using the threshold in theWorkers itself.
The given programs are an example of ETL and will be bottlenecked on thestring split happening on the Python VM.
The given programs are together an example of ELT that maximally utilizesthe Spark cluster and no further performance optimizations are required.
By moving the string split into PySpark driver program, I can get rid of thePython program and thus make the program an ELT.
Rewriting the string split to use native PySpark dataframe string functions willbe best suited for performance.
Correct answers
The given programs are an example of ETL and will be bottlenecked on thestring split happening on the Python VM.
Rewriting the string split to use native PySpark dataframe string functions willbe best suited for performance.
In a manufacturing facility, the supervisor is interested in improving efficiency of the assembly line using sensor data. While the modern machines in the facility are able to provide automated data feeds at 1Hz frequency (i.e. one data point every second) over a standard Ethernet port, the older machines have a screen showing the digital readings but no data feeds. She therefore instructs the operators to jot down readings from the screen. Which of the following are possible challenges she will have to overcome?
Ethernet is not a suitable protocol for machine data and so, for the modernmachines too, it is best to have operators read from the screen and jot down manually.
The rate of 1Hz for data feeds is too fast for available streaming technologies.It is best to reconfigure the machines to produce at 0.5Hz or lower.
Manual data entry is prone to error and so she cannot trust the data for theolder machines blindly.
The rate at which operators are able to jot down the readings will not matchthe 1Hz frequency of automated data collection. So, she has to reconfigure the modern machines to produce at 0.1Hz or lower.
The number of operators doing only data readings would be significantlyhigher if she wants to collect data at a reasonable frequency for being useful to increase efficiency.
Correct answers
Manual data entry is prone to error and so she cannot trust the data for theolder machines blindly.
The number of operators doing only data readings would be significantlyhigher if she wants to collect data at a reasonable frequency for being useful to increase efficiency.
