Quiz Space

Introduction to Big Data · Quiz 2 · 3 Dec 2023 · September 2023 term

Question 13: Consider the program outline as below running on a Spark…

Question 13

+1 markOne correct option

Consider the program outline as below running on a Spark cluster of 1 driver and 4 worker nodes with 2 executors per worker node:

Consider the below mutually-exclusive characterisations about the program:
i. Program will not run since RDD transformation operation map() runs on Executors that is referencing a variable train_master which is declared in the Spark driver
ii. Program will run successfully and produce the count of bookings that match for trains present in the train_master
For either of the 2 characterizations, consider the following (one or more) actions you could carry out in order to improve on the characterizations:
1. Program will need to be changed to bring the train_master into the Closure for executors to pick it up
2. Program will perform more poorly as the number of executors per worker node increases. Broadcast of train_master will need to be used to make it perform better.
3. Program will perform more poorly as the number of worker nodes increases (keeping number of executors per worker node constant). Broadcast of train_master will need to be used to make it perform better.
Which of the following combinations of statements are correct?

  1. A

    i & 1

  2. B

    i & 1 followed by 2

  3. C

    i & 1 followed by 2 & 3

  4. D

    ii & 2

  5. E

    ii & 2 & 3

  6. F

    ii & 3

Show answer

Correct answer

  • D

    ii & 2

See the whole paperTake it as a mock test

Question 13 of 26 in the IIT Madras BS Introduction to Big Data (Intro to Big Data) Quiz 2 paper sat on 3 Dec 2023, in the September 2023 term (IIT M DEGREE AN2 EXAM QDB2 03 Dec 2023). It carries 1 mark.

More questions from this paper

  1. Q1What best describes "big data"?
  2. Q2Which of these represent examples of divide-and-conquer?
  3. Q3Which of the following statements about Spark application architecture is correct?
  4. Q4Consider the problem of sorting a 1 petabyte file of numbers stored in a Hadoop cluster of 10 machines where the size o…
  5. Q5A website sees 1 Billion hits every month. The website owner wants to count average hits per customer in the latest mon…
  6. Q6An enterprise software designer wants to leverage the best of cloud to minimize the number of administrative overheads …
  7. Q7Consider an application that can scale from handling 1000 users to handling 100 million users by simply making copies o…
  8. Q8Linux is an example of an operating system. Which of the following is considered as the "operating system of a cluster …
  9. Q9Figure question
  10. Q10Figure question
  11. Q11What happens behind the scenes when the following code is run?
  12. Q12Consider a file “data.bin” which is formatted as follows: every data record is in the form of pairs of values of the fo…
  13. Q14What is the output of the following code?
  14. Q15What is the output of the following code when deployed on a spark cluster?
  15. Q16What is the output of the following code?
  16. Q17What is the purpose of the cache() function of an RDD in PySpark?
  17. Q18What role does metadata play in a Data Lake?
  18. Q19What option(s) best describe the differences between MapReduce and Spark?
  19. Q20Would IRCTC's railway ticket booking application be suitable for a serverless implementation?
  20. Q21You are provided with a Spark program that picks out a list of suspicious transactions based on the amount of the trans…
  21. Q22Which of the following types of data sources can you read successfully without missing data using a program that extrac…
  22. Q23Which of the following is true?
  23. Q24Which of the following statements is/ are true for Google Cloud Functions?
  24. Q25Which of the following is True ?
  25. Q26Which of the following is True about YARN ?