Join the discussion
Question 32/34
CORRECT TEXT
Problem Scenario 45 : You have been given 2 files , with the content as given Below
(spark12/technology.txt)
(spark12/salary.txt)
(spark12/technology.txt)
first,last,technology
Amit,Jain,java
Lokesh,kumar,unix
Mithun,kale,spark
Rajni,vekat,hadoop
Rahul,Yadav,scala
(spark12/salary.txt)
first,last,salary
Amit,Jain,100000
Lokesh,kumar,95000
Mithun,kale,150000
Rajni,vekat,154000
Rahul,Yadav,120000
Write a Spark program, which will join the data based on first and last name and save the joined results in following format, first Last.technology.salary
Problem Scenario 45 : You have been given 2 files , with the content as given Below
(spark12/technology.txt)
(spark12/salary.txt)
(spark12/technology.txt)
first,last,technology
Amit,Jain,java
Lokesh,kumar,unix
Mithun,kale,spark
Rajni,vekat,hadoop
Rahul,Yadav,scala
(spark12/salary.txt)
first,last,salary
Amit,Jain,100000
Lokesh,kumar,95000
Mithun,kale,150000
Rajni,vekat,154000
Rahul,Yadav,120000
Write a Spark program, which will join the data based on first and last name and save the joined results in following format, first Last.technology.salary
Correct Answer:
See the explanation for Step by Step Solution and configuration.
Explanation:
Solution :
Step 1 : Create 2 files first using Hue in hdfs.
Step 2 : Load all file as an RDD
val technology = sc.textFile(Msparkl2/technology.txt").map(e => e.splitf',")) val salary = sc.textFile("spark12/salary.txt").map(e => e.split("."))
Step 3 : Now create Key.value pair of data and join them.
val joined = technology.map(e=>((e(0),e(1)),e(2))).join(salary.map(e=>((e(0),e(1)),e(2))))
Step 4 : Save the results in a text file as below.
joined.repartition(1).saveAsTextFile("spark12/multiColumn Joined.txt")
Explanation:
Solution :
Step 1 : Create 2 files first using Hue in hdfs.
Step 2 : Load all file as an RDD
val technology = sc.textFile(Msparkl2/technology.txt").map(e => e.splitf',")) val salary = sc.textFile("spark12/salary.txt").map(e => e.split("."))
Step 3 : Now create Key.value pair of data and join them.
val joined = technology.map(e=>((e(0),e(1)),e(2))).join(salary.map(e=>((e(0),e(1)),e(2))))
Step 4 : Save the results in a text file as below.
joined.repartition(1).saveAsTextFile("spark12/multiColumn Joined.txt")
Add Comments
- Other Question (34q)
- Q1. CORRECT TEXT Problem Scenario 29 : Please accomplish the following exercises using HDFS co...
- Q2. CORRECT TEXT Problem Scenario 59 : You have been given below code snippet. val x = sc.para...
- Q3. CORRECT TEXT Problem Scenario 25 : You have been given below comma separated employee info...
- Q4. CORRECT TEXT Problem Scenario 50 : You have been given below code snippet (calculating an ...
- Q5. CORRECT TEXT Problem Scenario 95 : You have to run your Spark application on yarn with eac...
- Q6. CORRECT TEXT Problem Scenario 48 : You have been given below Python code snippet, with int...
- Q7. CORRECT TEXT Problem Scenario 37 : ABCTECH.com has done survey on their Exam Products feed...
- Q8. CORRECT TEXT Problem Scenario 40 : You have been given sample data as below in a file call...
- Q9. CORRECT TEXT Problem Scenario 9 : You have been given following mysql database details as ...
- Q10. CORRECT TEXT Problem Scenario 34 : You have given a file named spark6/user.csv. Data is gi...
- Q11. CORRECT TEXT Problem Scenario 41 : You have been given below code snippet. val aul = sc.pa...
- Q12. CORRECT TEXT Problem Scenario 53 : You have been given below code snippet. val a = sc.para...
- Q13. CORRECT TEXT Problem Scenario 51 : You have been given below code snippet. val a = sc.para...
- Q14. CORRECT TEXT Problem Scenario 2 : There is a parent organization called "ABC Group Inc", w...
- Q15. CORRECT TEXT Problem Scenario 19 : You have been given following mysql database details as...
- Q16. CORRECT TEXT Problem Scenario 56 : You have been given below code snippet. val a = sc.para...
- Q17. CORRECT TEXT Problem Scenario 93 : You have to run your Spark application with locally 8 t...
- Q18. CORRECT TEXT Problem Scenario 32 : You have given three files as below. spark3/sparkdir1/f...
- Q19. CORRECT TEXT Problem Scenario 18 : You have been given following mysql database details as...
- Q20. CORRECT TEXT Problem Scenario 22 : You have been given below comma separated employee info...
- Q21. CORRECT TEXT Problem Scenario 80 : You have been given MySQL DB with following details. us...
- Q22. CORRECT TEXT Problem Scenario 78 : You have been given MySQL DB with following details. us...
- Q23. CORRECT TEXT Problem Scenario 4: You have been given MySQL DB with following details. user...
- Q24. CORRECT TEXT Problem Scenario 20 : You have been given MySQL DB with following details. us...
- Q25. CORRECT TEXT Problem Scenario 68 : You have given a file as below. spark75/f ile1.txt File...
- Q26. CORRECT TEXT Problem Scenario 90 : You have been given below two files course.txt id,cours...
- Q27. CORRECT TEXT Problem Scenario 27 : You need to implement near real time solutions for coll...
- Q28. CORRECT TEXT Problem Scenario 81 : You have been given MySQL DB with following details. Yo...
- Q29. CORRECT TEXT Problem Scenario 74 : You have been given MySQL DB with following details. us...
- Q30. CORRECT TEXT Problem Scenario 10 : You have been given following mysql database details as...
- Q31. CORRECT TEXT Problem Scenario 79 : You have been given MySQL DB with following details. us...
- Q32. CORRECT TEXT Problem Scenario 45 : You have been given 2 files , with the content as given...
- Q33. CORRECT TEXT Problem Scenario 36 : You have been given a file named spark8/data.csv (type,...
- Q34. CORRECT TEXT Problem Scenario 5 : You have been given following mysql database details. us...
