我有一個包含許多文本檔案的檔案夾，我必須在一個 RDD 中讀取這些檔案并保存檔案名和上面的文字
例子 ：
doc1.txt :
" hello my name sam "
doc2.txt :
"hello world"
我需要傳遞檔案夾路徑，結果是：
（你好，doc1），（my，doc1），（world，doc2），......等
我試過這個：
val rddWhole = spark.sparkContext.wholeTextFiles("C:/tmp/files/*")
rddWhole.foreach(f=>{
println(f._1 "=>" f._2)
})
但它將檔案中的整個文本作為一個字串處理，任何人都知道我該如何解決它？
uj5u.com熱心網友回復：
根據我的假設，您希望提取檔案中的每個單詞，并將其與包含該單詞的檔案名相結合。正如您所提到的，spark 將檔案的全部內容作為單個字串提供給您。就像這是檔案內容：
hello
my name is
John Doe
你得到的價值是：
val fileString = "hello\nmy name is\nJohn Doe"
正確的？所以你需要用任意數量的空格或換行符分割字串值，如下所示：
val wordsSeparated = fileString.split("\\s |\\n ") // \\s means space, \\n means new line (in regexes, character escaping and stuff)
所以最后，你需要這樣的東西：
rddWhole.foreach { f =>
f._2.split("\\s |\\n ").foreach(word => println(f._1 " => " word))
}
這將是結果：
file:/tmp/spark-test/two.txt => and
file:/tmp/spark-test/two.txt => this
file:/tmp/spark-test/two.txt => would
file:/tmp/spark-test/one.txt => so
file:/tmp/spark-test/one.txt => hello
file:/tmp/spark-test/one.txt => my
file:/tmp/spark-test/one.txt => name
file:/tmp/spark-test/one.txt => is
file:/tmp/spark-test/one.txt => John
file:/tmp/spark-test/one.txt => Doe
file:/tmp/spark-test/two.txt => be
file:/tmp/spark-test/two.txt => the
file:/tmp/spark-test/two.txt => second
file:/tmp/spark-test/two.txt => text
file:/tmp/spark-test/two.txt => file
轉載請註明出處，本文鏈接：https://www.uj5u.com/ruanti/515732.html