Resources about the use of Spark:
https://amplab.cs.berkeley.edu/for-big-data-moores-law-means-better-decisions/
https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html
http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
https://github.com/rozim/ChessData (PGN source) https://en.wikipedia.org/wiki/Portable_Game_Notation (PGN wiki)
https://spark.apache.org/examples.html
Spark installation's issues on Window 10:
The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw- (on Windows)
How to fix it:
Be sure you have the right version of winutil.exe (its size should be > 100kb). The download link of it is here: http://letstalkspark.blogspot.com/2016/02/getting-started-with-spark-on-window-64.html
find out this line: e.g. - Download winutils.exe ( Put in C:\BigData\Hadoop\bin ) -- This is for 64-bit
Then execute this command:
C:\pathToWinutils\winutils.exe chmod -R 777 \tmp\hive
First task in Spark
(count word and sort them from default file e.g. README.md of spark package):
val textFile = sc.textFile("file:///spark/README.md")
val tokenizedFileData = textFile.flatMap(line => line.split(" "))
val countPrep = tokenizedFileData.map(word => (word, 1))
val counts = countPrep.reduceByKey((accumValue, newValue) => accumValue + newValue)
val sortedCounts = counts.sortBy(kvPair=>kvPair._2, false)
sortedCounts.saveAsTextFile("file:///PluralsightData/ReadMeWordCount")
Then checkout the files part-00000.txt and part-00001.txt stored in ReadMeWordCount folder in your FS.
Try other function on data set
tokenizedFileData.countByValue
