waali/dd-tp4-spark
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
16 commits 1 branch 0 tags 445 KiB
Find a file
Exact
username 9636e3cd75 feat: autocomplete algo
2024-10-13 22:51:34 +02:00
data 📝 update README and file struct 2024-10-09 18:50:17 +02:00
src/main/scala feat: autocomplete algo 2024-10-13 22:51:34 +02:00
.gitignore 📝 update README and file struct 2024-10-09 18:50:17 +02:00
AOL_README.txt feat: repetition percent 2024-10-08 16:27:20 +02:00
build.sbt 📝 update README + rename build .jar 2024-10-09 19:01:24 +02:00
README.md feat: autocomplete algo 2024-10-13 22:51:34 +02:00
TP-Spark-V2.pdf 📝 PDF 2024-10-07 11:47:08 +02:00

README.md

DD TP4 Spark

Setup

1. Download Spark in project

2. Set Spark path variable

SPARK_PATH=/home/username/workspace/DD/dd-tp4-spark/spark-3.5.3-bin-hadoop3

3. Import data

. root 
    | README.md 
    | src 
    | spark-3.5.3-bin-hadoop3 
    | data 
        | AOL-01.txt                < imported data
        | README.md                 < imported data

Word count

sbt package
$SPARK_PATH/bin/spark-submit \
  --class "WordCount" \
  --master local[4] \
  target/scala-2.12/dd-spark-tp_2.12-1.0.jar

AOL

sbt package
$SPARK_PATH/bin/spark-submit \
--class "AOL" \
--master local[4] \
target/scala-2.12/dd-spark-tp_2.12-1.0.jar