Support DESK

Follow

H6.4 - matchIT Hub for Spark - Running on a local cluster

Previous Article matchIT Hub Index Next Article

Prerequisites

Download and install Apache Spark from https://spark.apache.org/.

Download and install Java from http://java.com/en/download/manual.jsp.

Native Libraries

The native library files should be placed in folder, e.g. /opt/matchithub.lib or /usr/local/lib64, that can be synced with all the processing nodes. Execute the following steps:

cd matchithub-spark/lib
sudo cp lib*.so /usr/local/lib64

Use rsync to sync the /usr/local/lib64 folder with all processing nodes in the cluster. Ensure that spark-env.sh contains these lines:

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib64
SPARK_YARN_USER_ENV="LD_LIBRARY_PATH=$LD_LIBRARY_PATH"

Running Jobs

Each of the sample application folders contains a run.sh example script showing a spark-submit command.

Previous Article matchIT Hub Index Next Article
Was this article helpful?
0 out of 0 found this helpful

0 Comments

Please sign in to leave a comment.