trouble downloading seqspark and I cant find seqspark.jar

[dhynasah]

I am having trouble downloading and building seqspark. The error I keep getting is SeqSpark.jar is not found. So If you can send me the file that would be great. Or let me know what I am doing wrong.
this is the error:
./install --prefix ~/software/seqspark --db-dir /jamesdh/691_project/seqspark
Use of uninitialized value $prefix in string eq at ./install line 34.
Use of uninitialized value $prefix in concatenation (.) or string at ./install line 43.
Use of uninitialized value $prefix in concatenation (.) or string at ./install line 53.
Will install into , database files will be downloaded to /jamesdh/691_project/seqspark
Building seqspark ...
sh: sbt: command not found
Finished building seqspark.
Cannot find target/scala-2.11/SeqSpark.jar, compiling failed? at ./install line 84, <$fh> line 48.
[jamesdh@dc01 seqspark]$ ./install seqspark-db
Use of uninitialized value $ENV{"USR"} in concatenation (.) or string at ./install line 48.
No db-dir specified, will use /user//seqspark.
Building seqspark ...
sh: sbt: command not found
Finished building seqspark.
Cannot find target/scala-2.11/SeqSpark.jar, compiling failed? at ./install line 84, <$fh> line 48.
[jamesdh@dc01 seqspark]$ ^C
[jamesdh@dc01 seqspark]$ dnf reinstall sbt
-bash: dnf: command not found
[jamesdh@dc01 seqspark]$ which sbt
which: no sbt in (/usr/local/sbin:/usr/local/bin:/usr/bin:/lab/bin:/opt/cuda/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl)

[zhangdi-devel]

Hi @dhynasah
I saw two problems here.
First, you don't have sbt on your machine. Could you please download it here: https://www.scala-sbt.org/
Second, the $prefix not initialized issue. May I know what OS/Distro and version are you using? The install script is just a perl script. Could you show me the perl version: perl --version.

Di

[dhynasah]

I am quite new to this. I am running this on my school account virtually through putty and I simulated a cluster through the school machines. and I was just following your tutorial. https://statgenetics.github.io/seqspark/.
so are you saying I should use a different version of the script?

[zhangdi-devel]

I mean I am not sure if the OS or the Perl version cause the $prefix issues. The script should work on most Linux distros with any Perl 5. So I want to know if you are using a mainstream Linux and Perl 5.
If you're working on a school cluster, you can ask the admin about the system/Perl versions, and if they can help you install the prerequisites.

[dhynasah]

Ok, Finally I got someone to reply at my school. The problem was they did not have sbt installed on the global system. They added it and then I was able to install seqspark. I did get a few warnings when it was installing because of the prefix errors. They didn't really answer my questions about that. but I think for the most part it's fine. thanks for the help!

[dhynasah]

Hi. So I am having trouble again. I had to skip over a few of the things in the tutorial. The reason in that my school doesn't have Hadoop installed. They have a "hadoop like system". in order to run my files on all of the spark nodes I launch spark first, and then run the file. But I am having trouble because I cannot use the Seqspark commands. I am in the directory where the demo.conf file is. So is there a way for me to configure my environment to have these commands? Any help you could provide will be greatly appreciated.

[zhangdi-devel]

Have you added this line to your ~/.bashrc:
export PATH=~/software/seqspark/bin:$PATH

After doing this, execute bash or re-login.

[dhynasah]

Thanks! I definitely got somewhere. The commands are being recognized. But I am now getting another error.
Error: seqspark no longer takes run mode option, please check the latest document.
Usage: seqspark conf [opts]
conf -- File containing the parameters
opts -- Spark options
what do I do in this situation?

[zhangdi-devel]

Sorry about that. I guess you are trying to run the SingleStudy mode here. Just remove SingleStudy from the command line. I'll update the doc accordingly.

[dhynasah]

Thanks. That helped. I was able to get a little bit further. Although I keep running into more problems.
This is the error I am getting now:
[jamesdh@dc01 seqspark_files]$ seqspark demo.conf
conf file: demo.conf
spark options:
19/04/05 12:27:49 WARN util.NativeCodeLoader: Unable to load native-hadoop libra ry for your platform... using builtin-java classes where applicable
19/04/05 12:27:49 WARN internal.Logging$class: Local jar /home/jamesdh/seqspark/ SeqSpark.jar does not exist, skipping.
19/04/05 12:27:49 WARN internal.Logging$class: Failed to load org.dizhang.seqspa rk.SeqSpark.
java.lang.ClassNotFoundException: org.dizhang.seqspark.SeqSpark
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubm it$$runMain(SparkSubmit.scala:810)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167 )
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scal a:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

it doesn't make sense because the SesSpark.jar file is here. Should I specify the path or something...although im not sure how to do that.
and then the org.dizhand.seqspark.SeqSpark...i dont know what that error means. im not sure if that is supposed to be a file, or if its something else... i looked for it and couldn't find it.

[zhangdi-devel]

I noticed an extra space between ${SPARK_HOME} and SeqSpark.jar. Could you check the seqspark script? It is a bash script located at ${SPARK_HOME}/bin. If you are not certain, run this command and paste the output: cat $(which seqspark)

[dhynasah]

#!/usr/bin/env bash

Usage() {
echo "Usage: seqspark conf [opts] "
echo " conf -- File containing the parameters "
echo " opts -- Spark options "
}

if [[ -z $1 ]]
then
echo "Error: you must specify the conf file"
Usage
exit
fi

if [ "$1" == "SingleStudy" ] || [ "$1" == "MetaAnalysis" ]
then
echo "Error: seqspark no longer takes run mode option, please check the latest document."
Usage
exit
fi

Conf=$1
shift
Opts=$@

if [ -z $SEQSPARK_HOME ]
then
SEQSPARK_HOME=$(dirname $(dirname $0))
fi
CONFDIR=${SEQSPARK_HOME}/conf

echo "conf file: $Conf"
echo "spark options: $Opts"

spark-submit --driver-class-path $CONFDIR
--class org.dizhang.seqspark.SeqSpark $Opts
$SEQSPARK_HOME/SeqSpark.jar $Conf

exit 0

[zhangdi-devel]

Looks fine. This script does nothing but prepares the classpath for java. Could you try to run the command manually like this:
spark-submit --driver-class-path /home/jamesdh/seqspark/conf --class org.dizhang.seqspark.SeqSpark /home/jamesdh/seqspark/SeqSpark.jar demo.conf

[dhynasah]

I tried it and I am still getting the same error. should I reinstall spark? maybe that's the problem. Also I moved my spark into the seqspark directory because the spark submit commands were not recognized in that directory. and then I added this to my bash file:
export PATH=~/seqspark/spark/bin:$PATH

but this was before I had the current error.