Error in installing DeepRepeat

[Cesco16]

Hi there,

I would like to use try this interesting software.
I tried to install it using anaconda. I followed the instructions on the github page but, after typing

h5c++ -03 -std=c++11 [...]

I got the following error:

BamReader.c: In member function ‘int BamReadOption::set_w_qry_seq(bool)’:
BamReader.c:136:72: warning: no return statement in function returning non-void [-Wreturn-type]
int BamReadOption::set_w_qry_seq(bool mqryseq){ m_w_qry_seq = mqryseq; }
^
BamReader.c: In member function ‘int BamReadOption::set_w_map_detail(bool, bool, char*, int)’:
BamReader.c:143:1: warning: no return statement in function returning non-void [-Wreturn-type]
}
^
BamReader.c: In member function ‘int BamReadOption::set_w_qry_qual(bool)’:
BamReader.c:145:74: warning: no return statement in function returning non-void [-Wreturn-type]
int BamReadOption::set_w_qry_qual(bool mqryqual){ m_w_qry_qual=mqryqual; }
^
In file included from Fast5Reader.c:1:
Fast5Reader.h: In member function ‘virtual int F5Reader::readF5Channel()’:
Fast5Reader.h:133:33: warning: no return statement in function returning non-void [-Wreturn-type]
virtual int readF5Channel(){;}
^
Fast5Reader.h: In member function ‘virtual int F5Reader::readF5Fastq()’:
Fast5Reader.h:134:31: warning: no return statement in function returning non-void [-Wreturn-type]
virtual int readF5Fastq(){;}
^
Fast5Reader.h: In member function ‘virtual int F5Reader::readF5Events()’:
Fast5Reader.h:135:32: warning: no return statement in function returning non-void [-Wreturn-type]
virtual int readF5Events(){;}
^
Fast5Reader.h: In member function ‘virtual int F5Reader::readF5Signals()’:
Fast5Reader.h:136:33: warning: no return statement in function returning non-void [-Wreturn-type]
virtual int readF5Signals(){;}
^
Fast5Reader.c: In member function ‘int F5Reader::init(const char*, RunOption*)’:
Fast5Reader.c:219:1: warning: no return statement in function returning non-void [-Wreturn-type]
}
^
Fast5Reader.c: In member function ‘int F5Reader::del()’:
Fast5Reader.c:224:1: warning: no return statement in function returning non-void [-Wreturn-type]
}
^
Fast5Reader.c: In member function ‘int F5Reader::resetFast5File(const char*, RunOption*)’:
Fast5Reader.c:247:1: warning: no return statement in function returning non-void [-Wreturn-type]
}
^
Fast5Reader.c: In member function ‘int F5Reader::normalize_signal()’:
Fast5Reader.c:893:1: warning: control reaches end of non-void function [-Wreturn-type]
}
^
In file included from RepeatFeatExtract.h:7,
from RepeatFeatExtract.c:5:
Fast5Reader.h: In member function ‘virtual int F5Reader::readF5Channel()’:
Fast5Reader.h:133:33: warning: no return statement in function returning non-void [-Wreturn-type]
virtual int readF5Channel(){;}
^
Fast5Reader.h: In member function ‘virtual int F5Reader::readF5Fastq()’:
Fast5Reader.h:134:31: warning: no return statement in function returning non-void [-Wreturn-type]
virtual int readF5Fastq(){;}
^
Fast5Reader.h: In member function ‘virtual int F5Reader::readF5Events()’:
Fast5Reader.h:135:32: warning: no return statement in function returning non-void [-Wreturn-type]
virtual int readF5Events(){;}
^
Fast5Reader.h: In member function ‘virtual int F5Reader::readF5Signals()’:
Fast5Reader.h:136:33: warning: no return statement in function returning non-void [-Wreturn-type]
virtual int readF5Signals(){;}
^
In file included from RepeatFeatExtract.h:7,
from genomic1FE.c:14:
Fast5Reader.h: In member function ‘virtual int F5Reader::readF5Channel()’:
Fast5Reader.h:133:33: warning: no return statement in function returning non-void [-Wreturn-type]
virtual int readF5Channel(){;}
^
Fast5Reader.h: In member function ‘virtual int F5Reader::readF5Fastq()’:
Fast5Reader.h:134:31: warning: no return statement in function returning non-void [-Wreturn-type]
virtual int readF5Fastq(){;}
^
Fast5Reader.h: In member function ‘virtual int F5Reader::readF5Events()’:
Fast5Reader.h:135:32: warning: no return statement in function returning non-void [-Wreturn-type]
virtual int readF5Events(){;}
^
Fast5Reader.h: In member function ‘virtual int F5Reader::readF5Signals()’:
Fast5Reader.h:136:33: warning: no return statement in function returning non-void [-Wreturn-type]
virtual int readF5Signals(){;}
^
/usr/bin/ld: Fast5Reader.o: in function F5Reader::_init_(char const*, RunOption*)': Fast5Reader.c:(.text+0x16e7): undefined reference to H5::H5File::H5File(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, unsigned int, H5::FileCreatPropList const&, H5::FileAccPropList const&)'
/usr/bin/ld: Fast5Reader.o: in function F5Reader::_readF5Channel_(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)': Fast5Reader.c:(.text+0x3f34): undefined reference to H5::H5Location::openGroup(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const'
/usr/bin/ld: Fast5Reader.c:(.text+0x3fb0): undefined reference to H5::H5Object::openAttribute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const' /usr/bin/ld: Fast5Reader.c:(.text+0x405d): undefined reference to H5::H5Object::openAttribute(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const'
/usr/bin/ld: Fast5Reader.c:(.text+0x413b): undefined reference to H5::H5Object::openAttribute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const' /usr/bin/ld: Fast5Reader.c:(.text+0x421c): undefined reference to H5::H5Object::openAttribute(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const'
/usr/bin/ld: Fast5Reader.o: in function F5Reader::_readF5Fastq_(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)': Fast5Reader.c:(.text+0x49d5): undefined reference to H5::H5Location::openDataSet(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const'
/usr/bin/ld: Fast5Reader.c:(.text+0x4b06): undefined reference to H5::DataSet::read(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, H5::DataType const&, H5::DataSpace const&, H5::DataSpace const&, H5::DSetMemXferPropList const&) const' /usr/bin/ld: Fast5Reader.o: in function F5Reader::readF5Events(std::__cxx11::basic_string<char, std::char_traits, std::allocator >)':
Fast5Reader.c:(.text+0x5f0f): undefined reference to H5::H5Location::openDataSet(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const' /usr/bin/ld: Fast5Reader.c:(.text+0x60b1): undefined reference to H5::CompType::insertMember(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, unsigned long, H5::DataType const&) const'
/usr/bin/ld: Fast5Reader.c:(.text+0x6145): undefined reference to H5::CompType::insertMember(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, H5::DataType const&) const' /usr/bin/ld: Fast5Reader.c:(.text+0x61d9): undefined reference to H5::CompType::insertMember(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, unsigned long, H5::DataType const&) const'
/usr/bin/ld: Fast5Reader.c:(.text+0x626d): undefined reference to H5::CompType::insertMember(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, H5::DataType const&) const' /usr/bin/ld: Fast5Reader.c:(.text+0x6326): undefined reference to H5::CompType::insertMember(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, unsigned long, H5::DataType const&) const'
/usr/bin/ld: Fast5Reader.o:Fast5Reader.c:(.text+0x63c5): more undefined references to H5::CompType::insertMember(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, H5::DataType const&) const' follow /usr/bin/ld: Fast5Reader.o: in function F5Reader::readF5Signals(std::__cxx11::basic_string<char, std::char_traits, std::allocator >)':
Fast5Reader.c:(.text+0x6b7c): undefined reference to H5::H5Location::openDataSet(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const' /usr/bin/ld: Fast5Reader.c:(.text+0x6bb1): undefined reference to H5::DataSet::read(std::__cxx11::basic_string<char, std::char_traits, std::allocator >&, H5::DataType const&, H5::DataSpace const&, H5::DataSpace const&, H5::DSetMemXferPropList const&) const'
/usr/bin/ld: Fast5Reader.o: in function F5Reader1::readF5Signals()': Fast5Reader.c:(.text+0x70f8): undefined reference to H5::H5Location::openGroup(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const'
/usr/bin/ld: Fast5Reader.c:(.text+0x71e6): undefined reference to `H5::H5Location::getObjnameByIdx[abi:cxx11](unsigned long long) const'
collect2: error: ld returned 1 exit status

Any tips on how to fix it?
Thank you very much!

[liuqianhn]

@Cesco16 , Thank you for being interested in the tools.
Here are my suggestions:

1. You can try the docker docker run --rm genomicslab/deeprepeat:0.1.3 where you do not need to compile DeepRepeat.
2. The issues are from 2 sources: one is compiling option----please remove something like -W*error* from the compiling command; the other is the installation issue of hdf5. Please show the whole compiling command for investigating the detail (need compiling options or installation issue of hdf5?)

[Cesco16]

Hi, thank you!
I would like to solve the problem without using docker (if possible).
This is the whole command:

h5c++ -O3 -std=c++11 -I $DR_conda_base/include -L$DR_conda_base/lib -lhts -o genomic1FE ComFunction.c ComOption.c BamReader.c Fast5Index.c Fast5Reader.c RepeatFeatExtract.c genomic1FE.c $DR_conda_base//lib/libhdf5_hl_cpp.a $DR_conda_base//lib/libhdf5_cpp.a $DR_conda_base//lib/libhdf5_hl.a $DR_conda_base//lib/libhdf5.a -lz -ldl -lpthread

I just followed the installation instructions on github.
Thank you very much!

[Cesco16]

I tried to repeat the instructions in the github page.
Now, I created the conda environment using the environment.yml file provided.
Now, I got a different error message:

condapath/envs/py36deeprepeat/bin/h5c++: 1: eval: condapath/envs/py36deeprepeat/bin/x86_64-conda_cos6-linux-gnu-c++: not found

In fact, if I go to py36deeprepeat folder I found :

x86_64-conda_cos7-linux-gnu-c++

How to solve this? Maybe it is more simple than previous issue.
Thanks

[liuqianhn]

@Cesco16 It seems that it is the installation issue of gxx_linux-64. please try to reinstall "conda install gxx_linux-64".

[Cesco16]

Hi,
unfortunately, it did not work.
Which version of anaconda or miniconda do you use to run DeepRepeat?

[liuqianhn]

@Cesco16 I tested anaconda3 with python 3.6 on Ubuntu 20.04.4 LTS and CentOS 6 (not sure, @umahsn could you please help to check OS version on CHOP cluster?).

[umahsn]

Currently CHOP cluster is running "Red Hat Enterprise Linux 8.5 (Ootpa)", and I think it has been the OS for at least a year now.

[liuqianhn]

@umahsn Thank you very much. DeepRepeat was developed more than 1 year ago and CHOP cluster has the older OS(I remember the old OS is centos 6 or 7, but forgot which one).

[Cesco16]

Hi,
I tried to install it using a miniconda version of few years ago.
It gave me some warnings but no errors during compiling operations.
When I tried it to a sample, it gives me the following errors:

python DeepRepeat/bin/DeepRepeat.py Detect --bam Repeats/my_file.bam --o Repeats/ --repeat "chr19:45770061-45770481:TGC:3" --f5folder Repeats/fast5_pass

/path/miniconda3/envs/py36deeprepeat2/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/path/miniconda3/envs/py36deeprepeat2/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/path/miniconda3/envs/py36deeprepeat2/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/path/miniconda3/envs/py36deeprepeat2/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/path/miniconda3/envs/py36deeprepeat2/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/path/miniconda3/envs/py36deeprepeat2/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
The following options are used (included default):
UniqueID (mpred);
bam (Repeats/my_file.bam);
basecalled_path (workspace/pass/);
f5config (/path/DeepRepeat/bin/data/config/fast5_path.config);
f5folder (Repeats/fast5_pass);
f5i (Repeats/fast5_pass/f5.f5index);
f5i_basefile (f5);
feature_num (50);
label_size (4);
merge_gap (4.5);
mod_path (None);
mod_version (2);
multif5 (0);
nb_size (3);
nbsize (-1.5);
outlog (0);
outputfolder (Repeats/);
pcr (False);
repeat (chr19:45770061-45770481:TGC:3);
repeat_name (m_repeat);
repeat_pat (TGC);
rpg (/path/DeepRepeat/bin/data/trf.v0.bed);
summary_file (Repeats/fast5_pass/sequencing_summary.txt);

Generating f5index: /path/DeepRepeat/bin/scripts/IndexF5files Repeats/fast5_pass workspace/pass/ f5 Repeats/fast5_pass/sequencing_summary.txt 0
Options to be used:
base_index_path = Repeats/fast5_pass
basecalled_path = workspace/pass/
uniq_id = f5
seq_sum = Repeats/fast5_pass/sequencing_summary.txt
multifast5 = 0

The input sequencing summary are:
Repeats/fast5_pass/sequencing_summary.txt
The basecall path is <workspace/pass/>
The output index file is <Repeats/fast5_pass/f5.f5index>
<Repeats/fast5_pass/workspace/pass//.fast5> has 0 fast5 files

Base index is saved in <Repeats/fast5_pass/f5.f5index>.
Please keep this file together with its indexed fast5 folders.
You might need to provide this file for feature extraction and repeat detection.
Generating features: /path/DeepRepeat/bin/scripts/genomic1FE Repeats/my_file.bam Repeats//mpred.fs Repeats/fast5_pass/ Repeats/fast5_pass/f5.f5index chr19:45770061-45770481:TGC:3
/path/DeepRepeat/bin/data/trf.v0.bed /path/DeepRepeat/bin/data/config/fast5_path.config -1.5 100
p_f5_conf_file /path/DeepRepeat/bin/data/config/fast5_path.config
Input = Repeats/my_file.bam Repeats//mpred.fs
free(): invalid size
Aborted
Error! Cannot generate fs file: Repeats//mpred.fs

I did not understand

[liuqianhn]

Hi @Cesco16 The warning will not cause the error. The issue is that <Repeats/fast5_pass/workspace/pass//.fast5> has 0 fast5 files: it seems that no fast5 are found.

1. What is in the folder of Repeats/fast5_pass
2. Are you able to run a simple example from https://github.com/WGLab/DeepRepeat/blob/master/docs/Reproducibility.md

[Cesco16]

Hi @liuqianhn
In folder Repeats/fast5_pass I have 484 .fast5 files (obtained from MinION MK1C) and the sequencing_summary.txt file.
Ok, I can try to run an example and I will keep you updated.
Thank you!

[liuqianhn]

@Cesco16 Thank you for your update. It will be better you can share the folder structure under Repeats/fast5_pass, by default, it assumes that there is a subfolder workspace/pass/ under the basefold for fast5 files.

[Cesco16]

Hi @liuqianhn ,

Ok, I will try also to put the subfolder workspace/pass/

Anyway, I tried to run an example of the ones provided.
I downloaded the well-trained model and I created the folder structure shown in the github page.
I run the following command:

python DeepRepeat/bin/DeepRepeat.py Detect --gn hx1 --TempRem 0 --epchon 200 --repeat_relax_bp 20 --UniqueID AGAT_chr6_145272942-145273029 --is_pcr 0 --repeatName AGAT_chr6_145272942-145273029 --repeat chr6:145272942-145273029:AGAT:4 --f5i na12878_loci/AGAT_chr6_145272942-145273029/na.f5index --o test_op/AGAT_chr6_145272942-145273029 --bam na12878_loci/AGAT_chr6_145272942-145273029/AGAT_chr6_145272942-145273029.bam --f5folder na12878_loci/AGAT_chr6_145272942-145273029 --algLen 500 2>&1 | grep "GMM:"

It seems it was successful.
But the output folder test_op/AGAT_chr6_145272942-145273029 is empty.
What could be the error?

[liuqianhn]

please run python DeepRepeat/bin/DeepRepeat.py Detect --gn hx1 --TempRem 0 --epchon 200 --repeat_relax_bp 20 --UniqueID AGAT_chr6_145272942-145273029 --is_pcr 0 --repeatName AGAT_chr6_145272942-145273029 --repeat chr6:145272942-145273029:AGAT:4 --f5i na12878_loci/AGAT_chr6_145272942-145273029/na.f5index --o test_op/AGAT_chr6_145272942-145273029 --bam na12878_loci/AGAT_chr6_145272942-145273029/AGAT_chr6_145272942-145273029.bam --f5folder na12878_loci/AGAT_chr6_145272942-145273029 --algLen 500 , and you will see output from stdout.

[Cesco16]

Hi @liuqianhn , done, this is the result (error):

The following options are used (included default):
UniqueID (AGAT_chr6_145272942-145273029);
bam (na12878_loci/AGAT_chr6_145272942-145273029/AGAT_chr6_145272942-145273029.bam);
basecalled_path (workspace/pass/);
f5config (/path/DeepRepeat/bin/data/config/fast5_path.config);
f5folder (na12878_loci/AGAT_chr6_145272942-145273029);
f5i (na12878_loci/AGAT_chr6_145272942-145273029/na.f5index);
f5i_basefile (na);
feature_num (50);
label_size (4);
merge_gap (6.0);
mod_path (None);
mod_version (2);
multif5 (0);
nb_size (3);
nbsize (-1.5);
outlog (0);
outputfolder (test_op/AGAT_chr6_145272942-145273029);
pcr (True);
repeat (chr6:145272942-145273029:AGAT:4);
repeat_name (AGAT_chr6_145272942-145273029);
repeat_pat (AGAT);
rpg (/path/DeepRepeat/bin/data/trf.v0.bed);
summary_file (na12878_loci/AGAT_chr6_145272942-145273029/sequencing_summary.txt);

Generating features: path/DeepRepeat/bin/scripts/genomic1FE na12878_loci/AGAT_chr6_145272942-145273029/AGAT_chr6_145272942-145273029.bam test_op/AGAT_chr6_145272942-145273029/AGAT_chr6_145272942-145273029.fs na12878_loci/AGAT_chr6_145272942-145273029/ na12878_loci/AGAT_chr6_145272942-145273029/na.f5index chr6:145272942-145273029:AGAT:4 path/DeepRepeat/bin/data/trf.v0.bed path/DeepRepeat/bin/data/config/fast5_path.config -1.5 500
p_f5_conf_file path/DeepRepeat/bin/data/config/fast5_path.config
Input = na12878_loci/AGAT_chr6_145272942-145273029/AGAT_chr6_145272942-145273029.bam test_op/AGAT_chr6_145272942-145273029/AGAT_chr6_145272942-145273029.fs
free(): invalid size
Aborted
Error! Cannot generate fs file: test_op/AGAT_chr6_145272942-145273029/AGAT_chr6_145272942-145273029.fs