What version of MPI and gcc should I use to compile?

[GodloveD]

Can you please provide some guidance on what version of MPI / gcc you've used to compile and run bison?

[dpryan79]

It shouldn't much matter. I've generally used openMPI and gcc 5.3 or 5.4, but it shouldn't much matter. The Makefile will change a bit depending on the MPI source (openMPI, MPICH2, etc.), but that should be it.

[GodloveD]

Great. Can you give me the version number of openMPI too please? I'm seeing some unexpected behavior and I want to rule out differences in the build environment.

[dpryan79]

It looks like I most recently built it against 1.10.2

[GodloveD]

Thanks! I'll give it a go.

[dpryan79]

Feel free to post error messages if you run into any problems!

[GodloveD]

That didn't seem to help. Next question. Is there a specific version of bowtie2 I should be using?

[dpryan79]

I would try the most recent version, though really anything that's come out in the past few years should work. That's not needed for the installation, so if you're getting some sort of error at that point then that's not the cause. Just make sure some version of bowtie2 is in the PATH.

[GodloveD]

OK. I'm trying to run this on a Slurm cluster and I'm basically getting a different error every time I submit. I'm using data from your 2014 tutorial. Here is my script:

#!/bin/bash
echo ======
echo $HOSTNAME
echo ======
echo
module load bowtie/2-2.2.9 openmpi/1.10.3/gcc-5.3.0
cd /data/godlovedc/bison_test/bison_tutorial
mpiexec -n 3 /data/godlovedc/bison/62bf61f7/bison --directional -g genomes/E.Coli/ -1 reads/100_1.fq.gz -2 reads/100_2.fq.gz

Here is my submission command:

$ sbatch --ntasks=3 --ntasks-per-core=1 bisonjob

And here are the output files from 10 different runs (with ~10 different errors). Can you see something obvious I'm doing wrong?

======
cn3627
======

[-] Unloading openmpi 1.10.3 for GCC 5.3.0
[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn3627 has rank 0
cn3627 has rank 1
cn3627 has rank 2
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
fatal flex scanner internal error--end of buffer missed
[cn3627:27630] *** Process received signal ***
[cn3627:27630] Signal: Segmentation fault (11)
[cn3627:27630] Signal code: Invalid permissions (2)
[cn3627:27630] Failing at address: 0x2aaaac000000
[cn3627:27630] [ 0] /lib64/libpthread.so.0(+0xf7e0)[0x2aaaaaf617e0]
[cn3627:27630] [ 1] /lib64/libc.so.6(+0x79279)[0x2aaaab954279]
[cn3627:27630] [ 2] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-pal.so.13(opal_libevent2021_event_base_loop+0x7d9)[0x2aaaac1e18b9]
[cn3627:27630] [ 3] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-rte.so.12(+0x585fe)[0x2aaaabed95fe]
[cn3627:27630] [ 4] /lib64/libpthread.so.0(+0x7aa1)[0x2aaaaaf59aa1]
[cn3627:27630] [ 5] /lib64/libc.so.6(clone+0x6d)[0x2aaaab9c3aad]
[cn3627:27630] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 27630 on node cn3627 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
======
cn1816
======

[-] Unloading openmpi 1.10.3 for GCC 5.3.0
[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn1816 has rank 0
cn1816 has rank 1
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
[cn1816:17079] *** Process received signal ***
[cn1816:17079] Signal: Segmentation fault (11)
[cn1816:17079] Signal code: Address not mapped (1)
[cn1816:17079] Failing at address: 0x2aab92ede952
cn1816 has rank 2
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 17079 on node cn1816 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
======
cn1980
======

[-] Unloading openmpi 1.10.3 for GCC 5.3.0
[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn1980 has rank 0
cn1980 has rank 1
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
cn1980 has rank 2
[cn1980:17108] *** Process received signal ***
[cn1980:17108] Signal: Segmentation fault (11)
[cn1980:17108] Signal code: Address not mapped (1)
[cn1980:17108] Failing at address: 0x3c
[cn1980:17108] [ 0] --------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 17108 on node cn1980 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
======
cn1804
======

[-] Unloading openmpi 1.10.3 for GCC 5.3.0
[+] Loading openmpi 1.10.3 for GCC 5.3.0
Allocating space for 3000000000 characters
cn1804 has rank 2
cn1804 has rank 0
cn1804 has rank 1
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
[cn1804:49992] *** Process received signal ***
[cn1804:49992] Signal: Segmentation fault (11)
[cn1804:49992] Signal code: Address not mapped (1)
[cn1804:49992] Failing at address: 0x3c
--------------------------------------------------------------------------
An MPI process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your MPI job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

  Local host:          cn1804 (PID 49992)
  MPI_COMM_WORLD rank: 0

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------
[cn1804:49992] [ 0] /lib64/libpthread.so.0(+0xf7e0)[0x2aaaaaf617e0]
[cn1804:49992] [ 1] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-pal.so.13(opal_show_help_yylex+0x38a)[0x2aaaac1d38ca]
[cn1804:49992] [ 2] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-pal.so.13(opal_show_help_vstring+0x198)[0x2aaaac1d2c68]
[cn1804:49992] [ 3] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-rte.so.12(orte_show_help+0xb4)[0x2aaaabeb3494]
[cn1804:49992] [ 4] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libmpi.so.12(+0x7a0b2)[0x2aaaab1e90b2]
[cn1804:49992] [ 5] /lib64/libc.so.6(__libc_fork+0x55)[0x2aaaab987c65]
[cn1804:49992] [ 6] /lib64/libc.so.6(_IO_proc_open+0x137)[0x2aaaab9435b7]
[cn1804:49992] [ 7] /lib64/libc.so.6(popen+0x69)[0x2aaaab9438a9]
[cn1804:49992] [ 8] /data/godlovedc/bison/62bf61f7/bison[0x407080]
[cn1804:49992] [ 9] /lib64/libpthread.so.0(+0x7aa1)[0x2aaaaaf59aa1]
[cn1804:49992] [10] /lib64/libc.so.6(clone+0x6d)[0x2aaaab9c3aad]
[cn1804:49992] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 49992 on node cn1804 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
======
cn1805
======

[-] Unloading openmpi 1.10.3 for GCC 5.3.0
[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn1805 has rank 0
cn1805 has rank 1
cn1805 has rank 2
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
[cn1805:43889] *** Process received signal ***
[cn1805:43889] Signal: Segmentation fault (11)
[cn1805:43889] Signal code: Address not mapped (1)
[cn1805:43889] Failing at address: 0x2aab92ede952
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 43889 on node cn1805 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
======
cn1850
======

[-] Unloading openmpi 1.10.3 for GCC 5.3.0
[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn1850 has rank 0
cn1850 has rank 1
cn1850 has rank 2
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
fatal flex scanner internal error--end of buffer missed
[cn1850:31304] *** Process received signal ***
[cn1850:31304] Signal: Segmentation fault (11)
[cn1850:31304] Signal code: Address not mapped (1)
[cn1850:31304] Failing at address: 0x815000
[cn1850:31304] [ 0] /lib64/libpthread.so.0(+0xf7e0)[0x2aaaaaf617e0]
[cn1850:31304] [ 1] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-pal.so.13(opal_show_help_yylex+0x953)[0x2aaaac1d3e93]
[cn1850:31304] [ 2] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-pal.so.13(opal_show_help_vstring+0x198)[0x2aaaac1d2c68]
[cn1850:31304] [ 3] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-rte.so.12(orte_show_help+0xb4)[0x2aaaabeb3494]
[cn1850:31304] [ 4] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libmpi.so.12(+0x7a0b2)[0x2aaaab1e90b2]
[cn1850:31304] [ 5] /lib64/libc.so.6(__libc_fork+0x55)[0x2aaaab987c65]
[cn1850:31304] [ 6] /lib64/libc.so.6(_IO_proc_open+0x137)[0x2aaaab9435b7]
[cn1850:31304] [ 7] /lib64/libc.so.6(popen+0x69)[0x2aaaab9438a9]
[cn1850:31304] [ 8] /data/godlovedc/bison/62bf61f7/bison[0x407574]
[cn1850:31304] [ 9] /lib64/libpthread.so.0(+0x7aa1)[0x2aaaaaf59aa1]
[cn1850:31304] [10] /lib64/libc.so.6(clone+0x6d)[0x2aaaab9c3aad]
[cn1850:31304] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 31304 on node cn1850 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
======
cn2131
======

[-] Unloading openmpi 1.10.3 for GCC 5.3.0
[+] Loading openmpi 1.10.3 for GCC 5.3.0
Allocating space for 3000000000 characters
cn2131 has rank 1
cn2131 has rank 0
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
[cn2131:28366] *** Process received signal ***
[cn2131:28366] Signal: Segmentation fault (11)
[cn2131:28366] Signal code: Address not mapped (1)
[cn2131:28366] Failing at address: 0x3c
cn2131 has rank 2
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 28366 on node cn2131 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
======
cn2148
======

[-] Unloading openmpi 1.10.3 for GCC 5.3.0
[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn2148 has rank 0
cn2148 has rank 1
cn2148 has rank 2
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
[cn2148:16707] *** Process received signal ***
[cn2148:16707] Signal: Segmentation fault (11)
[cn2148:16707] Signal code: Address not mapped (1)
[cn2148:16707] Failing at address: 0x2aaa06e36b20
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 16707 on node cn2148 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
======
cn2150
======

[-] Unloading openmpi 1.10.3 for GCC 5.3.0
[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn2150 has rank 0
cn2150 has rank 2
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
[cn2150:18509] *** Process received signal ***
[cn2150:18509] Signal: Segmentation fault (11)
[cn2150:18509] Signal code: Address not mapped (1)
[cn2150:18509] Failing at address: 0x2aab92ede952
cn2150 has rank 1
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 18509 on node cn2150 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
======
cn2184
======

[-] Unloading openmpi 1.10.3 for GCC 5.3.0
[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn2184 has rank 0
cn2184 has rank 1
cn2184 has rank 2
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
[cn2184:28029] *** Process received signal ***
[cn2184:28029] Signal: Segmentation fault (11)
[cn2184:28029] Signal code: Invalid permissions (2)
[cn2184:28029] Failing at address: 0x2aaaad621008
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    mpi_init:warn-fork
from the file:
    help-mpi-runtime.txt
But I couldn't find that topic in the file.  Sorry!
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    mpi_init:warn-fork
from the file:
    help-mpi-runtime.txt
But I couldn't find that topic in the file.  Sorry!
--------------------------------------------------------------------------

[dpryan79]

That's a new one. What happens if you set -p 1? If that doesn't help then I'll have to have a look when I get into the office tomorrow.

[GodloveD]

Thanks for the suggestion. I ran 10 more jobs with the -p 1 option and here is the output. Still lots of errors and segfaults.

======
cn3119
======

[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn3119 has rank 0
cn3119 has rank 1
cn3119 has rank 2
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
--------------------------------------------------------------------------
An MPI process has executed an operation involving a call to the
An MPI process has executed an operation involving a call to the
other system errors; your MPI job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
data corruption.  The use of fork() (or system() or other calls that
data corruption.  The use of fork() (or system() or other calls that
The process that invoked fork was:
reate child processes) is strongly discouraged.
  Local host:          cn3119 (PID 19407)
and correctly survive a call to fork(), you may disable this warning

of its peer processes in the job will be killed properly.  You should
double check that everything has shut down cleanly.

  Reason:     (null)
  Local host: ▒▒M▒▒*
  PID:        8102928
--------------------------------------------------------------------------
[cn3119:19407] *** Process received signal ***
[cn3119:19407] Signal: Segmentation fault (11)
[cn3119:19407] Signal code: Invalid permissions (2)
[cn3119:19407] Failing at address: 0x2aaaac000000
[cn3119:19407] [ 0] /lib64/libpthread.so.0(+0xf7e0)[0x2aaaaaf617e0]
[cn3119:19407] [ 1] /lib64/libc.so.6(+0x79279)[0x2aaaab954279]
[cn3119:19407] [ 2] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-pal.so.13(opal_libevent2021_event_base_loop+0x7d9)[0x2aaaac1e18b9]
[cn3119:19407] [ 3] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-rte.so.12(+0x585fe)[0x2aaaabed95fe]
[cn3119:19407] [ 4] /lib64/libpthread.so.0(+0x7aa1)[0x2aaaaaf59aa1]
[cn3119:19407] [ 5] /lib64/libc.so.6(clone+0x6d)[0x2aaaab9c3aad]
[cn3119:19407] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 19407 on node cn3119 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
======
cn3295
======

[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn3295 has rank 0
cn3295 has rank 1
cn3295 has rank 2
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
[cn3295:28695] *** Process received signal ***
--------------------------------------------------------------------------
--------------------------------------------------------------------------
[cn3295:28695] Signal: Segmentation fault (11)
[cn3295:28695] Signal code: Address not mapped (1)
[cn3295:28695] Failing at address: 0x2c
[cn3295:28695] [ 0] /lib64/libpthread.so.0(+0xf7e0)[0x2aaaaaf617e0]
[cn3295:28695] [ 1] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-pal.so.13(opal_show_help_yylex+0x55b)[0x2aaaac1d3a9b]
[cn3295:28695] [ 2] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-pal.so.13(opal_show_help_vstring+0x221)[0x2aaaac1d2cf1]
[cn3295:28695] [ 3] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-rte.so.12(orte_show_help+0xb4)[0x2aaaabeb3494]
[cn3295:28695] [ 4] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libmpi.so.12(+0x7a0b2)[0x2aaaab1e90b2]
[cn3295:28695] [ 5] /lib64/libc.so.6(__libc_fork+0x55)[0x2aaaab987c65]
[cn3295:28695] [ 6] /lib64/libc.so.6(_IO_proc_open+0x137)[0x2aaaab9435b7]
[cn3295:28695] [ 7] /lib64/libc.so.6(popen+0x69)[0x2aaaab9438a9]
[cn3295:28695] [ 8] /data/godlovedc/bison/62bf61f7/bison[0x407080]
[cn3295:28695] [ 9] /lib64/libpthread.so.0(+0x7aa1)[0x2aaaaaf59aa1]
[cn3295:28695] [10] /lib64/libc.so.6(clone+0x6d)[0x2aaaab9c3aad]
[cn3295:28695] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 28695 on node cn3295 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
======
cn3365
======

[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn3365 has rank 0
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
cn3365 has rank 1
cn3365 has rank 2
[cn3365:22321] *** Process received signal ***
[cn3365:22321] Signal: Segmentation fault (11)
[cn3365:22321] Signal code: Address not mapped (1)
[cn3365:22321] Failing at address: 0x3c
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 22321 on node cn3365 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
======
cn3468
======

[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn3468 has rank 0
cn3468 has rank 2
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
cn3468 has rank 1
[cn3468:35400] *** Process received signal ***
--------------------------------------------------------------------------
An MPI process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your MPI job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

  Local host:          cn3468 (PID 35400)
  MPI_COMM_WORLD rank: 0

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------
[cn3468:35400] Signal: Segmentation fault (11)
[cn3468:35400] Signal code: Address not mapped (1)
[cn3468:35400] Failing at address: 0x2aab74ede55e
[cn3468:35400] [ 0] /lib64/libpthread.so.0(+0xf7e0)[0x2aaaaaf617e0]
[cn3468:35400] [ 1] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-pal.so.13(opal_show_help_yylex+0x1c1)[0x2aaaac1d3701]
[cn3468:35400] [ 2] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-pal.so.13(opal_show_help_vstring+0x198)[0x2aaaac1d2c68]
[cn3468:35400] [ 3] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-rte.so.12(orte_show_help+0xb4)[0x2aaaabeb3494]
[cn3468:35400] [ 4] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libmpi.so.12(+0x7a0b2)[0x2aaaab1e90b2]
[cn3468:35400] [ 5] /lib64/libc.so.6(__libc_fork+0x55)[0x2aaaab987c65]
[cn3468:35400] [ 6] /lib64/libc.so.6(_IO_proc_open+0x137)[0x2aaaab9435b7]
[cn3468:35400] [ 7] /lib64/libc.so.6(popen+0x69)[0x2aaaab9438a9]
[cn3468:35400] [ 8] /data/godlovedc/bison/62bf61f7/bison[0x407080]
[cn3468:35400] [ 9] /lib64/libpthread.so.0(+0x7aa1)[0x2aaaaaf59aa1]
[cn3468:35400] [10] /lib64/libc.so.6(clone+0x6d)[0x2aaaab9c3aad]
[cn3468:35400] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 35400 on node cn3468 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
======
cn3514
======

[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn3514 has rank 0
cn3514 has rank 2
cn3514 has rank 1
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
fatal flex scanner internal error--end of buffer missed
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    mpi_init:warn-fork
from the file:
    help-mpi-runtime.txt
But I couldn't find that topic in the file.  Sorry!
--------------------------------------------------------------------------
reads/100_1.fq.gz contained 100000 reads
======
cn3528
======

[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn3528 has rank 0
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
cn3528 has rank 1
cn3528 has rank 2
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    mpi_init:warn-fork
from the file:
    help-mpi-runtime.txt
But I couldn't find that topic in the file.  Sorry!
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    mpi_init:warn-fork
from the file:
    help-mpi-runtime.txt
But I couldn't find that topic in the file.  Sorry!
--------------------------------------------------------------------------
reads/100_2.fq.gz contained 100000 reads
reads/100_1.fq.gz contained 100000 reads
Reading in genomes/E.Coli/Escherichia_coli.GCA_000597845.1.23.dna.genome.fa
Finished genomes/E.Coli/Escherichia_coli.GCA_000597845.1.23.dna.genome.fa
Alignment metrics will be printed to reads/100_1.txt
Sending start to node 1
Sending start to node 2
Node 1 executing: bowtie2 -q --reorder  -p 1 --score-min 'L,-0.6,-0.6' --norc -x genomes/E.Coli/bisulfite_genome/CT_conversion/BS_CT -1 reads/100_1.CT.fq.gz -2 reads/100_2.GA.fq.gz
Node 2 executing: bowtie2 -q --reorder  -p 1 --score-min 'L,-0.6,-0.6' --nofw -x genomes/E.Coli/bisulfite_genome/GA_conversion/BS_GA -1 reads/100_1.CT.fq.gz -2 reads/100_2.GA.fq.gz
--------------------------------------------------------------------------
An MPI process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your MPI job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

  Local host:          cn3528 (PID 6724)
  MPI_COMM_WORLD rank: 2

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------
Node 2 began sending reads @Thu May 11 08:54:55 2017
Node 1 began sending reads @Thu May 11 08:54:55 2017
Started slurping @Thu May 11 08:54:55 2017
[cn3528:06720] 1 more process has sent help message help-mpi-runtime.txt / mpi_init:warn-fork
[cn3528:06720] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
100000 reads; of these:
  100000 (100.00%) were paired; of these:
    53464 (53.46%) aligned concordantly 0 times
    44030 (44.03%) aligned concordantly exactly 1 time
    2506 (2.51%) aligned concordantly >1 times
    ----
    53464 pairs aligned concordantly 0 times; of these:
      4550 (8.51%) aligned discordantly 1 time
    ----
    48914 pairs aligned 0 times concordantly or discordantly; of these:
      97828 mates make up the pairs; of these:
        96925 (99.08%) aligned 0 times
        169 (0.17%) aligned exactly 1 time
        734 (0.75%) aligned >1 times
51.54% overall alignment rate
Node 1 finished sending reads @Thu May 11 08:55:09 2017
        (14.000000 sec elapsed)
Exiting worker node 1
Returning from worker node 1
100000 reads; of these:
  100000 (100.00%) were paired; of these:
    53748 (53.75%) aligned concordantly 0 times
    43760 (43.76%) aligned concordantly exactly 1 time
    2492 (2.49%) aligned concordantly >1 times
    ----
    53748 pairs aligned concordantly 0 times; of these:
      4509 (8.39%) aligned discordantly 1 time
    ----
    49239 pairs aligned 0 times concordantly or discordantly; of these:
      98478 mates make up the pairs; of these:
        97582 (99.09%) aligned 0 times
        171 (0.17%) aligned exactly 1 time
        725 (0.74%) aligned >1 times
51.21% overall alignment rate
Node 2 finished sending reads @Thu May 11 08:55:09 2017
        (14.000000 sec elapsed)
Finished slurping @Thu May 11 08:55:09 2017
        (14.000000 seconds elapsed)
Exiting worker node 2
Returning from worker node 2
======
cn3542
======

[+] Loading openmpi 1.10.3 for GCC 5.3.0
Allocating space for 3000000000 characters
cn3542 has rank 0
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
cn3542 has rank 2
cn3542 has rank 1
fatal flex scanner internal error--end of buffer missed
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    mpi_init:warn-fork
from the file:
    help-mpi-runtime.txt
But I couldn't find that topic in the file.  Sorry!
--------------------------------------------------------------------------
reads/100_2.fq.gz contained 100000 reads
======
cn3129
======

[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn3129 has rank 0
cn3129 has rank 1
cn3129 has rank 2
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
[cn3129:19241] *** Process received signal ***
[cn3129:19241] Signal: Segmentation fault (11)
[cn3129:19241] Signal code: Address not mapped (1)
[cn3129:19241] Failing at address: 0x20
--------------------------------------------------------------------------
An MPI process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
perating in a condition that could result in memory corruption or
data corruption.  The use of fork() (or system() or other calls that


8F▒▒*
Local host:          cn3129 (PID 19241)
If you are *absolutely sure* that your application will successfully
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------
[cn3129:19241] [ 0] /lib64/libpthread.so.0(+0xf7e0)[0x2aaaaaf617e0]
[cn3129:19241] [ 1] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-pal.so.13(opal_show_help_yylex+0xc73)[0x2aaaac1d41b3]
[cn3129:19241] [ 2] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-pal.so.13(opal_show_help_vstring+0x198)[0x2aaaac1d2c68]
[cn3129:19241] [ 3] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libopen-rte.so.12(orte_show_help+0xb4)[0x2aaaabeb3494]
[cn3129:19241] [ 4] /usr/local/OpenMPI/1.10.3/gcc-5.3.0/lib/libmpi.so.12(+0x7a0b2)[0x2aaaab1e90b2]
[cn3129:19241] [ 5] /lib64/libc.so.6(__libc_fork+0x55)[0x2aaaab987c65]
[cn3129:19241] [ 6] /lib64/libc.so.6(_IO_proc_open+0x137)[0x2aaaab9435b7]
[cn3129:19241] [ 7] /lib64/libc.so.6(popen+0x69)[0x2aaaab9438a9]
[cn3129:19241] [ 8] /data/godlovedc/bison/62bf61f7/bison[0x407574]
[cn3129:19241] [ 9] /lib64/libpthread.so.0(+0x7aa1)[0x2aaaaaf59aa1]
[cn3129:19241] [10] /lib64/libc.so.6(clone+0x6d)[0x2aaaab9c3aad]
[cn3129:19241] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 19241 on node cn3129 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
======
cn3190
======

[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn3190 has rank 0
cn3190 has rank 1
cn3190 has rank 2
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    mpi_init:warn-fork
from the file:
    help-mpi-runtime.txt
But I couldn't find that topic in the file.  Sorry!
--------------------------------------------------------------------------
fatal flex scanner internal error--end of buffer missed
reads/100_2.fq.gz contained 100000 reads
======
cn3298
======

[+] Loading openmpi 1.10.3 for GCC 5.3.0
cn3298 has rank 0
cn3298 has rank 1
cn3298 has rank 2
Allocating space for 3000000000 characters
Will C->T convert reads/100_1.fq.gz and store the results in reads/100_1.CT.fq.gz.
Will G->A convert reads/100_2.fq.gz and store the results in reads/100_2.GA.fq.gz.
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    mpi_init:warn-fork
from the file:
    help-mpi-runtime.txt
But I couldn't find that topic in the file.  Sorry!
--------------------------------------------------------------------------
fatal flex scanner internal error--end of buffer missed
reads/100_2.fq.gz contained 100000 reads

[dpryan79]

I'll have to see if I can reproduce this locally then. Perhaps I tagged a bad release.

[dpryan79]

In the cluster in my office we're using Slurm with OpenMPI 1.10.2 and gcc 4.8.5. There, the following works:

#!/bin/bash
module load bowtie2 mpi slurm
mpiexec ~/bin/bison --directional -p 20 -g genomes/E.Coli/ -1 reads/100_1.fq.gz -2 reads/100_2.fq.gz

With the command sbatch -N 3 -c 20 -p bioinfo foo.sh I get a proper result. I'll try to get your version of GCC and openMPI installed and see if I can reproduce the issue with that.

[GodloveD]

What version of linux are you using and what glibc are you up to?

[dpryan79]

Our nodes are running CentOS7 I think, I don't recall what version of glibc that uses. If push comes to shove, I can try sending you some binaries to see if they work.

[GodloveD]

We're using CentOS 6 with glibc version 2.12. I'm guessing this may be the problem. I'm going to try building it in a Singularity container using CentOS 7 and see if that fixes the bug. But it might take a little time because I don't have any experience with MPI in a container yet.