loading system configuration loading override configuration from /zhome/89/0/38697/mxtest/mctest/../mccodelib/mcxtrace-test/McXtrace_CPU_MPICC_KISS/mccode_config.json INFO: Using directory: "3" INFO: Using existing c-file: ./Test_SAXS.c INFO: Using existing binary: ./Test_SAXS.out INFO: === INFO: Using system default number of mpirun -np processes -------------------------------------------------------------------------- By default, for Open MPI 4.0 and later, infiniband ports on a device are not used by default. The intent is to use UCX for these devices. You can override this policy by setting the btl_openib_allow_ib MCA parameter to true. Local host: n-62-29-34 Local adapter: mlx4_0 Local port: 1 -------------------------------------------------------------------------- -------------------------------------------------------------------------- WARNING: There was an error initializing an OpenFabrics device. Local host: n-62-29-34 Local device: mlx4_0 -------------------------------------------------------------------------- Simulation 'Test_SAXS' (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_5e7/Test_SAXS/Test_SAXS.instr): running on 10 nodes (master is 'n-62-29-34', MPI version 3.1). [Test_SAXS] Initialize [Test_SAXS] Initialize 3v03.pdb PDB file located... 3v03.pdb PDB file located... [Test_SAXS] Initialize 3v03.pdb PDB file located... [Test_SAXS] Initialize Calculating scattering from 1160 residues... Calculating scattering from 1160 residues... INFO (Sample7): Creating protein structure from file 3v03.pdb INFO (Sample7): Creating protein structure from file 3v03.pdb [Test_SAXS] Initialize [Test_SAXS] Initialize 3v03.pdb PDB file located... [Test_SAXS] Initialize 3v03.pdb PDB file located... Calculating scattering from 1160 residues... INFO (Sample7): Creating protein structure from file 3v03.pdb [Test_SAXS] Initialize 3v03.pdb PDB file located... 3v03.pdb PDB file located... 3v03.pdb PDB file located... [Test_SAXS] Initialize 3v03.pdb PDB file located... Calculating scattering from 1160 residues... INFO (Sample7): Creating protein structure from file 3v03.pdb Calculating scattering from 1160 residues... INFO (Sample7): Creating protein structure from file 3v03.pdb Calculating scattering from 1160 residues... INFO (Sample7): Creating protein structure from file 3v03.pdb Calculating scattering from 1160 residues... Calculating scattering from 1160 residues... INFO (Sample7): Creating protein structure from file 3v03.pdb INFO (Sample7): Creating protein structure from file 3v03.pdb Calculating scattering from 1160 residues... INFO (Sample7): Creating protein structure from file 3v03.pdb [Test_SAXS] Initialize 3v03.pdb PDB file located... Calculating scattering from 1160 residues... INFO (Sample7): Creating protein structure from file 3v03.pdb 6lyz.pdb PDB file located... 6lyz.pdb PDB file located... 6lyz.pdb PDB file located... Calculating scattering from 129 residues... INFO (Sample11): Creating protein structure from file 6lyz.pdb Calculating scattering from 129 residues... INFO (Sample11): Creating protein structure from file 6lyz.pdb Calculating scattering from 129 residues... INFO (Sample11): Creating protein structure from file 6lyz.pdb # : [pid 24420] Signal 11 detected [proc 1] SIGSEGV (Mem Error) # Simulation: Test_SAXS (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_5e7/Test_SAXS/Test_SAXS.instr) # Breakpoint: [_Sample11_init] component Sample11=SAXSPDBFast() INITIALISE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../contrib/SAXSPDBFast.comp:430] 0.00 % ( 0.0/50000000.0) # : [pid 24419] Signal 11 detected [proc 0] SIGSEGV (Mem Error) # Simulation: Test_SAXS (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_5e7/Test_SAXS/Test_SAXS.instr) # Breakpoint: [_Sample11_init] component Sample11=SAXSPDBFast() INITIALISE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../contrib/SAXSPDBFast.comp:430] 0.00 % ( 0.0/50000000.0) # Date: Thu Dec 1 03:44:15 2022 # Started: Thu Dec 1 03:44:12 2022 # Last I/O Error: No such file or directory # Date: Thu Dec 1 03:44:15 2022 # Started: Thu Dec 1 03:44:12 2022 # : Simulation stop (abort). # Last I/O Error: No such file or directory # : Simulation stop (abort). # : [pid 24421] Signal 11 detected [proc 2] SIGSEGV (Mem Error) # Simulation: Test_SAXS (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_5e7/Test_SAXS/Test_SAXS.instr) # Breakpoint: [_Sample11_init] component Sample11=SAXSPDBFast() INITIALISE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../contrib/SAXSPDBFast.comp:430] 0.00 % ( 0.0/50000000.0) # Date: Thu Dec 1 03:44:15 2022 # Started: Thu Dec 1 03:44:12 2022 # Last I/O Error: No such file or directory # : Simulation stop (abort). -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- 6lyz.pdb PDB file located... 6lyz.pdb PDB file located... # : [pid 32041] Signal 15 detected [proc 9] SIGTERM (Termination) # Simulation: Test_SAXS (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_5e7/Test_SAXS/Test_SAXS.instr) # Breakpoint: [_Sample10_init] component Sample10=SAXSNanodiscsWithTagsFast() INITIALISE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../contrib/SAXSNanodiscsWithTagsFast.comp:261] 0.00 % ( 0.0/50000000.0) # Date: Thu Dec 1 03:44:15 2022 # Started: Thu Dec 1 03:44:12 2022 # : Finishing simulation (save results and exit) # : [pid 32041] Signal 11 detected [proc 9] SIGSEGV (Mem Error) # Simulation: Test_SAXS (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_5e7/Test_SAXS/Test_SAXS.instr) # Breakpoint: [_Monitor_save] component Monitor=PSD_monitor() SAVE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/PSD_monitor.comp:125] 0.00 % ( 0.0/50000000.0) # Date: Thu Dec 1 03:44:15 2022 # Started: Thu Dec 1 03:44:12 2022 # : Simulation stop (abort). # Last I/O Error: No such file or directory # Last I/O Error: No such file or directory # Last I/O Error: No such file or directory Calculating scattering from 129 residues... INFO (Sample11): Creating protein structure from file 6lyz.pdb # : [pid 32039] Signal 11 detected [proc 7] SIGSEGV (Mem Error) # Simulation: Test_SAXS (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_5e7/Test_SAXS/Test_SAXS.instr) # Breakpoint: [_Sample11_init] component Sample11=SAXSPDBFast() INITIALISE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../contrib/SAXSPDBFast.comp:430] 0.00 % ( 0.0/50000000.0) # Date: Thu Dec 1 03:44:15 2022 # Started: Thu Dec 1 03:44:12 2022 # : Simulation stop (abort). Calculating scattering from 129 residues... INFO (Sample11): Creating protein structure from file 6lyz.pdb # : [pid 32040] Signal 11 detected [proc 8] SIGSEGV (Mem Error) # Simulation: Test_SAXS (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_5e7/Test_SAXS/Test_SAXS.instr) # Breakpoint: [_Sample11_init] component Sample11=SAXSPDBFast() INITIALISE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../contrib/SAXSPDBFast.comp:430] 0.00 % ( 0.0/50000000.0) # Date: Thu Dec 1 03:44:15 2022 # Started: Thu Dec 1 03:44:12 2022 # : Simulation stop (abort). # : [pid 31145] Signal 15 detected [proc 6] SIGTERM (Termination) # Simulation: Test_SAXS (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_5e7/Test_SAXS/Test_SAXS.instr) # Breakpoint: [_Sample10_init] component Sample10=SAXSNanodiscsWithTagsFast() INITIALISE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../contrib/SAXSNanodiscsWithTagsFast.comp:261] 0.00 % ( 0.0/50000000.0) # Date: Thu Dec 1 03:44:15 2022 # Started: Thu Dec 1 03:44:12 2022 # : Finishing simulation (save results and exit) # : [pid 31145] Signal 11 detected [proc 6] SIGSEGV (Mem Error) # Simulation: Test_SAXS (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_5e7/Test_SAXS/Test_SAXS.instr) # Breakpoint: [_Monitor_save] component Monitor=PSD_monitor() SAVE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/PSD_monitor.comp:125] 0.00 % ( 0.0/50000000.0) # Date: Thu Dec 1 03:44:15 2022 # Started: Thu Dec 1 03:44:12 2022 # : Simulation stop (abort). # Last I/O Error: No such file or directory 6lyz.pdb PDB file located... Calculating scattering from 129 residues... INFO (Sample11): Creating protein structure from file 6lyz.pdb # : [pid 31144] Signal 11 detected [proc 5] SIGSEGV (Mem Error) # Simulation: Test_SAXS (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_5e7/Test_SAXS/Test_SAXS.instr) # Breakpoint: [_Sample11_init] component Sample11=SAXSPDBFast() INITIALISE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../contrib/SAXSPDBFast.comp:430] 0.00 % ( 0.0/50000000.0) # Date: Thu Dec 1 03:44:15 2022 # Started: Thu Dec 1 03:44:12 2022 # : Simulation stop (abort). # Last I/O Error: No such file or directory # : [pid 31143] Signal 15 detected [proc 4] SIGTERM (Termination) # Simulation: Test_SAXS (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_5e7/Test_SAXS/Test_SAXS.instr) # Breakpoint: [_Sample10_init] component Sample10=SAXSNanodiscsWithTagsFast() INITIALISE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../contrib/SAXSNanodiscsWithTagsFast.comp:261] 0.00 % ( 0.0/50000000.0) # Date: Thu Dec 1 03:44:15 2022 # Started: Thu Dec 1 03:44:12 2022 # : Finishing simulation (save results and exit) # : [pid 31143] Signal 11 detected [proc 4] SIGSEGV (Mem Error) # Simulation: Test_SAXS (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_5e7/Test_SAXS/Test_SAXS.instr) # Breakpoint: [_Monitor_save] component Monitor=PSD_monitor() SAVE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/PSD_monitor.comp:125] 0.00 % ( 0.0/50000000.0) # Date: Thu Dec 1 03:44:15 2022 # Started: Thu Dec 1 03:44:12 2022 # : Simulation stop (abort). # Last I/O Error: No such file or directory # Last I/O Error: No such file or directory 6lyz.pdb PDB file located... Calculating scattering from 129 residues... INFO (Sample11): Creating protein structure from file 6lyz.pdb # : [pid 31142] Signal 15 detected [proc 3] SIGTERM (Termination) # Simulation: Test_SAXS (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_5e7/Test_SAXS/Test_SAXS.instr) # Breakpoint: [_Sample11_init] component Sample11=SAXSPDBFast() INITIALISE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../contrib/SAXSPDBFast.comp:430] 0.00 % ( 0.0/50000000.0) # Date: Thu Dec 1 03:44:15 2022 # Started: Thu Dec 1 03:44:12 2022 # : Finishing simulation (save results and exit) # : [pid 31142] Signal 11 detected [proc 3] SIGSEGV (Mem Error) # Simulation: Test_SAXS (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_5e7/Test_SAXS/Test_SAXS.instr) # Breakpoint: [_Monitor_save] component Monitor=PSD_monitor() SAVE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/PSD_monitor.comp:125] 0.00 % ( 0.0/50000000.0) # Date: Thu Dec 1 03:44:15 2022 # Started: Thu Dec 1 03:44:12 2022 # : Simulation stop (abort). -------------------------------------------------------------------------- mpirun noticed that process rank 2 with PID 0 on node n-62-29-34 exited on signal 3 (Quit). -------------------------------------------------------------------------- [n-62-29-34:24413] 9 more processes have sent help message help-mpi-btl-openib.txt / ib port not selected [n-62-29-34:24413] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages [n-62-29-34:24413] 9 more processes have sent help message help-mpi-btl-openib.txt / error in device init