loading system configuration
loading override configuration from /users/willend/mxtest/mctest/../mccodelib/mcxtrace-test/McXtrace_DUAL_A100_DMSC/mccode_config.json
INFO: Using directory: "4"
INFO: Using existing c-file: ./Test_SAXS.c
INFO: Using existing binary: ./Test_SAXS.out
INFO: ===
--------------------------------------------------------------------------
WARNING: No preset parameters were found for the device that Open MPI
detected:

  Local host:            r6n22
  Device name:           mlx5_0
  Device vendor ID:      0x02c9
  Device vendor part ID: 4123

Default device parameters will be used, which may result in lower
performance.  You can edit any of the files specified by the
btl_openib_device_param_files MCA parameter to set values for your
device.

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_no_device_params_found to 0.
--------------------------------------------------------------------------
Simulation 'Test_SAXS' (/users/willend/xTESTS/2022-12-01/McXtrace_DUAL_A100_DMSC_5e7/Test_SAXS/Test_SAXS.instr): running on 2 nodes (master is 'r6n22.esss.dk', MPI version 3.1).
Have found 2 GPU devices on rank 0. Will use device 0.
Have found 2 GPU devices on rank 1. Will use device 1.
[Test_SAXS] Initialize
[Test_SAXS] Initialize
3v03.pdb PDB file located... 
3v03.pdb PDB file located... 
Calculating scattering from 1160 residues...
INFO (Sample7): Creating protein structure from file 3v03.pdb
Calculating scattering from 1160 residues...
INFO (Sample7): Creating protein structure from file 3v03.pdb
6lyz.pdb PDB file located... 
Calculating scattering from 129 residues...
INFO (Sample11): Creating protein structure from file 6lyz.pdb
6lyz.pdb PDB file located... 
Calculating scattering from 129 residues...
INFO (Sample11): Creating protein structure from file 6lyz.pdb
[r6n22:3414511] *** Process received signal ***
[r6n22:3414511] Signal: Segmentation fault (11)
[r6n22:3414511] Signal code: Address not mapped (1)
[r6n22:3414511] Failing at address: (nil)
[r6n22:3414511] *** End of error message ***
[r6n22:3414510] *** Process received signal ***
[r6n22:3414510] Signal: Segmentation fault (11)
[r6n22:3414510] Signal code: Address not mapped (1)
[r6n22:3414510] Failing at address: (nil)
[r6n22:3414510] *** End of error message ***
/users/willend/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../bin/acc_gpu_bind: line 12: 3414510 Segmentation fault      (core dumped) "$@"
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[28508,1],1]
  Exit code:    139
--------------------------------------------------------------------------
[r6n22.esss.dk:3414504] 1 more process has sent help message help-mpi-btl-openib.txt / no device params found
[r6n22.esss.dk:3414504] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages