loading system configuration
loading override configuration from /users/willend/mxtest/mctest/../mccodelib/mcxtrace-test/McXtrace_DUAL_A100_DMSC/mccode_config.json
INFO: Using directory: "2"
INFO: Using existing c-file: ./Test_monitors.c
INFO: Using existing binary: ./Test_monitors.out
WARNING: Ignoring invalid parameter: "Test_monitors.instr"
INFO: ===
--------------------------------------------------------------------------
WARNING: No preset parameters were found for the device that Open MPI
detected:

  Local host:            r6n22
  Device name:           mlx5_0
  Device vendor ID:      0x02c9
  Device vendor part ID: 4123

Default device parameters will be used, which may result in lower
performance.  You can edit any of the files specified by the
btl_openib_device_param_files MCA parameter to set values for your
device.

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_no_device_params_found to 0.
--------------------------------------------------------------------------
Simulation 'Test_monitors' (/users/willend/xTESTS/2022-12-01/McXtrace_DUAL_A100_DMSC_5e7/Test_monitors/Test_monitors.instr): running on 2 nodes (master is 'r6n22.esss.dk', MPI version 3.1).
Have found 2 GPU devices on rank 0. Will use device 0.
Have found 2 GPU devices on rank 1. Will use device 1.
[Test_monitors] Initialize
[Test_monitors] Initialize
*** TRACE end *** 

Save [Test_monitors]
Detector: epsd1_I=1.81378 epsd1_ERR=0.000269354 epsd1_N=4.53446e+07 "epsd1_Sum.dat"
Detector: epsd1_I=0.0110486 epsd1_ERR=2.10225e-05 epsd1_N=276216 "epsd1_00.dat"
Detector: epsd1_I=0.0193634 epsd1_ERR=2.78305e-05 epsd1_N=484084 "epsd1_01.dat"
Detector: epsd1_I=0.0317173 epsd1_ERR=3.56187e-05 epsd1_N=792933 "epsd1_02.dat"
Detector: epsd1_I=0.0491261 epsd1_ERR=4.43288e-05 epsd1_N=1.22815e+06 "epsd1_03.dat"
Detector: epsd1_I=0.071388 epsd1_ERR=5.34371e-05 epsd1_N=1.7847e+06 "epsd1_04.dat"
Detector: epsd1_I=0.0974657 epsd1_ERR=6.2439e-05 epsd1_N=2.43664e+06 "epsd1_05.dat"
Detector: epsd1_I=0.12481 epsd1_ERR=7.06568e-05 epsd1_N=3.12024e+06 "epsd1_06.dat"
Detector: epsd1_I=0.150415 epsd1_ERR=7.75668e-05 epsd1_N=3.76038e+06 "epsd1_07.dat"
Detector: epsd1_I=0.170373 epsd1_ERR=8.25524e-05 epsd1_N=4.25932e+06 "epsd1_08.dat"
Detector: epsd1_I=0.181256 epsd1_ERR=8.51484e-05 epsd1_N=4.53141e+06 "epsd1_09.dat"
Detector: epsd1_I=0.181326 epsd1_ERR=8.51648e-05 epsd1_N=4.53315e+06 "epsd1_10.dat"
Detector: epsd1_I=0.170251 epsd1_ERR=8.2523e-05 epsd1_N=4.25628e+06 "epsd1_11.dat"
Detector: epsd1_I=0.150441 epsd1_ERR=7.75734e-05 epsd1_N=3.76102e+06 "epsd1_12.dat"
Detector: epsd1_I=0.124878 epsd1_ERR=7.06763e-05 epsd1_N=3.12196e+06 "epsd1_13.dat"
Detector: epsd1_I=0.0973506 epsd1_ERR=6.24021e-05 epsd1_N=2.43377e+06 "epsd1_14.dat"
Detector: epsd1_I=0.0712834 epsd1_ERR=5.33979e-05 epsd1_N=1.78209e+06 "epsd1_15.dat"
Detector: epsd1_I=0.0491637 epsd1_ERR=4.43458e-05 epsd1_N=1.22909e+06 "epsd1_16.dat"
Detector: epsd1_I=0.0317947 epsd1_ERR=3.56621e-05 epsd1_N=794868 "epsd1_17.dat"
Detector: epsd1_I=0.0192884 epsd1_ERR=2.77765e-05 epsd1_N=482210 "epsd1_18.dat"

Finally [Test_monitors: 2]. Time: 0 [s] 
Detector: epsd1_I=0.0110437 epsd1_ERR=2.10178e-05 epsd1_N=276093 "epsd1_19.dat"

Finally [Test_monitors: 2]. Time: 0 [s] 
[r6n22.esss.dk:3416488] 1 more process has sent help message help-mpi-btl-openib.txt / no device params found
[r6n22.esss.dk:3416488] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
INFO: Placing instr file copy /users/willend/xTESTS/2022-12-01/McXtrace_DUAL_A100_DMSC_5e7/Test_monitors/Test_monitors.instr in dataset 2