loading system configuration
loading override configuration from /zhome/89/0/38697/mxtest/mctest/../mccodelib/mcxtrace-test/McXtrace_CPU_MPICC_KISS/mccode_config.json
INFO: Using directory: "2"
INFO: Using existing c-file: ./Test_monitors.c
INFO: Using existing binary: ./Test_monitors.out
WARNING: Ignoring invalid parameter: "Test_monitors.instr"
INFO: ===
INFO: Using system default number of mpirun -np processes
--------------------------------------------------------------------------
By default, for Open MPI 4.0 and later, infiniband ports on a device
are not used by default.  The intent is to use UCX for these devices.
You can override this policy by setting the btl_openib_allow_ib MCA parameter
to true.

  Local host:              n-62-29-12
  Local adapter:           mlx4_0
  Local port:              1

--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   n-62-29-12
  Local device: mlx4_0
--------------------------------------------------------------------------
Simulation 'Test_monitors' (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_1e7/Test_monitors/Test_monitors.instr): running on 10 nodes (master is 'n-62-29-12', MPI version 3.1).
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
*** TRACE end *** 

Save [Test_monitors]
Detector: epsd1_I=1.814 epsd1_ERR=0.000602328 epsd1_N=9.06998e+06 "epsd1_Sum.dat"
Detector: epsd1_I=0.0110716 epsd1_ERR=4.70566e-05 epsd1_N=55358 "epsd1_00.dat"
Detector: epsd1_I=0.0191928 epsd1_ERR=6.19561e-05 epsd1_N=95964 "epsd1_01.dat"
Detector: epsd1_I=0.0318278 epsd1_ERR=7.97845e-05 epsd1_N=159139 "epsd1_02.dat"
Detector: epsd1_I=0.0490848 epsd1_ERR=9.90806e-05 epsd1_N=245424 "epsd1_03.dat"
Detector: epsd1_I=0.0713608 epsd1_ERR=0.000119466 epsd1_N=356804 "epsd1_04.dat"
Detector: epsd1_I=0.0972926 epsd1_ERR=0.000139494 epsd1_N=486463 "epsd1_05.dat"
Detector: epsd1_I=0.124628 epsd1_ERR=0.000157879 epsd1_N=623141 "epsd1_06.dat"
Detector: epsd1_I=0.150599 epsd1_ERR=0.000173551 epsd1_N=752996 "epsd1_07.dat"
Detector: epsd1_I=0.17041 epsd1_ERR=0.000184613 epsd1_N=852049 "epsd1_08.dat"
Detector: epsd1_I=0.181519 epsd1_ERR=0.000190536 epsd1_N=907596 "epsd1_09.dat"
Detector: epsd1_I=0.181261 epsd1_ERR=0.0001904 epsd1_N=906303 "epsd1_10.dat"
Detector: epsd1_I=0.170158 epsd1_ERR=0.000184476 epsd1_N=850788 "epsd1_11.dat"
Detector: epsd1_I=0.150295 epsd1_ERR=0.000173375 epsd1_N=751473 "epsd1_12.dat"
Detector: epsd1_I=0.12495 epsd1_ERR=0.000158082 epsd1_N=624748 "epsd1_13.dat"
Detector: epsd1_I=0.0973886 epsd1_ERR=0.000139563 epsd1_N=486943 "epsd1_14.dat"
Detector: epsd1_I=0.0715332 epsd1_ERR=0.00011961 epsd1_N=357666 "epsd1_15.dat"
Detector: epsd1_I=0.0491464 epsd1_ERR=9.91427e-05 epsd1_N=245732 "epsd1_16.dat"
Detector: epsd1_I=0.0317978 epsd1_ERR=7.97468e-05 epsd1_N=158989 "epsd1_17.dat"
Detector: epsd1_I=0.019397 epsd1_ERR=6.22848e-05 epsd1_N=96985 "epsd1_18.dat"


Detector: epsd1_I=0.0110834 epsd1_ERR=4.70816e-05 epsd1_N=55417 "epsd1_19.dat"
Finally [Test_monitors: 2]. Time: 1 [s] 
Finally [Test_monitors: 2]. Time: 1 [s] 

Finally [Test_monitors: 2]. Time: 1 [s] 

Finally [Test_monitors: 2]. Time: 1 [s] 

Finally [Test_monitors: 2]. Time: 1 [s] 


Finally [Test_monitors: 2]. Time: 1 [s] 

Finally [Test_monitors: 2]. Time: 1 [s] 

Finally [Test_monitors: 2]. Time: 1 [s] 
Finally [Test_monitors: 2]. Time: 1 [s] 

Finally [Test_monitors: 2]. Time: 1 [s] 
[n-62-29-12:25227] 9 more processes have sent help message help-mpi-btl-openib.txt / ib port not selected
[n-62-29-12:25227] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[n-62-29-12:25227] 9 more processes have sent help message help-mpi-btl-openib.txt / error in device init
INFO: Placing instr file copy /zhome/89/0/38697/xTESTS/2022-12-01/McXtrace_CPU_MPICC_KISS_1e7/Test_monitors/Test_monitors.instr in dataset 2