INFO: Using directory: "2"
INFO: Using existing c-file: ./Test_monitors.c
INFO: Using existing binary: ./Test_monitors.out
WARNING: Ignoring invalid parameter: "Test_monitors.instr"
INFO: ===
INFO: Using system default number of mpirun -np processes
--------------------------------------------------------------------------
By default, for Open MPI 4.0 and later, infiniband ports on a device
are not used by default.  The intent is to use UCX for these devices.
You can override this policy by setting the btl_openib_allow_ib MCA parameter
to true.

  Local host:              n-62-11-62
  Local adapter:           mlx5_0
  Local port:              1

--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   n-62-11-62
  Local device: mlx5_0
--------------------------------------------------------------------------
Simulation 'Test_monitors' (/zhome/89/0/38697/xTESTS/2022-06-01/McXtrace_CPU_MPICC_KISS_5e7/Test_monitors/Test_monitors.instr): running on 10 nodes (master is 'n-62-11-62', MPI version 3.1).
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
*** TRACE end *** 

Save [Test_monitors]
Detector: epsd1_I=1.81383 epsd1_ERR=0.000269357 epsd1_N=4.53459e+07 "epsd1_Sum.dat"
Detector: epsd1_I=0.0110606 epsd1_ERR=2.10339e-05 epsd1_N=276516 "epsd1_00.dat"
Detector: epsd1_I=0.0193646 epsd1_ERR=2.78313e-05 epsd1_N=484115 "epsd1_01.dat"
Detector: epsd1_I=0.0318286 epsd1_ERR=3.56811e-05 epsd1_N=795714 "epsd1_02.dat"
Detector: epsd1_I=0.0491511 epsd1_ERR=4.43401e-05 epsd1_N=1.22878e+06 "epsd1_03.dat"
Detector: epsd1_I=0.0712592 epsd1_ERR=5.33889e-05 epsd1_N=1.78148e+06 "epsd1_04.dat"
Detector: epsd1_I=0.0973705 epsd1_ERR=6.24085e-05 epsd1_N=2.43426e+06 "epsd1_05.dat"
Detector: epsd1_I=0.124785 epsd1_ERR=7.06498e-05 epsd1_N=3.11962e+06 "epsd1_06.dat"
Detector: epsd1_I=0.15054 epsd1_ERR=7.75991e-05 epsd1_N=3.76351e+06 "epsd1_07.dat"
Detector: epsd1_I=0.170422 epsd1_ERR=8.25644e-05 epsd1_N=4.26055e+06 "epsd1_08.dat"
Detector: epsd1_I=0.181191 epsd1_ERR=8.51331e-05 epsd1_N=4.52978e+06 "epsd1_09.dat"
Detector: epsd1_I=0.181227 epsd1_ERR=8.51416e-05 epsd1_N=4.53068e+06 "epsd1_10.dat"
Detector: epsd1_I=0.170259 epsd1_ERR=8.25249e-05 epsd1_N=4.25647e+06 "epsd1_11.dat"
Detector: epsd1_I=0.150364 epsd1_ERR=7.75537e-05 epsd1_N=3.75911e+06 "epsd1_12.dat"
Detector: epsd1_I=0.124864 epsd1_ERR=7.06723e-05 epsd1_N=3.12161e+06 "epsd1_13.dat"
Detector: epsd1_I=0.0974174 epsd1_ERR=6.24235e-05 epsd1_N=2.43544e+06 "epsd1_14.dat"
Detector: epsd1_I=0.0713546 epsd1_ERR=5.34245e-05 epsd1_N=1.78386e+06 "epsd1_15.dat"
Detector: epsd1_I=0.0491673 epsd1_ERR=4.43474e-05 epsd1_N=1.22918e+06 "epsd1_16.dat"
Detector: epsd1_I=0.0318317 epsd1_ERR=3.56829e-05 epsd1_N=795793 "epsd1_17.dat"
Detector: epsd1_I=0.019346 epsd1_ERR=2.7818e-05 epsd1_N=483651 "epsd1_18.dat"
Detector: epsd1_I=0.0110296 epsd1_ERR=2.10044e-05 epsd1_N=275741 "epsd1_19.dat"

Finally [Test_monitors: 2]. Time: 5 [s] 

Finally [Test_monitors: 2]. Time: 5 [s] 

Finally [Test_monitors: 2]. Time: 5 [s] 

Finally [Test_monitors: 2]. Time: 5 [s] 


Finally [Test_monitors: 2]. Time: 5 [s] 
Finally [Test_monitors: 2]. Time: 5 [s] 


Finally [Test_monitors: 2]. Time: 4 [s] 

Finally [Test_monitors: 2]. Time: 5 [s] 
Finally [Test_monitors: 2]. Time: 5 [s] 

Finally [Test_monitors: 2]. Time: 5 [s] 
[n-62-11-62:08578] 9 more processes have sent help message help-mpi-btl-openib.txt / ib port not selected
[n-62-11-62:08578] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[n-62-11-62:08578] 9 more processes have sent help message help-mpi-btl-openib.txt / error in device init
INFO: Placing instr file copy /zhome/89/0/38697/xTESTS/2022-06-01/McXtrace_CPU_MPICC_KISS_5e7/Test_monitors/Test_monitors.instr in dataset 2
loading system configuration
loading override configuration from /zhome/89/0/38697/mxtest/mctest/../mccodelib/mcxtrace-test/McXtrace_CPU_MPICC_KISS/mccode_config.json