INFO: Using directory: "2"
INFO: Using existing c-file: ./Test_monitors.c
INFO: Using existing binary: ./Test_monitors.out
WARNING: Ignoring invalid parameter: "Test_monitors.instr"
INFO: ===
INFO: Using system default number of mpirun -np processes
--------------------------------------------------------------------------
By default, for Open MPI 4.0 and later, infiniband ports on a device
are not used by default.  The intent is to use UCX for these devices.
You can override this policy by setting the btl_openib_allow_ib MCA parameter
to true.

  Local host:              n-62-29-12
  Local adapter:           mlx4_0
  Local port:              1

--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   n-62-29-12
  Local device: mlx4_0
--------------------------------------------------------------------------
Simulation 'Test_monitors' (/zhome/89/0/38697/xTESTS/2022-03-01/McXtrace-1.x_CPU_MPICC_5e7/Test_monitors/Test_monitors.instr): running on 10 nodes (master is 'n-62-29-34', MPI version 3.1).
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize
[Test_monitors] Initialize

Finally [Test_monitors: 2]. Time: 3 [s] 


Finally [Test_monitors: 2]. Time: 3 [s] 
Finally [Test_monitors: 2]. Time: 3 [s] 

Finally [Test_monitors: 2]. Time: 3 [s] 

Finally [Test_monitors: 2]. Time: 3 [s] 

Save [Test_monitors]
Detector: epsd1_I=1.81366 epsd1_ERR=0.000269344 epsd1_N=4.53414e+07 "epsd1_Sum.dat"

Finally [Test_monitors: 2]. Time: 3 [s] 

Finally [Test_monitors: 2]. Time: 3 [s] 
Detector: epsd1_I=0.0110442 epsd1_ERR=2.10183e-05 epsd1_N=276106 "epsd1_00.dat"
Detector: epsd1_I=0.0192929 epsd1_ERR=2.77798e-05 epsd1_N=482322 "epsd1_01.dat"
Detector: epsd1_I=0.0318157 epsd1_ERR=3.56739e-05 epsd1_N=795392 "epsd1_02.dat"

Finally [Test_monitors: 2]. Time: 3 [s] 
Detector: epsd1_I=0.0491787 epsd1_ERR=4.43525e-05 epsd1_N=1.22947e+06 "epsd1_03.dat"
Detector: epsd1_I=0.0713312 epsd1_ERR=5.34158e-05 epsd1_N=1.78328e+06 "epsd1_04.dat"
Detector: epsd1_I=0.0973421 epsd1_ERR=6.23994e-05 epsd1_N=2.43355e+06 "epsd1_05.dat"
Detector: epsd1_I=0.124783 epsd1_ERR=7.06492e-05 epsd1_N=3.11957e+06 "epsd1_06.dat"
Detector: epsd1_I=0.150488 epsd1_ERR=7.75855e-05 epsd1_N=3.76219e+06 "epsd1_07.dat"
Detector: epsd1_I=0.170365 epsd1_ERR=8.25505e-05 epsd1_N=4.25912e+06 "epsd1_08.dat"
Detector: epsd1_I=0.181379 epsd1_ERR=8.51772e-05 epsd1_N=4.53447e+06 "epsd1_09.dat"
Detector: epsd1_I=0.181132 epsd1_ERR=8.51192e-05 epsd1_N=4.5283e+06 "epsd1_10.dat"
Detector: epsd1_I=0.170377 epsd1_ERR=8.25535e-05 epsd1_N=4.25943e+06 "epsd1_11.dat"
Detector: epsd1_I=0.150414 epsd1_ERR=7.75665e-05 epsd1_N=3.76035e+06 "epsd1_12.dat"
Detector: epsd1_I=0.124782 epsd1_ERR=7.06491e-05 epsd1_N=3.11956e+06 "epsd1_13.dat"
Detector: epsd1_I=0.0973611 epsd1_ERR=6.24055e-05 epsd1_N=2.43403e+06 "epsd1_14.dat"
Detector: epsd1_I=0.0713205 epsd1_ERR=5.34118e-05 epsd1_N=1.78301e+06 "epsd1_15.dat"
Detector: epsd1_I=0.049118 epsd1_ERR=4.43252e-05 epsd1_N=1.22795e+06 "epsd1_16.dat"
Detector: epsd1_I=0.0317386 epsd1_ERR=3.56307e-05 epsd1_N=793465 "epsd1_17.dat"
Detector: epsd1_I=0.0193246 epsd1_ERR=2.78026e-05 epsd1_N=483115 "epsd1_18.dat"

Finally [Test_monitors: 2]. Time: 4 [s] 
Detector: epsd1_I=0.0110708 epsd1_ERR=2.10436e-05 epsd1_N=276770 "epsd1_19.dat"

Finally [Test_monitors: 2]. Time: 4 [s] 
[n-62-29-34:19434] 9 more processes have sent help message help-mpi-btl-openib.txt / ib port not selected
[n-62-29-34:19434] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[n-62-29-34:19434] 9 more processes have sent help message help-mpi-btl-openib.txt / error in device init
INFO: Placing instr file copy /zhome/89/0/38697/xTESTS/2022-03-01/McXtrace-1.x_CPU_MPICC_5e7/Test_monitors/Test_monitors.instr in dataset 2
loading system configuration
loading override configuration from /zhome/89/0/38697/mxtest/mctest/../mccodelib/mcxtrace-test/McXtrace-1.x_CPU_MPICC/mccode_config.json