loading system configuration
loading override configuration from /zhome/89/0/38697/mxtest/mctest/../mccodelib/mcxtrace-test/McXtrace-1.x_CPU_MPICC/mccode_config.json
INFO: Using directory: "1"
INFO: Using existing c-file: ./Template_Scatter_log_losses.c
INFO: Using existing binary: ./Template_Scatter_log_losses.out
WARNING: Ignoring invalid parameter: "Template_Scatter_log_losses"
INFO: ===
INFO: Using system default number of mpirun -np processes
--------------------------------------------------------------------------
By default, for Open MPI 4.0 and later, infiniband ports on a device
are not used by default.  The intent is to use UCX for these devices.
You can override this policy by setting the btl_openib_allow_ib MCA parameter
to true.

  Local host:              n-62-29-31
  Local adapter:           mlx4_0
  Local port:              1

--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   n-62-29-31
  Local device: mlx4_0
--------------------------------------------------------------------------
Simulation 'Template_Scatter_log_losses' (/zhome/89/0/38697/xTESTS/2022-12-01/McXtrace-1.x_CPU_MPICC_1e7/Template_Scatter_log_losses/Template_Scatter_log_losses.instr): running on 10 nodes (master is 'n-62-29-31', MPI version 3.1).
[Template_Scatter_log_losses] Initialize
Opening input file '/zhome/89/0/38697/McXtrace/mcxtrace/1.x-dev/tools/Python/mxrun/../mccodelib/../../../data/Rh.txt' (Table_Read_Offset)
[Template_Scatter_log_losses] Initialize
[Template_Scatter_log_losses] Initialize
[Template_Scatter_log_losses] Initialize
[Template_Scatter_log_losses] Initialize
[Template_Scatter_log_losses] Initialize
[Template_Scatter_log_losses] Initialize
[Template_Scatter_log_losses] Initialize
[Template_Scatter_log_losses] Initialize
[Template_Scatter_log_losses] Initialize

Finally [Template_Scatter_log_losses: 1]. Time: 2 [s] 
Warning: 248270 events were removed in Component[3] psd0=PSD_monitor()
         (negative time, miss next components, rounding errors, Nan, Inf).

Finally [Template_Scatter_log_losses: 1]. Time: 2 [s] 
Warning: 249561 events were removed in Component[3] psd0=PSD_monitor()
         (negative time, miss next components, rounding errors, Nan, Inf).

Finally [Template_Scatter_log_losses: 1]. Time: 2 [s] 
Warning: 248878 events were removed in Component[3] psd0=PSD_monitor()
         (negative time, miss next components, rounding errors, Nan, Inf).
Warning: 249007 events were removed in Component[3] psd0=PSD_monitor()
         (negative time, miss next components, rounding errors, Nan, Inf).

Finally [Template_Scatter_log_losses: 1]. Time: 2 [s] 

Finally [Template_Scatter_log_losses: 1]. Time: 2 [s] 
Warning: 249213 events were removed in Component[3] psd0=PSD_monitor()
         (negative time, miss next components, rounding errors, Nan, Inf).

Save [Template_Scatter_log_losses]

Finally [Template_Scatter_log_losses: 1]. Time: 2 [s] 
Warning: 249474 events were removed in Component[3] psd0=PSD_monitor()
         (negative time, miss next components, rounding errors, Nan, Inf).

Finally [Template_Scatter_log_losses: 1]. Time: 2 [s] 
Warning: 249102 events were removed in Component[3] psd0=PSD_monitor()
         (negative time, miss next components, rounding errors, Nan, Inf).
Detector: psd0_I=2.39018e-05 psd0_ERR=8.72212e-09 psd0_N=7.50958e+06 "psd0.dat"

Finally [Template_Scatter_log_losses: 1]. Time: 2 [s] 
Warning: 248770 events were removed in Component[3] psd0=PSD_monitor()
         (negative time, miss next components, rounding errors, Nan, Inf).
Detector: mnd1_I=8.81124e-06 mnd1_ERR=5.05478e-09 mnd1_N=4.42348e+06 "mnd1.dat"
Detector: mnd1abs_I=1.39391e-05 mnd1abs_ERR=6.47084e-09 mnd1abs_N=6.03457e+06 "mnd1abs.dat"

Finally [Template_Scatter_log_losses: 1]. Time: 2 [s] 
Warning: 249587 events were removed in Component[3] psd0=PSD_monitor()
         (negative time, miss next components, rounding errors, Nan, Inf).
Detector: mnd2_I=4.32446e-06 mnd2_ERR=3.4334e-09 mnd2_N=2.96013e+06 "mnd2.dat"
Detector: endPSD_I=9.96271e-06 endPSD_ERR=5.40516e-09 endPSD_N=5.8985e+06 "endpsd.dat"

Finally [Template_Scatter_log_losses: 1]. Time: 2 [s] 
Warning: 248553 events were removed in Component[3] psd0=PSD_monitor()
         (negative time, miss next components, rounding errors, Nan, Inf).
[n-62-29-31:06635] 9 more processes have sent help message help-mpi-btl-openib.txt / ib port not selected
[n-62-29-31:06635] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[n-62-29-31:06635] 9 more processes have sent help message help-mpi-btl-openib.txt / error in device init
INFO: Placing instr file copy /zhome/89/0/38697/xTESTS/2022-12-01/McXtrace-1.x_CPU_MPICC_1e7/Template_Scatter_log_losses/Template_Scatter_log_losses.instr in dataset 1