INFO: Using directory: "1" INFO: Using existing c-file: ./Template_Scatter_log_losses.c INFO: Using existing binary: ./Template_Scatter_log_losses.out WARNING: Ignoring invalid parameter: "Template_Scatter_log_losses" INFO: === INFO: Using system default number of mpirun -np processes -------------------------------------------------------------------------- By default, for Open MPI 4.0 and later, infiniband ports on a device are not used by default. The intent is to use UCX for these devices. You can override this policy by setting the btl_openib_allow_ib MCA parameter to true. Local host: n-62-29-12 Local adapter: mlx4_0 Local port: 1 -------------------------------------------------------------------------- -------------------------------------------------------------------------- WARNING: There was an error initializing an OpenFabrics device. Local host: n-62-29-12 Local device: mlx4_0 -------------------------------------------------------------------------- Simulation 'Template_Scatter_log_losses' (/zhome/89/0/38697/xTESTS/2022-03-01/McXtrace-1.x_CPU_MPICC_5e7/Template_Scatter_log_losses/Template_Scatter_log_losses.instr): running on 10 nodes (master is 'n-62-29-34', MPI version 3.1). [Template_Scatter_log_losses] Initialize [Template_Scatter_log_losses] Initialize [Template_Scatter_log_losses] Initialize Opening input file '/zhome/89/0/38697/McXtrace/mcxtrace/1.x-dev/tools/Python/mxrun/../mccodelib/../../../data/Rh.txt' (Table_Read_Offset) [Template_Scatter_log_losses] Initialize [Template_Scatter_log_losses] Initialize [Template_Scatter_log_losses] Initialize [Template_Scatter_log_losses] Initialize [Template_Scatter_log_losses] Initialize [Template_Scatter_log_losses] Initialize [Template_Scatter_log_losses] Initialize [n-62-29-34:16641] 9 more processes have sent help message help-mpi-btl-openib.txt / ib port not selected [n-62-29-34:16641] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages [n-62-29-34:16641] 9 more processes have sent help message help-mpi-btl-openib.txt / error in device init Finally [Template_Scatter_log_losses: 1]. Time: 6 [s] Warning: 1.24346e+06 events were removed in Component[3] psd0=PSD_monitor() (negative time, miss next components, rounding errors, Nan, Inf). Finally [Template_Scatter_log_losses: 1]. Time: 6 [s] Warning: 1.24393e+06 events were removed in Component[3] psd0=PSD_monitor() (negative time, miss next components, rounding errors, Nan, Inf). Finally [Template_Scatter_log_losses: 1]. Time: 6 [s] Warning: 1.24184e+06 events were removed in Component[3] psd0=PSD_monitor() (negative time, miss next components, rounding errors, Nan, Inf). Finally [Template_Scatter_log_losses: 1]. Time: 6 [s] Warning: 1.24206e+06 events were removed in Component[3] psd0=PSD_monitor() (negative time, miss next components, rounding errors, Nan, Inf). Finally [Template_Scatter_log_losses: 1]. Time: 7 [s] Warning: 1.24303e+06 events were removed in Component[3] psd0=PSD_monitor() (negative time, miss next components, rounding errors, Nan, Inf). Finally [Template_Scatter_log_losses: 1]. Time: 7 [s] Warning: 1.24251e+06 events were removed in Component[3] psd0=PSD_monitor() (negative time, miss next components, rounding errors, Nan, Inf). Finally [Template_Scatter_log_losses: 1]. Time: 7 [s] Warning: 1.24561e+06 events were removed in Component[3] psd0=PSD_monitor() (negative time, miss next components, rounding errors, Nan, Inf). Save [Template_Scatter_log_losses] Detector: psd0_I=2.39122e-05 psd0_ERR=3.9015e-09 psd0_N=3.75643e+07 "psd0.dat" Detector: mnd1_I=8.81667e-06 mnd1_ERR=2.26139e-09 mnd1_N=2.21262e+07 "mnd1.dat" Detector: mnd1abs_I=1.39479e-05 mnd1abs_ERR=2.89487e-09 mnd1abs_N=3.0187e+07 "mnd1abs.dat" Detector: mnd2_I=4.32554e-06 mnd2_ERR=1.53579e-09 mnd2_N=1.48024e+07 "mnd2.dat" Finally [Template_Scatter_log_losses: 1]. Time: 7 [s] Warning: 1.24473e+06 events were removed in Component[3] psd0=PSD_monitor() (negative time, miss next components, rounding errors, Nan, Inf). Finally [Template_Scatter_log_losses: 1]. Time: 7 [s] Warning: 1.24474e+06 events were removed in Component[3] psd0=PSD_monitor() (negative time, miss next components, rounding errors, Nan, Inf). Detector: endPSD_I=9.96423e-06 endPSD_ERR=2.41751e-09 endPSD_N=2.95034e+07 "endpsd.dat" Finally [Template_Scatter_log_losses: 1]. Time: 7 [s] Warning: 1.2438e+06 events were removed in Component[3] psd0=PSD_monitor() (negative time, miss next components, rounding errors, Nan, Inf). INFO: Placing instr file copy /zhome/89/0/38697/xTESTS/2022-03-01/McXtrace-1.x_CPU_MPICC_5e7/Template_Scatter_log_losses/Template_Scatter_log_losses.instr in dataset 1 loading system configuration loading override configuration from /zhome/89/0/38697/mxtest/mctest/../mccodelib/mcxtrace-test/McXtrace-1.x_CPU_MPICC/mccode_config.json