INFO: Using directory: "1" INFO: Using existing c-file: ./Test_Absorption.c INFO: Using existing binary: ./Test_Absorption.out WARNING: Ignoring invalid parameter: "Test_Absorption.instr" INFO: === INFO: Using system default number of mpirun -np processes -------------------------------------------------------------------------- By default, for Open MPI 4.0 and later, infiniband ports on a device are not used by default. The intent is to use UCX for these devices. You can override this policy by setting the btl_openib_allow_ib MCA parameter to true. Local host: n-62-29-17 Local adapter: mlx4_0 Local port: 1 -------------------------------------------------------------------------- -------------------------------------------------------------------------- WARNING: There was an error initializing an OpenFabrics device. Local host: n-62-29-17 Local device: mlx4_0 -------------------------------------------------------------------------- Simulation 'Test_Absorption' (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr): running on 10 nodes (master is 'n-62-29-17', MPI version 3.1). [Test_Absorption] Initialize [Test_Absorption] Initialize WARNING: (Table_Rebin): Rebinning table from Mn.txt would exceed 1M rows. Skipping. WARNING: (Table_Rebin): Rebinning table from Mn.txt would exceed 1M rows. Skipping. Warning: sample_scan2: Invalid inner volume specification. Ignoring. sample_scan1: Rebinning material data (Mn.txt) for faster tracing...done INFO: sample_scan2: Setting rho_o=7.43 from Mn.txt sample_scan1: Rebinning material data (Mn.txt) for faster tracing...done Warning: sample_scan2: Invalid inner volume specification. Ignoring. INFO: sample_scan2: Setting rho_o=7.43 from Mn.txt Error: Could not open input file 'input_small.dat' (Open_File) Error: sample_scan3: Could not open file "input_small.dat". Error: Could not open input file 'input_small.dat' (Open_File) Error: sample_scan3: Could not open file "input_small.dat". [Test_Absorption] Initialize WARNING: (Table_Rebin): Rebinning table from Mn.txt would exceed 1M rows. Skipping. Warning: sample_scan2: Invalid inner volume specification. Ignoring. sample_scan1: Rebinning material data (Mn.txt) for faster tracing...done INFO: sample_scan2: Setting rho_o=7.43 from Mn.txt Error: Could not open input file 'input_small.dat' (Open_File) Error: sample_scan3: Could not open file "input_small.dat". [Test_Absorption] Initialize [Test_Absorption] Initialize [Test_Absorption] Initialize Opening input file '/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../data/Mn.txt' (Table_Read_Offset) WARNING: (Table_Rebin): Rebinning table from Mn.txt would exceed 1M rows. Skipping. sample_scan1: Rebinning material data (Mn.txt) for faster tracing...done Warning: sample_scan2: Invalid inner volume specification. Ignoring. Reusing input file 'Mn.txt' (Table_Read_Offset) INFO: sample_scan2: Setting rho_o=7.43 from Mn.txt WARNING: (Table_Rebin): Rebinning table from Mn.txt would exceed 1M rows. Skipping. Reusing input file 'Mn.txt' (Table_Read_Offset) sample_scan1: Rebinning material data (Mn.txt) for faster tracing...done Warning: sample_scan2: Invalid inner volume specification. Ignoring. Loading geometry file (OFF/PLY): cube.off INFO: sample_scan2: Setting rho_o=7.43 from Mn.txt WARNING: (Table_Rebin): Rebinning table from Mn.txt would exceed 1M rows. Skipping. sample_scan1: Rebinning material data (Mn.txt) for faster tracing...done Warning: sample_scan2: Invalid inner volume specification. Ignoring. INFO: sample_scan2: Setting rho_o=7.43 from Mn.txt Number of vertices: 8 Number of polygons: 6 Warning: Aspect ratio of the geometry cube.off was modified. If you want to keep the original proportions, specifiy only one of the dimensions. Bounding box dimensions for geometry cube.off: Length=0.001000 (0.100%) Width= 0.001000 (0.100%) Depth= 0.000100 (0.010%) sample_scan3: Volume[1], material='Mn.txt' geometry='cube.off' sample_scan3: Volume[1], material='Mn.txt' geometry='cube.off' sample_scan3: Volume[1], material='Mn.txt' geometry='cube.off' [Test_Absorption] Initialize WARNING: (Table_Rebin): Rebinning table from Mn.txt would exceed 1M rows. Skipping. sample_scan1: Rebinning material data (Mn.txt) for faster tracing...done Warning: sample_scan2: Invalid inner volume specification. Ignoring. INFO: sample_scan2: Setting rho_o=7.43 from Mn.txt sample_scan3: Volume[1], material='Mn.txt' geometry='cube.off' [Test_Absorption] Initialize [Test_Absorption] Initialize WARNING: (Table_Rebin): Rebinning table from Mn.txt would exceed 1M rows. Skipping. sample_scan1: Rebinning material data (Mn.txt) for faster tracing...done Warning: sample_scan2: Invalid inner volume specification. Ignoring. INFO: sample_scan2: Setting rho_o=7.43 from Mn.txt sample_scan3: Volume[1], material='Mn.txt' geometry='cube.off' WARNING: (Table_Rebin): Rebinning table from Mn.txt would exceed 1M rows. Skipping. sample_scan1: Rebinning material data (Mn.txt) for faster tracing...done Warning: sample_scan2: Invalid inner volume specification. Ignoring. INFO: sample_scan2: Setting rho_o=7.43 from Mn.txt sample_scan3: Volume[1], material='Mn.txt' geometry='cube.off' [Test_Absorption] Initialize WARNING: (Table_Rebin): Rebinning table from Mn.txt would exceed 1M rows. Skipping. Warning: sample_scan2: Invalid inner volume specification. Ignoring. sample_scan1: Rebinning material data (Mn.txt) for faster tracing...done INFO: sample_scan2: Setting rho_o=7.43 from Mn.txt sample_scan3: Volume[1], material='Mn.txt' geometry='cube.off' [n-62-29-17:11209] 9 more processes have sent help message help-mpi-btl-openib.txt / ib port not selected [n-62-29-17:11209] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages [n-62-29-17:11209] 9 more processes have sent help message help-mpi-btl-openib.txt / error in device init *** TRACE end *** mpirun: Forwarding signal 12 to job # McXtrace 3.x-dev - Mar. 31, 2022: [pid 31726] Signal 12 detected [proc 2] SIGUSR2 (Save simulation) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # McXtrace 3.x-dev - Mar. 31, 2022: [pid 32324] Signal 12 detected [proc 4] SIGUSR2 (Save simulation) # McXtrace 3.x-dev - Mar. 31, 2022: [pid 30548] Signal 12 detected [proc 9] SIGUSR2 (Save simulation) # McXtrace 3.x-dev - Mar. 31, 2022: [pid 22104] Signal 12 detected [proc 3] SIGUSR2 (Save simulation) # McXtrace 3.x-dev - Mar. 31, 2022: [pid 5261] Signal 12 detected [proc 5] SIGUSR2 (Save simulation) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # McXtrace 3.x-dev - Mar. 31, 2022: [pid 21602] Signal 12 detected [proc 7] SIGUSR2 (Save simulation) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) [n-62-29-13:5261 :0:5261] ucp_worker.c:1425 UCX Assertion `worker->inprogress++ == 0' failed # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # Breakpoint: [_psd_trace] component psd=PSD_monitor() TRACE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/PSD_monitor.comp:83] 0.00 % ( 0.0/10000000.0) # Breakpoint: [_psd_trace] component psd=PSD_monitor() TRACE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/PSD_monitor.comp:83] 0.00 % ( 0.0/10000000.0) # McXtrace 3.x-dev - Mar. 31, 2022: [pid 3942] Signal 12 detected [proc 1] SIGUSR2 (Save simulation) # Breakpoint: [_sample_scan3_init] component sample_scan3=Abs_objects() INITIALISE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../samples/Abs_objects.comp:125] 0.00 % ( 0.0/50000000.0) # Breakpoint: [_sample_scan3_init] component sample_scan3=Abs_objects() INITIALISE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../samples/Abs_objects.comp:125] 0.00 % ( 0.0/50000000.0) # Breakpoint: [_psd_trace] component psd=PSD_monitor() TRACE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/PSD_monitor.comp:83] 0.00 % ( 0.0/10000000.0) # Breakpoint: [_psd_trace] component psd=PSD_monitor() TRACE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/PSD_monitor.comp:83] 0.00 % ( 0.0/10000000.0) # McXtrace 3.x-dev - Mar. 31, 2022: [pid 5262] Signal 12 detected [proc 6] SIGUSR2 (Save simulation) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # Date: Fri Apr 1 10:01:12 2022 # Started: Thu Jan 1 01:00:00 1970 # McXtrace 3.x-dev - Mar. 31, 2022: Saving data and resume simulation (continue) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # Date: Fri Apr 1 10:01:12 2022 # McXtrace 3.x-dev - Mar. 31, 2022: [pid 21603] Signal 12 detected [proc 8] SIGUSR2 (Save simulation) # Breakpoint: [_psd_trace] component psd=PSD_monitor() TRACE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/PSD_monitor.comp:83] 0.00 % ( 0.0/10000000.0) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # Breakpoint: [_psd_trace] component psd=PSD_monitor() TRACE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/PSD_monitor.comp:83] 0.00 % ( 0.0/10000000.0) # Date: Fri Apr 1 10:01:12 2022 # Date: Fri Apr 1 10:01:12 2022 # Started: Thu Jan 1 01:00:00 1970 # Breakpoint: [_sample_scan3_init] component sample_scan3=Abs_objects() INITIALISE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../samples/Abs_objects.comp:125] 0.00 % ( 0.0/50000000.0) # McXtrace 3.x-dev - Mar. 31, 2022: Saving data and resume simulation (continue) # Started: Thu Jan 1 01:00:00 1970 # Date: Fri Apr 1 10:01:12 2022 # Date: Fri Apr 1 10:01:12 2022 # Date: Fri Apr 1 10:01:12 2022 # Started: Thu Jan 1 01:00:00 1970 # McXtrace 3.x-dev - Mar. 31, 2022: Saving data and resume simulation (continue) # McXtrace 3.x-dev - Mar. 31, 2022: Saving data and resume simulation (continue) # Started: Thu Jan 1 01:00:00 1970 # Started: Thu Jan 1 01:00:00 1970 # Started: Thu Jan 1 01:00:00 1970 # McXtrace 3.x-dev - Mar. 31, 2022: Saving data and resume simulation (continue) # Date: Fri Apr 1 10:01:12 2022 # McXtrace 3.x-dev - Mar. 31, 2022: Saving data and resume simulation (continue) # McXtrace 3.x-dev - Mar. 31, 2022: Saving data and resume simulation (continue) # Date: Fri Apr 1 10:01:12 2022 # Started: Thu Jan 1 01:00:00 1970 # McXtrace 3.x-dev - Mar. 31, 2022: Saving data and resume simulation (continue) # Started: Thu Jan 1 01:00:00 1970 # McXtrace 3.x-dev - Mar. 31, 2022: [pid 31726] Signal 11 detected [proc 2] SIGSEGV (Mem Error) # McXtrace 3.x-dev - Mar. 31, 2022: Saving data and resume simulation (continue) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # McXtrace 3.x-dev - Mar. 31, 2022: [pid 32324] Signal 11 detected [proc 4] SIGSEGV (Mem Error) # Breakpoint: [_psd_save] component psd=PSD_monitor() SAVE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/PSD_monitor.comp:122] 0.00 % ( 0.0/50000000.0) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # McXtrace 3.x-dev - Mar. 31, 2022: [pid 21603] Signal 11 detected [proc 8] SIGSEGV (Mem Error) # Date: Fri Apr 1 10:01:12 2022 # Breakpoint: [_psd_save] component psd=PSD_monitor() SAVE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/PSD_monitor.comp:122] 0.00 % ( 0.0/50000000.0) [n-62-29-27:30548:0:30548] ucp_worker.c:1425 UCX Assertion `worker->inprogress++ == 0' failed [n-62-29-13:5262 :0:5262] ucp_worker.c:1425 UCX Assertion `worker->inprogress++ == 0' failed [n-62-29-18:3942 :0:3942] ucp_worker.c:1425 UCX Assertion `worker->inprogress++ == 0' failed # Last I/O Error: Resource temporarily unavailable # Last I/O Error: Resource temporarily unavailable # Last I/O Error: Resource temporarily unavailable # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # Started: Thu Jan 1 01:00:00 1970 # Date: Fri Apr 1 10:01:12 2022 # Breakpoint: [_psd_save] component psd=PSD_monitor() SAVE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/PSD_monitor.comp:122] 0.00 % ( 0.0/50000000.0) # Started: Thu Jan 1 01:00:00 1970 # Date: Fri Apr 1 10:01:12 2022 # McXtrace 3.x-dev - Mar. 31, 2022: Simulation stop (abort). # Started: Thu Jan 1 01:00:00 1970 # McXtrace 3.x-dev - Mar. 31, 2022: Simulation stop (abort). # McXtrace 3.x-dev - Mar. 31, 2022: Simulation stop (abort). # McXtrace 3.x-dev - Mar. 31, 2022: [pid 11215] Signal 12 detected [proc 0] SIGUSR2 (Save simulation) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # Breakpoint: [_psd_trace] component psd=PSD_monitor() TRACE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/PSD_monitor.comp:83] 0.00 % ( 0.0/10000000.0) # Date: Fri Apr 1 10:01:12 2022 # Started: Thu Jan 1 01:00:00 1970 # McXtrace 3.x-dev - Mar. 31, 2022: Saving data and resume simulation (continue) Save [Test_Absorption] [n-62-29-17:11215:0:11215] ucp_worker.c:1425 UCX Assertion `worker->inprogress++ == 0' failed -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- # McXtrace 3.x-dev - Mar. 31, 2022: [pid 5261] Signal 15 detected [proc 5] SIGTERM (Termination) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # Breakpoint: [_emon_save] component emon=E_monitor() SAVE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/E_monitor.comp:111] 0.00 % ( 0.0/10000000.0) # Date: Fri Apr 1 10:01:12 2022 # Started: Thu Jan 1 01:00:00 1970 # McXtrace 3.x-dev - Mar. 31, 2022: Finishing simulation (save results and exit) # McXtrace 3.x-dev - Mar. 31, 2022: [pid 5261] Signal 6 detected [proc 5] SIGABRT (Abort) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # Breakpoint: [_emon_save] component emon=E_monitor() SAVE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/E_monitor.comp:111] 0.00 % ( 0.0/10000000.0) # Date: Fri Apr 1 10:01:12 2022 # Started: Thu Jan 1 01:00:00 1970 # McXtrace 3.x-dev - Mar. 31, 2022: Simulation stop (abort). [n-62-29-13:5261 :0:5261] ucp_worker.c:1425 UCX Assertion `worker->inprogress++ == 0' failed ==== backtrace ==== 0 0x0000000000019e90 ucp_worker_progress() /tmp/sebox/ucx-1.5.1/src/ucp/core/ucp_worker.c:1425 1 0x0000000000003337 mca_pml_ucx_progress() ???:0 2 0x000000000002f02c opal_progress() ???:0 3 0x000000000004df85 ompi_request_default_wait() ???:0 4 0x00000000000a7793 ompi_coll_base_sendrecv_actual() ???:0 5 0x00000000000a7bbc ompi_coll_base_allreduce_intra_recursivedoubling() ???:0 6 0x0000000000065179 MPI_Allreduce() ???:0 7 0x0000000000402d1d mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:1397 8 0x0000000000405d20 mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:2127 9 0x00000000004068e5 mcdetector_out_1D() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:3262 10 0x000000000041c32b class_E_monitor_save() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:11546 11 0x000000000041c629 save() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:11652 12 0x000000000041eb09 finally() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:11895 13 0x000000000041ed15 sighandler() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:5409 14 0x0000000000036450 killpg() ???:0 15 0x0000000000096ab7 scan_unit_for_symbols() dwarf2.o:0 16 0x00000000000975d8 comp_unit_find_nearest_line() dwarf2.o:0 17 0x0000000000099a74 _bfd_dwarf2_find_nearest_line() :0 18 0x0000000000075316 _bfd_elf_find_nearest_line() :0 19 0x0000000000048c0e find_address_in_section() /tmp/sebox/ucx-1.5.1/src/ucs/debug/debug.c:264 20 0x000000000005591c bfd_map_over_sections() :0 21 0x0000000000048e43 get_line_info() /tmp/sebox/ucx-1.5.1/src/ucs/debug/debug.c:296 22 0x00000000000499dd ucs_debug_print_backtrace() /tmp/sebox/ucx-1.5.1/src/ucs/debug/debug.c:451 23 0x000000000004a954 ucs_handle_error() /tmp/sebox/ucx-1.5.1/src/ucs/debug/debug.c:994 24 0x000000000004886c ucs_fatal_error() /tmp/sebox/ucx-1.5.1/src/ucs/debug/assert.c:33 25 0x0000000000019e90 ucp_worker_progress() /tmp/sebox/ucx-1.5.1/src/ucp/core/ucp_worker.c:1425 26 0x0000000000003337 mca_pml_ucx_progress() ???:0 27 0x000000000002f02c opal_progress() ???:0 28 0x000000000004df85 ompi_request_default_wait() ???:0 29 0x00000000000a7793 ompi_coll_base_sendrecv_actual() ???:0 30 0x00000000000a7bbc ompi_coll_base_allreduce_intra_recursivedoubling() ???:0 31 0x0000000000065179 MPI_Allreduce() ???:0 32 0x0000000000402d1d mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:1397 33 0x0000000000405d20 mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:2127 34 0x00000000004068e5 mcdetector_out_1D() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:3262 35 0x000000000041c32b class_E_monitor_save() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:11546 36 0x000000000041c629 save() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:11652 37 0x000000000041efa1 sighandler() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:5401 38 0x0000000000036450 killpg() ???:0 39 0x00000000000374b4 uct_mm_iface_poll_fifo() /tmp/sebox/ucx-1.5.1/src/uct/sm/mm/mm_iface.c:233 40 0x00000000000374b4 uct_mm_iface_progress() /tmp/sebox/ucx-1.5.1/src/uct/sm/mm/mm_iface.c:263 41 0x0000000000019e32 ucs_callbackq_dispatch() /tmp/sebox/ucx-1.5.1/src/ucs/datastruct/callbackq.h:209 42 0x0000000000019e32 uct_worker_progress() /tmp/sebox/ucx-1.5.1/src/uct/api/uct.h:1677 43 0x0000000000019e32 ucp_worker_progress() /tmp/sebox/ucx-1.5.1/src/ucp/core/ucp_worker.c:1426 44 0x0000000000003337 mc # McXtrace 3.x-dev - Mar. 31, 2022: [pid 5262] Signal 6 detected [proc 6] SIGABRT (Abort) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # Breakpoint: [_emon_save] component emon=E_monitor() SAVE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/E_monitor.comp:111] 0.00 % ( 0.0/10000000.0) # Date: Fri Apr 1 10:01:12 2022 # Started: Thu Jan 1 01:00:00 1970 # McXtrace 3.x-dev - Mar. 31, 2022: Simulation stop (abort). # McXtrace 3.x-dev - Mar. 31, 2022: [pid 30548] Signal 6 detected [proc 9] SIGABRT (Abort) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # Breakpoint: [_emon_save] component emon=E_monitor() SAVE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/E_monitor.comp:111] 0.00 % ( 0.0/10000000.0) # Date: Fri Apr 1 10:01:12 2022 # Started: Thu Jan 1 01:00:00 1970 # McXtrace 3.x-dev - Mar. 31, 2022: Simulation stop (abort). ==== backtrace ==== 0 0x0000000000019e90 ucp_worker_progress() /tmp/sebox/ucx-1.5.1/src/ucp/core/ucp_worker.c:1425 1 0x0000000000003337 mca_pml_ucx_progress() ???:0 2 0x000000000002f02c opal_progress() ???:0 3 0x000000000004df85 ompi_request_default_wait() ???:0 4 0x00000000000a7793 ompi_coll_base_sendrecv_actual() ???:0 5 0x00000000000a7bbc ompi_coll_base_allreduce_intra_recursivedoubling() ???:0 6 0x0000000000065179 MPI_Allreduce() ???:0 7 0x0000000000402d1d mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:1397 8 0x0000000000405d20 mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:2127 9 0x00000000004068e5 mcdetector_out_1D() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:3262 10 0x000000000041c32b class_E_monitor_save() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:11546 11 0x000000000041c629 save() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:11652 12 0x000000000041efa1 sighandler() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:5401 13 0x0000000000036450 killpg() ???:0 14 0x00000000000292c8 uct_rc_verbs_iface_poll_rx_common() /tmp/sebox/ucx-1.5.1/src/uct/ib/rc/verbs/rc_verbs_common.h:180 15 0x00000000000292c8 uct_rc_verbs_iface_progress() /tmp/sebox/ucx-1.5.1/src/uct/ib/rc/verbs/rc_verbs_iface.c:111 16 0x0000000000019e32 ucs_callbackq_dispatch() /tmp/sebox/ucx-1.5.1/src/ucs/datastruct/callbackq.h:209 17 0x0000000000019e32 uct_worker_progress() /tmp/sebox/ucx-1.5.1/src/uct/api/uct.h:1677 18 0x0000000000019e32 ucp_worker_progress() /tmp/sebox/ucx-1.5.1/src/ucp/core/ucp_worker.c:1426 19 0x0000000000003337 mca_pml_ucx_progress() ???:0 20 0x000000000002f02c opal_progress() ???:0 21 0x000000000004df85 ompi_request_default_wait() ???:0 22 0x00000000000a7793 ompi_coll_base_sendrecv_actual() ???:0 23 0x00000000000a7bbc ompi_coll_base_allreduce_intra_recursivedoubling() ???:0 24 0x0000000000065179 MPI_Allreduce() ???:0 25 0x0000000000402d1d mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:1397 26 0x000000000042043e mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:1384 27 0x0000000000022555 __libc_start_main() ???:0 28 0x0000000000401ffe _start() ???:0 =================== # Last I/O Error: No such file or directory # McXtrace 3.x-dev - Mar. 31, 2022: [pid 3942] Signal 6 detected [proc 1] SIGABRT (Abort) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # Breakpoint: [_emon_save] component emon=E_monitor() SAVE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/E_monitor.comp:111] 0.00 % ( 0.0/10000000.0) # Date: Fri Apr 1 10:01:12 2022 # Started: Thu Jan 1 01:00:00 1970 # McXtrace 3.x-dev - Mar. 31, 2022: Simulation stop (abort). # McXtrace 3.x-dev - Mar. 31, 2022: [pid 11215] Signal 6 detected [proc 0] SIGABRT (Abort) # Simulation: Test_Absorption (/zhome/89/0/38697/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/Test_Absorption.instr) # Breakpoint: [_emon_save] component emon=E_monitor() SAVE [/zhome/89/0/38697/McXtrace/mcxtrace/3.x-dev/tools/Python/mxrun/../mccodelib/../../../monitors/E_monitor.comp:111] 0.00 % ( 0.0/10000000.0) # Date: Fri Apr 1 10:01:12 2022 # Started: Thu Jan 1 01:00:00 1970 # McXtrace 3.x-dev - Mar. 31, 2022: Simulation stop (abort). a_pml_ucx_progress() ???:0 45 0x000000000002f02c opal_progress() ???:0 46 0x000000000004df85 ompi_request_default_wait() ???:0 47 0x00000000000a7793 ompi_coll_base_sendrecv_actual() ???:0 48 0x00000000000a7bbc ompi_coll_base_allreduce_intra_recursivedoubling() ???:0 49 0x0000000000065179 MPI_Allreduce() ???:0 50 0x0000000000402d1d mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:1397 51 0x000000000042043e mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:1384 52 0x0000000000022555 __libc_start_main() ???:0 53 0x0000000000401ffe _start() ???:0 =================== # Last I/O Error: No such file or directory ==== backtrace ==== 0 0x0000000000019e90 ucp_worker_progress() /tmp/sebox/ucx-1.5.1/src/ucp/core/ucp_worker.c:1425 1 0x0000000000003337 mca_pml_ucx_progress() ???:0 2 0x000000000002f02c opal_progress() ???:0 3 0x000000000004df85 ompi_request_default_wait() ???:0 4 0x00000000000a7793 ompi_coll_base_sendrecv_actual() ???:0 5 0x00000000000a7bbc ompi_coll_base_allreduce_intra_recursivedoubling() ???:0 6 0x0000000000065179 MPI_Allreduce() ???:0 7 0x0000000000402d1d mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:1397 8 0x0000000000405d20 mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:2127 9 0x00000000004068e5 mcdetector_out_1D() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:3262 10 0x000000000041c32b class_E_monitor_save() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:11546 11 0x000000000041c629 save() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:11652 12 0x000000000041efa1 sighandler() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:5401 13 0x0000000000036450 killpg() ???:0 14 0x000000000000c573 pthread_spin_lock() ???:0 =================== # Last I/O Error: No such file or directory ==== backtrace ==== 0 0x0000000000019e90 ucp_worker_progress() /tmp/sebox/ucx-1.5.1/src/ucp/core/ucp_worker.c:1425 1 0x0000000000003de8 mca_pml_ucx_recv() ???:0 2 0x00000000000a806e ompi_coll_base_allreduce_intra_recursivedoubling() ???:0 3 0x0000000000065179 MPI_Allreduce() ???:0 4 0x0000000000402d1d mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:1397 5 0x0000000000405d20 mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:2127 6 0x00000000004068e5 mcdetector_out_1D() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:3262 7 0x000000000041c32b class_E_monitor_save() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:11546 8 0x000000000041c629 save() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:11652 9 0x000000000041efa1 sighandler() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:5401 10 0x0000000000036450 killpg() ???:0 11 0x000000000000c573 pthread_spin_lock() ???:0 =================== # Last I/O Error: No such file or directory ==== backtrace ==== 0 0x0000000000019e90 ucp_worker_progress() /tmp/sebox/ucx-1.5.1/src/ucp/core/ucp_worker.c:1425 1 0x0000000000003de8 mca_pml_ucx_recv() ???:0 2 0x00000000000a7de2 ompi_coll_base_allreduce_intra_recursivedoubling() ???:0 3 0x0000000000065179 MPI_Allreduce() ???:0 4 0x0000000000402d1d mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:1397 5 0x0000000000405d20 mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:2127 6 0x00000000004068e5 mcdetector_out_1D() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:3262 7 0x000000000041c32b class_E_monitor_save() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:11546 8 0x000000000041c629 save() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:11652 9 0x000000000041efa1 sighandler() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:5401 10 0x0000000000036450 killpg() ???:0 11 0x0000000000029652 uct_ib_poll_cq() /tmp/sebox/ucx-1.5.1/src/uct/ib/base/ib_device.h:297 12 0x0000000000029652 uct_rc_verbs_iface_progress() /tmp/sebox/ucx-1.5.1/src/uct/ib/rc/verbs/rc_verbs_iface.c:116 13 0x0000000000019e32 ucs_callbackq_dispatch() /tmp/sebox/ucx-1.5.1/src/ucs/datastruct/callbackq.h:209 14 0x0000000000019e32 uct_worker_progress() /tmp/sebox/ucx-1.5.1/src/uct/api/uct.h:1677 15 0x0000000000019e32 ucp_worker_progress() /tmp/sebox/ucx-1.5.1/src/ucp/core/ucp_worker.c:1426 16 0x0000000000003de8 mca_pml_ucx_recv() ???:0 17 0x00000000000a7de2 ompi_coll_base_allreduce_intra_recursivedoubling() ???:0 18 0x0000000000065179 MPI_Allreduce() ???:0 19 0x0000000000402d1d mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:1397 20 0x000000000042043e mc_MPI_Sum() /work1/pkwi/xTESTS/2022-04-01/McXtrace_CPU_MPICC_KISS_5e7/Test_Absorption/./Test_Absorption.c:1384 21 0x0000000000022555 __libc_start_main() ???:0 22 0x0000000000401ffe _start() ???:0 =================== # Last I/O Error: No such file or directory -------------------------------------------------------------------------- mpirun noticed that process rank 4 with PID 32324 on node n-62-29-35 exited on signal 3 (Quit). -------------------------------------------------------------------------- 3 total processes killed (some possibly by mpirun during cleanup)