Job Started: Mon Dec 28 09:53:18 CST 2015 Job Id: 8810474 12/28/15 09:53:18 Welcome to Task Manager Version 2015. 12/28/15 09:53:18 Writing '.tm_config' $config{"tm_domain"} = "boomer"; $config{"tm_region"} = "casa2015"; $config{"tm_root"} = "/home/kwthomas/rt2015"; $config{"tm_user"} = "/home/kwthomas/rt2015/user"; read_cfg: Reading /home/kwthomas/rt2015/config/boomer.cfg read_cfg: Reading /home/kwthomas/rt2015/user/config/casa2015.cfg 12/28/15 09:53:18 Writing '.rt_config' $config{"acct"} = ""; $config{"allow_mpi"} = "1"; $config{"casa"} = "1"; $config{"is_boomer"} = "1"; $config{"mpi_cmd"} = "/home/kwthomas/rt2015/scripts/mpiwrapper -np"; $config{"msg"} = "BOOMER CASA Edition"; $config{"ncarg_root"} = "/home/ywang/tools/ncarg"; $config{"non_mpi_cmd"} = ""; $config{"plot_queue"} = "caps_post"; $config{"procs_per_node"} = "16"; $config{"procs_per_node_limit"} = "1"; $config{"queue"} = "caps_forecast"; $config{"radar_mpi_cmd"} = "mpirun.lsf -np"; $config{"shm_post"} = "1"; $config{"submit_cmd"} = "bsub < "; $config{"submit_plot"} = "1"; $config{"subvers"} = "openmpi"; $config{"vers"} = "casa2015"; 12/28/15 09:53:18 BOOMER CASA Edition. 12/28/15 09:53:18 MODEL=arpspltcycle2 12/28/15 09:53:18 MODE=run 12/28/15 09:53:18 PREFIX=ar2015122815 12/28/15 09:53:18 DIR1=/work/arpsdata/arpscasa/fcst-casa2014_dfw/20151228/1545Z 12/28/15 09:53:18 DIR2=/work/arpsdata/arpscasa/fcst-casa2014_dfw/20151228/1545Z 12/28/15 09:53:18 CONFIG=casa2015 12/28/15 09:53:18 SOFTWARE=casa2015 12/28/15 09:53:18 VERSION=openmpi 12/28/15 09:53:18 MPI=enabled 12/28/15 09:53:18 REGION=fcst-casa2014_dfw 12/28/15 09:53:18 SOFTDIR=/home/kwthomas/rt2015/user/software/casa2015/openmpi 12/28/15 09:53:18 SCRIPTSDIR=/home/kwthomas/rt2015/scripts 12/28/15 09:53:18 USER_HOME=/home/kwthomas/rt2015/user 12/28/15 09:53:18 MPI_CMD=/home/kwthomas/rt2015/scripts/mpiwrapper -np 12/28/15 09:53:18 'arps' version is 'arps5.3.6_20150513' 12/28/15 09:53:18 'arw' version is not defined 12/28/15 09:53:18 'coamps' version is not defined 12/28/15 09:53:18 DELETE_TASK: arps3dvar_retrieval_run (not retrieving) 12/28/15 09:53:18 DELETE_TASK: arps_retrieval_run (not retrieving) 12/28/15 09:53:18 DELETE_TASK: adas_retrieval_run (not retrieving) 12/28/15 09:53:18 DELETE_TASK: arpsenkf_retrieval_run (not retrieving) 12/28/15 09:53:18 DELETE_TASK: extm_lbc_retrieval_run (not retrieving) 12/28/15 09:53:18 DELETE_TASK: extm_t0_retrieval_run (not retrieving) 12/28/15 09:53:18 CYCLES=2 12/28/15 09:53:18 Creation of '.cycle' was successful. 12/28/15 09:53:18 DELETE_TASK: arps_submit (non-remote task) 12/28/15 09:53:18 DELETE_TASK: wrf_submit (non-remote task) 12/28/15 09:53:18 DELETE_TASK: cycle_exit_enkf_run (no cycle exit) 12/28/15 09:53:18 DELETE_TASK: cycle_exit_wrf_run (no cycle exit) 12/28/15 09:53:18 DELETE_TASK: arps2coamps_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: wrf_nmm_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: install_namelist_wrf_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: plot_coamps_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: coamps_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: wrf_arw_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: wrf_nmm_ferrier_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: nmm4wrf_lbc_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: real_nmm_ferrier_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: wrfdaupdate_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: real_clean_arw_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: arw4wrf_lbc_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: plot_wrf_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: real_clean_nmm_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: real_nmm_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: arw4wrf_t0_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: real_arw_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: coamps_pre_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: install_namelist_real_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: nmm4wrf_t0_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: plot_adas_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: plot_arps3dvar_run (ARPS run) 12/28/15 09:53:18 DELETE_TASK: ens_ana_run (no input file) 12/28/15 09:53:18 DELETE_TASK: plot_arpsenkf_run (no input file) 12/28/15 09:53:18 DELETE_TASK: arw_pre_run (no input file) 12/28/15 09:53:18 DELETE_TASK: arpsenkf_run (no input file) 12/28/15 09:53:18 DELETE_TASK: adas_run (no input file) 12/28/15 09:53:18 DELETE_TASK: enkf_anal_run (no input file) 12/28/15 09:53:18 DELETE_TASK: local_run (nonexistent script 'run_fcst-casa2014_dfw') 12/28/15 09:53:18 DELETE_TASK: arpsintrp_lbc_run (no input file) 12/28/15 09:53:18 DELETE_TASK: arpsensic_run (no input file) 12/28/15 09:53:18 DELETE_TASK: satellite_run (no input file) 12/28/15 09:53:18 DELETE_TASK: arpstrn_run (no input file) 12/28/15 09:53:18 DELETE_TASK: verif_run (no input file) 12/28/15 09:53:18 DELETE_TASK: arpssfc_run (no input file) 12/28/15 09:53:18 DELETE_TASK: arpsintrp_t0_run (no input file) 12/28/15 09:53:18 DELETE_TASK: enkf_radar_run (no input file) 12/28/15 09:53:18 DELETE_TASK: enkf_watch_run (no input file) 12/28/15 09:53:18 DELETE_TASK: nmm_pre_run (no input file) 12/28/15 09:53:18 Will use '/home/kwthomas/rt2015/scripts/pre_task_boomer' for task 'pre_task' 12/28/15 09:53:18 TASK_CHAIN=pre_task_0_run find_extm_0_run extm_t0_0_run arpstintrp_0_run extm_lbc_0_run radar_0_run plot_arps_0_run arps3dvar_0_run arps_0_run radar_1_run arps3dvar_1_run arps_1_run nudge_outputs_0_run done 12/28/15 09:53:18 TASK=pre_task_0_run 12/28/15 09:53:18 TASK=find_extm_0_run 12/28/15 09:53:18 DEPENDS= pre_task_0_run 12/28/15 09:53:18 TASK=extm_t0_0_run 12/28/15 09:53:18 DEPENDS= find_extm_0_run pre_task_0_run 12/28/15 09:53:18 TASK=arpstintrp_0_run 12/28/15 09:53:18 DEPENDS= find_extm_0_run pre_task_0_run extm_t0_0_run 12/28/15 09:53:18 TASK=extm_lbc_0_run 12/28/15 09:53:18 DEPENDS= arpstintrp_0_run find_extm_0_run pre_task_0_run extm_t0_0_run 12/28/15 09:53:18 TASK=radar_0_run 12/28/15 09:53:18 DEPENDS= arpstintrp_0_run find_extm_0_run extm_lbc_0_run pre_task_0_run extm_t0_0_run 12/28/15 09:53:18 TASK=plot_arps_0_run 12/28/15 09:53:18 DEPENDS= arpstintrp_0_run radar_0_run extm_t0_0_run 12/28/15 09:53:18 TASK=arps3dvar_0_run 12/28/15 09:53:18 DEPENDS= arpstintrp_0_run find_extm_0_run extm_lbc_0_run radar_0_run pre_task_0_run extm_t0_0_run 12/28/15 09:53:18 TASK=arps_0_run 12/28/15 09:53:18 DEPENDS= arps3dvar_0_run arpstintrp_0_run find_extm_0_run extm_lbc_0_run radar_0_run pre_task_0_run extm_t0_0_run 12/28/15 09:53:18 TASK=radar_1_run 12/28/15 09:53:18 DEPENDS= arps3dvar_0_run arpstintrp_0_run find_extm_0_run extm_lbc_0_run radar_0_run pre_task_0_run extm_t0_0_run arps_0_run 12/28/15 09:53:18 TASK=arps3dvar_1_run 12/28/15 09:53:18 DEPENDS= arps3dvar_0_run extm_lbc_0_run radar_1_run pre_task_0_run extm_t0_0_run arpstintrp_0_run find_extm_0_run radar_0_run arps_0_run 12/28/15 09:53:18 TASK=arps_1_run 12/28/15 09:53:18 DEPENDS= arps3dvar_0_run arps3dvar_1_run extm_lbc_0_run radar_1_run pre_task_0_run extm_t0_0_run arpstintrp_0_run radar_0_run find_extm_0_run arps_0_run 12/28/15 09:53:18 TASK=nudge_outputs_0_run 12/28/15 09:53:18 DEPENDS= arps3dvar_0_run arps3dvar_1_run extm_lbc_0_run radar_1_run pre_task_0_run extm_t0_0_run arps_1_run arpstintrp_0_run radar_0_run find_extm_0_run arps_0_run 12/28/15 09:53:18 TASK=done 12/28/15 09:53:18 DEPENDS= arps3dvar_1_run arps3dvar_0_run radar_1_run extm_lbc_0_run nudge_outputs_0_run extm_t0_0_run pre_task_0_run arps_1_run arpstintrp_0_run plot_arps_0_run find_extm_0_run radar_0_run arps_0_run 12/28/15 09:53:18 Task 'pre_task_0 run' starts. 12/28/15 09:53:18 /home/kwthomas/rt2015/scripts/pre_task_boomer 12/28/15 09:53:18 read_cfg: Reading .tm_config $config{"tm_domain"} = "boomer"; $config{"tm_region"} = "casa2015"; $config{"tm_root"} = "/home/kwthomas/rt2015"; $config{"tm_user"} = "/home/kwthomas/rt2015/user"; 12/28/15 09:53:18 read_cfg: Reading .rt_config $config{"acct"} = ""; $config{"allow_mpi"} = "1"; $config{"casa"} = "1"; $config{"is_boomer"} = "1"; $config{"mpi_cmd"} = "/home/kwthomas/rt2015/scripts/mpiwrapper -np"; $config{"msg"} = "BOOMER CASA Edition"; $config{"ncarg_root"} = "/home/ywang/tools/ncarg"; $config{"non_mpi_cmd"} = ""; $config{"plot_queue"} = "caps_post"; $config{"procs_per_node"} = "16"; $config{"procs_per_node_limit"} = "1"; $config{"queue"} = "caps_forecast"; $config{"radar_mpi_cmd"} = "mpirun.lsf -np"; $config{"shm_post"} = "1"; $config{"submit_cmd"} = "bsub < "; $config{"submit_plot"} = "1"; $config{"subvers"} = "openmpi"; $config{"vers"} = "casa2015"; 12/28/15 09:53:18 pre_task_boomer: Pretask_Boomer 12/28/15 09:53:18 pre_task_boomer: Command is '/home/kwthomas/rt2015/scripts/boomer_clean_dev_shm' 12/28/15 09:53:18 Welcome to 'boomer_clean_dev_shm' 12/28/15 09:53:18 Checking c110 12/28/15 09:53:18 Checking c086 12/28/15 09:53:18 Checking c102 12/28/15 09:53:18 Checking c082 12/28/15 09:53:18 Checking c084 12/28/15 09:53:18 Checking c109 12/28/15 09:53:18 Checking c085 12/28/15 09:53:18 Checking c087 12/28/15 09:53:18 Checking c104 12/28/15 09:53:18 Checking c089 12/28/15 09:53:18 Checking c108 12/28/15 09:53:18 Checking c083 12/28/15 09:53:18 'boomer_clean_dev_shm' is finished 12/28/15 09:53:18 pre_task_boomer: Command is '/home/kwthomas/rt2015/scripts/boomer_clean_radar_jobs' 12/28/15 09:53:18 Welcome to 'boomer_clean_radar_jobs' 12/28/15 09:53:18 Checking c110 12/28/15 09:53:18 Checking c086 12/28/15 09:53:18 Checking c102 12/28/15 09:53:18 Checking c082 12/28/15 09:53:18 Checking c084 12/28/15 09:53:18 Checking c109 12/28/15 09:53:18 Checking c085 12/28/15 09:53:18 Checking c087 12/28/15 09:53:18 Checking c104 12/28/15 09:53:18 Checking c089 12/28/15 09:53:18 Checking c108 12/28/15 09:53:18 Checking c083 12/28/15 09:53:19 'boomer_clean_radar_jobs' is finished 12/28/15 09:53:19 pre_task_0 run: Exit Status 0. 12/28/15 09:53:19 Task 'pre_task_0 run' finishes. Runtime 00:00:01. 12/28/15 09:53:19 Task 'find_extm_0 run' starts. 12/28/15 09:53:19 /home/kwthomas/rt2015/scripts/find_extm 12/28/15 09:53:19 External data already present. 12/28/15 09:53:20 find_extm_0 run: Exit Status 0. 12/28/15 09:53:20 Task 'find_extm_0 run' finishes. Runtime 00:00:01. 12/28/15 09:53:20 Task 'extm_t0_0 run' starts. 12/28/15 09:53:20 template: reading em2015122815-t0.extin 12/28/15 09:53:20 match for 'nproc_x' on line 211 - nproc_x=12, 12/28/15 09:53:20 match for 'nproc_y' on line 212 - nproc_y=16, 12/28/15 09:53:20 nproc_x/nproc_y=12/16 12/28/15 09:53:20 /home/kwthomas/rt2015/scripts/mpiwrapper -np 192 /home/kwthomas/rt2015/user/software/casa2015/openmpi/arps/abin/ext2arps_mpi < em2015122815-t0.extin >> em2015122815-t0.extout 2>&1 12/28/15 09:53:50 extm_t0_0 run: Exit Status 0. 12/28/15 09:53:50 Task 'extm_t0_0 run' finishes. Runtime 00:00:30. 12/28/15 09:53:50 Task 'arpstintrp_0 run' starts. 12/28/15 09:53:50 template: reading ei2015122815-t0.tintrpin 12/28/15 09:53:50 match for 'nproc_x' on line 70 - nproc_x=12, 12/28/15 09:53:50 match for 'nproc_y' on line 71 - nproc_y=16, 12/28/15 09:53:50 match for 'nproc_x_in' on line 73 - nproc_x_in=12, 12/28/15 09:53:50 match for 'nproc_y_in' on line 74 - nproc_y_in=16, 12/28/15 09:53:50 nproc_x/nproc_y=12/16 12/28/15 09:53:50 /home/kwthomas/rt2015/scripts/mpiwrapper -np 192 /home/kwthomas/rt2015/user/software/casa2015/openmpi/arps/abin/arpstintrp_mpi < ei2015122815-t0.tintrpin >> ei2015122815-t0.tintrpout 2>&1 12/28/15 09:54:15 arpstintrp_0 run: Exit Status 0. 12/28/15 09:54:15 Task 'arpstintrp_0 run' finishes. Runtime 00:00:25. 12/28/15 09:54:15 Task 'extm_lbc_0 run' starts. 12/28/15 09:54:15 template: reading em2015122815-lbc.extin 12/28/15 09:54:15 match for 'nproc_x' on line 211 - nproc_x=12, 12/28/15 09:54:15 match for 'nproc_y' on line 212 - nproc_y=16, 12/28/15 09:54:15 nproc_x/nproc_y=12/16 12/28/15 09:54:15 /home/kwthomas/rt2015/scripts/mpiwrapper -np 192 /home/kwthomas/rt2015/user/software/casa2015/openmpi/arps/abin/ext2arps_mpi < em2015122815-lbc.extin >> em2015122815-lbc.extout 2>&1 12/28/15 09:54:48 extm_lbc_0 run: Exit Status 0. 12/28/15 09:54:48 Task 'extm_lbc_0 run' finishes. Runtime 00:00:33. 12/28/15 09:54:48 Found cycle1/.start 12/28/15 09:54:48 Run directory is cycle1. 12/28/15 09:54:48 Task 'radar_0 run' starts. 12/28/15 09:54:48 /home/kwthomas/rt2015/scripts/run_radar 12/28/15 09:54:48 read_cfg: Reading ../.tm_config $config{"tm_domain"} = "boomer"; $config{"tm_region"} = "casa2015"; $config{"tm_root"} = "/home/kwthomas/rt2015"; $config{"tm_user"} = "/home/kwthomas/rt2015/user"; 12/28/15 09:54:48 read_cfg: Reading ../.rt_config $config{"acct"} = ""; $config{"allow_mpi"} = "1"; $config{"casa"} = "1"; $config{"is_boomer"} = "1"; $config{"mpi_cmd"} = "/home/kwthomas/rt2015/scripts/mpiwrapper -np"; $config{"msg"} = "BOOMER CASA Edition"; $config{"ncarg_root"} = "/home/ywang/tools/ncarg"; $config{"non_mpi_cmd"} = ""; $config{"plot_queue"} = "caps_post"; $config{"procs_per_node"} = "16"; $config{"procs_per_node_limit"} = "1"; $config{"queue"} = "caps_forecast"; $config{"radar_mpi_cmd"} = "mpirun.lsf -np"; $config{"shm_post"} = "1"; $config{"submit_cmd"} = "bsub < "; $config{"submit_plot"} = "1"; $config{"subvers"} = "openmpi"; $config{"vers"} = "casa2015"; 12/28/15 09:54:48 Welcome to 'run_radar' 12/28/15 09:54:48 12/28/15 09:54:48 Cycle 1 12/28/15 09:54:48 template: reading casa/casa2015122815.casain 12/28/15 09:54:48 match for 'nproc_x' on line 88 - nproc_x=2, 12/28/15 09:54:48 match for 'nproc_y' on line 89 - nproc_y=8, 12/28/15 09:54:48 run_radar: nexrad0: No 'nids/*.r88din' file. 12/28/15 09:54:48 template: reading nexrad/r882015122815.r88din 12/28/15 09:54:48 match for 'nproc_x' on line 88 - nproc_x=2, 12/28/15 09:54:48 match for 'nproc_y' on line 89 - nproc_y=8, 12/28/15 09:54:48 template: reading nids/nd2015122815.nidsin 12/28/15 09:54:48 match for 'nproc_x' on line 88 - nproc_x=2, 12/28/15 09:54:48 match for 'nproc_y' on line 89 - nproc_y=8, 12/28/15 09:54:48 template: reading tdwr/tdwr2015122815.tdwrin 12/28/15 09:54:48 match for 'nproc_x' on line 88 - nproc_x=2, 12/28/15 09:54:48 match for 'nproc_y' on line 89 - nproc_y=8, 12/28/15 09:54:48 template: reading ../em2015122815-t0.extin 12/28/15 09:54:48 match for 'nproc_x' on line 211 - nproc_x=12, 12/28/15 09:54:48 match for 'nproc_y' on line 212 - nproc_y=16, 12/28/15 09:54:48 match for 'nproc_x_out' on line 216 - nproc_x_out=1, 12/28/15 09:54:48 match for 'nproc_y_out' on line 217 - nproc_y_out=1, 12/28/15 09:54:48 match for 'dmp_out_joined' on line 2160 - dmp_out_joined=0, 12/28/15 09:54:48 template: reading ../ei2015122815-t0.tintrpin 12/28/15 09:54:48 match for 'nproc_x' on line 70 - nproc_x=12, 12/28/15 09:54:48 match for 'nproc_y' on line 71 - nproc_y=16, 12/28/15 09:54:48 match for 'nproc_x_out' on line 76 - nproc_x_out=12, 12/28/15 09:54:48 match for 'nproc_y_out' on line 77 - nproc_y_out=16, 12/28/15 09:54:48 Splitfiles detected. 12/28/15 09:54:48 Making directory 'radar' 12/28/15 09:54:48 Number of processors available is 192. 12/28/15 09:54:48 Number of processors per node is 1. 12/28/15 09:54:48 nslots=12 12/28/15 09:54:48 njobs=11 12/28/15 09:54:48 nprocs=16 12/28/15 09:54:48 12/28/15 09:54:48 max=1 12/28/15 09:54:48 cmd=radar1_0_radar 12/28/15 09:54:48 cmd=radar2_0_radar 12/28/15 09:54:48 cmd=radar3_0_radar 12/28/15 09:54:48 cmd=radar4_0_radar 12/28/15 09:54:48 cmd=radar5_0_radar 12/28/15 09:54:48 cmd=radar6_0_radar 12/28/15 09:54:48 cmd=radar7_0_radar 12/28/15 09:54:48 cmd=radar8_0_radar 12/28/15 09:54:48 cmd=radar9_0_radar 12/28/15 09:54:48 cmd=radar10_0_radar 12/28/15 09:54:48 cmd=radar11_0_radar 12/28/15 09:54:48 cmd=radar12_0_radar 12/28/15 09:54:49 Return to parent directory. 12/28/15 09:55:36 BOOMER radar cleanup. 12/28/15 09:55:36 Welcome to 'boomer_clean_radar_jobs' 12/28/15 09:55:36 Checking c110 12/28/15 09:55:36 Checking c086 12/28/15 09:55:36 Checking c102 12/28/15 09:55:36 Checking c082 12/28/15 09:55:36 Checking c084 12/28/15 09:55:36 Checking c109 12/28/15 09:55:36 Checking c085 12/28/15 09:55:36 Checking c087 12/28/15 09:55:36 Checking c104 12/28/15 09:55:36 Checking c089 12/28/15 09:55:36 Checking c108 12/28/15 09:55:36 Checking c083 12/28/15 09:55:36 'boomer_clean_radar_jobs' is finished 12/28/15 09:55:36 Running /home/kwthomas/rt2015/scripts/casa_post for casa. 12/28/15 09:55:36 Running /home/kwthomas/rt2015/scripts/nexrad_post for nexrad. 12/28/15 09:55:37 Running /home/kwthomas/rt2015/scripts/radar_post for nids. 12/28/15 09:55:37 Running /home/kwthomas/rt2015/scripts/tdwr_post for tdwr. 12/28/15 09:55:37 NEXRAD/NIDS/CASA/TDWR processing is finished. 12/28/15 09:55:37 radar_0 run: Exit Status 0. 12/28/15 09:55:37 Task 'radar_0 run' finishes. Runtime 00:00:49. 12/28/15 09:55:37 Found cycle1/.start 12/28/15 09:55:37 Run directory is cycle1. 12/28/15 09:55:37 Task 'arps3dvar_0 run' starts. 12/28/15 09:55:37 template: reading ad201512281500.3dvarin 12/28/15 09:55:37 match for 'nproc_x' on line 211 - nproc_x=12, 12/28/15 09:55:37 match for 'nproc_y' on line 212 - nproc_y=16, 12/28/15 09:55:37 nproc_x/nproc_y=12/16 12/28/15 09:55:37 /home/kwthomas/rt2015/scripts/mpiwrapper -np 192 /home/kwthomas/rt2015/user/software/casa2015/openmpi/arps/abin/arps3dvar_mpi < ad201512281500.3dvarin >> ad201512281500.3dvarout 2>&1 12/28/15 09:55:38 Return to parent directory. 12/28/15 09:55:38 Task 'plot_arps_0 run' starts. 12/28/15 09:55:38 /home/kwthomas/rt2015/scripts/run_post 12/28/15 09:55:38 read_cfg: Reading .tm_config $config{"tm_domain"} = "boomer"; $config{"tm_region"} = "casa2015"; $config{"tm_root"} = "/home/kwthomas/rt2015"; $config{"tm_user"} = "/home/kwthomas/rt2015/user"; 12/28/15 09:55:38 read_cfg: Reading .rt_config $config{"acct"} = ""; $config{"allow_mpi"} = "1"; $config{"casa"} = "1"; $config{"is_boomer"} = "1"; $config{"mpi_cmd"} = "/home/kwthomas/rt2015/scripts/mpiwrapper -np"; $config{"msg"} = "BOOMER CASA Edition"; $config{"ncarg_root"} = "/home/ywang/tools/ncarg"; $config{"non_mpi_cmd"} = ""; $config{"plot_queue"} = "caps_post"; $config{"procs_per_node"} = "16"; $config{"procs_per_node_limit"} = "1"; $config{"queue"} = "caps_forecast"; $config{"radar_mpi_cmd"} = "mpirun.lsf -np"; $config{"shm_post"} = "1"; $config{"submit_cmd"} = "bsub < "; $config{"submit_plot"} = "1"; $config{"subvers"} = "openmpi"; $config{"vers"} = "casa2015"; 12/28/15 09:55:38 run_post: t_offset=300 t_end=7200 12/28/15 09:55:38 run_post: Read .cycle, value is '2'. 12/28/15 09:55:38 run_post: is_wrf=0 is_coamps=0 soundings=1 12/28/15 09:55:38 run_post: file=cycle1/ar201512281500.arpsin 12/28/15 09:55:38 run_post: match for 'nproc_x' on line 211 - nproc_x=12, 12/28/15 09:55:38 run_post: match for 'nproc_y' on line 212 - nproc_y=16, 12/28/15 09:55:38 run_post: match for 'runname' on line 248 - runname='ar2015122815', 12/28/15 09:55:38 run_post: match for 'initime' on line 417 - initime='2015-12-28.15:35:00', 12/28/15 09:55:38 run_post: match for 'nudgopt' on line 533 - nudgopt=2, 12/28/15 09:55:38 run_post: match for 'ndstop' on line 535 - ndstop=600, 12/28/15 09:55:38 run_post: match for 'dmp_out_joined' on line 2160 - dmp_out_joined=0, 12/28/15 09:55:38 run_post: match for 'hdmpfmt' on line 2161 - hdmpfmt=3, 12/28/15 09:55:38 run_post: match for 'dirname' on line 2344 - dirname='../arpsdata/', 12/28/15 09:55:38 run_post: match for 'readyfl' on line 2346 - readyfl=2, 12/28/15 09:55:38 run_post: NUDGE=600 12/28/15 09:55:38 run_post: 12/28/15 09:55:38 run_post: t_end modified to 7800 12/28/15 09:55:38 run_post: arps_subdirs=1 wrf_subdirs=0 12/28/15 09:55:38 run_post: Runname is ar2015122815 12/28/15 09:55:38 run_post: Data uses splitfiles. 12/28/15 09:55:38 run_post: match for 'nproc_x' on line 3780 - nproc_x=2, 12/28/15 09:55:38 run_post: match for 'nproc_y' on line 3780 - nproc_y=8, 12/28/15 09:55:39 run_post: Region is casa2014_dfw 12/28/15 09:55:39 run_post: t_current now 0 t_end=7800 12/28/15 09:55:39 run_post: Looking for 'arpsdata/ar2015122815.hdf000000_ready' 12/28/15 09:55:54 run_post: Looking for 'arpsdata/ar2015122815.hdf000000_ready' 12/28/15 09:56:09 run_post: Looking for 'arpsdata/ar2015122815.hdf000000_ready' 12/28/15 09:56:24 run_post: Looking for 'arpsdata/ar2015122815.hdf000000_ready' 12/28/15 09:56:39 run_post: Looking for 'arpsdata/ar2015122815.hdf000000_ready' 12/28/15 09:56:52 arps3dvar_0 run: Exit Status 0. 12/28/15 09:56:52 Task 'arps3dvar_0 run' finishes. Runtime 00:01:15. 12/28/15 09:56:52 Found cycle1/.start 12/28/15 09:56:52 Run directory is cycle1. 12/28/15 09:56:52 Task 'arps_0 run' starts. 12/28/15 09:56:52 template: reading ar201512281500.arpsin 12/28/15 09:56:52 match for 'nproc_x' on line 211 - nproc_x=12, 12/28/15 09:56:52 match for 'nproc_y' on line 212 - nproc_y=16, 12/28/15 09:56:52 nproc_x/nproc_y=12/16 12/28/15 09:56:52 /home/kwthomas/rt2015/scripts/mpiwrapper -np 192 /home/kwthomas/rt2015/user/software/casa2015/openmpi/arps/abin/arps_mpi < ar201512281500.arpsin >> ar201512281500.arpsout 2>&1 12/28/15 09:56:53 Return to parent directory. 12/28/15 09:56:54 run_post: Looking for 'arpsdata/ar2015122815.hdf000000_ready' 12/28/15 09:57:09 run_post: This nudging event not plotted. 12/28/15 09:57:09 run_post: t_current now 300 t_end=7800 12/28/15 09:57:09 run_post: Looking for 'arpsdata/ar2015122815.hdf000300_ready' 12/28/15 09:57:24 run_post: This nudging event not plotted. 12/28/15 09:57:24 run_post: t_current now 600 t_end=7800 12/28/15 09:57:24 run_post: Looking for 'arpsdata/ar2015122815.hdf000600_ready' 12/28/15 09:57:39 run_post: p_string=000000 h_string=0000 t_string=000600 12/28/15 09:57:39 run_post: Checking plot/0000/arpsplt_0000.nqs 12/28/15 09:57:39 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 09:57:39 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 09:57:39 run_post: Processing sfc.pltin 12/28/15 09:57:39 run_post: Processing sfcchar.pltin 12/28/15 09:57:39 run_post: Processing ua.pltin 12/28/15 09:57:39 run_post: Processing tmp.aein 12/28/15 09:57:39 run_post: Job <8810476> is submitted to queue . 12/28/15 09:57:40 run_post: t_current now 900 t_end=7800 12/28/15 09:57:40 run_post: Looking for 'arpsdata/ar2015122815.hdf000900_ready' 12/28/15 09:57:43 arps_0 run: Exit Status 0. 12/28/15 09:57:43 Task 'arps_0 run' finishes. Runtime 00:00:51. 12/28/15 09:57:43 Found cycle2/.start 12/28/15 09:57:43 Run directory is cycle2. 12/28/15 09:57:43 Task 'radar_1 run' starts. 12/28/15 09:57:43 /home/kwthomas/rt2015/scripts/run_radar 12/28/15 09:57:43 read_cfg: Reading ../.tm_config $config{"tm_domain"} = "boomer"; $config{"tm_region"} = "casa2015"; $config{"tm_root"} = "/home/kwthomas/rt2015"; $config{"tm_user"} = "/home/kwthomas/rt2015/user"; 12/28/15 09:57:43 read_cfg: Reading ../.rt_config $config{"acct"} = ""; $config{"allow_mpi"} = "1"; $config{"casa"} = "1"; $config{"is_boomer"} = "1"; $config{"mpi_cmd"} = "/home/kwthomas/rt2015/scripts/mpiwrapper -np"; $config{"msg"} = "BOOMER CASA Edition"; $config{"ncarg_root"} = "/home/ywang/tools/ncarg"; $config{"non_mpi_cmd"} = ""; $config{"plot_queue"} = "caps_post"; $config{"procs_per_node"} = "16"; $config{"procs_per_node_limit"} = "1"; $config{"queue"} = "caps_forecast"; $config{"radar_mpi_cmd"} = "mpirun.lsf -np"; $config{"shm_post"} = "1"; $config{"submit_cmd"} = "bsub < "; $config{"submit_plot"} = "1"; $config{"subvers"} = "openmpi"; $config{"vers"} = "casa2015"; 12/28/15 09:57:43 Welcome to 'run_radar' 12/28/15 09:57:43 12/28/15 09:57:43 Cycle 2 12/28/15 09:57:43 template: reading casa/casa2015122815.casain 12/28/15 09:57:43 match for 'nproc_x' on line 88 - nproc_x=2, 12/28/15 09:57:43 match for 'nproc_y' on line 89 - nproc_y=8, 12/28/15 09:57:43 run_radar: nexrad0: No 'nids/*.r88din' file. 12/28/15 09:57:43 template: reading nexrad/r882015122815.r88din 12/28/15 09:57:43 match for 'nproc_x' on line 88 - nproc_x=2, 12/28/15 09:57:43 match for 'nproc_y' on line 89 - nproc_y=8, 12/28/15 09:57:43 template: reading nids/nd2015122815.nidsin 12/28/15 09:57:43 match for 'nproc_x' on line 88 - nproc_x=2, 12/28/15 09:57:43 match for 'nproc_y' on line 89 - nproc_y=8, 12/28/15 09:57:43 template: reading tdwr/tdwr2015122815.tdwrin 12/28/15 09:57:43 match for 'nproc_x' on line 88 - nproc_x=2, 12/28/15 09:57:43 match for 'nproc_y' on line 89 - nproc_y=8, 12/28/15 09:57:43 template: reading ../em2015122815-t0.extin 12/28/15 09:57:43 match for 'nproc_x' on line 211 - nproc_x=12, 12/28/15 09:57:43 match for 'nproc_y' on line 212 - nproc_y=16, 12/28/15 09:57:43 match for 'nproc_x_out' on line 216 - nproc_x_out=1, 12/28/15 09:57:43 match for 'nproc_y_out' on line 217 - nproc_y_out=1, 12/28/15 09:57:44 match for 'dmp_out_joined' on line 2160 - dmp_out_joined=0, 12/28/15 09:57:44 template: reading ../ei2015122815-t0.tintrpin 12/28/15 09:57:44 match for 'nproc_x' on line 70 - nproc_x=12, 12/28/15 09:57:44 match for 'nproc_y' on line 71 - nproc_y=16, 12/28/15 09:57:44 match for 'nproc_x_out' on line 76 - nproc_x_out=12, 12/28/15 09:57:44 match for 'nproc_y_out' on line 77 - nproc_y_out=16, 12/28/15 09:57:44 Splitfiles detected. 12/28/15 09:57:44 Making directory 'radar' 12/28/15 09:57:44 Number of processors available is 192. 12/28/15 09:57:44 Number of processors per node is 1. 12/28/15 09:57:44 nslots=12 12/28/15 09:57:44 njobs=11 12/28/15 09:57:44 nprocs=16 12/28/15 09:57:44 12/28/15 09:57:44 max=1 12/28/15 09:57:44 cmd=radar1_0_radar 12/28/15 09:57:44 cmd=radar2_0_radar 12/28/15 09:57:44 cmd=radar3_0_radar 12/28/15 09:57:44 cmd=radar4_0_radar 12/28/15 09:57:44 cmd=radar5_0_radar 12/28/15 09:57:44 cmd=radar6_0_radar 12/28/15 09:57:44 cmd=radar7_0_radar 12/28/15 09:57:44 cmd=radar8_0_radar 12/28/15 09:57:44 cmd=radar9_0_radar 12/28/15 09:57:44 cmd=radar10_0_radar 12/28/15 09:57:44 cmd=radar11_0_radar 12/28/15 09:57:44 cmd=radar12_0_radar 12/28/15 09:57:44 Return to parent directory. 12/28/15 09:57:55 run_post: Looking for 'arpsdata/ar2015122815.hdf000900_ready' 12/28/15 09:58:10 run_post: Looking for 'arpsdata/ar2015122815.hdf000900_ready' 12/28/15 09:58:25 run_post: Looking for 'arpsdata/ar2015122815.hdf000900_ready' 12/28/15 09:58:29 BOOMER radar cleanup. 12/28/15 09:58:29 Welcome to 'boomer_clean_radar_jobs' 12/28/15 09:58:29 Checking c110 12/28/15 09:58:29 Checking c086 12/28/15 09:58:29 Checking c102 12/28/15 09:58:29 Checking c082 12/28/15 09:58:29 Checking c084 12/28/15 09:58:29 Checking c109 12/28/15 09:58:29 Checking c085 12/28/15 09:58:29 Checking c087 12/28/15 09:58:29 Checking c104 12/28/15 09:58:29 Checking c089 12/28/15 09:58:29 Checking c108 12/28/15 09:58:29 Checking c083 12/28/15 09:58:29 'boomer_clean_radar_jobs' is finished 12/28/15 09:58:29 Running /home/kwthomas/rt2015/scripts/casa_post for casa. 12/28/15 09:58:30 Running /home/kwthomas/rt2015/scripts/nexrad_post for nexrad. 12/28/15 09:58:30 Running /home/kwthomas/rt2015/scripts/radar_post for nids. 12/28/15 09:58:30 Running /home/kwthomas/rt2015/scripts/tdwr_post for tdwr. 12/28/15 09:58:30 NEXRAD/NIDS/CASA/TDWR processing is finished. 12/28/15 09:58:30 radar_1 run: Exit Status 0. 12/28/15 09:58:30 Task 'radar_1 run' finishes. Runtime 00:00:47. 12/28/15 09:58:30 Found cycle2/.start 12/28/15 09:58:30 Run directory is cycle2. 12/28/15 09:58:30 Task 'arps3dvar_1 run' starts. 12/28/15 09:58:30 Change '*.3dvarin' to 'ad201512281510.3dvarin' 12/28/15 09:58:30 Change 'arps/abin/arps3dvar_mpi < *.3dvarin >> *.3dvarout 2>&1' to 12/28/15 09:58:30 'arps/abin/arps3dvar_mpi < ad201512281510.3dvarin >> ad201512281510.3dvarout 2>&1' 12/28/15 09:58:30 template: reading ad201512281510.3dvarin 12/28/15 09:58:30 match for 'nproc_x' on line 211 - nproc_x=12, 12/28/15 09:58:30 match for 'nproc_y' on line 212 - nproc_y=16, 12/28/15 09:58:30 nproc_x/nproc_y=12/16 12/28/15 09:58:30 /home/kwthomas/rt2015/scripts/mpiwrapper -np 192 /home/kwthomas/rt2015/user/software/casa2015/openmpi/arps/abin/arps3dvar_mpi < ad201512281510.3dvarin >> ad201512281510.3dvarout 2>&1 12/28/15 09:58:31 Return to parent directory. 12/28/15 09:58:40 run_post: Looking for 'arpsdata/ar2015122815.hdf000900_ready' 12/28/15 09:58:55 run_post: Looking for 'arpsdata/ar2015122815.hdf000900_ready' 12/28/15 09:59:10 run_post: Looking for 'arpsdata/ar2015122815.hdf000900_ready' 12/28/15 09:59:25 run_post: Looking for 'arpsdata/ar2015122815.hdf000900_ready' 12/28/15 09:59:40 run_post: Looking for 'arpsdata/ar2015122815.hdf000900_ready' 12/28/15 09:59:46 arps3dvar_1 run: Exit Status 0. 12/28/15 09:59:46 Task 'arps3dvar_1 run' finishes. Runtime 00:01:16. 12/28/15 09:59:46 Found cycle2/.start 12/28/15 09:59:46 Run directory is cycle2. 12/28/15 09:59:46 Task 'arps_1 run' starts. 12/28/15 09:59:46 Change '*.arpsin' to 'ar201512281510.arpsin' 12/28/15 09:59:46 Change 'arps/abin/arps_mpi < *.arpsin >> *.arpsout 2>&1' to 12/28/15 09:59:46 'arps/abin/arps_mpi < ar201512281510.arpsin >> ar201512281510.arpsout 2>&1' 12/28/15 09:59:46 template: reading ar201512281510.arpsin 12/28/15 09:59:46 match for 'nproc_x' on line 211 - nproc_x=12, 12/28/15 09:59:46 match for 'nproc_y' on line 212 - nproc_y=16, 12/28/15 09:59:46 nproc_x/nproc_y=12/16 12/28/15 09:59:46 /home/kwthomas/rt2015/scripts/mpiwrapper -np 192 /home/kwthomas/rt2015/user/software/casa2015/openmpi/arps/abin/arps_mpi < ar201512281510.arpsin >> ar201512281510.arpsout 2>&1 12/28/15 09:59:47 Return to parent directory. 12/28/15 09:59:55 run_post: Looking for 'arpsdata/ar2015122815.hdf000900_ready' 12/28/15 10:00:10 run_post: Looking for 'arpsdata/ar2015122815.hdf000900_ready' 12/28/15 10:00:25 run_post: Looking for 'arpsdata/ar2015122815.hdf000900_ready' 12/28/15 10:00:40 run_post: p_string=000300 h_string=0005 t_string=000900 12/28/15 10:00:40 run_post: Checking plot/0005/arpsplt_0005.nqs 12/28/15 10:00:40 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:00:40 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:00:40 run_post: Processing sfc.pltin 12/28/15 10:00:40 run_post: Processing sfcchar.pltin 12/28/15 10:00:40 run_post: Processing ua.pltin 12/28/15 10:00:40 run_post: Processing tmp.aein 12/28/15 10:00:40 run_post: Job <8810478> is submitted to queue . 12/28/15 10:00:41 run_post: t_current now 1200 t_end=7800 12/28/15 10:00:41 run_post: Looking for 'arpsdata/ar2015122815.hdf001200_ready' 12/28/15 10:00:56 run_post: p_string=000600 h_string=0010 t_string=001200 12/28/15 10:00:56 run_post: Checking plot/0010/arpsplt_0010.nqs 12/28/15 10:00:56 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:00:56 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:00:56 run_post: Processing sfc.pltin 12/28/15 10:00:56 run_post: Processing sfcchar.pltin 12/28/15 10:00:56 run_post: Processing ua.pltin 12/28/15 10:00:56 run_post: Processing tmp.aein 12/28/15 10:00:56 run_post: Job <8810479> is submitted to queue . 12/28/15 10:00:58 run_post: t_current now 1500 t_end=7800 12/28/15 10:00:58 run_post: Looking for 'arpsdata/ar2015122815.hdf001500_ready' 12/28/15 10:01:13 run_post: p_string=000900 h_string=0015 t_string=001500 12/28/15 10:01:13 run_post: Checking plot/0015/arpsplt_0015.nqs 12/28/15 10:01:13 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:01:13 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:01:13 run_post: Processing sfc.pltin 12/28/15 10:01:13 run_post: Processing sfcchar.pltin 12/28/15 10:01:13 run_post: Processing ua.pltin 12/28/15 10:01:13 run_post: Processing tmp.aein 12/28/15 10:01:13 run_post: Job <8810480> is submitted to queue . 12/28/15 10:01:15 run_post: t_current now 1800 t_end=7800 12/28/15 10:01:15 run_post: Looking for 'arpsdata/ar2015122815.hdf001800_ready' 12/28/15 10:01:30 run_post: p_string=001200 h_string=0020 t_string=001800 12/28/15 10:01:30 run_post: Checking plot/0020/arpsplt_0020.nqs 12/28/15 10:01:30 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:01:30 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:01:30 run_post: Processing sfc.pltin 12/28/15 10:01:30 run_post: Processing sfcchar.pltin 12/28/15 10:01:30 run_post: Processing ua.pltin 12/28/15 10:01:30 run_post: Processing tmp.aein 12/28/15 10:01:30 run_post: Job <8810481> is submitted to queue . 12/28/15 10:01:32 run_post: t_current now 2100 t_end=7800 12/28/15 10:01:32 run_post: Looking for 'arpsdata/ar2015122815.hdf002100_ready' 12/28/15 10:01:47 run_post: p_string=001500 h_string=0025 t_string=002100 12/28/15 10:01:47 run_post: Checking plot/0025/arpsplt_0025.nqs 12/28/15 10:01:47 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:01:47 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:01:47 run_post: Processing sfc.pltin 12/28/15 10:01:47 run_post: Processing sfcchar.pltin 12/28/15 10:01:47 run_post: Processing ua.pltin 12/28/15 10:01:47 run_post: Processing tmp.aein 12/28/15 10:01:47 run_post: Job <8810482> is submitted to queue . 12/28/15 10:01:49 run_post: t_current now 2400 t_end=7800 12/28/15 10:01:49 run_post: Looking for 'arpsdata/ar2015122815.hdf002400_ready' 12/28/15 10:02:04 run_post: p_string=001800 h_string=0030 t_string=002400 12/28/15 10:02:04 run_post: Checking plot/0030/arpsplt_0030.nqs 12/28/15 10:02:04 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:02:04 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:02:04 run_post: Processing sfc.pltin 12/28/15 10:02:04 run_post: Processing sfcchar.pltin 12/28/15 10:02:04 run_post: Processing ua.pltin 12/28/15 10:02:04 run_post: Processing tmp.aein 12/28/15 10:02:04 run_post: Job <8810483> is submitted to queue . 12/28/15 10:02:06 run_post: t_current now 2700 t_end=7800 12/28/15 10:02:06 run_post: Looking for 'arpsdata/ar2015122815.hdf002700_ready' 12/28/15 10:02:21 run_post: p_string=002100 h_string=0035 t_string=002700 12/28/15 10:02:21 run_post: Checking plot/0035/arpsplt_0035.nqs 12/28/15 10:02:21 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:02:21 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:02:21 run_post: Processing sfc.pltin 12/28/15 10:02:21 run_post: Processing sfcchar.pltin 12/28/15 10:02:21 run_post: Processing ua.pltin 12/28/15 10:02:21 run_post: Processing tmp.aein 12/28/15 10:02:21 run_post: Job <8810484> is submitted to queue . 12/28/15 10:02:22 run_post: t_current now 3000 t_end=7800 12/28/15 10:02:22 run_post: Looking for 'arpsdata/ar2015122815.hdf003000_ready' 12/28/15 10:02:37 run_post: p_string=002400 h_string=0040 t_string=003000 12/28/15 10:02:37 run_post: Checking plot/0040/arpsplt_0040.nqs 12/28/15 10:02:37 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:02:37 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:02:37 run_post: Processing sfc.pltin 12/28/15 10:02:37 run_post: Processing sfcchar.pltin 12/28/15 10:02:38 run_post: Processing ua.pltin 12/28/15 10:02:38 run_post: Processing tmp.aein 12/28/15 10:02:38 run_post: Job <8810485> is submitted to queue . 12/28/15 10:02:39 run_post: t_current now 3300 t_end=7800 12/28/15 10:02:39 run_post: Looking for 'arpsdata/ar2015122815.hdf003300_ready' 12/28/15 10:02:54 run_post: p_string=002700 h_string=0045 t_string=003300 12/28/15 10:02:54 run_post: Checking plot/0045/arpsplt_0045.nqs 12/28/15 10:02:54 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:02:54 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:02:54 run_post: Processing sfc.pltin 12/28/15 10:02:54 run_post: Processing sfcchar.pltin 12/28/15 10:02:54 run_post: Processing ua.pltin 12/28/15 10:02:54 run_post: Processing tmp.aein 12/28/15 10:02:54 run_post: Job <8810486> is submitted to queue . 12/28/15 10:02:56 run_post: t_current now 3600 t_end=7800 12/28/15 10:02:56 run_post: p_string=003000 h_string=0050 t_string=003600 12/28/15 10:02:56 run_post: Checking plot/0050/arpsplt_0050.nqs 12/28/15 10:02:56 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:02:56 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:02:56 run_post: Processing sfc.pltin 12/28/15 10:02:56 run_post: Processing sfcchar.pltin 12/28/15 10:02:56 run_post: Processing ua.pltin 12/28/15 10:02:56 run_post: Processing tmp.aein 12/28/15 10:02:56 run_post: Job <8810487> is submitted to queue . 12/28/15 10:02:58 run_post: t_current now 3900 t_end=7800 12/28/15 10:02:58 run_post: Looking for 'arpsdata/ar2015122815.hdf003900_ready' 12/28/15 10:03:13 run_post: p_string=003300 h_string=0055 t_string=003900 12/28/15 10:03:13 run_post: Checking plot/0055/arpsplt_0055.nqs 12/28/15 10:03:13 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:03:13 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:03:13 run_post: Processing sfc.pltin 12/28/15 10:03:13 run_post: Processing sfcchar.pltin 12/28/15 10:03:13 run_post: Processing ua.pltin 12/28/15 10:03:13 run_post: Processing tmp.aein 12/28/15 10:03:13 run_post: Job <8810488> is submitted to queue . 12/28/15 10:03:14 run_post: t_current now 4200 t_end=7800 12/28/15 10:03:14 run_post: Looking for 'arpsdata/ar2015122815.hdf004200_ready' 12/28/15 10:03:29 run_post: p_string=003600 h_string=0100 t_string=004200 12/28/15 10:03:29 run_post: Checking plot/0100/arpsplt_0100.nqs 12/28/15 10:03:29 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:03:29 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:03:30 run_post: Processing sfc.pltin 12/28/15 10:03:30 run_post: Processing sfcchar.pltin 12/28/15 10:03:30 run_post: Processing ua.pltin 12/28/15 10:03:30 run_post: Processing tmp.aein 12/28/15 10:03:30 run_post: Job <8810489> is submitted to queue . 12/28/15 10:03:31 run_post: t_current now 4500 t_end=7800 12/28/15 10:03:31 run_post: Looking for 'arpsdata/ar2015122815.hdf004500_ready' 12/28/15 10:03:46 run_post: p_string=003900 h_string=0105 t_string=004500 12/28/15 10:03:46 run_post: Checking plot/0105/arpsplt_0105.nqs 12/28/15 10:03:46 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:03:46 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:03:46 run_post: Processing sfc.pltin 12/28/15 10:03:46 run_post: Processing sfcchar.pltin 12/28/15 10:03:46 run_post: Processing ua.pltin 12/28/15 10:03:47 run_post: Processing tmp.aein 12/28/15 10:03:47 run_post: Job <8810491> is submitted to queue . 12/28/15 10:03:48 run_post: t_current now 4800 t_end=7800 12/28/15 10:03:48 run_post: Looking for 'arpsdata/ar2015122815.hdf004800_ready' 12/28/15 10:04:03 run_post: p_string=004200 h_string=0110 t_string=004800 12/28/15 10:04:03 run_post: Checking plot/0110/arpsplt_0110.nqs 12/28/15 10:04:03 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:04:03 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:04:03 run_post: Processing sfc.pltin 12/28/15 10:04:03 run_post: Processing sfcchar.pltin 12/28/15 10:04:03 run_post: Processing ua.pltin 12/28/15 10:04:03 run_post: Processing tmp.aein 12/28/15 10:04:03 run_post: Job <8810492> is submitted to queue . 12/28/15 10:04:05 run_post: t_current now 5100 t_end=7800 12/28/15 10:04:05 run_post: Looking for 'arpsdata/ar2015122815.hdf005100_ready' 12/28/15 10:04:20 run_post: p_string=004500 h_string=0115 t_string=005100 12/28/15 10:04:20 run_post: Checking plot/0115/arpsplt_0115.nqs 12/28/15 10:04:20 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:04:20 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:04:20 run_post: Processing sfc.pltin 12/28/15 10:04:20 run_post: Processing sfcchar.pltin 12/28/15 10:04:20 run_post: Processing ua.pltin 12/28/15 10:04:20 run_post: Processing tmp.aein 12/28/15 10:04:20 run_post: Job <8810493> is submitted to queue . 12/28/15 10:04:22 run_post: t_current now 5400 t_end=7800 12/28/15 10:04:22 run_post: Looking for 'arpsdata/ar2015122815.hdf005400_ready' 12/28/15 10:04:37 run_post: p_string=004800 h_string=0120 t_string=005400 12/28/15 10:04:37 run_post: Checking plot/0120/arpsplt_0120.nqs 12/28/15 10:04:37 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:04:37 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:04:37 run_post: Processing sfc.pltin 12/28/15 10:04:37 run_post: Processing sfcchar.pltin 12/28/15 10:04:37 run_post: Processing ua.pltin 12/28/15 10:04:37 run_post: Processing tmp.aein 12/28/15 10:04:37 run_post: Job <8810494> is submitted to queue . 12/28/15 10:04:39 run_post: t_current now 5700 t_end=7800 12/28/15 10:04:39 run_post: Looking for 'arpsdata/ar2015122815.hdf005700_ready' 12/28/15 10:04:54 run_post: p_string=005100 h_string=0125 t_string=005700 12/28/15 10:04:54 run_post: Checking plot/0125/arpsplt_0125.nqs 12/28/15 10:04:54 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:04:54 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:04:54 run_post: Processing sfc.pltin 12/28/15 10:04:54 run_post: Processing sfcchar.pltin 12/28/15 10:04:54 run_post: Processing ua.pltin 12/28/15 10:04:54 run_post: Processing tmp.aein 12/28/15 10:04:54 run_post: Job <8810495> is submitted to queue . 12/28/15 10:04:56 run_post: t_current now 6000 t_end=7800 12/28/15 10:04:56 run_post: Looking for 'arpsdata/ar2015122815.hdf006000_ready' 12/28/15 10:05:11 run_post: Looking for 'arpsdata/ar2015122815.hdf006000_ready' 12/28/15 10:05:26 run_post: p_string=005400 h_string=0130 t_string=006000 12/28/15 10:05:26 run_post: Checking plot/0130/arpsplt_0130.nqs 12/28/15 10:05:26 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:05:26 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:05:26 run_post: Processing sfc.pltin 12/28/15 10:05:26 run_post: Processing sfcchar.pltin 12/28/15 10:05:26 run_post: Processing ua.pltin 12/28/15 10:05:26 run_post: Processing tmp.aein 12/28/15 10:05:26 run_post: Job <8810496> is submitted to queue . 12/28/15 10:05:28 run_post: t_current now 6300 t_end=7800 12/28/15 10:05:28 run_post: Looking for 'arpsdata/ar2015122815.hdf006300_ready' 12/28/15 10:05:43 run_post: p_string=005700 h_string=0135 t_string=006300 12/28/15 10:05:43 run_post: Checking plot/0135/arpsplt_0135.nqs 12/28/15 10:05:43 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:05:43 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:05:43 run_post: Processing sfc.pltin 12/28/15 10:05:43 run_post: Processing sfcchar.pltin 12/28/15 10:05:43 run_post: Processing ua.pltin 12/28/15 10:05:43 run_post: Processing tmp.aein 12/28/15 10:05:43 run_post: Job <8810497> is submitted to queue . 12/28/15 10:05:44 run_post: t_current now 6600 t_end=7800 12/28/15 10:05:44 run_post: Looking for 'arpsdata/ar2015122815.hdf006600_ready' 12/28/15 10:05:59 run_post: p_string=006000 h_string=0140 t_string=006600 12/28/15 10:05:59 run_post: Checking plot/0140/arpsplt_0140.nqs 12/28/15 10:05:59 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:05:59 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:05:59 run_post: Processing sfc.pltin 12/28/15 10:06:00 run_post: Processing sfcchar.pltin 12/28/15 10:06:00 run_post: Processing ua.pltin 12/28/15 10:06:00 run_post: Processing tmp.aein 12/28/15 10:06:00 run_post: Job <8810498> is submitted to queue . 12/28/15 10:06:01 run_post: t_current now 6900 t_end=7800 12/28/15 10:06:01 run_post: Looking for 'arpsdata/ar2015122815.hdf006900_ready' 12/28/15 10:06:16 run_post: Looking for 'arpsdata/ar2015122815.hdf006900_ready' 12/28/15 10:06:31 run_post: p_string=006300 h_string=0145 t_string=006900 12/28/15 10:06:31 run_post: Checking plot/0145/arpsplt_0145.nqs 12/28/15 10:06:31 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:06:31 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:06:31 run_post: Processing sfc.pltin 12/28/15 10:06:31 run_post: Processing sfcchar.pltin 12/28/15 10:06:31 run_post: Processing ua.pltin 12/28/15 10:06:31 run_post: Processing tmp.aein 12/28/15 10:06:31 run_post: Job <8810499> is submitted to queue . 12/28/15 10:06:33 run_post: t_current now 7200 t_end=7800 12/28/15 10:06:33 run_post: Looking for 'arpsdata/ar2015122815.hdf007200_ready' 12/28/15 10:06:48 run_post: p_string=006600 h_string=0150 t_string=007200 12/28/15 10:06:48 run_post: Checking plot/0150/arpsplt_0150.nqs 12/28/15 10:06:48 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:06:48 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:06:48 run_post: Processing sfc.pltin 12/28/15 10:06:48 run_post: Processing sfcchar.pltin 12/28/15 10:06:49 run_post: Processing ua.pltin 12/28/15 10:06:49 run_post: Processing tmp.aein 12/28/15 10:06:49 run_post: Job <8810500> is submitted to queue . 12/28/15 10:06:50 run_post: t_current now 7500 t_end=7800 12/28/15 10:06:50 run_post: p_string=006900 h_string=0155 t_string=007500 12/28/15 10:06:50 run_post: Checking plot/0155/arpsplt_0155.nqs 12/28/15 10:06:50 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:06:50 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:06:50 run_post: Processing sfc.pltin 12/28/15 10:06:50 run_post: Processing sfcchar.pltin 12/28/15 10:06:50 run_post: Processing ua.pltin 12/28/15 10:06:50 run_post: Processing tmp.aein 12/28/15 10:06:50 run_post: Job <8810501> is submitted to queue . 12/28/15 10:06:51 run_post: t_current now 7800 t_end=7800 12/28/15 10:06:51 run_post: Looking for 'arpsdata/ar2015122815.hdf007800_ready' 12/28/15 10:07:06 run_post: p_string=007200 h_string=0200 t_string=007800 12/28/15 10:07:06 run_post: Checking plot/0200/arpsplt_0200.nqs 12/28/15 10:07:06 run_post: template: reading /home/kwthomas/rt2015/templates/boomer_pre 12/28/15 10:07:06 run_post: template: reading /home/kwthomas/rt2015/templates/graphics.nqs_template 12/28/15 10:07:06 run_post: Processing sfc.pltin 12/28/15 10:07:07 run_post: Processing sfcchar.pltin 12/28/15 10:07:07 run_post: Processing ua.pltin 12/28/15 10:07:07 run_post: Processing tmp.aein 12/28/15 10:07:07 run_post: Job <8810502> is submitted to queue . 12/28/15 10:07:07 plot_arps_0 run: Exit Status 0. 12/28/15 10:07:07 Task 'plot_arps_0 run' finishes. Runtime 00:11:29. 12/28/15 10:07:10 arps_1 run: Exit Status 0. 12/28/15 10:07:10 Task 'arps_1 run' finishes. Runtime 00:07:24. 12/28/15 10:07:10 Task 'nudge_outputs_0 run' starts. 12/28/15 10:07:10 /home/kwthomas/rt2015/scripts/casa_outputs cmd=cat cycle1/ad201512281500.3dvarout cycle2/ad201512281510.3dvarout > ad2015122815.3dvarout cmd=cat cycle1/ar201512281500.arpsout cycle2/ar201512281510.arpsout > ar2015122815.arpsout cmd=cat cycle1/ad201512281500.adasstat cycle2/ad201512281510.adasstat > ad2015122815.adasstat 12/28/15 10:07:11 nudge_outputs_0 run: Exit Status 0. 12/28/15 10:07:11 Task 'nudge_outputs_0 run' finishes. Runtime 00:00:01. 12/28/15 10:07:11 Task Manager finishes. Runtime 00:13:53. Job run time: 00:13 ------------------------------------------------------------ Sender: LSF System Subject: Job 8810474: Done Job was submitted from host by user in cluster . Job was executed on host(s) <16*c110>, in queue , as user in cluster . <16*c086> <16*c102> <16*c082> <16*c084> <16*c109> <16*c085> <16*c087> <16*c104> <16*c089> <16*c108> <16*c083> was used as the home directory. was used as the working directory. Started at Mon Dec 28 09:52:41 2015 Results reported at Mon Dec 28 10:07:11 2015 Your job looked like: ------------------------------------------------------------ # LSBATCH: User input #!/bin/csh #BSUB -J casa2-122815-02 #BSUB -R "span[ptile=16]" #BSUB -W 0:40 #BSUB -a openmpi #BSUB -e ar2015122815.err #BSUB -n 192 #BSUB -o ar2015122815.run #BSUB -q caps_forecast #BSUB -x set time1=`perl -e "print time"` echo -n "Job Started: " date echo "Job Id: $LSB_JOBID" echo " " set echo chmod 644 *run *err cd /work/arpsdata/arpscasa/fcst-casa2014_dfw/20151228/1545Z /home/kwthomas/rt2015/tm/task_manager5 arpspltcycle2 local ar2015122815 /work/arpsdata/arpscasa/fcst-casa2014_dfw/20151228/1545Z /work/arpsdata/arpscasa/fcst-casa2014_dfw/20151228/1545Z casa2015 fgrep -q 'Thanks' ar2015122815.arpsout if ( $status != 0 ) then echo 'It appears that the analysis/forecast failed.' jobs -l > jobs test -s jobs if ( $status == 0 ) then cat jobs | cut -c8- | awk '{ print $1 }' | xargs kill -9 endif /bin/rm -rf jobs goto finish endif wait finish: unset echo set time2=`perl -e "print time"` @ diff = $time2 - $time1 @ hour = $diff / 3600 @ diff = $diff % 3600 @ min = $diff / 60 printf '\nJob run time: %02d:%02d\n' $hour $min ------------------------------------------------------------ Successfully completed. Resource usage summary: CPU time : 109187.73 sec. Max Memory : 5 MB Max Swap : 128 MB Max Processes : 5 Max Threads : 5 The output (if any) is above this job summary. PS: Read file for stderr output of this job.