Job Started: Thu Apr 11 02:48:11 CDT 2013 Job Id: 800266 04/11/13 02:48:12 Welcome to Task Manager Version 5. 04/11/13 02:48:12 Writing '.tm_config' $config{"tm_domain"} = "boomer"; $config{"tm_region"} = "casa2013"; $config{"tm_root"} = "/home/kwthomas/rt2013"; $config{"tm_user"} = "/home/kwthomas/rt2013/user"; read_cfg: Reading /home/kwthomas/rt2013/config/boomer.cfg read_cfg: Reading /home/kwthomas/rt2013/user/config/casa2013.cfg 04/11/13 02:48:12 Writing '.rt_config' $config{"acct"} = ""; $config{"allow_mpi"} = "1"; $config{"is_boomer"} = "1"; $config{"mpi_cmd"} = "/home/kwthomas/rt2013/scripts/mpiwrapper -np"; $config{"msg"} = "BOOMER CASA 2013 Edition"; $config{"non_mpi_cmd"} = ""; $config{"plot_queue"} = "caps_post"; $config{"procs_per_node"} = "16"; $config{"procs_per_node_limit"} = "1"; $config{"queue"} = "caps_forecast"; $config{"radar_mpi_cmd"} = "mpirun.lsf -np"; $config{"submit_cmd"} = "bsub < "; $config{"subvers"} = "openmpi"; $config{"vers"} = "casa2013"; 04/11/13 02:48:12 BOOMER CASA 2013 Edition. 04/11/13 02:48:12 MODEL=arpspltcycle2 04/11/13 02:48:12 MODE=run 04/11/13 02:48:12 PREFIX=ar2013041107 04/11/13 02:48:12 DIR1=/work/arpsdata/arpscasa/fcst-casa2013_dfw/20130411/0745Z 04/11/13 02:48:12 DIR2=/work/arpsdata/arpscasa/fcst-casa2013_dfw/20130411/0745Z 04/11/13 02:48:12 CONFIG=casa2013 04/11/13 02:48:12 SOFTWARE=casa2013 04/11/13 02:48:12 VERSION=openmpi 04/11/13 02:48:12 MPI=enabled 04/11/13 02:48:12 REGION=fcst-casa2013_dfw 04/11/13 02:48:12 SOFTDIR=/home/kwthomas/rt2013/user/software/casa2013/openmpi 04/11/13 02:48:12 SCRIPTSDIR=/home/kwthomas/rt2013/scripts 04/11/13 02:48:12 USER_HOME=/home/kwthomas/rt2013/user 04/11/13 02:48:12 MPI_CMD=/home/kwthomas/rt2013/scripts/mpiwrapper -np 04/11/13 02:48:12 'arps' version is 'arps5.3.4_20130402' 04/11/13 02:48:12 'arw' version is 'WRFV3.4.1_CAPS_20130329' 04/11/13 02:48:12 DELETE_TASK: arps3dvar_retrieval_run (not retrieving) 04/11/13 02:48:12 DELETE_TASK: arps_retrieval_run (not retrieving) 04/11/13 02:48:12 DELETE_TASK: adas_retrieval_run (not retrieving) 04/11/13 02:48:12 DELETE_TASK: arpsenkf_retrieval_run (not retrieving) 04/11/13 02:48:12 DELETE_TASK: extm_lbc_retrieval_run (not retrieving) 04/11/13 02:48:12 DELETE_TASK: extm_t0_retrieval_run (not retrieving) 04/11/13 02:48:12 CYCLES=2 04/11/13 02:48:12 Creation of '.cycle' was successful. 04/11/13 02:48:12 DELETE_TASK: arps_submit (non-remote task) 04/11/13 02:48:12 DELETE_TASK: wrf_submit (non-remote task) 04/11/13 02:48:12 DELETE_TASK: arps2coamps_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: wrf_nmm_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: install_namelist_wrf_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: plot_coamps_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: coamps_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: wrf_arw_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: wrf_nmm_ferrier_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: nmm4wrf_lbc_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: real_nmm_ferrier_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: real_clean_arw_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: arw4wrf_lbc_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: plot_wrf_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: real_clean_nmm_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: real_nmm_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: arw4wrf_t0_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: real_arw_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: coamps_pre_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: install_namelist_real_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: nmm4wrf_t0_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: plot_adas_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: plot_arps3dvar_run (ARPS run) 04/11/13 02:48:12 DELETE_TASK: ens_ana_run (no input file) 04/11/13 02:48:12 DELETE_TASK: plot_arpsenkf_run (no input file) 04/11/13 02:48:12 DELETE_TASK: arw_pre_run (no input file) 04/11/13 02:48:12 DELETE_TASK: arpsenkf_run (no input file) 04/11/13 02:48:12 DELETE_TASK: adas_run (no input file) 04/11/13 02:48:12 DELETE_TASK: local_run (nonexistent script 'run_fcst-casa2013_dfw') 04/11/13 02:48:12 DELETE_TASK: arpsintrp_lbc_run (no input file) 04/11/13 02:48:12 DELETE_TASK: arpsensic_run (no input file) 04/11/13 02:48:12 DELETE_TASK: satellite_run (no input file) 04/11/13 02:48:12 DELETE_TASK: arpstrn_run (no input file) 04/11/13 02:48:12 DELETE_TASK: verif_run (no input file) 04/11/13 02:48:12 DELETE_TASK: arpssfc_run (no input file) 04/11/13 02:48:12 DELETE_TASK: arpsintrp_t0_run (no input file) 04/11/13 02:48:12 DELETE_TASK: nmm_pre_run (no input file) CHOICE=/home/kwthomas/rt2013/scripts/find_extm_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/find_extm CHOICE=/home/kwthomas/rt2013/user/scripts/find_extm_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/find_extm_casa2013 CHOICE=/home/kwthomas/rt2013/scripts/casa_outputs_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/casa_outputs CHOICE=/home/kwthomas/rt2013/user/scripts/casa_outputs_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/casa_outputs_casa2013 CHOICE=/home/kwthomas/rt2013/scripts/pre_task_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/pre_task CHOICE=/home/kwthomas/rt2013/user/scripts/pre_task_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/pre_task_casa2013 04/11/13 02:48:12 Will use '/home/kwthomas/rt2013/scripts/pre_task_boomer' for task 'pre_task' CHOICE=/home/kwthomas/rt2013/scripts/run_radar_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/run_radar CHOICE=/home/kwthomas/rt2013/user/scripts/run_radar_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/run_radar_casa2013 CHOICE=/home/kwthomas/rt2013/scripts/run_post_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/run_post CHOICE=/home/kwthomas/rt2013/user/scripts/run_post_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/run_post_casa2013 04/11/13 02:48:12 TASK_CHAIN=pre_task_0_run find_extm_0_run extm_t0_0_run arpstintrp_0_run extm_lbc_0_run radar_0_run plot_arps_0_run arps3dvar_0_run arps_0_run radar_1_run arps3dvar_1_run arps_1_run nudge_outputs_0_run done 04/11/13 02:48:12 TASK=pre_task_0_run 04/11/13 02:48:12 TASK=find_extm_0_run 04/11/13 02:48:12 DEPENDS= pre_task_0_run 04/11/13 02:48:12 TASK=extm_t0_0_run 04/11/13 02:48:12 DEPENDS= find_extm_0_run pre_task_0_run 04/11/13 02:48:12 TASK=arpstintrp_0_run 04/11/13 02:48:12 DEPENDS= find_extm_0_run pre_task_0_run extm_t0_0_run 04/11/13 02:48:12 TASK=extm_lbc_0_run 04/11/13 02:48:12 DEPENDS= arpstintrp_0_run find_extm_0_run pre_task_0_run extm_t0_0_run 04/11/13 02:48:12 TASK=radar_0_run 04/11/13 02:48:12 DEPENDS= arpstintrp_0_run find_extm_0_run extm_lbc_0_run pre_task_0_run extm_t0_0_run 04/11/13 02:48:12 TASK=plot_arps_0_run 04/11/13 02:48:12 DEPENDS= arpstintrp_0_run radar_0_run extm_t0_0_run 04/11/13 02:48:12 TASK=arps3dvar_0_run 04/11/13 02:48:12 DEPENDS= arpstintrp_0_run find_extm_0_run extm_lbc_0_run radar_0_run pre_task_0_run extm_t0_0_run 04/11/13 02:48:12 TASK=arps_0_run 04/11/13 02:48:12 DEPENDS= arps3dvar_0_run arpstintrp_0_run find_extm_0_run extm_lbc_0_run radar_0_run pre_task_0_run extm_t0_0_run 04/11/13 02:48:12 TASK=radar_1_run 04/11/13 02:48:12 DEPENDS= arps3dvar_0_run arpstintrp_0_run find_extm_0_run extm_lbc_0_run radar_0_run pre_task_0_run extm_t0_0_run arps_0_run 04/11/13 02:48:12 TASK=arps3dvar_1_run 04/11/13 02:48:12 DEPENDS= arps3dvar_0_run extm_lbc_0_run radar_1_run pre_task_0_run extm_t0_0_run arpstintrp_0_run find_extm_0_run radar_0_run arps_0_run 04/11/13 02:48:12 TASK=arps_1_run 04/11/13 02:48:12 DEPENDS= arps3dvar_0_run arps3dvar_1_run extm_lbc_0_run radar_1_run pre_task_0_run extm_t0_0_run arpstintrp_0_run radar_0_run find_extm_0_run arps_0_run 04/11/13 02:48:12 TASK=nudge_outputs_0_run 04/11/13 02:48:12 DEPENDS= arps3dvar_0_run arps3dvar_1_run extm_lbc_0_run radar_1_run pre_task_0_run extm_t0_0_run arps_1_run arpstintrp_0_run radar_0_run find_extm_0_run arps_0_run 04/11/13 02:48:12 TASK=done 04/11/13 02:48:12 DEPENDS= arps3dvar_1_run arps3dvar_0_run radar_1_run extm_lbc_0_run nudge_outputs_0_run extm_t0_0_run pre_task_0_run arps_1_run arpstintrp_0_run plot_arps_0_run find_extm_0_run radar_0_run arps_0_run 04/11/13 02:48:12 Task 'pre_task_0 run' starts. 04/11/13 02:48:12 /home/kwthomas/rt2013/scripts/pre_task_boomer 04/11/13 02:48:12 Welcome to 'boomer_clean_dev_shm' 04/11/13 02:48:12 Checking c020 04/11/13 02:48:12 Checking c008 04/11/13 02:48:12 Checking c019 04/11/13 02:48:12 Checking c006 04/11/13 02:48:12 Checking c003 04/11/13 02:48:12 Checking c002 04/11/13 02:48:12 Checking c059 04/11/13 02:48:12 Checking c056 04/11/13 02:48:12 Checking c022 04/11/13 02:48:12 Checking c101 04/11/13 02:48:12 Checking c418 04/11/13 02:48:12 Checking c016 04/11/13 02:48:13 'boomer_clean_dev_shm' is finished 04/11/13 02:48:13 Welcome to 'boomer_clean_radar_jobs' 04/11/13 02:48:13 Checking c020 04/11/13 02:48:13 Checking c008 04/11/13 02:48:13 Checking c019 04/11/13 02:48:13 Checking c006 04/11/13 02:48:13 Checking c003 04/11/13 02:48:13 Checking c002 04/11/13 02:48:13 Checking c059 04/11/13 02:48:13 Checking c056 04/11/13 02:48:13 Checking c022 04/11/13 02:48:13 Checking c101 04/11/13 02:48:13 Checking c418 04/11/13 02:48:13 Checking c016 04/11/13 02:48:13 'boomer_clean_radar_jobs' is finished 04/11/13 02:48:13 pre_task_0 run: Exit Status 0. 04/11/13 02:48:13 Task 'pre_task_0 run' finishes. Runtime 00:00:01. 04/11/13 02:48:13 Task 'find_extm_0 run' starts. 04/11/13 02:48:13 /home/kwthomas/rt2013/scripts/find_extm 04/11/13 02:48:13 External data already present. 04/11/13 02:48:13 find_extm_0 run: Exit Status 0. 04/11/13 02:48:13 Task 'find_extm_0 run' finishes. Runtime 00:00:00. 04/11/13 02:48:13 Task 'extm_t0_0 run' starts. 04/11/13 02:48:13 template: reading em2013041107-t0.extin 04/11/13 02:48:13 match for 'nproc_x' on line 211 - nproc_x=12, 04/11/13 02:48:13 match for 'nproc_y' on line 212 - nproc_y=16, 04/11/13 02:48:13 nproc_x/nproc_y=12/16 04/11/13 02:48:13 /home/kwthomas/rt2013/scripts/mpiwrapper -np 192 /home/kwthomas/rt2013/user/software/casa2013/openmpi/arps/abin/ext2arps_mpi < em2013041107-t0.extin >> em2013041107-t0.extout 2>&1 04/11/13 02:48:51 extm_t0_0 run: Exit Status 0. 04/11/13 02:48:51 Task 'extm_t0_0 run' finishes. Runtime 00:00:38. 04/11/13 02:48:51 Task 'arpstintrp_0 run' starts. 04/11/13 02:48:51 template: reading ei2013041107-t0.tintrpin 04/11/13 02:48:51 match for 'nproc_x' on line 70 - nproc_x=12, 04/11/13 02:48:51 match for 'nproc_y' on line 71 - nproc_y=16, 04/11/13 02:48:51 nproc_x/nproc_y=12/16 04/11/13 02:48:51 /home/kwthomas/rt2013/scripts/mpiwrapper -np 192 /home/kwthomas/rt2013/user/software/casa2013/openmpi/arps/abin/arpstintrp_mpi < ei2013041107-t0.tintrpin >> ei2013041107-t0.tintrpout 2>&1 04/11/13 02:49:18 arpstintrp_0 run: Exit Status 0. 04/11/13 02:49:18 Task 'arpstintrp_0 run' finishes. Runtime 00:00:27. 04/11/13 02:49:18 Task 'extm_lbc_0 run' starts. 04/11/13 02:49:18 template: reading em2013041107-lbc.extin 04/11/13 02:49:18 match for 'nproc_x' on line 211 - nproc_x=12, 04/11/13 02:49:18 match for 'nproc_y' on line 212 - nproc_y=16, 04/11/13 02:49:18 nproc_x/nproc_y=12/16 04/11/13 02:49:18 /home/kwthomas/rt2013/scripts/mpiwrapper -np 192 /home/kwthomas/rt2013/user/software/casa2013/openmpi/arps/abin/ext2arps_mpi < em2013041107-lbc.extin >> em2013041107-lbc.extout 2>&1 04/11/13 02:49:55 extm_lbc_0 run: Exit Status 0. 04/11/13 02:49:55 Task 'extm_lbc_0 run' finishes. Runtime 00:00:37. 04/11/13 02:49:55 Found cycle1/.start 04/11/13 02:49:55 Run directory is cycle1. 04/11/13 02:49:55 Task 'radar_0 run' starts. 04/11/13 02:49:55 Return to parent directory. 04/11/13 02:49:55 /home/kwthomas/rt2013/scripts/run_radar 04/11/13 02:49:55 read_cfg: Reading ../.tm_config $config{"tm_domain"} = "boomer"; $config{"tm_region"} = "casa2013"; $config{"tm_root"} = "/home/kwthomas/rt2013"; $config{"tm_user"} = "/home/kwthomas/rt2013/user"; 04/11/13 02:49:55 read_cfg: Reading ../.rt_config $config{"acct"} = ""; $config{"allow_mpi"} = "1"; $config{"is_boomer"} = "1"; $config{"mpi_cmd"} = "/home/kwthomas/rt2013/scripts/mpiwrapper -np"; $config{"msg"} = "BOOMER CASA 2013 Edition"; $config{"non_mpi_cmd"} = ""; $config{"plot_queue"} = "caps_post"; $config{"procs_per_node"} = "16"; $config{"procs_per_node_limit"} = "1"; $config{"queue"} = "caps_forecast"; $config{"radar_mpi_cmd"} = "mpirun.lsf -np"; $config{"submit_cmd"} = "bsub < "; $config{"subvers"} = "openmpi"; $config{"vers"} = "casa2013"; 04/11/13 02:49:55 Welcome to 'run_radar'...BOOMER Edition. CHOICE=/home/kwthomas/rt2013/scripts/boomer_clean_radar_jobs_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/boomer_clean_radar_jobs CHOICE=/home/kwthomas/rt2013/user/scripts/boomer_clean_radar_jobs_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/boomer_clean_radar_jobs_casa2013 04/11/13 02:49:55 Network interface 'openmpi'. 04/11/13 02:49:55 04/11/13 02:49:55 Cycle 1 04/11/13 02:49:55 run_radar: casa: No 'casa/*.casain' file. 04/11/13 02:49:55 template: reading nids/r882013041107.r88din 04/11/13 02:49:55 match for 'nproc_x' on line 88 - nproc_x=2, 04/11/13 02:49:55 match for 'nproc_y' on line 89 - nproc_y=8, 04/11/13 02:49:55 template: reading nids/nd2013041107.nidsin 04/11/13 02:49:55 match for 'nproc_x' on line 88 - nproc_x=2, 04/11/13 02:49:55 match for 'nproc_y' on line 89 - nproc_y=8, 04/11/13 02:49:55 template: reading tdwr/tdwr2013041107.tdwrin 04/11/13 02:49:55 match for 'nproc_x' on line 88 - nproc_x=2, 04/11/13 02:49:55 match for 'nproc_y' on line 89 - nproc_y=8, 04/11/13 02:49:55 template: reading ../em2013041107-t0.extin 04/11/13 02:49:55 match for 'nproc_x' on line 211 - nproc_x=12, 04/11/13 02:49:55 match for 'nproc_y' on line 212 - nproc_y=16, 04/11/13 02:49:55 match for 'dmp_out_joined' on line 2140 - dmp_out_joined=0, 04/11/13 02:49:55 template: reading ../ei2013041107-t0.tintrpin 04/11/13 02:49:55 match for 'nproc_x' on line 70 - nproc_x=12, 04/11/13 02:49:55 match for 'nproc_y' on line 71 - nproc_y=16, 04/11/13 02:49:55 match for 'nproc_x_out' on line 76 - nproc_x_out=12, 04/11/13 02:49:55 match for 'nproc_y_out' on line 77 - nproc_y_out=16, 04/11/13 02:49:55 Splitfiles detected. 04/11/13 02:49:55 Making directory 'radar' 04/11/13 02:49:55 Number of processors available is 192. 04/11/13 02:49:55 Number of processors per node is 16. 04/11/13 02:49:55 nslots=12 04/11/13 02:49:55 njobs=6 04/11/13 02:49:55 nprocs=16 04/11/13 02:49:55 04/11/13 02:49:55 max=1 04/11/13 02:49:55 cmd=radar1_0_radar 04/11/13 02:49:55 cmd=radar2_0_radar 04/11/13 02:49:55 cmd=radar3_0_radar 04/11/13 02:49:55 cmd=radar4_0_radar 04/11/13 02:49:55 cmd=radar5_0_radar 04/11/13 02:49:55 cmd=radar6_0_radar 04/11/13 02:49:55 cmd=radar7_0_radar 04/11/13 02:49:55 cmd=radar8_0_radar 04/11/13 02:49:55 cmd=radar9_0_radar 04/11/13 02:49:55 cmd=radar10_0_radar 04/11/13 02:49:55 cmd=radar11_0_radar 04/11/13 02:49:55 cmd=radar12_0_radar 04/11/13 02:50:37 BOOMER radar cleanup. 04/11/13 02:50:37 Welcome to 'boomer_clean_radar_jobs' 04/11/13 02:50:37 Checking c020 04/11/13 02:50:37 Checking c008 04/11/13 02:50:37 Checking c019 04/11/13 02:50:37 Checking c006 04/11/13 02:50:37 Checking c003 04/11/13 02:50:37 Checking c002 04/11/13 02:50:37 Checking c059 04/11/13 02:50:37 Checking c056 04/11/13 02:50:37 Checking c022 04/11/13 02:50:37 Checking c101 04/11/13 02:50:37 Checking c418 04/11/13 02:50:37 Checking c016 04/11/13 02:50:37 'boomer_clean_radar_jobs' is finished 04/11/13 02:50:37 CHOICE=/home/kwthomas/rt2013/scripts/radar_post_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/radar_post CHOICE=/home/kwthomas/rt2013/user/scripts/radar_post_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/radar_post_casa2013 Running /home/kwthomas/rt2013/scripts/radar_post. 04/11/13 02:50:37 CHOICE=/home/kwthomas/rt2013/scripts/radar_post_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/radar_post CHOICE=/home/kwthomas/rt2013/user/scripts/radar_post_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/radar_post_casa2013 Running /home/kwthomas/rt2013/scripts/radar_post. 04/11/13 02:50:37 CHOICE=/home/kwthomas/rt2013/scripts/tdwr_post_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/tdwr_post CHOICE=/home/kwthomas/rt2013/user/scripts/tdwr_post_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/tdwr_post_casa2013 Running /home/kwthomas/rt2013/scripts/tdwr_post. 04/11/13 02:50:37 NEXRAD/NIDS/CASA/TDWR processing is finished. 04/11/13 02:50:37 radar_0 run: Exit Status 0. 04/11/13 02:50:37 Task 'radar_0 run' finishes. Runtime 00:00:42. 04/11/13 02:50:37 Found cycle1/.start 04/11/13 02:50:37 Run directory is cycle1. 04/11/13 02:50:37 Task 'arps3dvar_0 run' starts. 04/11/13 02:50:37 template: reading ad201304110700.3dvarin 04/11/13 02:50:38 match for 'nproc_x' on line 211 - nproc_x=12, 04/11/13 02:50:38 match for 'nproc_y' on line 212 - nproc_y=16, 04/11/13 02:50:38 nproc_x/nproc_y=12/16 04/11/13 02:50:38 Return to parent directory. 04/11/13 02:50:38 Task 'plot_arps_0 run' starts. 04/11/13 02:50:38 /home/kwthomas/rt2013/scripts/mpiwrapper -np 192 /home/kwthomas/rt2013/user/software/casa2013/openmpi/arps/abin/arps3dvar_mpi < ad201304110700.3dvarin >> ad201304110700.3dvarout 2>&1 04/11/13 02:50:38 /home/kwthomas/rt2013/scripts/run_post 04/11/13 02:50:38 read_cfg: Reading .tm_config $config{"tm_domain"} = "boomer"; $config{"tm_region"} = "casa2013"; $config{"tm_root"} = "/home/kwthomas/rt2013"; $config{"tm_user"} = "/home/kwthomas/rt2013/user"; 04/11/13 02:50:38 read_cfg: Reading .rt_config $config{"acct"} = ""; $config{"allow_mpi"} = "1"; $config{"is_boomer"} = "1"; $config{"mpi_cmd"} = "/home/kwthomas/rt2013/scripts/mpiwrapper -np"; $config{"msg"} = "BOOMER CASA 2013 Edition"; $config{"non_mpi_cmd"} = ""; $config{"plot_queue"} = "caps_post"; $config{"procs_per_node"} = "16"; $config{"procs_per_node_limit"} = "1"; $config{"queue"} = "caps_forecast"; $config{"radar_mpi_cmd"} = "mpirun.lsf -np"; $config{"submit_cmd"} = "bsub < "; $config{"subvers"} = "openmpi"; $config{"vers"} = "casa2013"; 04/11/13 02:50:38 run_post: t_offset=300 t_end=7200 04/11/13 02:50:38 run_post: Read .cycle, value is '2'. 04/11/13 02:50:38 run_post: is_wrf=0 is_coamps=0 soundings=1 04/11/13 02:50:38 run_post: file=cycle1/ar201304110700.arpsin 04/11/13 02:50:38 run_post: match for 'nproc_x' on line 211 - nproc_x=12, 04/11/13 02:50:38 run_post: match for 'nproc_y' on line 212 - nproc_y=16, 04/11/13 02:50:38 run_post: match for 'runname' on line 245 - runname='ar2013041107', 04/11/13 02:50:38 run_post: match for 'initime' on line 414 - initime='2013-04-11.07:35:00', 04/11/13 02:50:38 run_post: match for 'nudgopt' on line 520 - nudgopt=2, 04/11/13 02:50:38 run_post: match for 'ndstop' on line 522 - ndstop=600, 04/11/13 02:50:38 run_post: match for 'dmp_out_joined' on line 2140 - dmp_out_joined=0, 04/11/13 02:50:38 run_post: match for 'hdmpfmt' on line 2141 - hdmpfmt=3, 04/11/13 02:50:38 run_post: match for 'dirname' on line 2325 - dirname='../arpsdata/', 04/11/13 02:50:38 run_post: NUDGE=600 04/11/13 02:50:38 run_post: 04/11/13 02:50:38 run_post: t_end modified to 7800 04/11/13 02:50:38 run_post: arps_subdirs=1 wrf_subdirs=0 04/11/13 02:50:38 run_post: Runname is ar2013041107 04/11/13 02:50:38 run_post: Data uses splitfiles. 04/11/13 02:50:38 run_post: match for 'nproc_x' on line 3622 - nproc_x=2, 04/11/13 02:50:38 run_post: match for 'nproc_y' on line 3622 - nproc_y=8, 04/11/13 02:50:38 run_post: Region is casa2013_dfw 04/11/13 02:50:38 run_post: t_current now 0 t_end=7800 04/11/13 02:50:38 run_post: Looking for 'arpsdata/ar2013041107.hdf000000_001001_ready' 04/11/13 02:50:53 run_post: Looking for 'arpsdata/ar2013041107.hdf000000_001001_ready' 04/11/13 02:51:08 run_post: Looking for 'arpsdata/ar2013041107.hdf000000_001001_ready' 04/11/13 02:51:23 run_post: Looking for 'arpsdata/ar2013041107.hdf000000_001001_ready' 04/11/13 02:51:38 run_post: Looking for 'arpsdata/ar2013041107.hdf000000_001001_ready' 04/11/13 02:51:39 arps3dvar_0 run: Exit Status 0. 04/11/13 02:51:39 Task 'arps3dvar_0 run' finishes. Runtime 00:01:01. 04/11/13 02:51:39 Found cycle1/.start 04/11/13 02:51:39 Run directory is cycle1. 04/11/13 02:51:39 Task 'arps_0 run' starts. 04/11/13 02:51:39 template: reading ar201304110700.arpsin 04/11/13 02:51:39 match for 'nproc_x' on line 211 - nproc_x=12, 04/11/13 02:51:39 match for 'nproc_y' on line 212 - nproc_y=16, 04/11/13 02:51:39 nproc_x/nproc_y=12/16 04/11/13 02:51:39 Return to parent directory. 04/11/13 02:51:39 /home/kwthomas/rt2013/scripts/mpiwrapper -np 192 /home/kwthomas/rt2013/user/software/casa2013/openmpi/arps/abin/arps_mpi < ar201304110700.arpsin >> ar201304110700.arpsout 2>&1 04/11/13 02:51:53 run_post: Looking for 'arpsdata/ar2013041107.hdf000000_001001_ready' 04/11/13 02:52:08 run_post: This nudging event not plotted. 04/11/13 02:52:08 run_post: t_current now 300 t_end=7800 04/11/13 02:52:08 run_post: Looking for 'arpsdata/ar2013041107.hdf000300_001001_ready' 04/11/13 02:52:23 run_post: This nudging event not plotted. 04/11/13 02:52:23 run_post: t_current now 600 t_end=7800 04/11/13 02:52:23 run_post: Looking for 'arpsdata/ar2013041107.hdf000600_001001_ready' 04/11/13 02:52:34 arps_0 run: Exit Status 0. 04/11/13 02:52:34 Task 'arps_0 run' finishes. Runtime 00:00:55. 04/11/13 02:52:34 Looking for file cycle2/.start 04/11/13 02:52:39 run_post: p_string=000000 h_string=0000 t_string=000600 04/11/13 02:52:39 run_post: Checking plot/0000/arpsplt_0000.nqs 04/11/13 02:52:39 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 02:52:39 run_post: Processing sfc.pltin 04/11/13 02:52:39 run_post: Processing sfcchar.pltin 04/11/13 02:52:39 run_post: Processing ua.pltin 04/11/13 02:52:39 run_post: Processing tmp.aein 04/11/13 02:52:39 run_post: Job <800269> is submitted to queue . 04/11/13 02:52:39 run_post: t_current now 900 t_end=7800 04/11/13 02:52:39 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:52:49 Looking for file cycle2/.start 04/11/13 02:52:54 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:53:04 Looking for file cycle2/.start 04/11/13 02:53:09 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:53:19 Looking for file cycle2/.start 04/11/13 02:53:24 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:53:34 Looking for file cycle2/.start 04/11/13 02:53:39 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:53:49 Looking for file cycle2/.start 04/11/13 02:53:54 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:54:04 Looking for file cycle2/.start 04/11/13 02:54:09 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:54:19 Looking for file cycle2/.start 04/11/13 02:54:24 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:54:34 Looking for file cycle2/.start 04/11/13 02:54:39 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:54:49 Looking for file cycle2/.start 04/11/13 02:54:54 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:55:04 Looking for file cycle2/.start 04/11/13 02:55:09 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:55:19 Found cycle2/.start 04/11/13 02:55:19 Run directory is cycle2. 04/11/13 02:55:19 Task 'radar_1 run' starts. 04/11/13 02:55:19 Return to parent directory. 04/11/13 02:55:19 /home/kwthomas/rt2013/scripts/run_radar 04/11/13 02:55:19 read_cfg: Reading ../.tm_config $config{"tm_domain"} = "boomer"; $config{"tm_region"} = "casa2013"; $config{"tm_root"} = "/home/kwthomas/rt2013"; $config{"tm_user"} = "/home/kwthomas/rt2013/user"; 04/11/13 02:55:19 read_cfg: Reading ../.rt_config $config{"acct"} = ""; $config{"allow_mpi"} = "1"; $config{"is_boomer"} = "1"; $config{"mpi_cmd"} = "/home/kwthomas/rt2013/scripts/mpiwrapper -np"; $config{"msg"} = "BOOMER CASA 2013 Edition"; $config{"non_mpi_cmd"} = ""; $config{"plot_queue"} = "caps_post"; $config{"procs_per_node"} = "16"; $config{"procs_per_node_limit"} = "1"; $config{"queue"} = "caps_forecast"; $config{"radar_mpi_cmd"} = "mpirun.lsf -np"; $config{"submit_cmd"} = "bsub < "; $config{"subvers"} = "openmpi"; $config{"vers"} = "casa2013"; 04/11/13 02:55:19 Welcome to 'run_radar'...BOOMER Edition. CHOICE=/home/kwthomas/rt2013/scripts/boomer_clean_radar_jobs_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/boomer_clean_radar_jobs CHOICE=/home/kwthomas/rt2013/user/scripts/boomer_clean_radar_jobs_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/boomer_clean_radar_jobs_casa2013 04/11/13 02:55:19 Network interface 'openmpi'. 04/11/13 02:55:19 04/11/13 02:55:19 Cycle 2 04/11/13 02:55:19 run_radar: casa: No 'casa/*.casain' file. 04/11/13 02:55:19 template: reading nids/r882013041107.r88din 04/11/13 02:55:19 match for 'nproc_x' on line 88 - nproc_x=2, 04/11/13 02:55:19 match for 'nproc_y' on line 89 - nproc_y=8, 04/11/13 02:55:19 template: reading nids/nd2013041107.nidsin 04/11/13 02:55:19 match for 'nproc_x' on line 88 - nproc_x=2, 04/11/13 02:55:19 match for 'nproc_y' on line 89 - nproc_y=8, 04/11/13 02:55:19 template: reading tdwr/tdwr2013041107.tdwrin 04/11/13 02:55:19 match for 'nproc_x' on line 88 - nproc_x=2, 04/11/13 02:55:19 match for 'nproc_y' on line 89 - nproc_y=8, 04/11/13 02:55:19 template: reading ../em2013041107-t0.extin 04/11/13 02:55:19 match for 'nproc_x' on line 211 - nproc_x=12, 04/11/13 02:55:19 match for 'nproc_y' on line 212 - nproc_y=16, 04/11/13 02:55:20 match for 'dmp_out_joined' on line 2140 - dmp_out_joined=0, 04/11/13 02:55:20 template: reading ../ei2013041107-t0.tintrpin 04/11/13 02:55:20 match for 'nproc_x' on line 70 - nproc_x=12, 04/11/13 02:55:20 match for 'nproc_y' on line 71 - nproc_y=16, 04/11/13 02:55:20 match for 'nproc_x_out' on line 76 - nproc_x_out=12, 04/11/13 02:55:20 match for 'nproc_y_out' on line 77 - nproc_y_out=16, 04/11/13 02:55:20 Splitfiles detected. 04/11/13 02:55:20 Making directory 'radar' 04/11/13 02:55:20 Number of processors available is 192. 04/11/13 02:55:20 Number of processors per node is 16. 04/11/13 02:55:20 nslots=12 04/11/13 02:55:20 njobs=6 04/11/13 02:55:20 nprocs=16 04/11/13 02:55:20 04/11/13 02:55:20 max=1 04/11/13 02:55:20 cmd=radar1_0_radar 04/11/13 02:55:20 cmd=radar2_0_radar 04/11/13 02:55:20 cmd=radar3_0_radar 04/11/13 02:55:20 cmd=radar4_0_radar 04/11/13 02:55:20 cmd=radar5_0_radar 04/11/13 02:55:20 cmd=radar6_0_radar 04/11/13 02:55:20 cmd=radar7_0_radar 04/11/13 02:55:20 cmd=radar8_0_radar 04/11/13 02:55:20 cmd=radar9_0_radar 04/11/13 02:55:20 cmd=radar10_0_radar 04/11/13 02:55:20 cmd=radar11_0_radar 04/11/13 02:55:20 cmd=radar12_0_radar 04/11/13 02:55:24 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:55:39 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:55:54 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:56:09 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:56:10 BOOMER radar cleanup. 04/11/13 02:56:10 Welcome to 'boomer_clean_radar_jobs' 04/11/13 02:56:10 Checking c020 04/11/13 02:56:10 Checking c008 04/11/13 02:56:10 Checking c019 04/11/13 02:56:10 Checking c006 04/11/13 02:56:10 Checking c003 04/11/13 02:56:10 Checking c002 04/11/13 02:56:10 Checking c059 04/11/13 02:56:10 Checking c056 04/11/13 02:56:10 Checking c022 04/11/13 02:56:10 Checking c101 04/11/13 02:56:10 Checking c418 04/11/13 02:56:10 Checking c016 04/11/13 02:56:11 'boomer_clean_radar_jobs' is finished 04/11/13 02:56:11 CHOICE=/home/kwthomas/rt2013/scripts/radar_post_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/radar_post CHOICE=/home/kwthomas/rt2013/user/scripts/radar_post_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/radar_post_casa2013 Running /home/kwthomas/rt2013/scripts/radar_post. 04/11/13 02:56:11 CHOICE=/home/kwthomas/rt2013/scripts/radar_post_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/radar_post CHOICE=/home/kwthomas/rt2013/user/scripts/radar_post_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/radar_post_casa2013 Running /home/kwthomas/rt2013/scripts/radar_post. 04/11/13 02:56:11 CHOICE=/home/kwthomas/rt2013/scripts/tdwr_post_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/tdwr_post CHOICE=/home/kwthomas/rt2013/user/scripts/tdwr_post_boomer CHOICE=/home/kwthomas/rt2013/user/scripts/tdwr_post_casa2013 Running /home/kwthomas/rt2013/scripts/tdwr_post. 04/11/13 02:56:11 NEXRAD/NIDS/CASA/TDWR processing is finished. 04/11/13 02:56:11 radar_1 run: Exit Status 0. 04/11/13 02:56:11 Task 'radar_1 run' finishes. Runtime 00:00:52. 04/11/13 02:56:11 Found cycle2/.start 04/11/13 02:56:11 Run directory is cycle2. 04/11/13 02:56:11 Task 'arps3dvar_1 run' starts. 04/11/13 02:56:11 Change '*.3dvarin' to 'ad201304110710.3dvarin' 04/11/13 02:56:11 Change 'arps/abin/arps3dvar_mpi < *.3dvarin >> *.3dvarout 2>&1' to 04/11/13 02:56:11 'arps/abin/arps3dvar_mpi < ad201304110710.3dvarin >> ad201304110710.3dvarout 2>&1' 04/11/13 02:56:11 template: reading ad201304110710.3dvarin 04/11/13 02:56:11 match for 'nproc_x' on line 211 - nproc_x=12, 04/11/13 02:56:11 match for 'nproc_y' on line 212 - nproc_y=16, 04/11/13 02:56:11 nproc_x/nproc_y=12/16 04/11/13 02:56:11 Return to parent directory. 04/11/13 02:56:11 /home/kwthomas/rt2013/scripts/mpiwrapper -np 192 /home/kwthomas/rt2013/user/software/casa2013/openmpi/arps/abin/arps3dvar_mpi < ad201304110710.3dvarin >> ad201304110710.3dvarout 2>&1 04/11/13 02:56:24 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:56:39 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:56:54 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:57:09 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:57:12 arps3dvar_1 run: Exit Status 0. 04/11/13 02:57:12 Task 'arps3dvar_1 run' finishes. Runtime 00:01:01. 04/11/13 02:57:12 Found cycle2/.start 04/11/13 02:57:12 Run directory is cycle2. 04/11/13 02:57:12 Task 'arps_1 run' starts. 04/11/13 02:57:12 Change '*.arpsin' to 'ar201304110710.arpsin' 04/11/13 02:57:12 Change 'arps/abin/arps_mpi < *.arpsin >> *.arpsout 2>&1' to 04/11/13 02:57:12 'arps/abin/arps_mpi < ar201304110710.arpsin >> ar201304110710.arpsout 2>&1' 04/11/13 02:57:12 template: reading ar201304110710.arpsin 04/11/13 02:57:12 match for 'nproc_x' on line 211 - nproc_x=12, 04/11/13 02:57:12 match for 'nproc_y' on line 212 - nproc_y=16, 04/11/13 02:57:12 nproc_x/nproc_y=12/16 04/11/13 02:57:12 Return to parent directory. 04/11/13 02:57:12 /home/kwthomas/rt2013/scripts/mpiwrapper -np 192 /home/kwthomas/rt2013/user/software/casa2013/openmpi/arps/abin/arps_mpi < ar201304110710.arpsin >> ar201304110710.arpsout 2>&1 04/11/13 02:57:24 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:57:39 run_post: Looking for 'arpsdata/ar2013041107.hdf000900_001001_ready' 04/11/13 02:57:54 run_post: p_string=000300 h_string=0005 t_string=000900 04/11/13 02:57:54 run_post: Checking plot/0005/arpsplt_0005.nqs 04/11/13 02:57:54 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 02:57:54 run_post: Processing sfc.pltin 04/11/13 02:57:54 run_post: Processing sfcchar.pltin 04/11/13 02:57:54 run_post: Processing ua.pltin 04/11/13 02:57:54 run_post: Processing tmp.aein 04/11/13 02:57:54 run_post: Job <800274> is submitted to queue . 04/11/13 02:57:54 run_post: t_current now 1200 t_end=7800 04/11/13 02:57:54 run_post: Looking for 'arpsdata/ar2013041107.hdf001200_001001_ready' 04/11/13 02:58:10 run_post: p_string=000600 h_string=0010 t_string=001200 04/11/13 02:58:10 run_post: Checking plot/0010/arpsplt_0010.nqs 04/11/13 02:58:10 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 02:58:10 run_post: Processing sfc.pltin 04/11/13 02:58:10 run_post: Processing sfcchar.pltin 04/11/13 02:58:10 run_post: Processing ua.pltin 04/11/13 02:58:10 run_post: Processing tmp.aein 04/11/13 02:58:10 run_post: Job <800275> is submitted to queue . 04/11/13 02:58:10 run_post: t_current now 1500 t_end=7800 04/11/13 02:58:10 run_post: Looking for 'arpsdata/ar2013041107.hdf001500_001001_ready' 04/11/13 02:58:25 run_post: p_string=000900 h_string=0015 t_string=001500 04/11/13 02:58:25 run_post: Checking plot/0015/arpsplt_0015.nqs 04/11/13 02:58:25 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 02:58:25 run_post: Processing sfc.pltin 04/11/13 02:58:25 run_post: Processing sfcchar.pltin 04/11/13 02:58:25 run_post: Processing ua.pltin 04/11/13 02:58:25 run_post: Processing tmp.aein 04/11/13 02:58:25 run_post: Job <800276> is submitted to queue . 04/11/13 02:58:26 run_post: t_current now 1800 t_end=7800 04/11/13 02:58:26 run_post: Looking for 'arpsdata/ar2013041107.hdf001800_001001_ready' 04/11/13 02:58:41 run_post: p_string=001200 h_string=0020 t_string=001800 04/11/13 02:58:41 run_post: Checking plot/0020/arpsplt_0020.nqs 04/11/13 02:58:41 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 02:58:41 run_post: Processing sfc.pltin 04/11/13 02:58:41 run_post: Processing sfcchar.pltin 04/11/13 02:58:41 run_post: Processing ua.pltin 04/11/13 02:58:41 run_post: Processing tmp.aein 04/11/13 02:58:41 run_post: Job <800277> is submitted to queue . 04/11/13 02:58:41 run_post: t_current now 2100 t_end=7800 04/11/13 02:58:41 run_post: p_string=001500 h_string=0025 t_string=002100 04/11/13 02:58:41 run_post: Checking plot/0025/arpsplt_0025.nqs 04/11/13 02:58:41 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 02:58:41 run_post: Processing sfc.pltin 04/11/13 02:58:42 run_post: Processing sfcchar.pltin 04/11/13 02:58:42 run_post: Processing ua.pltin 04/11/13 02:58:42 run_post: Processing tmp.aein 04/11/13 02:58:42 run_post: Job <800278> is submitted to queue . 04/11/13 02:58:42 run_post: t_current now 2400 t_end=7800 04/11/13 02:58:42 run_post: Looking for 'arpsdata/ar2013041107.hdf002400_001001_ready' 04/11/13 02:58:57 run_post: p_string=001800 h_string=0030 t_string=002400 04/11/13 02:58:57 run_post: Checking plot/0030/arpsplt_0030.nqs 04/11/13 02:58:57 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 02:58:57 run_post: Processing sfc.pltin 04/11/13 02:58:57 run_post: Processing sfcchar.pltin 04/11/13 02:58:57 run_post: Processing ua.pltin 04/11/13 02:58:57 run_post: Processing tmp.aein 04/11/13 02:58:57 run_post: Job <800279> is submitted to queue . 04/11/13 02:58:58 run_post: t_current now 2700 t_end=7800 04/11/13 02:58:58 run_post: Looking for 'arpsdata/ar2013041107.hdf002700_001001_ready' 04/11/13 02:59:13 run_post: p_string=002100 h_string=0035 t_string=002700 04/11/13 02:59:13 run_post: Checking plot/0035/arpsplt_0035.nqs 04/11/13 02:59:13 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 02:59:13 run_post: Processing sfc.pltin 04/11/13 02:59:13 run_post: Processing sfcchar.pltin 04/11/13 02:59:13 run_post: Processing ua.pltin 04/11/13 02:59:13 run_post: Processing tmp.aein 04/11/13 02:59:13 run_post: Job <800280> is submitted to queue . 04/11/13 02:59:13 run_post: t_current now 3000 t_end=7800 04/11/13 02:59:13 run_post: Looking for 'arpsdata/ar2013041107.hdf003000_001001_ready' 04/11/13 02:59:28 run_post: p_string=002400 h_string=0040 t_string=003000 04/11/13 02:59:28 run_post: Checking plot/0040/arpsplt_0040.nqs 04/11/13 02:59:28 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 02:59:28 run_post: Processing sfc.pltin 04/11/13 02:59:28 run_post: Processing sfcchar.pltin 04/11/13 02:59:28 run_post: Processing ua.pltin 04/11/13 02:59:28 run_post: Processing tmp.aein 04/11/13 02:59:28 run_post: Job <800281> is submitted to queue . 04/11/13 02:59:29 run_post: t_current now 3300 t_end=7800 04/11/13 02:59:29 run_post: Looking for 'arpsdata/ar2013041107.hdf003300_001001_ready' 04/11/13 02:59:44 run_post: p_string=002700 h_string=0045 t_string=003300 04/11/13 02:59:44 run_post: Checking plot/0045/arpsplt_0045.nqs 04/11/13 02:59:44 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 02:59:44 run_post: Processing sfc.pltin 04/11/13 02:59:44 run_post: Processing sfcchar.pltin 04/11/13 02:59:44 run_post: Processing ua.pltin 04/11/13 02:59:44 run_post: Processing tmp.aein 04/11/13 02:59:44 run_post: Job <800282> is submitted to queue . 04/11/13 02:59:44 run_post: t_current now 3600 t_end=7800 04/11/13 02:59:44 run_post: Looking for 'arpsdata/ar2013041107.hdf003600_001001_ready' 04/11/13 02:59:59 run_post: p_string=003000 h_string=0050 t_string=003600 04/11/13 02:59:59 run_post: Checking plot/0050/arpsplt_0050.nqs 04/11/13 02:59:59 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 02:59:59 run_post: Processing sfc.pltin 04/11/13 02:59:59 run_post: Processing sfcchar.pltin 04/11/13 02:59:59 run_post: Processing ua.pltin 04/11/13 02:59:59 run_post: Processing tmp.aein 04/11/13 02:59:59 run_post: Job <800283> is submitted to queue . 04/11/13 03:00:00 run_post: t_current now 3900 t_end=7800 04/11/13 03:00:00 run_post: Looking for 'arpsdata/ar2013041107.hdf003900_001001_ready' 04/11/13 03:00:15 run_post: p_string=003300 h_string=0055 t_string=003900 04/11/13 03:00:15 run_post: Checking plot/0055/arpsplt_0055.nqs 04/11/13 03:00:15 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 03:00:15 run_post: Processing sfc.pltin 04/11/13 03:00:16 run_post: Processing sfcchar.pltin 04/11/13 03:00:16 run_post: Processing ua.pltin 04/11/13 03:00:17 run_post: Processing tmp.aein 04/11/13 03:00:17 run_post: Job <800284> is submitted to queue . 04/11/13 03:00:17 run_post: t_current now 4200 t_end=7800 04/11/13 03:00:17 run_post: p_string=003600 h_string=0100 t_string=004200 04/11/13 03:00:17 run_post: Checking plot/0100/arpsplt_0100.nqs 04/11/13 03:00:17 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 03:00:17 run_post: Processing sfc.pltin 04/11/13 03:00:17 run_post: Processing sfcchar.pltin 04/11/13 03:00:17 run_post: Processing ua.pltin 04/11/13 03:00:17 run_post: Processing tmp.aein 04/11/13 03:00:17 run_post: Job <800285> is submitted to queue . 04/11/13 03:00:18 run_post: t_current now 4500 t_end=7800 04/11/13 03:00:18 run_post: Looking for 'arpsdata/ar2013041107.hdf004500_001001_ready' 04/11/13 03:00:33 run_post: p_string=003900 h_string=0105 t_string=004500 04/11/13 03:00:33 run_post: Checking plot/0105/arpsplt_0105.nqs 04/11/13 03:00:33 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 03:00:33 run_post: Processing sfc.pltin 04/11/13 03:00:33 run_post: Processing sfcchar.pltin 04/11/13 03:00:33 run_post: Processing ua.pltin 04/11/13 03:00:33 run_post: Processing tmp.aein 04/11/13 03:00:33 run_post: Job <800286> is submitted to queue . 04/11/13 03:00:33 run_post: t_current now 4800 t_end=7800 04/11/13 03:00:33 run_post: Looking for 'arpsdata/ar2013041107.hdf004800_001001_ready' 04/11/13 03:00:48 run_post: p_string=004200 h_string=0110 t_string=004800 04/11/13 03:00:48 run_post: Checking plot/0110/arpsplt_0110.nqs 04/11/13 03:00:48 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 03:00:48 run_post: Processing sfc.pltin 04/11/13 03:00:48 run_post: Processing sfcchar.pltin 04/11/13 03:00:48 run_post: Processing ua.pltin 04/11/13 03:00:49 run_post: Processing tmp.aein 04/11/13 03:00:49 run_post: Job <800287> is submitted to queue . 04/11/13 03:00:49 run_post: t_current now 5100 t_end=7800 04/11/13 03:00:49 run_post: Looking for 'arpsdata/ar2013041107.hdf005100_001001_ready' 04/11/13 03:01:04 run_post: p_string=004500 h_string=0115 t_string=005100 04/11/13 03:01:04 run_post: Checking plot/0115/arpsplt_0115.nqs 04/11/13 03:01:04 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 03:01:04 run_post: Processing sfc.pltin 04/11/13 03:01:04 run_post: Processing sfcchar.pltin 04/11/13 03:01:04 run_post: Processing ua.pltin 04/11/13 03:01:04 run_post: Processing tmp.aein 04/11/13 03:01:04 run_post: Job <800288> is submitted to queue . 04/11/13 03:01:04 run_post: t_current now 5400 t_end=7800 04/11/13 03:01:04 run_post: Looking for 'arpsdata/ar2013041107.hdf005400_001001_ready' 04/11/13 03:01:20 run_post: p_string=004800 h_string=0120 t_string=005400 04/11/13 03:01:20 run_post: Checking plot/0120/arpsplt_0120.nqs 04/11/13 03:01:20 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 03:01:20 run_post: Processing sfc.pltin 04/11/13 03:01:20 run_post: Processing sfcchar.pltin 04/11/13 03:01:20 run_post: Processing ua.pltin 04/11/13 03:01:20 run_post: Processing tmp.aein 04/11/13 03:01:20 run_post: Job <800289> is submitted to queue . 04/11/13 03:01:20 run_post: t_current now 5700 t_end=7800 04/11/13 03:01:20 run_post: Looking for 'arpsdata/ar2013041107.hdf005700_001001_ready' 04/11/13 03:01:35 run_post: p_string=005100 h_string=0125 t_string=005700 04/11/13 03:01:35 run_post: Checking plot/0125/arpsplt_0125.nqs 04/11/13 03:01:35 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 03:01:35 run_post: Processing sfc.pltin 04/11/13 03:01:35 run_post: Processing sfcchar.pltin 04/11/13 03:01:35 run_post: Processing ua.pltin 04/11/13 03:01:35 run_post: Processing tmp.aein 04/11/13 03:01:35 run_post: Job <800290> is submitted to queue . 04/11/13 03:01:36 run_post: t_current now 6000 t_end=7800 04/11/13 03:01:36 run_post: Looking for 'arpsdata/ar2013041107.hdf006000_001001_ready' 04/11/13 03:01:51 run_post: p_string=005400 h_string=0130 t_string=006000 04/11/13 03:01:51 run_post: Checking plot/0130/arpsplt_0130.nqs 04/11/13 03:01:51 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 03:01:51 run_post: Processing sfc.pltin 04/11/13 03:01:51 run_post: Processing sfcchar.pltin 04/11/13 03:01:51 run_post: Processing ua.pltin 04/11/13 03:01:51 run_post: Processing tmp.aein 04/11/13 03:01:51 run_post: Job <800291> is submitted to queue . 04/11/13 03:01:51 run_post: t_current now 6300 t_end=7800 04/11/13 03:01:51 run_post: Looking for 'arpsdata/ar2013041107.hdf006300_001001_ready' 04/11/13 03:02:06 run_post: p_string=005700 h_string=0135 t_string=006300 04/11/13 03:02:06 run_post: Checking plot/0135/arpsplt_0135.nqs 04/11/13 03:02:06 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 03:02:06 run_post: Processing sfc.pltin 04/11/13 03:02:06 run_post: Processing sfcchar.pltin 04/11/13 03:02:06 run_post: Processing ua.pltin 04/11/13 03:02:06 run_post: Processing tmp.aein 04/11/13 03:02:06 run_post: Job <800292> is submitted to queue . 04/11/13 03:02:07 run_post: t_current now 6600 t_end=7800 04/11/13 03:02:07 run_post: Looking for 'arpsdata/ar2013041107.hdf006600_001001_ready' 04/11/13 03:02:22 run_post: p_string=006000 h_string=0140 t_string=006600 04/11/13 03:02:22 run_post: Checking plot/0140/arpsplt_0140.nqs 04/11/13 03:02:22 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 03:02:22 run_post: Processing sfc.pltin 04/11/13 03:02:22 run_post: Processing sfcchar.pltin 04/11/13 03:02:22 run_post: Processing ua.pltin 04/11/13 03:02:22 run_post: Processing tmp.aein 04/11/13 03:02:22 run_post: Job <800293> is submitted to queue . 04/11/13 03:02:22 run_post: t_current now 6900 t_end=7800 04/11/13 03:02:22 run_post: Looking for 'arpsdata/ar2013041107.hdf006900_001001_ready' 04/11/13 03:02:38 run_post: p_string=006300 h_string=0145 t_string=006900 04/11/13 03:02:38 run_post: Checking plot/0145/arpsplt_0145.nqs 04/11/13 03:02:38 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 03:02:38 run_post: Processing sfc.pltin 04/11/13 03:02:38 run_post: Processing sfcchar.pltin 04/11/13 03:02:38 run_post: Processing ua.pltin 04/11/13 03:02:38 run_post: Processing tmp.aein 04/11/13 03:02:38 run_post: Job <800295> is submitted to queue . 04/11/13 03:02:38 run_post: t_current now 7200 t_end=7800 04/11/13 03:02:38 run_post: Looking for 'arpsdata/ar2013041107.hdf007200_001001_ready' 04/11/13 03:02:53 run_post: p_string=006600 h_string=0150 t_string=007200 04/11/13 03:02:53 run_post: Checking plot/0150/arpsplt_0150.nqs 04/11/13 03:02:53 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 03:02:53 run_post: Processing sfc.pltin 04/11/13 03:02:53 run_post: Processing sfcchar.pltin 04/11/13 03:02:53 run_post: Processing ua.pltin 04/11/13 03:02:53 run_post: Processing tmp.aein 04/11/13 03:02:53 run_post: Job <800297> is submitted to queue . 04/11/13 03:02:54 run_post: t_current now 7500 t_end=7800 04/11/13 03:02:54 run_post: Looking for 'arpsdata/ar2013041107.hdf007500_001001_ready' 04/11/13 03:03:09 run_post: p_string=006900 h_string=0155 t_string=007500 04/11/13 03:03:09 run_post: Checking plot/0155/arpsplt_0155.nqs 04/11/13 03:03:09 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 03:03:09 run_post: Processing sfc.pltin 04/11/13 03:03:09 run_post: Processing sfcchar.pltin 04/11/13 03:03:09 run_post: Processing ua.pltin 04/11/13 03:03:09 run_post: Processing tmp.aein 04/11/13 03:03:09 run_post: Job <800299> is submitted to queue . 04/11/13 03:03:09 run_post: t_current now 7800 t_end=7800 04/11/13 03:03:09 run_post: Looking for 'arpsdata/ar2013041107.hdf007800_001001_ready' 04/11/13 03:03:17 arps_1 run: Exit Status 0. 04/11/13 03:03:17 Task 'arps_1 run' finishes. Runtime 00:06:05. 04/11/13 03:03:17 Task 'nudge_outputs_0 run' starts. 04/11/13 03:03:17 /home/kwthomas/rt2013/scripts/casa_outputs cmd=cat cycle1/ad201304110700.3dvarout cycle2/ad201304110710.3dvarout > ad2013041107.3dvarout cmd=cat cycle1/ar201304110700.arpsout cycle2/ar201304110710.arpsout > ar2013041107.arpsout cmd=cat cycle1/ad201304110700.adasstat cycle2/ad201304110710.adasstat > ad2013041107.adasstat 04/11/13 03:03:17 nudge_outputs_0 run: Exit Status 0. 04/11/13 03:03:17 Task 'nudge_outputs_0 run' finishes. Runtime 00:00:00. 04/11/13 03:03:24 run_post: p_string=007200 h_string=0200 t_string=007800 04/11/13 03:03:24 run_post: Checking plot/0200/arpsplt_0200.nqs 04/11/13 03:03:24 run_post: template: reading /home/kwthomas/rt2013/templates/graphics.nqs_template 04/11/13 03:03:24 run_post: Processing sfc.pltin 04/11/13 03:03:24 run_post: Processing sfcchar.pltin 04/11/13 03:03:24 run_post: Processing ua.pltin 04/11/13 03:03:24 run_post: Processing tmp.aein 04/11/13 03:03:24 run_post: Job <800300> is submitted to queue . 04/11/13 03:03:25 plot_arps_0 run: Exit Status 0. 04/11/13 03:03:25 Task 'plot_arps_0 run' finishes. Runtime 00:12:47. 04/11/13 03:03:25 Task Manager finishes. Runtime 00:15:13. Job run time: 00:15 ------------------------------------------------------------ Sender: LSF System Subject: Job 800266: Done Job was submitted from host by user in cluster . Job was executed on host(s) <16*c020>, in queue , as user in cluster . <16*c008> <16*c019> <16*c006> <16*c003> <16*c002> <16*c059> <16*c056> <16*c022> <16*c101> <16*c418> <16*c016> was used as the home directory. was used as the working directory. Started at Thu Apr 11 02:48:06 2013 Results reported at Thu Apr 11 03:03:25 2013 Your job looked like: ------------------------------------------------------------ # LSBATCH: User input #!/bin/csh #BSUB -J casa2-041107-02 #BSUB -R "span[ptile=16]" #BSUB -W 0:40 #BSUB -a openmpi #BSUB -e ar2013041107.err #BSUB -n 192 #BSUB -o ar2013041107.run #BSUB -q caps_forecast #BSUB -x set time1=`perl -e "print time"` echo -n "Job Started: " date echo "Job Id: $LSB_JOBID" echo " " set echo cd /work/arpsdata/arpscasa/fcst-casa2013_dfw/20130411/0745Z restart ar2013041107 7200 0 /home/kwthomas/rt2013/tm/task_manager5 arpspltcycle2 local ar2013041107 /work/arpsdata/arpscasa/fcst-casa2013_dfw/20130411/0745Z /work/arpsdata/arpscasa/fcst-casa2013_dfw/20130411/0745Z casa2013 fgrep -q 'Thanks' ar2013041107.arpsout if ( $status != 0 ) then echo 'It appears that the analysis/forecast failed.' jobs -l > jobs test -s jobs if ( $status == 0 ) then cat jobs | cut -c8- | awk '{ print $1 }' | xargs kill -9 endif /bin/rm -rf jobs goto finish endif wait finish: unset echo set time2=`perl -e "print time"` @ diff = $time2 - $time1 @ hour = $diff / 3600 @ diff = $diff % 3600 @ min = $diff / 60 printf '\nJob run time: %02d:%02d\n' $hour $min ------------------------------------------------------------ Successfully completed. Resource usage summary: CPU time : 86394.56 sec. Max Memory : 1 MB Max Swap : 49 MB Max Processes : 1 Max Threads : 1 The output (if any) is above this job summary. PS: Read file for stderr output of this job.