Create reformatting support for MODE

[bikegeek]

Describe the Enhancement

Provide reformatting support for MET MODE output. MODE has two output files: CTS and object attribute. Support formats that can be read in by METcalcpy agg_stat_bootstrap.py module. Refer to the R script agg_stat_boostrap.info_tmpl for creating the config file for agg_stat_bootstrap.py.

Sample XML files

• on 'seneca'
• /d1/persona/mkalb/MODE_plots copied to /d1/personal/mwin/MODE_NRL/MODE_from_Tina
• Begin with supporting plots in the Fcst_vs_Obs directory
• APCP and REFC variables
  Stats of interest:
• acsi ( plot_icperts_mode_apcp_afbias.png)

associated XML config file from METviewer:

 `<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<plot_spec>
   <connection>
       <host>mohawk</host>
       <database>mv_rrfse_icpert_stoch_mode_fvo</database>
       <user>******</user>
       <password>******</password>
       <management_system>mariadb</management_system>
   </connection>
   <rscript>/usr/local/R/bin/Rscript</rscript>
   <folders>
       <r_tmpl>/opt/vxwww/tomcat/webapps/metviewer//R_tmpl</r_tmpl>
       <r_work>/opt/vxwww/tomcat/webapps/metviewer//R_work</r_work>
       <plots>/d2/www/dtcenter/met/metviewer_output//plots</plots>
       <data>/d2/www/dtcenter/met/metviewer_output//data</data>
       <scripts>/d2/www/dtcenter/met/metviewer_output//scripts</scripts>
   </folders>
   <plot>
       <template>series_plot.R_tmpl</template>
       <dep>
           <dep1>
               <fcst_var name="APCP_01_ENS_NEP_ge2.54_NBRHD729">
                   <stat>AREARAT_FSA_OSA</stat>
               </fcst_var>
           </dep1>
           <dep2/>
       </dep>
       <series1>
           <field name="model">
               <val>nostoch</val>
               <val>stoch</val>
           </field>
       </series1>
       <series2/>
       <plot_fix/>
       <plot_cond/>
       <indep equalize="false" name="fcst_lead">
           <val label="1" plot_val="">10000</val>
           <val label="2" plot_val="">20000</val>
           <val label="3" plot_val="">30000</val>
           <val label="4" plot_val="">40000</val>
           <val label="5" plot_val="">50000</val>
           <val label="6" plot_val="">60000</val>
           <val label="7" plot_val="">70000</val>
           <val label="8" plot_val="">80000</val>
           <val label="9" plot_val="">90000</val>
           <val label="10" plot_val="">100000</val>
           <val label="11" plot_val="">110000</val>
           <val label="12" plot_val="">120000</val>
           <val label="13" plot_val="">130000</val>
           <val label="14" plot_val="">140000</val>
           <val label="15" plot_val="">150000</val>
           <val label="16" plot_val="">160000</val>
           <val label="17" plot_val="">170000</val>
           <val label="18" plot_val="">180000</val>
           <val label="19" plot_val="">190000</val>
           <val label="20" plot_val="">200000</val>
           <val label="21" plot_val="">210000</val>
           <val label="22" plot_val="">220000</val>
           <val label="23" plot_val="">230000</val>
           <val label="24" plot_val="">240000</val>
           <val label="25" plot_val="">250000</val>
           <val label="26" plot_val="">260000</val>
           <val label="27" plot_val="">270000</val>
           <val label="28" plot_val="">280000</val>
           <val label="29" plot_val="">290000</val>
           <val label="30" plot_val="">300000</val>
           <val label="31" plot_val="">310000</val>
           <val label="32" plot_val="">320000</val>
           <val label="33" plot_val="">330000</val>
           <val label="34" plot_val="">340000</val>
           <val label="35" plot_val="">350000</val>
           <val label="36" plot_val="">360000</val>
       </indep>
       <agg_stat>
           <agg_mode>true</agg_mode>
           <boot_repl>1</boot_repl>
           <boot_random_seed/>
           <boot_ci>perc</boot_ci>
           <eveq_dis>false</eveq_dis>
           <cache_agg_stat>false</cache_agg_stat>
           <circular_block_bootstrap>true</circular_block_bootstrap>
       </agg_stat>
       <plot_stat>median</plot_stat>
       <tmpl>
           <data_file>plot_icperts_mode_apcp_afbias.data</data_file>
           <plot_file>plot_icperts_mode_apcp_afbias.png</plot_file>
           <r_file>plot_icperts_mode_apcp_afbias.R</r_file>
           <title>1h Accum Precipitation NEP (&gt;= 2.54 mm) Object Area Weighted Frequency Bias</title>
           <x_label>Forecast Lead Time (h)</x_label>
           <y1_label>Frequency Bias</y1_label>
           <y2_label/>
           <caption/>
           <job_title>icperts_mode_apcp_afbias</job_title>
           <keep_revisions>false</keep_revisions>
           <listdiffseries1>list()</listdiffseries1>
           <listdiffseries2>list()</listdiffseries2>
       </tmpl>
       <execution_type>Python</execution_type>
       <event_equal>false</event_equal>
       <vert_plot>false</vert_plot>
       <x_reverse>false</x_reverse>
       <num_stats>false</num_stats>
       <indy1_stag>false</indy1_stag>
       <indy2_stag>false</indy2_stag>
       <start_from_zero>false</start_from_zero>
       <grid_on>true</grid_on>
       <sync_axes>false</sync_axes>
       <dump_points1>false</dump_points1>
       <dump_points2>false</dump_points2>
       <log_y1>false</log_y1>
       <log_y2>false</log_y2>
       <varianceinflationfactor>false</varianceinflationfactor>
       <plot_type>png16m</plot_type>
       <plot_height>8.5</plot_height>
       <plot_width>11</plot_width>
       <plot_res>72</plot_res>
       <plot_units>in</plot_units>
       <mar>c(8,4,5,4)</mar>
       <mgp>c(1,1,0)</mgp>
       <cex>1</cex>
       <title_weight>2</title_weight>
       <title_size>1.4</title_size>
       <title_offset>-2</title_offset>
       <title_align>0.5</title_align>
       <xtlab_orient>1</xtlab_orient>
       <xtlab_perp>-0.75</xtlab_perp>
       <xtlab_horiz>0.5</xtlab_horiz>
       <xtlab_freq>0</xtlab_freq>
       <xtlab_size>2</xtlab_size>
       <xlab_weight>1</xlab_weight>
       <xlab_size>3</xlab_size>
       <xlab_offset>2</xlab_offset>
       <xlab_align>0.5</xlab_align>
       <ytlab_orient>1</ytlab_orient>
       <ytlab_perp>0.5</ytlab_perp>
       <ytlab_horiz>0.5</ytlab_horiz>
       <ytlab_size>2</ytlab_size>
       <ylab_weight>1</ylab_weight>
       <ylab_size>3</ylab_size>
       <ylab_offset>-7</ylab_offset>
       <ylab_align>0.5</ylab_align>
       <grid_lty>3</grid_lty>
       <grid_col>#cccccc</grid_col>
       <grid_lwd>1</grid_lwd>
       <grid_x>listX</grid_x>
       <x2tlab_orient>1</x2tlab_orient>
       <x2tlab_perp>1</x2tlab_perp>
       <x2tlab_horiz>0.5</x2tlab_horiz>
       <x2tlab_size>0.8</x2tlab_size>
       <x2lab_size>0.8</x2lab_size>
       <x2lab_offset>-0.5</x2lab_offset>
       <x2lab_align>0.5</x2lab_align>
       <y2tlab_orient>1</y2tlab_orient>
       <y2tlab_perp>0.5</y2tlab_perp>
       <y2tlab_horiz>0.5</y2tlab_horiz>
       <y2tlab_size>1</y2tlab_size>
       <y2lab_size>1</y2lab_size>
       <y2lab_offset>1</y2lab_offset>
       <y2lab_align>0.5</y2lab_align>
       <legend_box>o</legend_box>
       <legend_inset>c(0, -.25)</legend_inset>
       <legend_ncol>3</legend_ncol>
       <legend_size>1.2</legend_size>
       <caption_weight>1</caption_weight>
       <caption_col>#333333</caption_col>
       <caption_size>0.8</caption_size>
       <caption_offset>3</caption_offset>
       <caption_align>0</caption_align>
       <ci_alpha>0.05</ci_alpha>
       <eqbound_low>-0.001</eqbound_low>
       <eqbound_high>0.001</eqbound_high>
       <lines>
           <line color="#000000" line_pos="1" lty="1" lwd="1" type="horiz_line"/>
       </lines>
       <plot_ci>c("none","none")</plot_ci>
       <show_signif>c(FALSE,FALSE)</show_signif>
       <plot_disp>c(TRUE,TRUE)</plot_disp>
       <colors>c("#ff0000FF","#0000ffFF")</colors>
       <pch>c(20,20)</pch>
       <type>c("b","b")</type>
       <lty>c(1,1)</lty>
       <lwd>c(2,2)</lwd>
       <con_series>c(1,1)</con_series>
       <order_series>c(1,2)</order_series>
       <plot_cmd/>
       <legend>c("no stoch","stoch")</legend>
       <create_html>TRUE</create_html>
       <y1_lim>c()</y1_lim>
       <x1_lim>c()</x1_lim>
       <y1_bufr>0.04</y1_bufr>
       <y2_lim>c()</y2_lim>
   </plot>
</plot_spec>

• afbias
• matched object centroid distance (m_cendist)
• matched and unmatched objects intensity (mu_int90)
• object area (mu_area)
• object counts (mu_ocounts)

Time Estimate

Estimate the amount of work required here.
3-5 days per sub-issue

Sub-Issues

Consider breaking the enhancement down into sub-issues.

• Reformat object attribute output (stat_name, stat_values, stat_btcl, stat_btcu, n_stats)
• Reformat object attribute output for input to METcalcpy agg_stat_bootstrap.py module
• Reformat the CTS output (stat_name, stat_values, stat_btcl, stat_btcu,??)
• Generate config file for running METcalcpy agg_stat_bootstrap.py to calculate the boostrap statistics
• Verify that plots like the above example can be generated
• Generate tests to verify that reformatting is correct
• Update User's Guide in METdataio

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

NRL Year 1 7730022

Define the Metadata

Assignee

• Select engineer(s) Minna Win
• Select scientist(s) Tina Kalb

Labels

• Review default alert labels
• Select component(s)
• Select priority
• Select requestor(s)

Milestone and Projects

• Select Milestone as a METdataio-X.Y.Z version, Consider for Next Release, or Backlog of Development Ideas
• For a METdataio-X.Y.Z version, select the METdataio-X.Y.Z Development project

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Add any new Python packages to the METplus Components Python Requirements table.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issue
  Select: Milestone as the next official version
  Select: METdataio-X.Y.Z Development project for development toward the next official release
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.