geowatch.cli.run_metrics_framework module¶

Script for running the iarpa metrics code with a bit of post-processing.

Todo

[ ] Rename to run_polygon_evaluation.py? Or run_iarpa_metrics.py?

Note currently depends on iarpa_smart_metrics, which can be obtained from:

https://smartgitlab.com/TE/metrics-and-test-framework or https://gitlab.kitware.com/smart/metrics-and-test-framework

Example

>>> # xdoctest: +REQUIRES(module:iarpa_smart_metrics)
>>> from geowatch.cli.run_metrics_framework import *  # NOQA
>>> from geowatch.demo.metrics_demo.generate_demodata import generate_demo_metrics_framework_data
>>> cmdline = 0
>>> base_dpath = ub.Path.appdir('geowatch', 'tests', 'test-iarpa-metrics2')
>>> data_dpath = base_dpath / 'inputs'
>>> dpath = base_dpath / 'outputs'
>>> demo_info1 = generate_demo_metrics_framework_data(
>>>     roi='DR_R001',
>>>     num_sites=5, num_observations=10, noise=2, p_observe=0.5,
>>>     p_transition=0.3, drop_noise=0.5, drop_limit=0.5)
>>> demo_info2 = generate_demo_metrics_framework_data(
>>>     roi='DR_R002',
>>>     num_sites=7, num_observations=10, noise=1, p_observe=0.5,
>>>     p_transition=0.1, drop_noise=0.8, drop_limit=0.5)
>>> demo_info3 = generate_demo_metrics_framework_data(
>>>     roi='DR_R003',
>>>     num_sites=11, num_observations=10, noise=3, p_observe=0.5,
>>>     p_transition=0.2, drop_noise=0.3, drop_limit=0.5)
>>> print('demo_info1 = {}'.format(ub.urepr(demo_info1, nl=1)))
>>> print('demo_info2 = {}'.format(ub.urepr(demo_info2, nl=1)))
>>> print('demo_info3 = {}'.format(ub.urepr(demo_info3, nl=1)))
>>> out_dpath = dpath / 'region_metrics'
>>> merge_fpath = dpath / 'merged.json'
>>> out_dpath.delete()
>>> kwargs = {
>>>     'pred_sites': demo_info1['pred_site_dpath'],
>>>     'true_region_dpath': demo_info1['true_region_dpath'],
>>>     'true_site_dpath': demo_info1['true_site_dpath'],
>>>     'merge': True,
>>>     'merge_fpath': merge_fpath,
>>>     'out_dir': out_dpath,
>>> }
>>> main(cmdline=False, **kwargs)
>>> # TODO: visualize

Example

>>> # xdoctest: +REQUIRES(module:iarpa_smart_metrics)
>>> from geowatch.cli.run_metrics_framework import *  # NOQA
>>> from geowatch.demo.metrics_demo.generate_demodata import generate_demo_metrics_framework_data
>>> # Test single region case
>>> cmdline = 0
>>> base_dpath = ub.Path.appdir('geowatch', 'tests', 'test-iarpa-metrics5')
>>> data_dpath = (base_dpath / 'inputs').ensuredir()
>>> dpath = (base_dpath / 'outputs').ensuredir()
>>> demo_info1 = generate_demo_metrics_framework_data(
>>>     roi='DR_R004',
>>>     num_sites=5, num_observations=10, noise=2, p_observe=0.5,
>>>     p_transition=0.3, drop_noise=0.5, drop_limit=0.5, outdir=data_dpath)
>>> print('demo_info1 = {}'.format(ub.urepr(demo_info1, nl=1)))
>>> out_dpath = dpath / 'poly_eval'
>>> merge_fpath = dpath / 'poly_eval.json'
>>> out_dpath.delete()
>>> kwargs = {
>>>     'pred_sites': demo_info1['pred_site_dpath'],
>>>     'true_region_dpath': demo_info1['true_region_dpath'],
>>>     'true_site_dpath': demo_info1['true_site_dpath'],
>>>     'merge': True,
>>>     'merge_fpath': merge_fpath,
>>>     'out_dir': out_dpath,
>>> }
>>> main(cmdline=False, **kwargs)
>>> # TODO: visualize

class geowatch.cli.run_metrics_framework.MetricsConfig(*args, **kwargs)[source]¶

Bases: DataConfig

Score IARPA site model GeoJSON files using IARPA’s metrics-and-test-framework

Valid options: []

Parameters:

*args – positional arguments for this data config
**kwargs – keyword arguments for this data config

default = {'enable_sc_viz': <Value(False)>, 'enable_viz': <Value(False)>, 'gt_dpath': <Value(None)>, 'load_workers': <Value(0)>, 'merge': <Value('overwrite')>, 'merge_fbetas': <Value([])>, 'merge_fpath': <Value(None)>, 'name': <Value('unknown')>, 'out_dir': <Value(None)>, 'parallel': <Value(False)>, 'performer': <Value('kit')>, 'pred_sites': <Value(None)>, 'rho': <Value(0.5)>, 'tau': <Value(0.2)>, 'tmp_dir': <Value(None)>, 'true_region_dpath': <Value(None)>, 'true_site_dpath': <Value(None)>}¶

main(**kwargs)¶

CommandLine

xdoctest -m geowatch.cli.run_metrics_framework main

geowatch.cli.run_metrics_framework.ensure_thumbnails(image_root, region_id, sites)[source]¶

Symlink and organize images in the format the metrics framework expects

For the region visualizations: > image_list = glob(f”{self.image_path}/ > {self.region_model.id.replace(‘_’, ‘/’)}/images///*.jp2”)

For the site visualizations: > image_list = glob(f”{self.image_path}/ > {gt_ann_id.replace(‘_’, ‘/’)}/crops/*.tif”)

Which becomes: {country_code}/

{region_num}/

images/

*/

*/
*.jp2

{site_num}/

crops/
*.tif

Parameters:

image_root – root directory to save under
region_id – ex. ‘KR_R001’
sites – proposed sites with image paths in the ‘source’ field TODO change to ‘misc_info’ field

geowatch.cli.run_metrics_framework.main(cmdline=True, **kwargs)[source]¶

CommandLine

xdoctest -m geowatch.cli.run_metrics_framework main