geowatch.tasks.fusion.evaluate module

Compute semantic segmentation evaluation metrics

TODO:: - RRMSE (relative root mean squared error) RMSE normalized by root mean sqare value where each residual is scaled against the actual value

sqrt((1 / n) * sum((y - y_hat) ** 2) / sum(y ** 2))

Todo

• [ ] Move to kwcoco proper

class geowatch.tasks.fusion.evaluate.SegmentationEvalConfig(*args, **kwargs)

Bases: DataConfig

Evaluation script for change/segmentation task

Valid options: []

Parameters:

• *args – positional arguments for this data config

• **kwargs – keyword arguments for this data config

default = {'balance_area': <Value(False)>, 'draw_curves': <Value('auto')>, 'draw_heatmaps': <Value('auto')>, 'draw_workers': <Value('auto')>, 'eval_dpath': <Value(None)>, 'eval_fpath': <Value(None)>, 'pred_dataset': <Value(None)>, 'resolution': <Value(None)>, 'score_space': <Value('auto')>, 'true_dataset': <Value(None)>, 'viz_thresh': <Value('auto')>, 'workers': <Value('auto')>}

geowatch.tasks.fusion.evaluate.main(cmdline=True, **kwargs)

Entry point: todo: doctest and CLI structure

todo: ProcessContext to track resource usage

geowatch.tasks.fusion.evaluate.single_image_segmentation_metrics(pred_coco_img, true_coco_img, true_classes, true_dets, video1=None, thresh_bins=None, config=None)

Parameters:

• true_coco_img (kwcoco.CocoImage) – detatched true coco image

• pred_coco_img (kwcoco.CocoImage) – detatched predicted coco image

• thresh_bins (int) – if specified rounds scores into this many bins to make calculating metrics more efficient

geowatch.tasks.fusion.evaluate.draw_confusion_image(pred, target)

geowatch.tasks.fusion.evaluate.colorize_class_probs(probs, classes)

probs = pred_cat_ohe classes = pred_classes

geowatch.tasks.fusion.evaluate.draw_truth_borders(true_dets, canvas, alpha=1.0, color=None)

geowatch.tasks.fusion.evaluate.dump_chunked_confusion(full_classes, true_coco_imgs, chunk_info, heatmap_dpath, title=None, config=None)

Draw a a sequence of true/pred image predictions

geowatch.tasks.fusion.evaluate.evaluate_segmentations(true_coco, pred_coco, eval_dpath=None, eval_fpath=None, config=None)

Todo

• [ ] Fold non-critical options into the config

CommandLine

XDEV_PROFILE=1 xdoctest -m geowatch.tasks.fusion.evaluate evaluate_segmentations

Example

>>> from geowatch.tasks.fusion.evaluate import *  # NOQA
>>> from kwcoco.coco_evaluator import CocoEvaluator
>>> from kwcoco.demo.perterb import perterb_coco
>>> import kwcoco
>>> true_coco1 = kwcoco.CocoDataset.demo('vidshapes2', image_size=(64, 64))
>>> true_coco2 = kwcoco.CocoDataset.demo('shapes2', image_size=(64, 64))
>>> #true_coco1 = kwcoco.CocoDataset.demo('vidshapes9')
>>> #true_coco2 = kwcoco.CocoDataset.demo('shapes128')
>>> true_coco = kwcoco.CocoDataset.union(true_coco1, true_coco2)
>>> kwargs = {
>>>     'box_noise': 0.5,
>>>     'n_fp': (0, 10),
>>>     'n_fn': (0, 10),
>>>     'with_probs': True,
>>>     'with_heatmaps': True,
>>>     'verbose': 1,
>>> }
>>> # TODO: it would be nice to demo the soft metrics
>>> # functionality by adding "salient_prob" or "class_prob"
>>> # auxiliary channels to this demodata.
>>> print('perterbing')
>>> pred_coco = perterb_coco(true_coco, **kwargs)
>>> eval_dpath = ub.Path.appdir('geowatch/tests/fusion_eval').ensuredir()
>>> print('eval_dpath = {!r}'.format(eval_dpath))
>>> config = {}
>>> config['score_space'] = 'image'
>>> draw_curves = 'auto'
>>> draw_heatmaps = 'auto'
>>> #draw_heatmaps = False
>>> config['workers'] = 'min(avail-2,6)'
>>> #workers = 0
>>> evaluate_segmentations(true_coco, pred_coco, eval_dpath, config=config)

Example

>>> # xdoctest: +REQUIRES(env:SLOW_DOCTEST)
>>> from geowatch.tasks.fusion.evaluate import *  # NOQA
>>> from kwcoco.coco_evaluator import CocoEvaluator
>>> from kwcoco.demo.perterb import perterb_coco
>>> import kwcoco
>>> true_coco = kwcoco.CocoDataset.demo('vidshapes2', image_size=(64, 64))
>>> kwargs = {
>>>     'box_noise': 0.5,
>>>     'n_fp': (0, 10),
>>>     'n_fn': (0, 10),
>>>     'with_probs': True,
>>>     'with_heatmaps': True,
>>>     'verbose': 1,
>>> }
>>> # TODO: it would be nice to demo the soft metrics
>>> # functionality by adding "salient_prob" or "class_prob"
>>> # auxiliary channels to this demodata.
>>> print('perterbing')
>>> pred_coco = perterb_coco(true_coco, **kwargs)
>>> eval_dpath = ub.Path.appdir('geowatch/tests/fusion_eval-video').ensuredir()
>>> print('eval_dpath = {!r}'.format(eval_dpath))
>>> config = {}
>>> config['score_space'] = 'video'
>>> config['balance_area'] = True
>>> draw_curves = 'auto'
>>> draw_heatmaps = 'auto'
>>> #draw_heatmaps = False
>>> config['workers'] = 'min(avail-2,6)'
>>> #workers = 0
>>> evaluate_segmentations(true_coco, pred_coco, eval_dpath, config=config)