geowatch.tasks.fusion.evaluate module

Compute semantic segmentation evaluation metrics

TODO:: - RRMSE (relative root mean squared error) RMSE normalized by root mean sqare value where each residual is scaled against the actual value

sqrt((1 / n) * sum((y - y_hat) ** 2) / sum(y ** 2))

Todo

  • [ ] Move to kwcoco proper

class geowatch.tasks.fusion.evaluate.SegmentationEvalConfig(*args, **kwargs)[source]

Bases: DataConfig

Evaluation script for change/segmentation task

Valid options: []

Parameters:

  • *args – positional arguments for this data config

  • **kwargs – keyword arguments for this data config

default = {'balance_area': <Value(False)>, 'draw_curves': <Value('auto')>, 'draw_heatmaps': <Value('auto')>, 'draw_legend': <Value(True)>, 'draw_weights': <Value(False)>, 'draw_workers': <Value('auto')>, 'eval_dpath': <Value(None)>, 'eval_fpath': <Value(None)>, 'pred_dataset': <Value(None)>, 'resolution': <Value(None)>, 'score_space': <Value('auto')>, 'thresh_bins': <Value(1024)>, 'true_dataset': <Value(None)>, 'viz_thresh': <Value('auto')>, 'workers': <Value('auto')>}
geowatch.tasks.fusion.evaluate.main(cmdline=True, **kwargs)[source]

Entry point: todo: doctest and CLI structure

geowatch.tasks.fusion.evaluate.single_image_segmentation_metrics(pred_coco_img, true_coco_img, true_classes, true_dets, video1=None, thresh_bins=None, config=None, salient_channel='salient')[source]
Parameters:

  • true_coco_img (kwcoco.CocoImage) – detatched true coco image

  • pred_coco_img (kwcoco.CocoImage) – detatched predicted coco image

  • thresh_bins (int) – if specified rounds scores into this many bins to make calculating metrics more efficient

CommandLine

xdoctest -m geowatch.tasks.fusion.evaluate single_image_segmentation_metrics

Example

>>> from geowatch.tasks.fusion.evaluate import *  # NOQA
>>> from kwcoco.coco_evaluator import CocoEvaluator
>>> from kwcoco.demo.perterb import perterb_coco
>>> import kwcoco
>>> # TODO: kwcoco demodata with easy dummy heatmap channels
>>> true_coco = kwcoco.CocoDataset.demo('vidshapes2', image_size=(64, 64))
>>> # Score an image against itself
>>> true_coco_img = true_coco.images()[0:1].coco_images[0]
>>> pred_coco_img = true_coco.images()[0:1].coco_images[0]
>>> config = {}
>>> true_dets = true_coco_img.annots().detections
>>> video1 = true_coco_img.video
>>> true_classes = true_coco.object_categories()
>>> salient_channel = 'r'  # pretend red is the salient channel
>>> thresh_bins = np.linspace(0, 255, 1024)
>>> info = single_image_segmentation_metrics(
>>>    pred_coco_img, true_coco_img, true_classes, true_dets,
>>>    thresh_bins=thresh_bins, config=config, video1=video1, salient_channel=salient_channel)
geowatch.tasks.fusion.evaluate.draw_confusion_image(pred, target)[source]
geowatch.tasks.fusion.evaluate.colorize_class_probs(probs, classes)[source]

probs = pred_cat_ohe classes = pred_classes

geowatch.tasks.fusion.evaluate.draw_truth_borders(true_dets, canvas, alpha=1.0, color=None)[source]
geowatch.tasks.fusion.evaluate.dump_chunked_confusion(full_classes, true_coco_imgs, chunk_info, heatmap_dpath, title=None, config=None)[source]

Draw a a sequence of true/pred image predictions

geowatch.tasks.fusion.evaluate.evaluate_segmentations(true_coco, pred_coco, eval_dpath=None, eval_fpath=None, config=None)[source]

Todo

  • [ ] Fold non-critical options into the config

CommandLine

XDEV_PROFILE=1 xdoctest -m geowatch.tasks.fusion.evaluate evaluate_segmentations

Example

>>> from geowatch.tasks.fusion.evaluate import *  # NOQA
>>> from kwcoco.coco_evaluator import CocoEvaluator
>>> from kwcoco.demo.perterb import perterb_coco
>>> import kwcoco
>>> true_coco1 = kwcoco.CocoDataset.demo('vidshapes2', image_size=(64, 64))
>>> true_coco2 = kwcoco.CocoDataset.demo('shapes2', image_size=(64, 64))
>>> #true_coco1 = kwcoco.CocoDataset.demo('vidshapes9')
>>> #true_coco2 = kwcoco.CocoDataset.demo('shapes128')
>>> true_coco = kwcoco.CocoDataset.union(true_coco1, true_coco2)
>>> kwargs = {
>>>     'box_noise': 0.5,
>>>     'n_fp': (0, 10),
>>>     'n_fn': (0, 10),
>>>     'with_probs': True,
>>>     'with_heatmaps': True,
>>>     'verbose': 1,
>>> }
>>> # TODO: it would be nice to demo the soft metrics
>>> # functionality by adding "salient_prob" or "class_prob"
>>> # auxiliary channels to this demodata.
>>> print('perterbing')
>>> pred_coco = perterb_coco(true_coco, **kwargs)
>>> eval_dpath = ub.Path.appdir('geowatch/tests/fusion_eval').ensuredir()
>>> print('eval_dpath = {!r}'.format(eval_dpath))
>>> config = {}
>>> config['score_space'] = 'image'
>>> draw_curves = 'auto'
>>> draw_heatmaps = 'auto'
>>> #draw_heatmaps = False
>>> config['workers'] = 'min(avail-2,6)'
>>> #workers = 0
>>> evaluate_segmentations(true_coco, pred_coco, eval_dpath, config=config)

Example

>>> # xdoctest: +REQUIRES(env:SLOW_DOCTEST)
>>> from geowatch.tasks.fusion.evaluate import *  # NOQA
>>> from kwcoco.coco_evaluator import CocoEvaluator
>>> from kwcoco.demo.perterb import perterb_coco
>>> import kwcoco
>>> true_coco = kwcoco.CocoDataset.demo('vidshapes2', image_size=(64, 64))
>>> kwargs = {
>>>     'box_noise': 0.5,
>>>     'n_fp': (0, 10),
>>>     'n_fn': (0, 10),
>>>     'with_probs': True,
>>>     'with_heatmaps': True,
>>>     'verbose': 1,
>>> }
>>> # TODO: it would be nice to demo the soft metrics
>>> # functionality by adding "salient_prob" or "class_prob"
>>> # auxiliary channels to this demodata.
>>> print('perterbing')
>>> pred_coco = perterb_coco(true_coco, **kwargs)
>>> eval_dpath = ub.Path.appdir('geowatch/tests/fusion_eval-video').ensuredir()
>>> print('eval_dpath = {!r}'.format(eval_dpath))
>>> config = {}
>>> config['score_space'] = 'video'
>>> config['balance_area'] = True
>>> draw_curves = 'auto'
>>> draw_heatmaps = 'auto'
>>> #draw_heatmaps = False
>>> config['workers'] = 'min(avail-2,6)'
>>> #workers = 0
>>> evaluate_segmentations(true_coco, pred_coco, eval_dpath, config=config)
class geowatch.tasks.fusion.evaluate.MaxQueuePool(max_queue_size=None, mode='thread', max_workers=0)[source]

Bases: object

This Class wraps a concurrent.futures.Executor limiting the size of its task queue. If max_queue_size tasks are submitted, the next call to submit will block until a previously submitted one is completed.

References

submit(function, *args, **kwargs)[source]

Submits a new task to the pool, blocks if Pool queue is full.

pool_queue_callback(_)[source]

Called once task is done, releases one queue slot.

shutdown()[source]

Calls the shutdown function of the underlying backend.