SMART Ensemble Tutorial

TODO: finish me.

Note: if you are following this tutorial, please help me make it better as you play with it and learn from it!

Ensembles are driven by the watch/cli/coco_average_features.py script, which can be accessed via

geowatch average_features --help

While this tutorial is still being written you can check out doctests inside the CLI file for more examples.

It takes multiple kwcoco files with either saliency or class predictions and then writes them to a new kwcoco file where those channels are averaged together. This new kwcoco file can be sent to the tracker for polygon extraction as if it was a kwcoco file from a single model. The rests of the pipeline can run from these polygons as-is.

Currently running this in mlops is not possible because there is an assumption that only one model is used at a time. A new specialized ensemble pipeline will likely need to be defined to use this effectively, but what you can do is use mlops to generate the commands required for a single model evaluation and then manually add in this step to produce the averaged output. Then just plug that into the tracker inputs and the rest of it should be straight forward to execute.

For example say you have predicted saliency with two models so you have:

• model1/pred.kwcoco.json and

• model2/pred.kwcoco.json

You could ensemble them like:

geowatch average_features \
    --kwcoco_file_paths \
        model1/pred.kwcoco.zip \
        model2/pred.kwcoco.zip \
    --output_kwcoco_path model_ensemble/averaged.kwcoco.zip \
    --channel_name saliency \
    --sensors all

Then for reference you could grab a template for bas commands like:

geowatch schedule --params="
    matrix:
        bas_pxl.package_fpath:
            - MY_PACKAGE.pt
        bas_pxl.test_dataset:
            - MY_DATASET.kwcoco.zip
        bas_pxl.chip_overlap: 0.3
        bas_pxl.chip_dims: auto
        bas_pxl.time_span: auto
        bas_pxl.time_sampling: soft4
        bas_poly.thresh:
            - 0.4
        bas_poly.inner_window_size: 1y
        bas_poly.inner_agg_fn: mean
        bas_poly.norm_ord: inf
        bas_poly.polygon_simplify_tolerance: 1
        bas_poly.agg_fn: probs
        bas_poly.time_thresh:
            - 0.8
        bas_poly.resolution: 10GSD
        bas_poly.moving_window_size: null
        bas_poly.poly_merge_method: 'v2'
        bas_poly.min_area_square_meters: 7200
        bas_poly.max_area_square_meters: 8000000
        bas_poly.boundary_region: $DVC_DATA_DPATH/annotations/drop6/region_models
        bas_poly_eval.true_site_dpath: $DVC_DATA_DPATH/annotations/drop6/site_models
        bas_poly_eval.true_region_dpath: $DVC_DATA_DPATH/annotations/drop6/region_models
        bas_pxl.enabled: 1
        bas_pxl_eval.enabled: 0
        bas_poly.enabled: 1
        bas_poly_eval.enabled: 1
        bas_poly_viz.enabled: 0
    " \
    --root_dpath="$DVC_EXPT_DPATH/_reference" \
    --backend=serial --queue_name "_reference" \
    --pipeline=bas \
    --skip_existing=0 \
    --run=0

And using this as a reference you might construct a set of command that look like this:

# I copied the pixel predict step twice  and put in some custom paths
python -m geowatch.tasks.fusion.predict \
    --package_fpath=MY_MODEL1.pt \
    --test_dataset=MY_DATASET.kwcoco.zip \
    --pred_dataset=./_reference/pred/flat/bas_pxl/model1_preds/pred.kwcoco.zip \
    --chip_overlap=0.3 \
    --chip_dims=auto \
    --time_span=auto \
    --time_sampling=soft4 \
    --drop_unused_frames=True  \
    --num_workers=2 \
    --devices=0, \
    --batch_size=1 \
    --with_saliency=True \
    --with_class=False \
    --with_change=False

python -m geowatch.tasks.fusion.predict \
    --package_fpath=MY_MODEL2.pt \
    --test_dataset=MY_DATASET.kwcoco.zip \
    --pred_dataset=./_reference/pred/flat/bas_pxl/model2_preds/pred.kwcoco.zip \
    --chip_overlap=0.3 \
    --chip_dims=auto \
    --time_span=auto \
    --time_sampling=soft4 \
    --drop_unused_frames=True  \
    --num_workers=2 \
    --devices=0, \
    --batch_size=1 \
    --with_saliency=True \
    --with_class=False \
    --with_change=False

# Inserting the custom average feature script here.

geowatch average_features \
   --kwcoco_file_paths \
       ./_reference/pred/flat/bas_pxl/model1_preds/pred.kwcoco.zip \
       ./_reference/pred/flat/bas_pxl/model2_preds/pred.kwcoco.zip \
   --output_kwcoco_path "./_reference/pred/flat/bas_ensemble/bas_ensemble_custom/pred.kwcoco.zip" \
   --channel_name saliency \
   --sensors all

# The rest of the tracking + eval part of the pipeline is unchanged.

python -m geowatch.cli.run_tracker \
    --in_file "./_reference/pred/flat/bas_ensemble/bas_ensemble_custom/pred.kwcoco.zip" \
    --default_track_fn saliency_heatmaps \
    --track_kwargs '{"agg_fn": "probs", "thresh": 0.4, "inner_window_size": "1y", "inner_agg_fn": "mean", "norm_ord": "inf", "polygon_simplify_tolerance": 1, "time_thresh": 0.8, "resolution": "10GSD", "moving_window_size": null, "poly_merge_method": "v2", "min_area_square_meters": 7200, "max_area_square_meters": 8000000}' \
    --clear_annots=True \
    --site_summary 'None' \
    --boundary_region './annotations/drop6/region_models' \
    --out_site_summaries_fpath "./_reference/pred/flat/bas_poly/bas_poly_id_custom/site_summaries_manifest.json" \
    --out_site_summaries_dir "./_reference/pred/flat/bas_poly/bas_poly_id_custom/site_summaries" \
    --out_sites_fpath "./_reference/pred/flat/bas_poly/bas_poly_id_custom/sites_manifest.json" \
    --out_sites_dir "./_reference/pred/flat/bas_poly/bas_poly_id_custom/sites" \
    --out_kwcoco "./_reference/pred/flat/bas_poly/bas_poly_id_custom/poly.kwcoco.zip"
#
python -m geowatch.cli.run_metrics_framework \
    --merge=True \
    --name "some-name" \
    --true_site_dpath "./annotations/drop6/site_models" \
    --true_region_dpath "./annotations/drop6/region_models" \
    --pred_sites "./_reference/pred/flat/bas_poly/bas_poly_id_custom/sites_manifest.json" \
    --tmp_dir "./_reference/eval/flat/bas_poly_eval/bas_poly_eval_id_custom/tmp" \
    --out_dir "./_reference/eval/flat/bas_poly_eval/bas_poly_eval_id_custom" \
    --merge_fpath "./_reference/eval/flat/bas_poly_eval/bas_poly_eval_id_custom/poly_eval.json"

Note: you could do a similar thing with the more complex bas_building_and_depth_vali pipeline.

Note: I do plan to eventually support ensembles in mlops, but the above should work in the meantime, and showing positive results would make me prioritize it higher.