geowatch.cli.coco_time_combine module¶

SeeAlso:: ~/code/watch/geowatch/cli/queue_cli/prepare_time_combined_dataset.py

CommandLine

DVC_DATA_DPATH=$(geowatch_dvc --tags='phase2_data' --hardware=auto)

python -m geowatch.cli.coco_time_combine \
    --input_kwcoco_fpath="$DVC_DATA_DPATH/Drop6/imgonly-KR_R002.kwcoco.zip" \
    --output_kwcoco_fpath="$DVC_DATA_DPATH/Drop6_MeanYear/imgonly-KR_R002.kwcoco.zip" \
    --channels="red|green|blue|nir|swir16|swir22" \
    --resolution=10GSD \
    --time_window=1year \
    --merge_method=mean \
    --workers=4

python -m geowatch reproject_annotations \
    --src $DVC_DATA_DPATH/Drop6_MeanYear/imgonly-KR_R002.kwcoco.zip \
    --dst $DVC_DATA_DPATH/Drop6_MeanYear/imganns-KR_R002.kwcoco.zip \
    --site_models="$DVC_DATA_DPATH/annotations/drop6/site_models/*.geojson"

Example

>>> # Toydata example for CI
>>> import geowatch
>>> from geowatch.cli import coco_time_combine
>>> import ubelt as ub
>>> dpath = ub.Path.appdir('geowatch/tests/cli/time_combine/t0')
>>> dset = geowatch.coerce_kwcoco(
>>>     'geowatch-msi', geodata=True,
>>>     dates={'start_time': '2020-01-01', 'end_time': '2020-06-01'},
>>>     image_size=(32, 32)
>>> )
>>> dpath.delete().ensuredir()
>>> output_fpath = dpath / 'time_combined/data.kwcoco.json'
>>> gsd = dset.videos().objs[0]['target_gsd']
>>> kwargs = coco_time_combine.TimeCombineConfig(
>>>     input_kwcoco_fpath=dset.fpath,
>>>     output_kwcoco_fpath=output_fpath,
>>>     time_window='2month',
>>>     merge_method='mean',
>>>     resolution=f'{gsd}GSD',
>>>     start_time='2019-06-01',
>>> )
>>> cmdline = 0
>>> coco_time_combine.main(cmdline=cmdline, **kwargs)
>>> import kwcoco
>>> out_dset = kwcoco.CocoDataset(output_fpath)
>>> assert len(out_dset.videos()) == len(dset.videos())
>>> assert out_dset.n_images < dset.n_images

from geowatch.cli import coco_visualize_videos coco_visualize_videos.main(cmdline=0, src=output_fpath, stack=’only’, workers=’avail’)

class geowatch.cli.coco_time_combine.TimeCombineConfig(*args, **kwargs)[source]¶

Bases: DataConfig

Averages kwcoco images over a sliding temporal window in a video.

Valid options: []

Parameters:

*args – positional arguments for this data config
**kwargs – keyword arguments for this data config

default = {'assets_dname': <Value('_assets')>, 'avoid_quality_values': <Value(['cloud'])>, 'channels': <Value('*')>, 'exclude_sensors': <Value(None)>, 'filter_season': <Value([])>, 'include_sensors': <Value(None)>, 'input_kwcoco_fpath': <Value(None)>, 'mask_low_quality': <Value(True)>, 'max_images_per_group': <Value(None)>, 'merge_method': <Value('mean')>, 'output_kwcoco_fpath': <Value(None)>, 'resolution': <Value('10GSD')>, 'select_images': <Value(None)>, 'select_videos': <Value(None)>, 'sensor_weights': <Value(0)>, 'separate_sensors': <Value(True)>, 'spatial_tile_size': <Value(None)>, 'start_time': <Value(None)>, 'time_window': <Value('1month')>, 'workers': <Value(0)>}¶

geowatch.cli.coco_time_combine.main(cmdline=1, **kwargs)[source]¶

CommandLine

DEVEL_TEST=1 xdoctest -m geowatch.cli.coco_time_combine main

from geowatch.cli.coco_time_combine import *  # NOQA
import geowatch
data_dvc_dpath = geowatch.find_dvc_dpath(tags='phase2_data', hardware='auto')
cmdline = 0
channels='red|green|blue'
kwargs = dict(
    input_kwcoco_fpath=data_dvc_dpath / 'Drop6/imgonly-KR_R001.kwcoco.zip',
    output_kwcoco_fpath=data_dvc_dpath / 'TestAveDrop6/test-timeave-valid_split1_1yr_mean_test.kwcoco.zip',
    workers=4,
    mask_low_quality=True,
    time_window='1 year',
    channels=channels,
)
output_coco_dset = main(cmdline=cmdline, **kwargs)
coco_visualize_videos.main(cmdline=cmdline, src=kwargs['output_kwcoco_fpath'], smart=True)

Example

>>> # 0: Baseline run.
>>> # xdoctest: +REQUIRES(env:DEVEL_TEST)
>>> from geowatch.cli.coco_time_combine import *  # NOQA
>>> import geowatch
>>> data_dvc_dpath = geowatch.find_dvc_dpath(tags='phase2_data', hardware='auto')
>>> cmdline = 0
>>> kwargs = dict(
>>>     input_kwcoco_fpath=data_dvc_dpath / 'Drop6/imgonly-KR_R001.kwcoco.zip',
>>>     output_kwcoco_fpath=data_dvc_dpath / 'Drop6/test-timeave-imgonly-KR_R001.kwcoco.zip',
>>>     workers=11,
>>>     mask_low_quality=False,
>>>     time_window='1 year',
>>>     channels='red|green|blue',
>>> )
>>> output_coco_dset = main(cmdline=cmdline, **kwargs)
>>> from geowatch.cli import coco_visualize_videos
>>> coco_visualize_videos.main(cmdline=0, src=kwargs['output_kwcoco_fpath'], smart=True)

Example

>>> # 1: Check cloudmasking.
>>> # xdoctest: +REQUIRES(env:DEVEL_TEST)
>>> from geowatch.cli.coco_time_combine import *  # NOQA
>>> import geowatch
>>> data_dvc_dpath = geowatch.find_dvc_dpath(tags='phase2_data', hardware='auto')
>>> cmdline = 0
>>> kwargs = dict(
>>>     input_kwcoco_fpath=data_dvc_dpath / 'Drop6/imgonly-KR_R001.kwcoco.zip',
>>>     output_kwcoco_fpath=data_dvc_dpath / 'Drop6/test-timeave-KR_R001-cloudmask.kwcoco.zip',
>>>     workers=11,
>>>     time_window='1 year',
>>>     channels='red|green|blue',
>>>     mask_low_quality=True,
>>> )
>>> output_coco_dset = main(cmdline=cmdline, **kwargs)
>>> from geowatch.cli import coco_visualize_videos
>>> coco_visualize_videos.main(cmdline=0, src=kwargs['output_kwcoco_fpath'], smart=True)

Example

>>> # 2: Check that resolution can be updated.
>>> # xdoctest: +REQUIRES(env:DEVEL_TEST)
>>> from geowatch.cli.coco_time_combine import *  # NOQA
>>> import geowatch
>>> data_dvc_dpath = geowatch.find_dvc_dpath(tags='phase2_data', hardware='auto')
>>> cmdline = 0
>>> kwargs = dict(
>>>     input_kwcoco_fpath=data_dvc_dpath / 'Drop6/imgonly-KR_R001.kwcoco.zip',
>>>     output_kwcoco_fpath=data_dvc_dpath / 'Drop6/test-timeave-KR_R001-cloudmask-5GSD.kwcoco.zip',
>>>     workers=11,
>>>     time_window='1 year',
>>>     channels='red|green|blue',
>>>     mask_low_quality=True,
>>>     resolution='5GSD',
>>> )
>>> output_coco_dset = main(cmdline=cmdline, **kwargs)
>>> from geowatch.cli import coco_visualize_videos
>>> coco_visualize_videos.main(cmdline=0, src=kwargs['output_kwcoco_fpath'], smart=True)

Example

>>> # 3: Median combining.
>>> # xdoctest: +REQUIRES(env:DEVEL_TEST)
>>> from geowatch.cli.coco_time_combine import *  # NOQA
>>> import geowatch
>>> data_dvc_dpath = geowatch.find_dvc_dpath(tags='phase2_data', hardware='auto')
>>> cmdline = 0
>>> kwargs = dict(
>>>     input_kwcoco_fpath=data_dvc_dpath / 'Drop6/imgonly-KR_R001.kwcoco.zip',
>>>     output_kwcoco_fpath=data_dvc_dpath / 'Drop6/test-timeave-KR_R001-median.kwcoco.zip',
>>>     workers=11,
>>>     merge_method='median',
>>>     time_window='1 year',
>>>     channels='red|green|blue',
>>>     mask_low_quality=False,
>>>     resolution='10GSD',
>>> )
>>> output_coco_dset = main(cmdline=cmdline, **kwargs)
>>> from geowatch.cli import coco_visualize_videos
>>> coco_visualize_videos.main(cmdline=0, src=kwargs['output_kwcoco_fpath'], smart=True)

Example

>>> # 4: Median combining with cloudmask.
>>> # xdoctest: +REQUIRES(env:DEVEL_TEST)
>>> from geowatch.cli.coco_time_combine import *  # NOQA
>>> import geowatch
>>> data_dvc_dpath = geowatch.find_dvc_dpath(tags='phase2_data', hardware='auto')
>>> cmdline = 0
>>> kwargs = dict(
>>>     input_kwcoco_fpath=data_dvc_dpath / 'Drop6/imganns-KR_R001.kwcoco.zip',
>>>     output_kwcoco_fpath=data_dvc_dpath / 'Drop6/test-timeave-KR_R001-cloudmask-median.kwcoco.zip',
>>>     workers=11,
>>>     merge_method='median',
>>>     time_window='1 year',
>>>     channels='red|green|blue',
>>>     mask_low_quality=True,
>>>     resolution='10GSD',
>>>     separate_sensors=True,
>>> )
>>> output_coco_dset = main(cmdline=cmdline, **kwargs)
>>> from geowatch.cli import coco_visualize_videos
>>> coco_visualize_videos.main(cmdline=0, src=kwargs['output_kwcoco_fpath'], smart=True)

Example

>>> # 5: Dont separate sensors.
>>> # xdoctest: +REQUIRES(env:DEVEL_TEST)
>>> from geowatch.cli.coco_time_combine import *  # NOQA
>>> import geowatch
>>> data_dvc_dpath = geowatch.find_dvc_dpath(tags='phase2_data', hardware='auto')
>>> cmdline = 0
>>> kwargs = dict(
>>>     input_kwcoco_fpath=data_dvc_dpath / 'Drop6/imgonly-KR_R001.kwcoco.zip',
>>>     output_kwcoco_fpath=data_dvc_dpath / 'Drop6/test-timeave-KR_R001-cloudmask-no_sensor_separate.kwcoco.zip',
>>>     workers=11,
>>>     merge_method='mean',
>>>     time_window='1 year',
>>>     channels='red|green|blue',
>>>     mask_low_quality=True,
>>>     resolution='10GSD',
>>>     separate_sensors=False,
>>> )
>>> output_coco_dset = main(cmdline=cmdline, **kwargs)
>>> from geowatch.cli import coco_visualize_videos
>>> coco_visualize_videos.main(cmdline=0, src=kwargs['output_kwcoco_fpath'], smart=True)

Example

>>> # 6: Adjust the effect of S2 imagery.
>>> from geowatch.cli.coco_time_combine import *  # NOQA
>>> # xdoctest: +REQUIRES(env:DEVEL_TEST)
>>> import geowatch
>>> data_dvc_dpath = geowatch.find_dvc_dpath(tags='phase2_data', hardware='auto')
>>> cmdline = 0
>>> kwargs = dict(
>>>     input_kwcoco_fpath=data_dvc_dpath / 'Drop6/imganns-KR_R001.kwcoco.zip',
>>>     output_kwcoco_fpath=data_dvc_dpath / 'Drop6/test-timeave-test_6-KR_R001.kwcoco.zip',
>>>     workers=11,
>>>     merge_method='mean',
>>>     time_window='1 year',
>>>     channels='red|green|blue',
>>>     mask_low_quality=True,
>>>     resolution='10GSD',
>>>     separate_sensors=False,
>>>     sensor_weights=dict(S2=2.5, L8=1.0, WV=5.0)
>>> )
>>> output_coco_dset = main(cmdline=cmdline, **kwargs)
>>> from geowatch.cli import coco_visualize_videos
>>> coco_visualize_videos.main(cmdline=0, src=kwargs['output_kwcoco_fpath'], smart=True)

Example

>>> # 7: Tile images instead of computing average all at once.
>>> from geowatch.cli.coco_time_combine import *  # NOQA
>>> # xdoctest: +REQUIRES(env:DEVEL_TEST)
>>> import geowatch
>>> data_dvc_dpath = geowatch.find_dvc_dpath(tags='phase2_data', hardware='auto')
>>> cmdline = 0
>>> kwargs = dict(
>>>     input_kwcoco_fpath=data_dvc_dpath / 'Drop6/imganns-KR_R001.kwcoco.zip',
>>>     output_kwcoco_fpath=data_dvc_dpath / 'Drop6/test-timeave-test_7-KR_R001.kwcoco.zip',
>>>     workers=11,
>>>     merge_method='median',
>>>     time_window='1 year',
>>>     channels='red|green|blue',
>>>     mask_low_quality=True,
>>>     resolution='10GSD',
>>>     separate_sensors=False,
>>>     spatial_tile_size=200,
>>> )
>>> output_coco_dset = main(cmdline=cmdline, **kwargs)
>>> from geowatch.cli import coco_visualize_videos
>>> coco_visualize_videos.main(cmdline=0, src=kwargs['output_kwcoco_fpath'], smart=True)

Example

>>> # 8: Tile images on large kwcoco file.
>>> # xdoctest: +REQUIRES(env:DEVEL_TEST)
>>> from geowatch.cli.coco_time_combine import *  # NOQA
>>> import geowatch
>>> data_dvc_dpath = geowatch.find_dvc_dpath(tags='phase2_data', hardware='auto')
>>> cmdline = 0
>>> kwargs = dict(
>>>     input_kwcoco_fpath=data_dvc_dpath / 'Drop6/data_train_split1.kwcoco.zip',
>>>     output_kwcoco_fpath=data_dvc_dpath / 'Drop6/test-timeave-test_8-train_split1.kwcoco.zip',
>>>     workers=4,
>>>     merge_method='median',
>>>     time_window='1 year',
>>>     channels='red|green|blue',
>>>     mask_low_quality=True,
>>>     resolution='10GSD',
>>>     separate_sensors=False,
>>>     spatial_tile_size=400,
>>>     include_sensors=['S2', 'L8'],
>>> )
>>> output_coco_dset = main(cmdline=cmdline, **kwargs)
>>> from geowatch.cli import coco_visualize_videos
>>> coco_visualize_videos.main(cmdline=0, src=kwargs['output_kwcoco_fpath'], smart=True)

Example

>>> # 9: Exclude winter seasons for time average.
>>> from geowatch.cli.coco_time_combine import *  # NOQA
>>> # xdoctest: +REQUIRES(env:DEVEL_TEST)
>>> import geowatch
>>> data_dvc_dpath = geowatch.find_dvc_dpath(tags='phase2_data', hardware='auto')
>>> cmdline = 0
>>> kwargs = dict(
>>>     input_kwcoco_fpath=data_dvc_dpath / 'Drop6/imganns-KR_R001.kwcoco.zip',
>>>     output_kwcoco_fpath=data_dvc_dpath / 'Drop6/test-timeave-test_9-KR_R001.kwcoco.zip',
>>>     workers=11,
>>>     merge_method='mean',
>>>     time_window='1 year',
>>>     channels='red|green|blue',
>>>     mask_low_quality=True,
>>>     resolution='10GSD',
>>>     separate_sensors=False,
>>>     filter_season='winter',
>>> )
>>> output_coco_dset = main(cmdline=cmdline, **kwargs)
>>> from geowatch.cli import coco_visualize_videos
>>> coco_visualize_videos.main(cmdline=0, src=kwargs['output_kwcoco_fpath'], smart=True)

geowatch.cli.coco_time_combine.combine_kwcoco_channels_temporally(config)[source]¶

Combine spatial data within a temporal window from a kwcoco dataset and save the result to a new kwcoco dataset.

High level steps: 1. Load kwcoco dataset. 2. Divide the dataset into temporal windows. 3. For each temporal window, combine the spatial data from each channel. 4. Save the combined image result to a new kwcoco dataset.

geowatch.cli.coco_time_combine.get_quality_mask(coco_image, space, resolution, avoid_quality_values=None, crop_slice=None, qa_scale=None)[source]¶

Get a binary mask of the quality data.

Parameters:

coco_image (kwcoco.coco_image.CocoImage) – Object that contains references to the image and assets including the quality mask.
space (str) – The space that the quality mask will be loaded in. Choices: ‘image’, ‘video’, ‘asset’
resolution (str | int) – The resolution that the quality mask will be loaded in. E.g. ‘10GSD’.
avoid_quality_values (list | None) – The values to include as bad quality according to the bitmask. Defaults to [‘cloud’, ‘cloud_shadow’, ‘cloud_adjacent’].
crop_slice (Tuple[slice, slice] | None) – The height and width crop slices to load from quality mask. Defaults to None, which loads full quality mask.

Returns:

A binary numpy array of shape [H, W, 1] where the 1 values corresponds to a quality pixel vice versa for 0 values.

Return type:

np.ndarray

geowatch.cli.coco_time_combine.estimate_top_quality_images(window_coco_images, max_images_per_group, avoid_quality_values)[source]¶: Heuristic to subselect top images.

geowatch.cli.coco_time_combine.merge_images(window_coco_images, merge_method, requested_chans, space, resolution, new_bundle_dpath, mask_low_quality, sensor_weights, og_kwcoco_fpath, spatial_tile_size, config)[source]¶

Parameters:: window_coco_images (List[kwcoco.CocoImage]) – images with channels to merge
Returns:: a new coco image that points at the merged image on disk.
Return type:: Dict

geowatch.cli.coco_time_combine.generate_frames(window_coco_images, crop_slice, resolution, space, merge_chans, mask_low_quality, avoid_quality_values)[source]¶

geowatch.cli.coco_time_combine.filter_image_ids_by_season(coco_dset, image_ids, filtered_seasons, ignore_winter_torrid_zone=True)[source]¶

Filter a sequence of image ids by season and geolocation.

Parameters:

coco_dset (kwcoco.CocoDataset) – A KWCOCO dataset object.
image_ids (List(int)) – A list of image ids that belong in coco_dset.
filtered_seasons (str | List(str) | None) – Which seasons to not include in the returned image ids.
ignore_winter_torrid_zone (bool, optional) – Do not filter images within the Torrid region when winter is one of the filtered seasons. Defaults to True.

Raises:

ValueError – Check if the filtered seasons varible is a correctable type.
ValueError – Check if one of the filtered seasons is not a valid season.

Returns:

A list of filtered image ids. Should never be longer than the input: image ids.

Return type:

List[int]

Example

>>> # 0: Baseline run.
>>> # xdoctest: +REQUIRES(env:DEVEL_TEST)
>>> from geowatch.cli.coco_time_combine import *  # NOQA
>>> import geowatch
>>> data_dvc_dpath = geowatch.find_dvc_dpath(tags='phase2_data', hardware='auto')
>>> # Load KWCOCO dataset.
>>> input_kwcoco_fpath = data_dvc_dpath / 'Drop6/imgonly-AE_C001.kwcoco.json'
>>> coco_dset = kwcoco.CocoDataset(input_kwcoco_fpath)
>>> image_ids = coco_dset.images().gids
>>> ignore_torrid_regions_gids = filter_image_ids_by_season(coco_dset,
>>>                                image_ids,
>>>                                filtered_seasons='winter',
>>>                                ignore_winter_torrid_zone=True)
>>> all_filtered_gids = filter_image_ids_by_season(coco_dset,
>>>                       image_ids,
>>>                       filtered_seasons='winter',
>>>                       ignore_winter_torrid_zone=True)
>>> assert len(all_filtered_gids) > len(ignore_torrid_regions_gids)