geowatch.tasks.cold.prepare_kwcoco module

This is step 1 / 4 in predict.py

SeeAlso:

predict.py

prepare_kwcoco.py *

tile_processing_kwcoco.py

export_cold_result_kwcoco.py

assemble_cold_result_kwcoco.py

This is a proof-of-concept for converting kwcoco files into the expected data structures for pycold.

Relevant functions:

• grab_demo_kwcoco_dataset - downloads a small kwcoco dataset for testing

• stack_kwcoco - runs the stacking process on an entire kwcoco file

• process_one_coco_image - runs the stacking for a single coco image.

Limitations:

• Currently only handles Landsat-8

• The quality bands are not exactly what I was expecting them to be, some of the quality filtering is stubbed out or disabled.

• Not setup for an HPC environment yet, but that extension shouldn’t be too hard.

• Nodata values are currently not masked or handled

• Configurations are hard-coded

Todo

• [ ] Incorporate geowatch/tasks/fusion/datamodules/qa_bands.py

class geowatch.tasks.cold.prepare_kwcoco.PrepareKwcocoConfig(*args, **kwargs)

Bases: DataConfig

The docstring will be the description in the CLI help

Valid options: []

Parameters:

• *args – positional arguments for this data config

• **kwargs – keyword arguments for this data config

default = {'adj_cloud': <Value(False)>, 'coco_fpath': <Value(None)>, 'method': <Value(None)>, 'out_dpath': <Value(None)>, 'resolution': <Value('30GSD')>, 'resolution_PD': <Value('3GSD')>, 'sensors': <Value('L8')>, 'workers': <Value(0)>}

geowatch.tasks.cold.prepare_kwcoco.prepare_kwcoco_main(cmdline=1, **kwargs)

_summary_

Parameters:

cmdline (int, optional) – _description_. Defaults to 1.

Example

>>> # xdoctest: +REQUIRES(env:TEST_COLD)
>>> from geowatch.tasks.cold.prepare_kwcoco import prepare_kwcoco_main
>>> from geowatch.tasks.cold.prepare_kwcoco import *
>>> kwargs= dict(
>>>   coco_fpath = ub.Path('/home/jws18003/data/dvc-repos/smart_data_dvc/Drop6/data_vali_split1_KR_R001.kwcoco.json'),
>>>   out_dpath = ub.Path.appdir('/gpfs/scratchfs1/zhz18039/jws18003/kwcoco'),
>>>   sensors = 'L8,S2',
>>>   adj_cloud = False,
>>>   method = None,
>>> )
>>> cmdline=0
>>> prepare_kwcoco_main(cmdline, **kwargs)

geowatch.tasks.cold.prepare_kwcoco.qa_decoding(qa_array)

This function is modified from qabitval_array_HLS function (https://github.com/GERSL/pycold/blob/c5b380eccc2916e5c3aec0bbd2b1982e114b75b1/src/python/pycold/imagetool/prepare_ard.py#L74)

geowatch.tasks.cold.prepare_kwcoco.qa_decoding_no_boundary(qa_array)

This function is modified from qabitval_array_HLS function (https://github.com/GERSL/pycold/blob/c5b380eccc2916e5c3aec0bbd2b1982e114b75b1/src/python/pycold/imagetool/prepare_ard.py#L74)

geowatch.tasks.cold.prepare_kwcoco.setup_logging()

geowatch.tasks.cold.prepare_kwcoco.hist_cut(band, mask, fill_value=-9999, k=3, minmax='std')

geowatch.tasks.cold.prepare_kwcoco.minmax_norm(band, mask, fill_value=-9999)

geowatch.tasks.cold.prepare_kwcoco.artificial_surface_index(Blue, Green, Red, NIR, SWIR1, SWIR2, Scale, MaskValid_Obs, fillV)

geowatch.tasks.cold.prepare_kwcoco.stack_kwcoco(coco_fpath, out_dir, sensors, adj_cloud, method, pman=None, workers=0, resolution=None)

Parameters:

• coco_fpath (str | PathLike | CocoDataset) – the kwcoco dataset to convert

• out_dir (str | PathLike) – path to write the data

Returns:

a list of dictionary result objects

Return type:

List[Dict]

Example

>>> # xdoctest: +SKIP
>>> # TODO: readd this doctest
>>> from pycold.imagetool.prepare_kwcoco import *  # NOQA
>>> setup_logging()
>>> coco_dset = geowatch.coerce_kwcoco('geowatch-msi')
>>> coco_fpath = coco_dset.fpath
>>> #coco_fpath = grab_demo_kwcoco_dataset()
>>> dpath = ub.Path.appdir('pycold/tests/stack_kwcoco').ensuredir()
>>> out_dir = dpath / 'stacked'
>>> results = stack_kwcoco(coco_fpath, out_dir)

geowatch.tasks.cold.prepare_kwcoco.stack_kwcoco_PD(coco_fpath, out_dir, sensors, adj_cloud, method, pman=None, workers=0, resolution=None)

Parameters:

• coco_fpath (str | PathLike | CocoDataset) – the kwcoco dataset to convert

• out_dir (str | PathLike) – path to write the data

Returns:

a list of dictionary result objects

Return type:

List[Dict]

Example

>>> # xdoctest: +SKIP
>>> # TODO: readd this doctest
>>> from pycold.imagetool.prepare_kwcoco import *  # NOQA
>>> setup_logging()
>>> coco_dset = geowatch.coerce_kwcoco('geowatch-msi')
>>> coco_fpath = coco_dset.fpath
>>> #coco_fpath = grab_demo_kwcoco_dataset()
>>> dpath = ub.Path.appdir('pycold/tests/stack_kwcoco').ensuredir()
>>> out_dir = dpath / 'stacked'
>>> results = stack_kwcoco(coco_fpath, out_dir)

geowatch.tasks.cold.prepare_kwcoco.process_one_coco_image(coco_image, out_dir, adj_cloud, method, resolution)

Parameters:

• coco_image (kwcoco.CocoImage) – the image to process

• out_dir (Path) – path to write the image data

• resolution (str | None) – resolution to process at (e.g. 30GSD).

Returns:

result dictionary with keys:

status (str) : either a string passed or failed fpaths (List[str]): a list of files that were written

Return type:

Dict