geowatch.cli.queue_cli.prepare_splits module¶

Todo

move to queue_cli

CommandLine

xdoctest -m geowatch.cli.queue_cli.prepare_splits __doc__

Example

>>> from geowatch.cli.queue_cli.prepare_splits import *  # NOQA
>>> dpath = ub.Path.appdir('geowatch', 'tests', 'prep_splits').ensuredir()
>>> (dpath / 'KR_R001.kwcoco.zip').touch()
>>> (dpath / 'KR_R002.kwcoco.zip').touch()
>>> (dpath / 'BR_R002.kwcoco.zip').touch()
>>> config = {
>>>     'base_fpath': dpath / '*.kwcoco.zip',
>>>     'virtualenv_cmd': 'conda activate geowatch',
>>>     'constructive_mode': True,
>>>     'run': 0,
>>>     'cache': False,
>>>     'backend': 'serial',
>>>     'splits': 'split6',
>>>     'verbose': 0,
>>> }
>>> queue = prep_splits(cmdline=False, **config)
>>> config['backend'] = 'slurm'
>>> queue = prep_splits(cmdline=False, **config)
>>> queue.print_commands()
>>> config['backend'] = 'tmux'
>>> queue = prep_splits(cmdline=False, **config)
>>> queue.print_commands()
>>> config['backend'] = 'serial'
>>> queue = prep_splits(cmdline=False, **config)
>>> queue.print_commands()

CommandLine

DVC_DATA_DPATH=$(geowatch_dvc --tags=phase2_data --hardware="hdd")
python -m geowatch.cli.queue_cli.prepare_splits         --src_kwcocos="$DVC_DATA_DPATH"/Drop7-MedianNoWinter10GSD-NoMask/*/imganns-*.kwcoco.zip         --dst_dpath "$DVC_DATA_DPATH"/Drop7-MedianNoWinter10GSD-NoMask         --suffix=rawbands         --backend=tmux --tmux_workers=6         --splits=split6         --run=0

class geowatch.cli.queue_cli.prepare_splits.PrepareSplitsConfig(*args, **kwargs)[source]¶

Bases: DataConfig

This generates the bash commands necessary to split a base kwcoco file into the standard train / validation splits.

Valid options: []

Parameters:

*args – positional arguments for this data config
**kwargs – keyword arguments for this data config

default = {'add_detail_suffix': <Value(True)>, 'backend': <Value('tmux')>, 'cache': <Value(0)>, 'constructive_mode': <Value(True)>, 'dst_dpath': <Value(None)>, 'other_session_handler': <Value('ask')>, 'run': <Value(True)>, 'splits': <Value('*')>, 'src_kwcocos': <Value(None)>, 'suffix': <Value('')>, 'verbose': <Value(1)>, 'virtualenv_cmd': <Value(None)>, 'with_textual': <Value('auto')>, 'workers': <Value(2)>}¶

geowatch.cli.queue_cli.prepare_splits.prep_splits(cmdline=False, **kwargs)[source]¶

The idea is that we should have a lightweight scheduler. I think something fairly minimal can be implemented with tmux, but it would be nice to have a more robust slurm extension.

Todo

[ ] Option to just dump the serial bash script that does everything.

geowatch.cli.queue_cli.prepare_splits.main(cmdline=False, **kwargs)¶

The idea is that we should have a lightweight scheduler. I think something fairly minimal can be implemented with tmux, but it would be nice to have a more robust slurm extension.

Todo

[ ] Option to just dump the serial bash script that does everything.