geowatch.cli.queue_cli.prepare_teamfeats module¶

The following example simply produces the script under different variations.

CommandLine

xdoctest -m geowatch.cli.queue_cli.prepare_teamfeats __doc__

SeeAlso:

../tasks/invariants/predict.py ../tasks/landcover/predict.py ../tasks/depth/predict.py ../tasks/cold/predict.py

~/code/watch/dev/poc/prepare_time_combined_dataset.py

Example

>>> from geowatch.cli.queue_cli.prepare_teamfeats import *  # NOQA
>>> config = {
>>>     'src_kwcocos': './pretend_bundle/data.kwcoco.json',
>>>     'gres': [0, 1],
>>>     'expt_dvc_dpath': './pretend_expt_dvc',
>>> #
>>>     'virtualenv_cmd': 'conda activate geowatch',
>>> #
>>>     #'with_s2_landcover': 1,
>>>     #'with_materials': 1,
>>>     #'with_invariants2': 1,
>>>     'with_mae': 1,
>>> #
>>>     'run': 0,
>>>     'check': False,
>>>     'skip_existing': False,
>>>     'backend': 'serial',
>>> }
>>> config['backend'] = 'slurm'
>>> outputs = prep_feats(cmdline=False, **config)
>>> outputs['queue'].print_commands(0, 0)
>>> config['backend'] = 'tmux'
>>> outputs = prep_feats(cmdline=False, **config)
>>> outputs['queue'].print_commands(0, 0)
>>> config['backend'] = 'serial'
>>> outputs = prep_feats(cmdline=False, **config)
>>> outputs['queue'].print_commands(0, 0)

Example

>>> # Test landcover commands
>>> from geowatch.cli.queue_cli.prepare_teamfeats import *  # NOQA
>>> config = {
>>>     'src_kwcocos': './PRETEND_BUNDLE/data.kwcoco.json',
>>>     'gres': [0, 1],
>>>     'expt_dvc_dpath': './PRETEND_EXPT_DVC',
>>>     'virtualenv_cmd': 'conda activate geowatch',
>>>     'with_s2_landcover': 1,
>>>     'with_wv_landcover': 1,
>>>     'num_wv_landcover_hidden': 0,
>>>     'num_s2_landcover_hidden': 0,
>>>     'run': 0,
>>>     'check': False,
>>>     'skip_existing': False,
>>>     'backend': 'serial',
>>> }
>>> config['backend'] = 'serial'
>>> outputs = prep_feats(cmdline=False, **config)
>>> outputs['queue'].print_commands(0, 0)
>>> output_paths = outputs['final_output_paths']
>>> print('output_paths = {}'.format(ub.urepr(output_paths, nl=1)))

Example

>>> # Test COLD commands
>>> from geowatch.cli.queue_cli.prepare_teamfeats import *  # NOQA
>>> cold_config = ub.codeblock(
    '''
    enabled: 1
    conse: 42
    sensors: 'L8,S2,WV'
    resolution: 3GSD
    ''')
>>> config = {
>>>     'src_kwcocos': './PRETEND_BUNDLE/data.kwcoco.json',
>>>     'gres': [0, 1],
>>>     'expt_dvc_dpath': './PRETEND_EXPT_DVC',
>>>     'virtualenv_cmd': 'conda activate geowatch',
>>>     'cold_config': cold_config,
>>>     'run': 0,
>>>     'check': False,
>>>     'skip_existing': False,
>>>     'backend': 'serial',
>>> }
>>> config['backend'] = 'serial'
>>> outputs = prep_feats(cmdline=False, **config)
>>> outputs['queue'].print_commands(0, 0)
>>> output_paths = outputs['final_output_paths']
>>> print('output_paths = {}'.format(ub.urepr(output_paths, nl=1)))

Example

>>> # Test COLD commands
>>> from geowatch.cli.queue_cli.prepare_teamfeats import *  # NOQA
>>> cold_config = None
>>> config = {
>>>     'src_kwcocos': './PRETEND_BUNDLE/data.kwcoco.json',
>>>     'gres': [0, 1],
>>>     'expt_dvc_dpath': './PRETEND_EXPT_DVC',
>>>     'virtualenv_cmd': 'conda activate geowatch',
>>>     'cold_config': cold_config,
>>>     'with_cold': cold_config,
>>>     'run': 0,
>>>     'check': False,
>>>     'skip_existing': False,
>>>     'backend': 'serial',
>>> }
>>> config['backend'] = 'serial'
>>> outputs = prep_feats(cmdline=False, **config)
>>> outputs['queue'].print_commands(0, 0)
>>> output_paths = outputs['final_output_paths']
>>> print('output_paths = {}'.format(ub.urepr(output_paths, nl=1)))

class geowatch.cli.queue_cli.prepare_teamfeats.TeamFeaturePipelineConfig(*args, **kwargs)[source]¶

Bases: CMDQueueConfig

This generates the bash commands necessary to run team feature computation, followed by aggregation and then splitting out train / val datasets.

Note

The models and parameters to use are hard coded in this script.

Todo

[ ] jsonargparse use-case: specifying parmeters of the subalgos

Valid options: []

Parameters:

*args – positional arguments for this data config
**kwargs – keyword arguments for this data config

default = {'assets_dname': <Value('_teamfeats')>, 'backend': <Value('tmux')>, 'check': <Value(True)>, 'cold_config': <Value(None)>, 'cold_workermode': <Value('process')>, 'cold_workers': <Value(2)>, 'data_workers': <Value(2)>, 'depth_workers': <Value(2)>, 'expt_dvc_dpath': <Value('auto')>, 'gres': <Value('auto')>, 'invariant_pca': <Value(0)>, 'invariant_resolution': <Value('10GSD')>, 'invariant_segmentation': <Value(False)>, 'kwcoco_ext': <Value('.kwcoco.zip')>, 'num_s2_landcover_hidden': 32, 'num_wv_landcover_hidden': 32, 'other_session_handler': <Value('ask')>, 'print_commands': <Value('auto')>, 'print_queue': <Value('auto')>, 'queue_name': <Value(None)>, 'run': <Value(False)>, 'skip_existing': <Value(True)>, 'slurm_options': <Value(None)>, 'src_kwcocos': <Value(None)>, 'tmux_workers': <Value(8)>, 'virtualenv_cmd': <Value(None)>, 'with_cold': <Value(False)>, 'with_depth': <Value(False)>, 'with_invariants2': <Value(False)>, 'with_mae': <Value(False)>, 'with_materials': <Value(False)>, 'with_s2_landcover': <Value(False)>, 'with_sam': <Value(False)>, 'with_textual': <Value('auto')>, 'with_wv_landcover': <Value(False)>}¶

geowatch.cli.queue_cli.prepare_teamfeats.prep_feats(cmdline=True, **kwargs)[source]¶

The idea is that we should have a lightweight scheduler. I think something fairly minimal can be implemented with tmux, but it would be nice to have a more robust slurm extension.

Todo

[ ] Option to just dump the serial bash script that does everything.

geowatch.cli.queue_cli.prepare_teamfeats.main(cmdline=True, **kwargs)¶

The idea is that we should have a lightweight scheduler. I think something fairly minimal can be implemented with tmux, but it would be nice to have a more robust slurm extension.

Todo

[ ] Option to just dump the serial bash script that does everything.