geowatch.mlops.schedule_evaluation module

Helper for scheduling a set of prediction + evaluation jobs.

This is the main entrypoint for running a bunch of evaluation jobs over a grid of parameters. We currently expect that pipelines are predefined in smart_pipeline.py but in the future they will likely be an external resource file.

Todo

• [ ] Differentiate between pixel models for different tasks.

• [ ] Allow the output of tracking to feed into activity classification

• [ ] Rename to “schedule”. The pipeline does not have to be an evaluation.

Example

# Dummy inputs, just for demonstration

python -m geowatch.mlops.schedule_evaluation

–params=”

matrix:

bas_pxl.package_fpath:

• ./my_bas_model1.pt

• ./my_bas_model2.pt

bas_pxl.test_dataset:

• ./my_test_dataset/bas_ready_data.kwcoco.json

bas_pxl.window_space_scale: 15GSD bas_pxl.time_sampling:

• ‘auto’

bas_pxl.input_space_scale:

• ‘15GSD’

bas_poly_eval.true_site_dpath: null bas_poly_eval.true_region_dpath: null bas_poly.moving_window_size: bas_poly.thresh:

• 0.1

• 0.2

sc_pxl.test_dataset:

• crop.dst

sc_pxl.window_space_scale:

• auto

sc_poly.thresh:

• 0.1

sc_poly.use_viterbi:

• 0

sc_pxl.package_fpath:

• my_sc_model1.pt

• my_sc_model2.pt

sc_poly_viz.enabled:

• false

“ –root_dpath=./my_dag_runs –devices=”0,1” –tmux_workers=2 –backend=serial –skip_existing=0 –pipeline=joint_bas_sc –run=0

python -m geowatch.mlops.schedule_evaluation

–params=”

matrix:

bas_pxl.package_fpath:

• ./my_bas_model1.pt

• ./my_bas_model2.pt

bas_pxl.test_dataset:

• ./my_test_dataset/bas_ready_data.kwcoco.json

bas_pxl.window_space_scale: 15GSD bas_pxl.time_sampling:

• ‘auto’

bas_pxl.input_space_scale:

• ‘15GSD’

bas_poly.moving_window_size: bas_poly.thresh:

• 0.1

• 0.2

bas_pxl.enabled: 1 bas_poly_eval.true_site_dpath: true-site bas_poly_eval.true_region_dpath: true-region

“ –root_dpath=./my_dag_runs –devices=”0,1” –backend=serial –skip_existing=0 –pipeline=bas –run=0

# Real inputs, this actually will run something given the DVC repos DVC_DATA_DPATH=$(geowatch_dvc –tags=’phase2_data’ –hardware=auto) DVC_EXPT_DPATH=$(geowatch_dvc –tags=’phase2_expt’ –hardware=auto)

SC_MODEL=$DVC_EXPT_DPATH/models/fusion/Drop4-SC/packages/Drop4_tune_V30_8GSD_V3/Drop4_tune_V30_8GSD_V3_epoch=2-step=17334.pt.pt BAS_MODEL=$DVC_EXPT_DPATH/models/fusion/Drop4-BAS/packages/Drop4_TuneV323_BAS_30GSD_BGRNSH_V2/package_epoch0_step41.pt.pt

python -m geowatch.mlops.schedule_evaluation

–params=”

matrix:

bas_pxl.package_fpath:

• $BAS_MODEL

bas_pxl.test_dataset:

• $DVC_DATA_DPATH/Drop4-BAS/KR_R001.kwcoco.json

bas_pxl.window_space_scale: 15GSD bas_pxl.time_sampling:

• “auto”

bas_pxl.input_space_scale:

• “15GSD”

bas_poly.moving_window_size: bas_poly.thresh:

• 0.1

sc_pxl.test_dataset:

• crop.dst

sc_pxl.window_space_scale:

• auto

sc_poly.thresh:

• 0.1

sc_poly.use_viterbi:

• 0

sc_pxl.package_fpath:

• $SC_MODEL

sc_poly_viz.enabled:

• false

“ –root_dpath=./my_dag_runs –devices=”0,1” –queue_size=2 –backend=serial –skip_existing=0 –pipeline=joint_bas_sc_nocrop –run=0

Example

# Real data DVC_DATA_DPATH=$(geowatch_dvc –tags=’phase2_data’ –hardware=auto) DVC_EXPT_DPATH=$(geowatch_dvc –tags=’phase2_expt’ –hardware=auto)

python -m geowatch.mlops.schedule_evaluation

–params=”

matrix:

bas_pxl.package_fpath:

# - $DVC_EXPT_DPATH/models/fusion/Drop4-BAS/packages/Drop4_TuneV323_BAS_30GSD_BGRNSH_V2/package_epoch0_step41.pt.pt - $DVC_EXPT_DPATH/package_epoch10_step200000.pt

bas_pxl.test_dataset:

• $DVC_DATA_DPATH/Drop4-BAS/KR_R001.kwcoco.json

# - $DVC_DATA_DPATH/Drop4-BAS/KR_R002.kwcoco.json

bas_pxl.window_space_scale:

• auto

# - “15GSD” # - “30GSD”

# bas_pxl.chip_dims: # - “256,256” bas_pxl.time_sampling:

• “auto”

# bas_pxl.input_space_scale: # - “window” bas_poly.moving_window_size:

• null

# - 100 # - 200

bas_poly.thresh:

• 0.1

# - 0.13 # - 0.2

sc_pxl.window_space_scale:

• auto

sc_pxl.input_space_scale:

• “window”

sc_pxl.chip_dims:

• “256,256”

sc_poly.thresh:

• 0.1

sc_poly.use_viterbi:

• 0

sc_pxl.package_fpath:

• $DVC_EXPT_DPATH/models/fusion/Drop4-SC/packages/Drop4_tune_V30_8GSD_V3/Drop4_tune_V30_8GSD_V3_epoch=2-step=17334.pt.pt

bas_poly_eval.enabled: 1 bas_pxl_eval.enabled: 1 bas_poly_viz.enabled: 1 sc_poly_eval.enabled: 1 sc_pxl_eval.enabled: 1 sc_poly_viz.enabled: 1

“ –root_dpath=$DVC_EXPT_DPATH/_testpipe –enable_links=1 –devices=”0,1” –queue_size=2 –backend=serial –pipeline=bas –cache=1 –rprint=1 –run=1

–pipeline=joint_bas_sc

xdev tree –dirblocklist “_*” my_expt_dir/_testpipe/ –max_files=1

class geowatch.mlops.schedule_evaluation.ScheduleEvaluationConfig(*args, **kwargs)

Bases: CMDQueueConfig

Driver for GeoWATCH mlops evaluation scheduling

Builds commands and optionally executes them via slurm, tmux, or serial (i.e. one at a time). This is a [link=https://gitlab.kitware.com/computer-vision/cmd_queue]cmd_queue[/link] CLI.

Valid options: []

Parameters:

• *args – positional arguments for this data config

• **kwargs – keyword arguments for this data config

default = {'backend': <Value('tmux')>, 'cache': <Value(True)>, 'devices': <Value(None)>, 'draw_curves': <Value(1)>, 'draw_heatmaps': <Value(1)>, 'enable_links': <Value(True)>, 'max_configs': <Value(None)>, 'other_session_handler': <Value('ask')>, 'params': <Value(None)>, 'pipeline': <Value('joint_bas_sc')>, 'pred_workers': <Value(4)>, 'print_commands': <Value('auto')>, 'print_queue': <Value('auto')>, 'print_varied': <Value('auto')>, 'queue_name': <Value(None)>, 'queue_size': <Value(None)>, 'root_dpath': <Value('auto')>, 'run': <Value(False)>, 'skip_existing': <Value(False)>, 'slurm_options': <Value(None)>, 'tmux_workers': <Value(8)>, 'virtualenv_cmd': <Value(None)>, 'with_textual': <Value('auto')>}

main(**kwargs)

normalize()

geowatch.mlops.schedule_evaluation.main(cmdline=True, **kwargs)

geowatch.mlops.schedule_evaluation.schedule_evaluation(config)

First ensure that models have been copied to the DVC repo in the appropriate path. (as noted by model_dpath)

geowatch.mlops.schedule_evaluation.ensure_iterable(inputs)