Leaderboards

Contents

Leaderboards#

Task 1#

	nrmse	lambda_x_isotrop	lambda_x_spacetime	lambda_t_spacetime	method	task
0	0.927	88.18 km	138.64 km	10.43 days	bfn	DC2020 OSSE Gulfstream Nadir
1	0.942	64.59 km	116.63 km	7.74 days	4dvarnet	DC2020 OSSE Gulfstream Nadir
2	0.915	122.62 km	174.49 km	10.77 days	duacs	DC2020 OSSE Gulfstream Nadir
3	0.911	118.61 km	156.46 km	11.50 days	dymost	DC2020 OSSE Gulfstream Nadir
4	0.927	99.85 km	157.37 km	10.07 days	miost	DC2020 OSSE Gulfstream Nadir
5	0.932	91.73 km	152.92 km	11.37 days	nerf_ffn	DC2020 OSSE Gulfstream Nadir
6	0.92	90.76 km	164.83 km	8.15 days	nerf_mlp	DC2020 OSSE Gulfstream Nadir
7	0.914	97.77 km	156.76 km	13.07 days	nerf_siren	DC2020 OSSE Gulfstream Nadir

Task 2#

	nrmse	lambda_x_isotrop	lambda_x_spacetime	lambda_t_spacetime	method	task
0	0.935	64.45 km	118.11 km	35.95 days	bfn	DC2020 OSSE Gulfstream Nadir + Swot
1	0.959	47.00 km	77.35 km	5.60 days	4dvarnet	DC2020 OSSE Gulfstream Nadir + Swot
2	0.921	105.68 km	138.61 km	11.65 days	duacs	DC2020 OSSE Gulfstream Nadir + Swot
3	0.926	107.75 km	135.28 km	9.93 days	dymost	DC2020 OSSE Gulfstream Nadir + Swot
4	0.938	87.64 km	131.21 km	10.13 days	miost	DC2020 OSSE Gulfstream Nadir + Swot
5	0.937	90.45 km	140.77 km	8.18 days	nerf_ffn	DC2020 OSSE Gulfstream Nadir + Swot
6	0.936	87.61 km	153.43 km	9.36 days	nerf_mlp	DC2020 OSSE Gulfstream Nadir + Swot
7	0.934	87.47 km	136.72 km	8.35 days	nerf_siren	DC2020 OSSE Gulfstream Nadir + Swot

Task 3#

	nrmse	lambda_x_isotrop	lambda_x_spacetime	lambda_t_spacetime	method	task
0	0.946	45.70 km	137.66 km	4.14 days	musti	OSSE Gulfstream Nadir + SST
1	0.959	45.62 km	87.32 km	3.72 days	4dvarnet	OSSE Gulfstream Nadir + SST

Task 4#

	nrmse	lambda_x_alongtrack	method	task
0	0.884	121.65 km	bfn	DC2021 OSE Gulfstream
1	0.909	97.90 km	4dvarnet	DC2021 OSE Gulfstream
2	0.894	113.41 km	convlstm	DC2021 OSE Gulfstream
3	0.902	107.54 km	convlstm_sst	DC2021 OSE Gulfstream
4	0.903	112.09 km	musti	DC2021 OSE Gulfstream
5	0.882	150.91 km	duacs	DC2021 OSE Gulfstream
6	0.898	130.60 km	dymost	DC2021 OSE Gulfstream
7	0.896	135.33 km	miost	DC2021 OSE Gulfstream

from pathlib import Path
if Path('.').absolute().name == 'notebooks':
    %cd ..

/raid/localscratch/qfebvre/oceanbench

%env LOGURU_LEVEL=WARNING
import xarray as xr
import hydra

env: LOGURU_LEVEL=WARNING

import yaml
import inspect
from IPython.display import Markdown, display
from omegaconf import OmegaConf

def pprint_cfg(cfg):
    display(Markdown("""```yaml\n\n""" +yaml.dump(OmegaConf.to_container(cfg), default_flow_style=None, indent=2)+"""\n\n```"""))

def task_cfg(task_name):
    with hydra.initialize('../config', version_base='1.3'):
        cfg_task = hydra.compose(f'task/{task_name}/task').task.outputs
    pprint_cfg(cfg_task)

Task 1: OSSE GF Nadirs#

task_cfg('osse_gf_nadir')

data: {obs: '${...data.outputs.nadir_gridded}', ssh: '${...data.outputs.ssh}'}
domain:
  lat: [33, 43]
  lon: [-65, -55]
name: DC2020 OSSE Gulfstream Nadir
splits:
  test: ['2012-10-22', '2012-12-02']
  trainval: ['2013-01-01', '2013-09-30']

%run oceanbench/__main__.py -m task=osse_gf_nadir/leaderboard

[2023-10-09 12:26:08,098][HYDRA] Launching 8 jobs locally
[2023-10-09 12:26:08,099][HYDRA] 	#0 : method=bfn task=osse_gf_nadir/leaderboard
[2023-10-09 12:26:16,550][HYDRA] 	#1 : method=4dvarnet task=osse_gf_nadir/leaderboard
[2023-10-09 12:26:23,625][HYDRA] 	#2 : method=duacs task=osse_gf_nadir/leaderboard
[2023-10-09 12:26:31,505][HYDRA] 	#3 : method=dymost task=osse_gf_nadir/leaderboard
[2023-10-09 12:26:38,834][HYDRA] 	#4 : method=miost task=osse_gf_nadir/leaderboard
[2023-10-09 12:26:45,435][HYDRA] 	#5 : method=nerf_ffn task=osse_gf_nadir/leaderboard
[2023-10-09 12:26:53,715][HYDRA] 	#6 : method=nerf_mlp task=osse_gf_nadir/leaderboard
[2023-10-09 12:27:03,211][HYDRA] 	#7 : method=nerf_siren task=osse_gf_nadir/leaderboard
|    |   nrmse | lambda_x_isotrop   | lambda_x_spacetime   | lambda_t_spacetime   | method     | task                         |
|---:|--------:|:-------------------|:---------------------|:---------------------|:-----------|:-----------------------------|
|  0 |   0.927 | 88.18 km           | 138.64 km            | 10.43 days           | bfn        | DC2020 OSSE Gulfstream Nadir |
|  1 |   0.944 | 64.80 km           | 97.51 km             | 7.84 days            | 4dvarnet   | DC2020 OSSE Gulfstream Nadir |
|  2 |   0.915 | 122.62 km          | 174.49 km            | 10.77 days           | duacs      | DC2020 OSSE Gulfstream Nadir |
|  3 |   0.911 | 118.61 km          | 156.46 km            | 11.50 days           | dymost     | DC2020 OSSE Gulfstream Nadir |
|  4 |   0.927 | 99.85 km           | 157.37 km            | 10.07 days           | miost      | DC2020 OSSE Gulfstream Nadir |
|  5 |   0.932 | 91.73 km           | 152.92 km            | 11.37 days           | nerf_ffn   | DC2020 OSSE Gulfstream Nadir |
|  6 |   0.92  | 90.76 km           | 164.83 km            | 8.15 days            | nerf_mlp   | DC2020 OSSE Gulfstream Nadir |
|  7 |   0.914 | 97.77 km           | 156.76 km            | 13.07 days           | nerf_siren | DC2020 OSSE Gulfstream Nadir |

Task 2: OSSE GF Nadirs + SWOT#

task_cfg('osse_gf_nadirswot')

data: {obs: '${...data.outputs.swot_gridded}', ssh: '${...data.outputs.ssh}'}
domain:
  lat: [33, 43]
  lon: [-65, -55]
name: DC2020 OSSE Gulfstream Nadir + Swot
splits:
  test: ['2012-10-22', '2012-12-02']
  trainval: ['2013-01-01', '2013-09-30']

%run oceanbench/__main__.py -m task=osse_gf_nadirswot/leaderboard

[2023-10-09 11:55:26,012][HYDRA] Launching 8 jobs locally
[2023-10-09 11:55:26,012][HYDRA] 	#0 : method=bfn task=osse_gf_nadirswot/leaderboard
[2023-10-09 11:55:34,297][HYDRA] 	#1 : method=4dvarnet task=osse_gf_nadirswot/leaderboard
[2023-10-09 11:55:41,771][HYDRA] 	#2 : method=duacs task=osse_gf_nadirswot/leaderboard
[2023-10-09 11:55:48,604][HYDRA] 	#3 : method=dymost task=osse_gf_nadirswot/leaderboard
[2023-10-09 11:55:55,942][HYDRA] 	#4 : method=miost task=osse_gf_nadirswot/leaderboard
[2023-10-09 11:56:03,289][HYDRA] 	#5 : method=nerf_ffn task=osse_gf_nadirswot/leaderboard
[2023-10-09 11:56:10,677][HYDRA] 	#6 : method=nerf_mlp task=osse_gf_nadirswot/leaderboard
[2023-10-09 11:56:20,857][HYDRA] 	#7 : method=nerf_siren task=osse_gf_nadirswot/leaderboard
|    |   nrmse | lambda_x_isotrop   | lambda_x_spacetime   | lambda_t_spacetime   | method     | task                                |
|---:|--------:|:-------------------|:---------------------|:---------------------|:-----------|:------------------------------------|
|  0 |   0.935 | 64.45 km           | 118.11 km            | 35.95 days           | bfn        | DC2020 OSSE Gulfstream Nadir + Swot |
|  1 |   0.959 | 47.00 km           | 77.35 km             | 5.60 days            | 4dvarnet   | DC2020 OSSE Gulfstream Nadir + Swot |
|  2 |   0.921 | 105.68 km          | 138.61 km            | 11.65 days           | duacs      | DC2020 OSSE Gulfstream Nadir + Swot |
|  3 |   0.926 | 107.75 km          | 135.28 km            | 9.93 days            | dymost     | DC2020 OSSE Gulfstream Nadir + Swot |
|  4 |   0.938 | 87.64 km           | 131.21 km            | 10.13 days           | miost      | DC2020 OSSE Gulfstream Nadir + Swot |
|  5 |   0.937 | 90.45 km           | 140.77 km            | 8.18 days            | nerf_ffn   | DC2020 OSSE Gulfstream Nadir + Swot |
|  6 |   0.936 | 87.61 km           | 153.43 km            | 9.36 days            | nerf_mlp   | DC2020 OSSE Gulfstream Nadir + Swot |
|  7 |   0.934 | 87.47 km           | 136.72 km            | 8.35 days            | nerf_siren | DC2020 OSSE Gulfstream Nadir + Swot |

Task 3: OSSE GF: Nadir + SST#

task_cfg('osse_gf_nadir_sst')

data: {obs: '${...data.outputs.nadir_gridded}', ssh: '${...data.outputs.ssh}', sst: '${...data.outputs.sst}'}
domain:
  lat: [33, 43]
  lon: [-65, -55]
name: OSSE Gulfstream Nadir + SST
splits:
  test: ['2012-10-22', '2012-12-02']
  trainval: ['2013-01-01', '2013-09-30']

%run oceanbench/__main__.py -m task=osse_gf_nadir_sst/leaderboard

[2023-10-09 11:56:38,993][HYDRA] Launching 2 jobs locally
[2023-10-09 11:56:38,994][HYDRA] 	#0 : method=musti task=osse_gf_nadir_sst/leaderboard
[2023-10-09 11:56:45,548][HYDRA] 	#1 : method=4dvarnet task=osse_gf_nadir_sst/leaderboard
|    |   nrmse | lambda_x_isotrop   | lambda_x_spacetime   | lambda_t_spacetime   | method   | task                        |
|---:|--------:|:-------------------|:---------------------|:---------------------|:---------|:----------------------------|
|  0 |   0.946 | 45.70 km           | 137.66 km            | 4.14 days            | musti    | OSSE Gulfstream Nadir + SST |
|  1 |   0.959 | 45.62 km           | 87.32 km             | 3.72 days            | 4dvarnet | OSSE Gulfstream Nadir + SST |

Task 4: OSE GF#

task_cfg('ose_gf')

data:
  test: {c2: '${....data.outputs.c2}'}
  train: {alg: '${....data.outputs.alg}', h2g: '${....data.outputs.h2g}', j2g: '${....data.outputs.j2g}',
    j2n: '${....data.outputs.j2n}', j3: '${....data.outputs.j3}', s3a: '${....data.outputs.s3a}'}
domain:
  lat: [33, 43]
  lon: [-65, -55]
name: DC2021 OSE Gulfstream
splits:
  test: ['2017-01-01', '2017-12-31']
  train: ['2016-12-01', '2018-01-31']

%run oceanbench/__main__.py -m task=ose_gf/leaderboard

[2023-10-09 12:18:24,644][HYDRA] Launching 8 jobs locally
[2023-10-09 12:18:24,644][HYDRA] 	#0 : method=bfn task=ose_gf/leaderboard
[2023-10-09 12:18:44,043][HYDRA] 	#1 : method=4dvarnet task=ose_gf/leaderboard
[2023-10-09 12:18:52,652][HYDRA] 	#2 : method=convlstm task=ose_gf/leaderboard
[2023-10-09 12:18:58,752][HYDRA] 	#3 : method=convlstm_sst task=ose_gf/leaderboard
[2023-10-09 12:19:03,596][HYDRA] 	#4 : method=musti task=ose_gf/leaderboard
[2023-10-09 12:19:11,127][HYDRA] 	#5 : method=duacs task=ose_gf/leaderboard
[2023-10-09 12:19:15,762][HYDRA] 	#6 : method=dymost task=ose_gf/leaderboard
[2023-10-09 12:19:20,311][HYDRA] 	#7 : method=miost task=ose_gf/leaderboard
|    |   nrmse | lambda_x_alongtrack   | method       | task                  |
|---:|--------:|:----------------------|:-------------|:----------------------|
|  0 |   0.884 | 121.65 km             | bfn          | DC2021 OSE Gulfstream |
|  1 |   0.909 | 97.90 km              | 4dvarnet     | DC2021 OSE Gulfstream |
|  2 |   0.894 | 113.41 km             | convlstm     | DC2021 OSE Gulfstream |
|  3 |   0.902 | 107.54 km             | convlstm_sst | DC2021 OSE Gulfstream |
|  4 |   0.903 | 112.09 km             | musti        | DC2021 OSE Gulfstream |
|  5 |   0.882 | 150.91 km             | duacs        | DC2021 OSE Gulfstream |
|  6 |   0.898 | 130.60 km             | dymost       | DC2021 OSE Gulfstream |
|  7 |   0.896 | 135.33 km             | miost        | DC2021 OSE Gulfstream |