Skip to content

AVG_CELLS result differs from xarray .mean() on ERA5 subset #8

Description

@jayendra13

Summary

AVG_CELLS on an ERA5 spatial+temporal subset returns a slightly different value than the equivalent xarray .mean(skipna=True) computation on the same data.

arrDB:   290.4389 (float64 accumulation)
xarray:  290.4131 (float32)

Difference: ~0.026 K

Query

SELECT AVG_CELLS("2m_temperature"[latitude(33.25:68.0), longitude(215.25:256.0), time('2021-06-20':'2021-07-03')]) FROM era5;

Equivalent xarray:

ds["2m_temperature"].sel(
    latitude=slice(68.0, 33.25),  # descending
    longitude=slice(215.25, 256.0),
    time=slice("2021-06-20", "2021-07-03"),
).mean(skipna=True).compute()

Data source

ERA5 ARCO: gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3
Variable: 2m_temperature
Case: EWB Case 1 — 2021 Pacific Northwest Heat Wave

Possible causes (to investigate)

  1. Boundary inclusion difference — arrDB's lookup_range_f64 (inclusive both ends, snap inward) may select a different number of lat/lon/time gridpoints than xarray's slice. Need to compare actual shapes (arrDB EXPLAIN: 140 lat × 164 lon × 313 time).

  2. Float accumulation order — arrDB accumulates f32→f64 per chunk, then merges partial f64 sums across 313 chunks. xarray/numpy may accumulate in f32 or use a different reduction order. Test: subset.astype("float64").mean(skipna=True) in xarray should get closer to arrDB.

  3. Chunk boundary padding — arrDB's ScanOp.trim_boundary_padding() trims NaN fill values from the last chunk along each dimension. If this includes/excludes cells differently from zarrs, the cell count changes.

Verification steps

  • Compare shapes: print xarray subset .shape vs arrDB EXPLAIN Trim ranges
  • Compare with f64 accumulation in xarray: subset.astype("float64").mean(skipna=True)
  • Compare non-NaN cell counts: xarray subset.count().compute() vs arrDB COUNT_CELLS
  • If shapes differ, check exact coordinate endpoints in both systems

Context

Discovered while building the ExtremeWeatherBench (EWB) benchmarking harness for arrDB. Numeric agreement between arrDB and xarray is a validation requirement (atol=1e-4).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions