feat(transport)!: page large results in parallel by offset; remove parallel_chunks

.importlinter

@@ -54,17 +54,12 @@ type = protected
 ; The root ``dataretrieval`` package is deliberately NOT an allowed importer.
 ; ``allowed_importers`` is matched with ``as_packages``, so naming the root here
 ; would make every module in the distribution an allowed importer and the
-; contract could never fail. Its two real imports are listed as explicit
-; exceptions below instead -- narrow, visible, and they fail if they go stale.
+; contract could never fail.
 protected_modules =
     dataretrieval.ogc
 allowed_importers =
     dataretrieval.ngwmn
     dataretrieval.waterdata
-ignore_imports =
-; The package __init__ re-exports the parallel-chunks context manager; it is
-; part of the documented public surface, not a service reaching into OGC.
-    dataretrieval -> dataretrieval.ogc.chunking
 
 [importlinter:contract:ogc-facade]
 name = Facade-only OGC consumers, never its internals (ADR 0007)

NEWS.md

@@ -8,6 +8,8 @@
 
 **08/06/2026:** Fan-out execution is now shared across services. Chunking is how a query is divided structurally (a Water Data/NGWMN URL over the byte limit); fan-out is how the pieces are distributed operationally. Only the first is protocol-specific, so the executor moved to `dataretrieval.transport.fanout` (`FanOut`, over a three-member `FanOutPlan` protocol) while chunk planning stays in `dataretrieval.ogc`. Water Use no longer re-implements the fan-out gather and inherits resume, progress reporting, and `API_USGS_CONCURRENT`: a multi-location pull interrupted by a rate limit now raises a resumable interruption whose `.call.resume()` re-issues only the locations that did not finish, instead of discarding every completed one. The interruption taxonomy moved to the `dataretrieval.interruptions` leaf and its base class is now `FanOutInterrupted`; **`ChunkInterrupted` is a permanent alias of the same class**, so `except ChunkInterrupted` keeps working. **Breaking change:** a Water Use fan-out interrupted by a 5xx, 429, or recoverable connection failure now raises `ServiceInterrupted`/`QuotaExhausted` rather than `ServiceUnavailable`/`RateLimited`/`NetworkError` — all remain `DataRetrievalError`, so broad handlers are unaffected, but narrow handlers around a Water Use call must widen. **Breaking change:** `wateruse.MAX_CONCURRENT_REQUESTS` is removed; set `API_USGS_CONCURRENT` (which now outranks any service default) or read `wateruse.DEFAULT_CONCURRENT_REQUESTS`.
 
+**08/06/2026:** Large Water Data pulls are now paged in parallel automatically. A multi-page result previously cost one round trip per page, because a cursor page's URL only exists once the previous page has been parsed. Where the API honors `offset`, every page's URL is computable up front, so the pages are fetched concurrently — the request count is unchanged, only their timing, and since the USGS quota is volume-based the speedup costs no extra quota. Measured 2.1× on 8 sites × 2 years (16,000 rows) and 3.1–4.2× on a single site's full daily history (~19,000 rows). **Breaking change:** `parallel_chunks(n)` is removed (along with `ChunkPlan.max_chunks`), because it bought parallelism the other way — splitting a request that already fit into more sub-requests, which spent extra quota and did nothing at all for a single-site query, the case with no multi-value axis to split. Delete the `with parallel_chunks(...):` wrapper; the pages inside it are now overlapped without it. Byte-driven chunking is unchanged and still required for correctness (the ~8 KB URL limit is real). `API_USGS_CONCURRENT` now also caps the page-fetch wave width; set it to `1` to page strictly sequentially. Two fallbacks keep results correct rather than merely fast: a server that ignores `offset` (a non-standard extension, so ignoring it is conventional) is detected before any rows are returned and the query re-runs via standard `next`-link paging, and the API's hard `offset` ceiling of 40,000 hands the remainder off to the sequential walk, so an arbitrarily deep pull is still returned in full.
+
 **08/03/2026:** Split the Water Data implementation into focused time-series, metadata, measurements, reference, samples, and CQL collection-family modules behind the unchanged `waterdata.api` facade. Active service modules now declare explicit exports; public Water Data imports, signatures, function identities, deprecations, and return contracts are protected by executable contract snapshots. OGC ambient context and schema/queryables execution are separated from request construction, adapter-to-adapter reach-through is prohibited by architecture tests, and service-specific output shapes are documented rather than forced into one model.
 
 **08/02/2026:** Added an internal API-neutral transport layer for guarded HTTP clients, host-scoped authentication, cursor pagination, bounded retry, response aggregation, progress, and sync-over-async dispatch. Water Use and the non-OGC Statistics API now consume transport directly instead of private OGC execution helpers; WQP, NLDI, and StreamStats opt into bounded transient retry while deprecated NWIS behavior remains unchanged. OGC retains CQL2, request construction, feature shaping, chunk planning, resumable calls, and interruption types, with compatibility imports at previous private paths. Failed pagination and fan-out still raise rather than returning partial data, and no public signatures or return shapes changed.

README.md

@@ -107,57 +107,52 @@ df, metadata = waterdata.get_continuous(
 print(f"Retrieved {len(df)} continuous gage height measurements")
 ```
 
-#### Speeding up large downloads with `parallel_chunks`
-
-By default the getters split a multi-value request only as far as the server's
-~8 KB URL limit forces — the fewest sub-requests. For a **large, paginated**
-pull, that default is needlessly conservative: every sub-request pages through
-its own results, so dividing the query into more, smaller sub-requests lets
-those pages be fetched **in parallel**. `parallel_chunks(n)` opts a single call
-into that finer split, fanning it out into `n` sub-requests. The finer split
-pays off only when the result is large enough to span many pages *and* the query
-has a multi-value argument to divide, such as a list of monitoring locations. On
-a small query — or one with nothing to split — it only adds requests, so
-`parallel_chunks` is a deliberate, scoped `with` block, never the default.
+#### Large Water Data downloads are paged in parallel automatically
+
+A large result arrives one page at a time. Standard OGC cursor pagination is
+inherently sequential: page *N+1*'s URL is revealed only by page *N*. The Water
+Data API also accepts `offset`, which makes page URLs computable up front
+(`offset = i * limit`). `dataretrieval` therefore overlaps pages automatically;
+there is no context manager to enable.
 
 ```python
 from dataretrieval import waterdata
 
-# All stream gages in Ohio, then 20 years of their daily discharge — large
-# enough to span many pages, so it profits from a finer split.
-sites, _ = waterdata.get_monitoring_locations(state="Ohio", site_type_code="ST")
-
-with waterdata.parallel_chunks(32):  # fan out into 32 sub-requests
-    df, md = waterdata.get_daily(
-        monitoring_location_id=sites["monitoring_location_id"],
-        parameter_code="00060",  # discharge
-        time="2004-01-01/2023-12-31",
-    )
+# A deep daily-value history can span many pages even for one location.
+df, md = waterdata.get_daily(
+    monitoring_location_id="USGS-01646500",
+    parameter_code="00060",  # discharge
+    time="2004-01-01/2023-12-31",
+    limit=2000,
+)
 ```
 
-`n` is the number of sub-requests to fan the call out into, capped by how many
-values there are to split. Each sub-request costs a request against your hourly
-[rate limit](https://api.waterdata.usgs.gov/signup/). How many run *at once* is
-capped separately by `API_USGS_CONCURRENT` (default 32), so the useful range is
-roughly `2` up to that value.
-
-Benchmark — a fixed 271-site subset of Ohio stream gages
-(`get_daily`, `parameter_code="00060"`), with a small fixed page size
-(`limit=250`) so every run fetches roughly the same number of pages (isolating
-the effect of parallelism). Each `n` ran against its own cold 1-year time
-window, so no run is served from the server's data-window cache:
-
-| `n`  | parallelism | pages | wall-clock              | speedup |
-| ---- | ----------- | ----- | ----------------------- | ------- |
-| off  | 1           | ~30   | 9.5 s / 9.1 s (2 runs)  | 1×      |
-| `8`  | 8           | ~32   | 2.2 s / 1.9 s           | ~4.5×   |
-| `32` | 32          | 54    | 1.2 s                   | ~8×     |
-
-The gain comes from overlapping each sub-request's per-page latency and
-server-side work. The exact multiplier therefore scales with how many pages the
-pull spans: a larger pull (more pages) has more parallelism to exploit. The
-extra sub-requests each cost quota, so reserve a large `n` for pulls you know
-are large.
+At a **fixed `limit`**, this changes timing rather than deliberately splitting
+the query into extra requests. The unknown page count is discovered with
+ramped speculative waves (1, 2, 4, ... requests), so the final wave can probe
+past the end, but total requests remain below twice the pages needed. Reducing
+`limit` creates more pages and therefore spends more hourly quota; the measured
+speedups are not free if you shrink `limit` to obtain them.
+
+The default `limit` is 50,000 while Water Data accepts offsets only through
+40,000. At that default there is no useful offset fan-out: the unbounded tail is
+walked through standard cursors. Material speedups therefore require an
+explicit paging-friendly `limit` below the offset ceiling, such as the `2000`
+above. `API_USGS_CONCURRENT` (default 32) bounds the page-wave width; set it to
+`1` to use cursor pagination strictly sequentially.
+
+Because `offset` is a Water Data extension rather than part of OGC API -
+Features, the walk is defensive. It verifies that the server honors the
+parameter before trusting rows and falls back to standard cursor pagination if
+not. At the API's 40,000-row offset ceiling it rewinds one page and follows the
+standard `next` links, keeping arbitrarily deep results complete without a gap
+or duplicate seam.
+
+The removed `parallel_chunks(n)` API split a fitting multi-value query into more
+chunks, spending extra quota and doing nothing for a single-location query.
+Delete that wrapper when migrating. Byte-driven chunking remains automatic and
+unchanged: queries above the service's ~8 KB request limit are still split for
+correctness.
 
 Visit the
 [API Reference](https://doi-usgs.github.io/dataretrieval-python/reference/waterdata.html)
@@ -176,8 +171,8 @@ logging.basicConfig(level=logging.DEBUG)
 ### National Ground-Water Monitoring Network (NGWMN)
 
 Access groundwater data aggregated from many state, federal, and local
-agencies. NGWMN uses the same OGC engine as the Water Data API,
-so chunking and pagination behave the same way:
+agencies. NGWMN uses the same OGC engine and byte-driven chunker as Water Data. It
+uses standard cursor pagination unless its API dialect declares offset support:
 
 ```python
 from dataretrieval import ngwmn

dataretrieval/__init__.py

@@ -62,11 +62,6 @@
     ServiceInterrupted,
 )
 
-# Parallel-chunks control (a context manager). Defined with the chunker in
-# ``dataretrieval.ogc.chunking``; surfaced here for a stable public path
-# ``from dataretrieval import parallel_chunks``.
-from dataretrieval.ogc.chunking import parallel_chunks
-
 from . import (
     exceptions,
     ngwmn,
@@ -109,7 +104,5 @@
     "FanOutInterrupted",
     "QuotaExhausted",
     "ServiceInterrupted",
-    # parallel-chunks control (defined in ogc.chunking)
-    "parallel_chunks",
     "__version__",
 ]