fetcher.timeout.queue silently drops URLs

[dpol1]

When a URL waits in the queue longer than fetcher.timeout.queue, FetcherThread just logs and moves on: no status, no metric, tuple acked. The comment assumes the tuple already failed at Storm level, but that's only true if topology.message.timeout.secs is shorter — otherwise the URL vanishes and the status index never hears about it.

Emit FETCH_ERROR? At least a metric?

Noticed while planning the characterization tests for #2057 — those will pin the current behaviour for now, the fix would be its own PR.

[jnioche]

Emit FETCH_ERROR? At least a metric?

it should not be a fetch error - as far as we know the fetch has not been attempted.

Could add a metric to make things more visible, we already log

[dpol1]

Agreed, nothing was attempted so FETCH_ERROR would be wrong. Metric it is.

The URL comes back anyway (no status update = old nextFetchDate, spout picks it up again), so visibility is really all that's missing. Maybe also a line in the docs saying fetcher.timeout.queue should stay below topology.message.timeout.secs, the code kind of assumes that.