Long-term solution: Move artifact hash calculation from Orchestrator-side to container-side.
Short-term solution: Introduce proper timeouts on artifact hash calculation.
A GCS directory with ~100000 files takes almost 40 seconds to list (100 HTTP requests), but sometimes the real number can jump to 2000+ seconds.
GCS client has timeout and retry/deadline parameters, but they do not work properly for list_blobs method due to paging. In fact, reducing timeout may increases the total time due to retries.
Long-term solution: Move artifact hash calculation from Orchestrator-side to container-side.
Short-term solution: Introduce proper timeouts on artifact hash calculation.
A GCS directory with ~100000 files takes almost 40 seconds to list (100 HTTP requests), but sometimes the real number can jump to 2000+ seconds.
GCS client has
timeoutandretry/deadlineparameters, but they do not work properly forlist_blobsmethod due to paging. In fact, reducing timeout may increases the total time due to retries.