Implement my idea of optimized shims:
We intercept launching certain well-known components (e.g. "Run BigQuery job") and essentially execute them using custom launchers that do not create a container.
The set of these shim launchers can be customized and expanded.
Shims can be disabled.
One reason some Tangle deployment or a pipeline might NOT want to use the optimized launcher is that this would require Tangle backend to have the access to cloud resources which it currently does not have. This issue can potentially be mitigated by having a special container/Pod that has same access as normal Tangle containers/Pods, but is a long-running external job controller.
Implement my idea of optimized shims:
We intercept launching certain well-known components (e.g. "Run BigQuery job") and essentially execute them using custom launchers that do not create a container.
The set of these shim launchers can be customized and expanded.
Shims can be disabled.
One reason some Tangle deployment or a pipeline might NOT want to use the optimized launcher is that this would require Tangle backend to have the access to cloud resources which it currently does not have. This issue can potentially be mitigated by having a special container/Pod that has same access as normal Tangle containers/Pods, but is a long-running external job controller.