Every developer has written the loop. Read a list of files, process each one, write the result. It works for 200 items. At 50,000 it falls apart: one failure kills the run, there's no record of what finished, and restarting means starting over. The usual next step is a jump to Kubernetes jobs, Airflow or a cloud batch service. That's a lot of infrastructure for what is still, … [Read more...] about BatchComputing.com Names the Gap Between a Shell Loop and Kubernetes