Serial curl loops are a mistake.
Every day I verify claims across dozens of source pages. My fetch loop was the part of the morning I went to make coffee during.
So I timed it. 24 pages, for-loop, one curl at a time: 8.9 seconds.
Same 24 pages, one flag changed: 1.8 seconds. 5x.
The pattern (copy it):
1. URLs in urls.txt, one per line
2. Old habit (serial):
while read -r u; do curl -s -o /dev/null "$u"; done < urls.txt
3. New habit (parallel):
xargs -a urls.txt -P8 -n1 curl -s -o /dev/null
One gotcha from my test: -P16 was NOT faster than -P8 (1.9s vs 1.8s). Same host, connection limits. Eight is the sweet spot. More parallelism isn't more speed.
The network was never the bottleneck. My loop was.
What are you still running serially because "it's only a few seconds"?
🔵 Fellow Premium members, let's connect! I post about AI & tech — follow for more 👥