Download a file using multiple parallel HTTP range requests, then reassemble the file in the correct order, showing overall progress.

chunked_curl.sh

#!/usr/bin/env bash
#
# chunked_curl.sh
#
# Download a file using multiple parallel HTTP range requests, then
# reassemble the file in the correct order, showing overall progress.
#
# Bill Church - November 2025

set -euo pipefail

chunks=16
url=""
output_file=""

usage() {
  echo "Usage: $0 -u URL [-o output_file] [-n chunks]" >&2
  exit 1
}

get_remote_size() {
  local remote_url="$1"
  local size

  size=$(
    curl -sI "$remote_url" \
      | awk '/Content-Length/ {print $2}' \
      | tr -d '\r'
  )

  if [ -z "$size" ]; then
    echo "Could not determine Content-Length for $remote_url" >&2
    exit 1
  fi

  echo "$size"
}

derive_output_name() {
  local remote_url="$1"
  local name

  name=$(basename "$remote_url")
  if [ -z "$name" ] || [ "$name" = "/" ]; then
    name="output.bin"
  fi

  echo "$name"
}

get_local_size() {
  local file="$1"
  local size

  if stat -f "%z" "$file" >/dev/null 2>&1; then
    size=$(stat -f "%z" "$file")
  else
    size=$(stat -c "%s" "$file")
  fi

  echo "$size"
}

###############################################################################
# Monitor total downloaded bytes across all part files
# Now exits when current >= total so the script does not hang.
###############################################################################
monitor_bytes() {
  local tmpdir="$1"
  local total="$2"

  while true; do
    sleep 1

    local current=0
    local size
    local f

    for f in "$tmpdir"/part-*; do
      [ -f "$f" ] || continue
      size=$(
        stat -f "%z" "$f" 2>/dev/null \
          || stat -c "%s" "$f" 2>/dev/null \
          || echo 0
      )
      current=$(( current + size ))
    done

    if [ "$total" -gt 0 ]; then
      if [ "$current" -ge "$total" ]; then
        printf "\rDownloaded: %d/%d bytes (100%%)\n" \
          "$total" "$total"
        break
      fi

      local percent=$(( current * 100 / total ))
      printf "\rDownloaded: %d/%d bytes (%d%%)" \
        "$current" "$total" "$percent"
    else
      printf "\rDownloaded: %d bytes" "$current"
    fi
  done
}

while getopts "u:o:n:h" opt; do
  case "$opt" in
    u) url="$OPTARG" ;;
    o) output_file="$OPTARG" ;;
    n) chunks="$OPTARG" ;;
    h|*) usage ;;
  esac
done

if [ -z "$url" ]; then
  usage
fi

if ! [[ "$chunks" =~ ^[0-9]+$ ]] || [ "$chunks" -le 0 ]; then
  echo "Chunks must be a positive integer" >&2
  exit 1
fi

if [ -z "$output_file" ]; then
  output_file=$(derive_output_name "$url")
fi

echo "URL:           $url"
echo "Output file:   $output_file"
echo "Chunks:        $chunks"

remote_size=$(get_remote_size "$url")
echo "Remote size:   $remote_size bytes"

chunk_size=$(( remote_size / chunks ))
if [ "$chunk_size" -le 0 ]; then
  echo "Chunk size computed as zero, reduce number of chunks" >&2
  exit 1
fi

echo "Chunk size:    $chunk_size bytes"

tmpdir=$(mktemp -d -t chunked_curl.XXXXXX)
echo "Temp dir:      $tmpdir"

cleanup() {
  rm -rf "$tmpdir"
}
trap cleanup EXIT

echo "Starting parallel downloads..."

i=0
last_index=$(( chunks - 1 ))
pids=()

while [ "$i" -lt "$chunks" ]; do
  start=$(( i * chunk_size ))
  if [ "$i" -eq "$last_index" ]; then
    range="${start}-"
  else
    end=$(( (i + 1) * chunk_size - 1 ))
    range="${start}-${end}"
  fi

  part_file="$tmpdir/part-$i"
  echo "  [part $i] bytes=$range"

  curl -s \
    -H "Range: bytes=${range}" \
    "$url" \
    -o "$part_file" &

  pids+=( "$!" )
  i=$(( i + 1 ))
done

# Start progress monitor in background
monitor_bytes "$tmpdir" "$remote_size" &
monitor_pid=$!

# Wait only for curl jobs, not for the monitor explicitly
for pid in "${pids[@]}"; do
  wait "$pid"
done

# Once all chunks are done, the monitor will naturally hit 100 percent
# and exit on its own shortly. Give it a moment then move on.
wait "$monitor_pid" 2>/dev/null || true
echo "All chunks downloaded."

echo "Reassembling into $output_file..."
: > "$output_file"

i=0
while [ "$i" -lt "$chunks" ]; do
  part_file="$tmpdir/part-$i"
  if [ ! -f "$part_file" ]; then
    echo "Missing part file: $part_file" >&2
    exit 1
  fi

  cat "$part_file" >> "$output_file"
  i=$(( i + 1 ))
done

local_size=$(get_local_size "$output_file")
echo "Local size:    $local_size bytes"

if [ "$local_size" -ne "$remote_size" ]; then
  echo "Warning: local size does not match remote size" >&2
  echo "  Remote: $remote_size" >&2
  echo "  Local:  $local_size" >&2
  exit 1
fi

echo "Success. File downloaded and verified."

Chunked cURL Downloader

A shell script that downloads large files using multiple parallel HTTP range requests, bypassing per-connection rate limits.

This is useful when a server throttles each connection to a slow speed, but allows multiple simultaneous connections. In my case I had a 1GB file which was rate-limited to 256Kb/sec with an ETA over an hour. I originally chunked it with curl manually to get it in a few minutes, while the original was still downloading, I was able to then refine it into the script above.

Description

• Determines the file size via Content-Length
• Splits the file into n chunks
• Downloads each chunk in parallel using curl range requests
• Reassembles all chunks in numeric order into the final file
• Verifies that the reassembled file matches the expected byte size

No external dependencies beyond standard Unix utilities.

Features

• Parallel downloads using HTTP Range headers
• Automatic chunk splitting
• Automatic output file naming
• Safe numeric reassembly
• Works on macOS and Linux
• Uses only standard tools: curl, awk, stat, mktemp

Usage

Basic

./chunked_curl.sh -u "<https://example.com/file.zip>"

This downloads the file into the current directory using 16 chunks.

Specify output file

./chunked_curl.sh \
  -u "<https://example.com/file.zip>" \
  -o file.zip

Change number of chunks

Example with 32 chunks:

./chunked_curl.sh \
  -u "<https://example.com/file.zip>" \
  -o file.zip \
  -n 32

More chunks usually means faster downloads if the server limits per-connection throughput.

#3 Full Options

Flag	Description
-u URL	Required. The URL of the file to download.
-o OUTPUT	Optional. Output filename (defaults to the basename of the URL).
-n CHUNKS	Optional. Number of parallel chunks (default: 16).
-h	Show help.

Example

Downloading a throttled S3 file with 32 parallel connections:

./chunked_curl.sh \
  -u "https://prod-ratta-firmware.s3.ap-northeast-1.amazonaws.com/239749/update.zip" \
  -o update.zip \
  -n 32

If the server throttles each connection to 256 KB/s, using 32 chunks can effectively push the composite rate > 8 MB/s depending on network conditions.

Output

During download, you’ll see lines like:

URL:           https://prod-ratta-firmware.s3.ap-northeast-1.amazonaws.com/239749/update.zip
Output file:   update.zip
Chunks:        32
Remote size:   1158345490 bytes
Chunk size:    36198296 bytes
Temp dir:      /var/folders/1c/4vzb8jgj0xn30c2p97pk0nj40000gn/T/chunked_curl.XXXXXX.qevvSOFja2
Starting parallel downloads...
  [part 0] bytes=0-36198295
  [part 1] bytes=36198296-72396591
  [part 2] bytes=72396592-108594887
...
  [part 30] bytes=1085948880-1122147175
  [part 31] bytes=1122147176-

When complete:

Downloaded: 1158345490/1158345490 bytes (100%)
All chunks downloaded.
Reassembling into update.zip...

Notes

• Files are assembled in the correct numeric order (avoids part-10 sorting before part-2).
• Temporary chunk files are stored in a secure ephemeral directory and removed automatically.
• Works on any server that supports HTTP Range requests (most do).

License

MIT License.
Feel free to copy, modify, or embed into your own tools.