When uploading a 1GB binary file using s3cmd from a Linode instance with 50Mb/s bandwidth capacity:

# s3cmd put 1gb.bin s3://my-bucket/1gb.bin
1gb.bin -> s3://my-bucket/1gb.bin  [1 of 1]
  366706688 of 1073741824    34% in  371s   963.22 kB/s

SCP transfer to EC2 shows significantly better performance (~44Mb/s), indicating the issue is S3-specific rather than general network limitations.

1. Multipart Upload Optimization

For large files, AWS recommends multipart uploads. Here's how to implement it with AWS CLI:

aws configure set default.s3.multipart_threshold 64MB
aws configure set default.s3.multipart_chunksize 16MB
aws s3 cp largefile.bin s3://my-bucket/ --recursive

2. Parallel Thread Configuration

Increase concurrent transfers in s3cmd:

# ~/.s3cfg configuration
multipart_chunk_size_mb = 50
max_concurrent_requests = 20

3. Geographic Endpoint Selection

The traceroute shows routing through IAD (Virginia) endpoint:

traceroute to s3-1-w.amazonaws.com. (72.21.194.32)
6  equinix02-iad2.amazon.com (206.223.115.35)  9.393 ms
7  72.21.220.41 (72.21.220.41)  32.610 ms

Consider testing alternative endpoints closer to your Linode location.

Using S3 Transfer Acceleration

Enable acceleration and use the special endpoint:

aws s3api put-bucket-accelerate-configuration \
    --bucket my-bucket \
    --accelerate-configuration Status=Enabled

aws s3 cp file.txt s3://my-bucket/file.txt \
    --endpoint-url http://s3-accelerate.amazonaws.com

Network Stack Optimization

Adjust TCP stack parameters on Linux:

# /etc/sysctl.conf
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_window_scaling = 1

Benchmark different clients for your use case:

# AWS CLI
time aws s3 cp 1gb.bin s3://my-bucket/

# rclone
rclone copy 1gb.bin :s3:my-bucket/ --s3-upload-concurrency 8

# gof3r
gof3r put -b my-bucket -k 1gb.bin < 1gb.bin

Use network monitoring during transfers:

# Bandwidth monitoring
iftop -nNP -i eth0

# Detailed packet analysis
tcpdump -i eth0 -w s3_upload.pcap 'host s3.amazonaws.com'

---

When uploading large files to Amazon S3 using s3cmd, many developers encounter unexpectedly slow transfer rates. In this case, a 1GB file upload from a Linode server with 50Mb/s bandwidth cap shows only 963.22 kB/s transfer speed - significantly below the expected capacity.

The traceroute reveals a reasonable latency until hop 9 (10ms), suggesting the network path itself isn't the primary bottleneck:

traceroute to s3-1-w.amazonaws.com. (72.21.194.32):
 1  207.99.1.13 0.635 ms
 9  72.21.218.3 10.245 ms

Several factors could contribute to slow S3 uploads:

• TCP Window Scaling: Suboptimal TCP window sizes for long-distance transfers
• SSL/TLS Overhead: s3cmd uses HTTPS by default
• Single-threaded Transfer: No parallel chunking in basic s3cmd put
• DNS Resolution: Improper S3 endpoint selection

1. Enable Multipart Uploads

The most effective solution is using S3's multipart upload API. Here's how to implement it with the AWS CLI:

aws configure set default.s3.multipart_threshold 64MB
aws configure set default.s3.multipart_chunksize 16MB
aws s3 cp largefile.bin s3://my-bucket/ --debug

2. Switch to AWS CLI

The AWS CLI often outperforms s3cmd:

# Install AWS CLI v2
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

# Configure and test
aws configure
time aws s3 cp 1gb.bin s3://my-bucket/

3. Optimize s3cmd Configuration

Edit ~/.s3cfg:

[default]
bucket_location = us-east-1
multipart_chunk_size_mb = 50
max_concurrent_requests = 20

4. Network Stack Tuning

Adjust TCP parameters for high-latency transfers:

# Temporary settings
sudo sysctl -w net.ipv4.tcp_window_scaling=1
sudo sysctl -w net.ipv4.tcp_timestamps=1
sudo sysctl -w net.ipv4.tcp_sack=1

# Permanent settings
echo "net.ipv4.tcp_window_scaling = 1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

Comparative transfer speeds using different tools:

Method	Avg. Speed	Configuration
s3cmd (default)	~1 MB/s	Default settings
AWS CLI (single)	~3 MB/s	Default multipart
s3cmd (tuned)	~2.5 MB/s	50MB chunks
Parallel S3 CLI	~5 MB/s	GNU parallel + 8 threads

For maximum throughput, split the file and upload in parallel:

#!/bin/bash
FILE="1gb.bin"
BUCKET="my-bucket"
CHUNKS=8

split -n $CHUNKS $FILE chunk_

for f in chunk_*; do
  aws s3api upload-part --bucket $BUCKET \
    --key $FILE \
    --part-number $(echo $f | cut -d'_' -f2) \
    --body $f &
done

wait
# Complete multipart upload...

Consider these specialized tools for massive transfers:

• rclone: rclone copy --transfers=16 --s3-upload-concurrency=8
• s4cmd: Python-based parallel S3 tool
• Cyberduck: GUI client with optimized transfers