My server says no space left on device. Is my data gone?
No — and this is almost always reclaimable garbage, not your data. Here is the complete safe recovery order (do the steps in this order):
Step 1 — Confirm what is actually full
df -h / /var/lib/docker docker system df
If /var/lib/docker is the culprit, continue. If / is full but
/var/lib/docker is small, jump to journal/system logs (journalctl --disk-usage).
Step 2 — Safe prune (keeps named volumes & tagged images)
docker system prune -f
Removes stopped containers, dangling images, unused networks, build cache. Your named volumes and tagged images are untouched. This alone usually frees gigabytes.
Step 3 — Still tight? Aggressive but still-safe prune
docker system prune -a --volumes -f # ⚠ removes ALL unused images AND anonymous volumes
Warning: only run when containers are down or you accept re-pull costs. Tagged-but-unused images go too. Named volumes survive unless listed explicitly.
Step 4 — The silent killers most guides miss
# container stdout logs can be tens of GB: du -sh /var/lib/docker/containers/*/*-json.log | sort -h | tail -5 truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log # journald eating the root fs: journalctl --vacuum-size=200M # old kernels + apt cache on the host: sudo apt-get clean && sudo apt autoremove -y
Step 5 — Stop it happening again
daemon.json: {"log-driver":"local","log-opts":{"max-size":"10m","max-file":"3"}}
Restart docker after editing /etc/docker/daemon.json. Then add a cron:
0 4 * * 1 docker system prune -f.
Why this works: Docker never deletes anything eagerly — every pulled layer, exited container, build cache entry and unbounded JSON log stays until told otherwise. The commands above reclaim in safe→aggressive order without touching running workloads.
Unlock the full guide — founding rate
First 20 buyers lock $2-equivalent forever (then $5). Remaining founding slots are counted only from verified settlements — no fake counters here.
Send exactly 2.65 USDC on BASE to:
| Amount | 2.65 USDC |
| Network | BASE |
| Address | 0x05FE364d9Ee3Dc0063878C286aF8b3074819Ca5B |
| Order ID | 29c22f17e241 |
| Status | waiting for your transaction… (auto-checks every 15 s — no refresh needed) · raw status |
The moment settlement lands on-chain, this page unlocks itself — the full 54-scenario cookbook fades in below. No email, no account, no refresh.
Full guide unlocked — 54 disk-full rescue scenarios
Order 29c22f17e241 verified settled. Thank you — founding rate locked.
Image & Layer Cleanup (10)
1 · Remove dangling images only
docker image prune -f
2 · Remove ALL images not used by a container
docker image prune -a -f
3 · Delete one specific image that refuses to die
docker rmi -f <image>
4 · Find which images eat the most space
docker images --format '{{.Size}}\t{{.Repository}}:{{.Tag}}' | sort -rh | head -205 · Images stuck as "<none>:<none>" everywhere
docker rmi $(docker images -f dangling=true -q)
6 · Force-delete an image used by a stopped container
docker rm <container> && docker rmi <image>
7 · Multi-arch/attestation manifests bloating the store
docker buildx prune -af
8 · Keep only the last N versions of an app image
docker images myapp --format '{{.ID}}' | tail -n +4 | xargs -r docker rmi -f9 · Verify layers actually got reclaimed
docker system df -v | head -30
10 · Image pull fails mid-way: "no space left" during pull
docker system prune -f && docker pull <image>
Build Cache Disasters (8)
11 · Build cache grew to 40 GB
docker builder prune -af
12 · Prune build cache older than 72h only
docker builder prune -af --filter 'until=72h'
13 · BuildKit cache keeps regrowing in CI
docker builder prune -af --keep-storage 5GB
Keeps a 5 GB floor so builds stay fast.
14 · "no space left" DURING docker build
docker builder prune -f && docker build --squash …
15 · buildx multi-stage temp containers piling up
docker rm $(docker ps -aq --filter label=com.docker.compose.project=buildx)
16 · Cache mounts (RUN --mount=type=cache) never evict
docker builder prune -af --filter type=exec.cachemount
17 · See exactly what build cache holds per stage
docker system df -v | grep -A20 'Build Cache'
18 · CI runners fill disk daily — permanent fix
cron: 0 */6 * * * docker builder prune -af --keep-storage 8GB
Volume Emergencies (8)
19 · Anonymous volumes accumulating for years
docker volume prune -f
20 · Find which volumes are the biggest
du -sh /var/lib/docker/volumes/*/ _data 2>/dev/null | sort -rh | head
21 · One specific volume is huge but needed — shrink contents
docker run --rm -v myvol:/data alpine sh -c 'find /data -size +100M -exec ls -lh {} \;'22 · Volume prune deleted a volume I needed
restore from backup: docker run --rm -v myvol:/data -v $PWD:/bak alpine tar xzf /bak/myvol.tgz -C /data
Anonymous-only prune spares named volumes — always name prod volumes.
23 · Postgres volume full: WAL bloat
docker exec -it db psql -U postgres -c "CHECKPOINT;" && docker exec -it db psql -U postgres -c "SELECT pg_switch_wal();"
24 · Dangling volumes from removed compose stacks
docker volume ls -qf dangling=true | xargs -r docker volume rm
25 · Move Docker data dir to a bigger disk
systemctl stop docker && rsync -aP /var/lib/docker/ /mnt/big/docker/ && daemon.json {"data-root":"/mnt/big/docker"} && systemctl start docker26 · Backup a volume before risky cleanup
docker run --rm -v myvol:/data -v $PWD:/bak alpine tar czf /bak/myvol.tgz -C /data .
Container Log Floods (8)
27 · Truncate a runaway json-log NOW (no restart)
truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log
28 · Find the loudest container logs instantly
du -sh /var/lib/docker/containers/*/*-json.log | sort -rh | head
29 · Cap logs for ALL future containers
daemon.json {"log-driver":"local","log-opts":{"max-size":"10m","max-file":"3"}} then systemctl restart docker30 · Cap logs per-container at runtime
docker run --log-opt max-size=10m --log-opt max-file=3 …
31 · Compose service spamming logs
compose: logging:{driver:"local",options:{max-size:"10m",max-file:"3"}}32 · Rotate logs without truncating mid-write corruption
docker logs <c> > /dev/null 2>&1 || true # then truncate — local driver handles rotation atomically
33 · journald (not docker) is the real hog
journalctl --vacuum-size=200M && echo 'SystemMaxUse=300M' >> /etc/systemd/journald.conf
34 · Log shipper (fluentd/vector) backlog filling disk
systemctl stop vector && rm /var/lib/vector/buffer/* && systemctl start vector
overlay2 / Filesystem Level (8)
35 · /var/lib/docker/overlay2 gigantic — is it safe to touch?
NEVER rm inside overlay2 by hand — always via docker prune/rmi/rm
36 · Map overlay2 dirs back to images/containers
docker ps -aq | xargs docker inspect --format '{{.Id}} {{.GraphDriver.Data.MergedDir}}'37 · Orphaned overlay dirs after crashed docker
systemctl restart docker && docker system prune -f
38 · Switch storage driver to save space (vfs→overlay2)
daemon.json {"storage-driver":"overlay2"} — requires re-pull of all images39 · Inodes exhausted, df shows space free
df -i /var/lib/docker && docker system prune -af
40 · Deleted file still held open by a container process
lsof +L1 | grep -i docker → restart the offending container
41 · XFS project quota exceeded on overlay
xfs_quota -x -c 'report -h' /var/lib/docker
42 · Emergency: need 2 GB RIGHT NOW before prune finishes
journalctl --vacuum-size=100M && apt-get clean && rm -rf /var/log/*.gz /var/log/*.[0-9]
Registry & Pull Failures (6)
43 · docker push fails "no space left" on the CLIENT
docker builder prune -af && retry push
44 · Self-hosted registry disk full
registry garbage-collect /etc/docker/registry/config.yml
45 · Registry GC refuses while read-only
set readonly:false temporarily or run GC with --delete-untagged=false offline
46 · Pull loop leaving partial layers each attempt
docker system prune -f once, ensure 2× image size free, then pull
47 · Hub rate-limit + retries duplicating layers
docker login first; authenticated pulls dedupe and cache cleanly
48 · Check registry blob usage per repo
du -sh /var/lib/registry/docker/registry/v2/repositories/* | sort -rh
Prevention & Monitoring (6)
49 · Weekly self-healing cron
0 4 * * 1 docker system prune -f --filter 'until=168h'
50 · Alert before disk-full kills prod
node_exporter + alert: node_filesystem_avail_bytes / size < 0.15
51 · Track Docker's own view of usage over time
watch -n3600 'docker system df >> /var/log/docker-df.log'
52 · Set hard limits so one container can't eat the disk
docker run --storage-opt size=20G … (requires overlay2 + xfs pquota)
53 · Compose projects leaving dead containers behind
docker compose ls --all && docker compose -p <dead> down --remove-orphans
54 · Full audit one-liner: where did my disk go?
sudo du -x -d2 -h /var/lib/docker | sort -rh | head -15