Scenario ^
I have a server that I use for making backups. I also send backups from that server into Amazon S3 at the “Infrequent Access” storage class. That class is cheaper to store but expensive to access. It’s intended for backups of last resort that you only access in an emergency. I use Duplicity to handle the S3 part.
(I could save a bit more by using one of the “Glacier” classes but at the moment the cost is minimal and I’m not that brave.)
I recently decided to change which server I use for the backups. I noticed that renting a server with only IPv6 connectivity was cheaper, and as all the hosts I back up have IPv6 connectivity I decided to give that a go.
This mostly worked fine. The only thing I really noticed was when I tried to install some software from GitHub. GitHub doesn’t support IPv6, so I had to piggy back that download through another host.
Then I came to set up Duplicity again and found that I needed to make some non-obvious changes to make it work with S3 over IPv6-only.
S3 endpoint ^
The main issue is that the default S3 endpoint URL is https://s3.<region>.amazonaws.com, and this host only has an A (IPv4) record! For example:
$ host s3.us-east-1.amazonaws.com s3.us-east-1.amazonaws.com has address 52.216.89.254 |
If you run Duplicity with a target like s3://yourbucketname/path/to/backup then it will try that endpoint, get only an IPv4 address, and return Network unreachable.
S3 does actually support IPv6, but for that to work you need to use a dual stack endpoint! They look like this:
$ host s3.dualstack.us-east-1.amazonaws.com s3.dualstack.us-east-1.amazonaws.com has address 54.231.129.0 s3.dualstack.us-east-1.amazonaws.com has IPv6 address 2600:1fa0:80dc:5101:34d9:451e:: |
So we need to specify the S3 endpoint to use.
Specifying the S3 endpoint ^
In order to do this you need to switch Duplicity to the “boto3” backend. Assuming you’ve installed the correct package (python3-boto3 on Debian), this is as simple as changing the target from s3://… to boto3+s3://….
That then allows you to use the command line arguments --s3-region-name and --s3-endpoint-url so you can tell it which host to talk to. That ends up giving you both an IPv4 and an IPv6 address and your system correctly chooses the IPv6 one.
The full script ^
The new, working script now looks something like this:
export PASSPHRASE="highlysecret"
export AWS_ACCESS_KEY_ID="notquiteassecret"
export AWS_SECRET_ACCESS_KEY="extremelysecret"
# Somewhere with plenty of free space.
export TMPDIR=/var/tmp
duplicity --encrypt-key ABCDEF0123456789 \
--asynchronous-upload \
-v 4 \
--archive-dir=/path/to/your/duplicity/archives \
--s3-use-ia \
--s3-use-multiprocessing \
--s3-use-new-style \
--s3-region-name "us-east-1" \
--s3-endpoint-url "https://s3.dualstack.us-east-1.amazonaws.com" \
incr \
--full-if-older-than 30D \
/stuff/you/want/backed/up \
"boto3+s3://yourbucketname/path/to/backups" |
The previous version of the script looked a bit like:
# All the exports stayed the same
duplicity --encrypt-key ABCDEF0123456789 \
--asynchronous-upload \
-v 4 \
--archive-dir=/path/to/your/duplicity/archives \
--s3-use-ia \
--s3-use-multiprocessing \
incr \
--full-if-older-than 30D \
/stuff/you/want/backed/up \
"s3+http://yourbucketname/path/to/backups" |
