[SOLVED] How to sync a lot of buckets from shell or python


This Content is from Stack Overflow. Question asked by yixuan

1.I can use “aws s3 ls” to list buckts
likes this

$aws s3 ls --endpoint http://XXXXXX
2022-08-22 16:47:37 cluster-back
2022-08-23 11:54:46 dog
2022-09-08 10:22:41 kesci-annotations
2022-09-08 10:22:59 kesci-datasets-ng
2022-09-08 10:23:13 kesci-datasets-ng-parsed-meta
2022-09-08 10:25:07 kesci-model-service-input
2022-09-08 10:25:39 kesci-model-service-output
2022-09-08 10:26:09 kesci-workspace-ng
2022-09-08 10:26:39 kesci-workspace-ng-parsed-meta

now how can I use shell or python to sync all buckts without buckets name

2.then I want to use this shell or python to kubernetes`s cronjob


The aws s3 sync command will copy from one source path to one destination path. The source path would include the bucket name. Therefore, you would need to run one sync command per bucket.

You could first run a command to retrieve all the bucket names:

aws s3api list-buckets --query 'Buckets[].[Name]' --output text

This will output each bucket name on a separate line.

You could then pass these bucket names into xargs to generate multiple aws s3 sync commands that will copy all the buckets.

This Question was asked in StackOverflow by yixuan and Answered by John Rotenstein It is licensed under the terms of CC BY-SA 2.5. - CC BY-SA 3.0. - CC BY-SA 4.0.

people found this article helpful. What about you?