Issue

This Content is from Stack Overflow. Question asked by yixuan

1.I can use “aws s3 ls” to list buckts
likes this

$aws s3 ls --endpoint http://XXXXXX
...
2022-08-22 16:47:37 cluster-back
2022-08-23 11:54:46 dog
2022-09-08 10:22:41 kesci-annotations
2022-09-08 10:22:59 kesci-datasets-ng
2022-09-08 10:23:13 kesci-datasets-ng-parsed-meta
2022-09-08 10:25:07 kesci-model-service-input
2022-09-08 10:25:39 kesci-model-service-output
2022-09-08 10:26:09 kesci-workspace-ng
2022-09-08 10:26:39 kesci-workspace-ng-parsed-meta
...

now how can I use shell or python to sync all buckts without buckets name

2.then I want to use this shell or python to kubernetes`s cronjob

Solution

The aws s3 sync command will copy from one source path to one destination path. The source path would include the bucket name. Therefore, you would need to run one sync command per bucket.

You could first run a command to retrieve all the bucket names:

aws s3api list-buckets --query 'Buckets[].[Name]' --output text

This will output each bucket name on a separate line.

You could then pass these bucket names into xargs to generate multiple aws s3 sync commands that will copy all the buckets.

This Question was asked in StackOverflow by yixuan and Answered by John Rotenstein It is licensed under the terms of CC BY-SA 2.5. - CC BY-SA 3.0. - CC BY-SA 4.0.