1.I can use “aws s3 ls” to list buckts
$aws s3 ls --endpoint http://XXXXXX ... 2022-08-22 16:47:37 cluster-back 2022-08-23 11:54:46 dog 2022-09-08 10:22:41 kesci-annotations 2022-09-08 10:22:59 kesci-datasets-ng 2022-09-08 10:23:13 kesci-datasets-ng-parsed-meta 2022-09-08 10:25:07 kesci-model-service-input 2022-09-08 10:25:39 kesci-model-service-output 2022-09-08 10:26:09 kesci-workspace-ng 2022-09-08 10:26:39 kesci-workspace-ng-parsed-meta ...
now how can I use shell or python to sync all buckts without buckets name
2.then I want to use this shell or python to kubernetes`s cronjob
aws s3 sync command will copy from one source path to one destination path. The source path would include the bucket name. Therefore, you would need to run one
sync command per bucket.
You could first run a command to retrieve all the bucket names:
aws s3api list-buckets --query 'Buckets.[Name]' --output text
This will output each bucket name on a separate line.
You could then pass these bucket names into
xargs to generate multiple
aws s3 sync commands that will copy all the buckets.