Restore Data from Azure Cool Tier Using Rclone

Follow this procedure to set up data restoration from Azure Cool Tier to Scality RING using rclone, both directly and through a Kubernetes job.

Download the Installer

Download the rclone installer from the rclone website. Choose the installer that matches your OS environment.

Set Up the Profiles

Create a configuration file that can be used by rclone to represent the source Microsoft Azure Blobstore, and the destination Scality RING.

The following example shows the configuration to set up the two profiles azure-src and scality-dest. These respectively represent the source and destination configurations.

[root@artesca ~]# cat ~/.config/rclone/rclone.conf
[azure-src]
type = azureblob
account = <azure account name>
key= <azure key>

[scality-dest]
type = s3
provider = Other
access_key_id = <access key>
secret_access_key = <secret access key>
region = us-east-1
endpoint = <endpoint>

Restore from Microsoft Azure to Scality RING

Copy objects of a specific prefix from a container on Azure Blobstore to Scality RING.

rclone copy \
azure-src:my-azure-container/my-prefix \
scality-dest:dr-restore

In the above example:

• azure-src and scality-dest represent the profiles created in ~/.config/rclone/rclone.conf.

• my-azure-container is the container in Azure Blobstore.

• my-prefix is a prefix under my-azure-container, that can be used optionally to restore from a specific prefix.

• dr-restore is a bucket in Scality RING.

Restore a Job Using Kubernetes

1. Create a yaml file for the rclone configuration similar to the following.

   [root@artesca ~]# cat rclone-secret.yaml
   apiVersion: v1
   kind: Secret
   metadata:
      name: rclone-config
   type: Opaque
   stringData:
     rclone.conf: |
      [azure-src]
      type = azureblob
      account = <azure account name>
      key= <azure key>
   
      [scality-dest]
      type = s3
      provider = Other
      access_key_id = <access key>
      secret_access_key = <secret access key>
      region = us-east-1
      endpoint = <endpoint>
   

2. Create a Kubernetes secret object using the rclone-secret.yaml file.

   sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf apply -f ./rclone-secret.yaml
   

3. Create a yaml file defining a Kubernetes job to launch the rclone command.

   [root@artesca ~]# cat rclone-job.yaml
      apiVersion: batch/v1
   kind: Job
   metadata:
     name: rclone-restore
   spec:
     template:
      spec:
      hostAliases:
      - ip: "10.100.7.181"
         hostnames:
         - "s3.scality.com"
      containers:
      - name: rclone-restore
         image: metalk8s-registry-from-config.invalid/zenko-2.4.24/rclone:1.59
         command: ["rclone"]
         args: ["--config=/config/rclone/rclone.conf", "copy", "my-azure-container/my-prefix", "scality-dest:dr-restore","-vv"]
         volumeMounts:
            - name: rclone-vol
            mountPath: /config/rclone
            readOnly: false
            - name: rclone-data
            mountPath: "/data"
            readOnly: false
      restartPolicy: Never
      volumes:
      - name: rclone-vol
         secret:
            secretName: rclone-config
            optional: false
      - name: rclone-data
         hostPath:
            path: /tmp
   

   Note

   The hostAliases section is optional. It can be used if there are issues resolving DNS for Scality RING S3C endpoint from the servers for ARTESCA.

4. Create a Kubernetes job using the yaml file created in the previous step.

   sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf apply -f ./rclone-job.yaml
   

5. Check that a pod has been created by the job.

   sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get pods | grep rclone
   

6. View the output of the job using the pod’s logs.

   sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf logs -f <rclone pod name>