Resize Service Volumes

This set of procedures aims at guiding you through the process of resizing Service Logical Volumes (LV) in two different scenarios: with or without the addition of Physical Volumes (PV).

Note

This procedure applies to service disk volumes which are provisioned as LVM LV under the LVM VG vgartesca. This does not apply to data disk volumes which can not be resized.

Find a Service Logical Volume (LV) that Needs to be Resized

1. Recognizing a Volume Filling Up alert

   When a Logical Volume (LV) utilized by an ARTESCA component is nearing capacity, the ARTESCA monitoring system will trigger a KubePersistentVolumeFillingUp alert.

   In this scenario, you will observe the Volumes icon on the Platform Status tile of the ARTESCA Overview page turning yellow (or red in critical cases).

   

   By clicking on the Volumes icon, you will be redirected to the Alerts page, where you should see the KubePersistentVolumeFillingUp alert as follows:

   

2. Looking for affected Volume(s)

   1. Click the Platform tab in the top navigation bar

      

   2. Click the Volumes tab in the left navigation bar

      

   3. Click twice on the Usage column at the center of the page to sort the volumes by usage in descending order (highest usage first).

      

   4. In the Volumes table, all service disk volumes having a usage percentage higher than 80% are candidates to be resized.

      

3. For each volume candidate

   1. Click the volume to access its details.

      Important

      Ensure that the VG Name is vgartesca (like in the area surrounded in yellow). Otherwise, it is not a service volume and should not be resized by following these procedures.

      Note

      We are looking for the Volume Name, that is in the first line of the screen preceded by the status icon (like in the area surrounded in green).

      

   2. Set the VOLUME_NAME variable

      Note

      Replace <VOLUME_NAME> by the name of the actual volume identified in the previous step.

      Example:

      export VOLUME_NAME='keycloak-postgresql-ip-10-0-253-145.eu-north-1.compute.internal'

      From your cluster bootstrap node, run the following command:

      [artesca-os@bootstrap ~]$
      

      export VOLUME_NAME='<VOLUME_NAME>'
      

   3. Set the LV_SELECTOR variable

      From your cluster bootstrap node, run the following command:

      [artesca-os@bootstrap ~]$
      

      export LV_SELECTOR=$(sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get volumes "$VOLUME_NAME" -o jsonpath='{.spec.template.metadata.labels}')
      

   4. Find all Logical Volume (LV) names and their associated nodes

      Note

      In multi-node clusters the same service is replicated on different nodes.

      So, we need to resize the Service Logical Volume (LV) on all nodes that a given service is present to make sure that all volumes have the same size.

      From your cluster bootstrap node, run the following command:

      [artesca-os@bootstrap ~]$
      

      sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get volumes -o json | jq -r --argjson selector "$LV_SELECTOR" '.items[] | select(.spec.template.metadata.labels | contains($selector)) | "NODE_NAME=\"\(.spec.nodeName)\" LV_NAME=\"\(.metadata.name)\""'
      

      NODE_NAME="ip-10-0-211-26.eu-north-1.compute.internal" LV_NAME="keycloak-postgresql-ip-10-0-211-26.eu-north-1.compute.internal"
      NODE_NAME="ip-10-0-253-145.eu-north-1.compute.internal" LV_NAME="keycloak-postgresql-ip-10-0-253-145.eu-north-1.compute.internal"
      NODE_NAME="ip-10-0-3-126.eu-north-1.compute.internal" LV_NAME="keycloak-postgresql-ip-10-0-3-126.eu-north-1.compute.internal"
      

   5. For each NODE_NAME returned by the previous command

      1. Connect to the proper node

         Note

         The NODE_NAME is the first column in the previous command output.

      2. Set the LV_NAME variable

         Note

         Replace <LV_NAME> by the name of the actual Logical Volume (LV) identified in the second column returned in the previous command output.

         Example:

         export LV_NAME="keycloak-postgresql-ip-10-0-211-26.eu-north-1.compute.internal"

         [artesca-os@bootstrap ~]$
         

         export LV_NAME="<LV_NAME>"
         

      3. Continue by checking the available space in the Volume Group (VG) (just below).

Check the Available Space in the Volume Group (VG)

1. Get the free space available of vgartesca Volume Group (VG)

   [artesca-os@bootstrap ~]$
   

   sudo vgs /dev/vgartesca --unit g -o free
   

   VFree
   600.00g
   

2. Evaluate which procedure to use based on the available space

   1. If the available space is sufficient to extend the Logical Volume (LV) to the desired size, follow the Simple Volume Resizing procedure just below.

   2. If the available space IS NOT enough to extend the Logical Volume (LV) to the desired size, follow the Extend Volume By Adding a New Disks procedure down in this page.

Simple Volume Resizing

Note

Check the Available Space in the Volume Group (VG) to make sure that the Volume Group (VG) has enough free space to extend the Logical Volume (LV).

1. Set the LV_EXPANSION_SIZE variable

   Important

   Adapt the LV_EXPANSION_SIZE value to fit your needs.

   Estimations can be done by observing the volume usage over time combined with data retention set to the given service.

   If you are not sure about the size you should use to extend the volume, contact Scality support.

   [artesca-os@bootstrap ~]$
   

   export LV_EXPANSION_SIZE="10G"
   

2. Resize the Logical Volume (LV).

   [artesca-os@bootstrap ~]$
   

   sudo lvextend --size +$LV_EXPANSION_SIZE --resizefs /dev/vgartesca/$LV_NAME
   

   Size of logical volume vgartesca/<LV_NAME> changed from 20,00 GiB (2560 extents) to 30,00 GiB (5120 extents).
   Logical volume vgartesca/<LV_NAME> successfully resized.
   resize2fs 1.42.9 (28-Dec-2013)
   Filesystem at /dev/mapper/vgartesca-<LV_NAME> is mounted on <MOUNTPOINT>; on-line resizing required
   old_desc_blocks = 2, new_desc_blocks = 3
   The filesystem on /dev/mapper/vgartesca-<LV_NAME> is now 5242880 blocks long.
   

3. Patch the Volume object.

   Important

   This step is mandatory to ensure that the volume size is correctly updated in the Kubernetes API.

   [artesca-os@bootstrap ~]$
   

   export LV_SIZE="$(sudo lvs --unit g -o size /dev/vgartesca/$LV_NAME --noheading --nosuffix | cut -d. -f1 | awk '{print $1}')G" && \
   sudo kubectl --kubeconfig /etc/kubernetes/admin.conf patch volumes $LV_NAME --type json --patch \
   "[{\"op\":\"replace\",\"path\":\"/spec/lvmLogicalVolume/size\",\"value\":\"$LV_SIZE\"}]"
   

   volume.storage.metalk8s.scality.com/<LV_NAME> patched
   

Extend Volume By Adding a New Disks

1. Plug the new disk into your server

   Important

   The Service Disk Preparation procedure will guide you to correctly set up the new disk and know its path to be used in next steps.

   If you are in doubt about the above procedure, contact Scality support.

2. Set the NEW_DISK variable

   Note

   Replace <PATH_TO_NEW_DISK> by the actual path to the new disk you found in previous step.

   Example 1: export NEW_DISK="/dev/sdb"

   Example 2: export NEW_DISK="/dev/nvme0n1"

   [artesca-os@bootstrap ~]$
   

   export NEW_DISK="<PATH_TO_NEW_DISK>"
   

3. Create a new Physical Volume (PV).

   [artesca-os@bootstrap ~]$
   

   sudo pvcreate $NEW_DISK
   

     Physical volume "<NEW_DISK>" successfully created.
   

4. Extend vgartesca Volume Group (VG).

   [artesca-os@bootstrap ~]$
   

   sudo vgextend vgartesca $NEW_DISK
   

     Volume group "vgartesca" successfully extended
   

5. Resize the Logical Volume (LV)

   Continue by following the Simple Volume Resizing procedure above.