Replace a Service Disk¶
Note
In the following commands, replace <NODE_NAME> by the name of the
faulty node.
Software RAID¶
Important
Plese note that devide id, partition number and disk name are
only examples. You need to replace them with the actual values of your
system.
The commands provided in this section are dangerous and can cause data loss.
Find the RAID arrays that must be repaired.
[artesca-os@<NODE_NAME> ~]$md_diag() {\ ROOT_BLK_DEV=$(sudo df "$1" --output=source | tail -n1); \ echo "Root block device: $ROOT_BLK_DEV"; \ ROOT_BLK_TYPE=$(sudo lsblk -no TYPE $ROOT_BLK_DEV); \ echo "Root block device type: $ROOT_BLK_TYPE"; \ if [ "$ROOT_BLK_TYPE" != "lvm" ]; then \ echo "Root block device is not an LVM volume"; \ echo "It differs to recommendations in Artesca Documentation"; \ echo "Please contact Artesca Support"; \ exit 1; \ fi; \ ROOT_VG=$(basename $ROOT_BLK_DEV | cut -d "-" -f 1); \ echo "Root volume group: $ROOT_VG"; \ declare -a ROOT_PVS=$(sudo pvs --reportformat json | jq -r ".report[] | .pv[] | select(.vg_name == \"$ROOT_VG\") | (.pv_name)"); \ echo "Root physical volumes: $(echo $ROOT_PVS | xargs)"; \ ROOT_PVS_TYPES=$(sudo lsblk -nldo TYPE ${ROOT_PVS[@]} | sort -u); \ if [ "$ROOT_PVS_TYPES" != "raid1" ]; then \ echo "Root physical volumes are not RAID1 Arrays"; \ echo "It differs to recommendations in Artesca Documentation"; \ echo "Please contact Artesca Support"; \ exit 1; \ fi; \ echo "Root physical volumes types: $ROOT_PVS_TYPES"; \ declare -a ROOT_DISKS=$(echo "$(for md in $ROOT_PVS; do sudo lsblk -o NAME,TYPE -J | jq -r ".blockdevices[] | select(.type == \"disk\" and .children != null and ((.children[].children != null and .children[].children[]?.name == \"$(echo $md | sed 's/\/dev\///g')\") or (.children[]?.name == \"$(echo $md | sed 's/\/dev\///g')\"))) | (.name)"; done)" | sort -u); \ echo "Root disks: $(echo $ROOT_DISKS | xargs)"; \ declare -a ALL_MDS=$(for disk in $ROOT_DISKS; do sudo lsblk -o NAME,TYPE -J /dev/$disk | jq -r '.blockdevices[] | select(.children != null) | .children[] | select(.type == "part" or .type == "raid1") | (.name)'; done | sort -u); \ echo "All md devices using root disks: $(echo $ALL_MDS | xargs)"; \ declare -a MDS_TO_REPAIR=$(for md in $ALL_MDS; do echo "/dev/$md"; done); \ for md in ${MDS_TO_REPAIR[@]}; do \ MOUNTPOINTS=$(sudo lsblk -o NAME,MOUNTPOINT -J $md | jq -r '.blockdevices[] | if .mountpoint != null then . else .children[] | select(.mountpoint != null) end | (.name + " mounted at " + .mountpoint)'); \ if [ -z "$MOUNTPOINTS" ]; then \ echo "No mountpoints found for $md"; \ else \ echo "Mountpoints using $md: $(echo $MOUNTPOINTS | xargs)"; \ fi; \ DEVICES=$(sudo mdadm -v --detail --scan $md | awk -F= '/^[ ]+devices/ {print $2}'); \ echo "Device or partition using $md: $(echo $DEVICES | xargs)"; \ done; \ echo "RAID arrays to repair: $(echo $MDS_TO_REPAIR | xargs)"; \ }; \ md_diag "/dev/vgartesca/$(ls /dev/vgartesca/ | head -n 1)"
Root block device: /dev/mapper/vgartesca-keycloak-postgresql-node2 Root block device type: lvm Root volume group: vgartesca Root physical volumes: /dev/md124 Root physical volumes types: raid1 Root disks: nvme0n3 All md devices using root disks: md124 Mountpoints using /dev/md124: vgartesca-varlibetcd mounted at /var/lib/etcd Device or partition using /dev/md124: /dev/nvme0n3 RAID arrays to repair: /dev/md124
Note
We are interested to the
RAID arrays to repairand values.Repair RAID array:
Identify the failed logical drive (State = degraded).
You have to remove the faulty disk from RAID array find in the previous step.
Note
- The disk can either:
be absent (removed) if it is no longer detected.
be in a faulty state. In this case, you have to manually remove it using the example command below.
[artesca-os@<NODE_NAME> ~]$sudo mdadm --detail /dev/md124
/dev/md124: Version : 1.2 Creation Time : Thu Sep 5 13:44:01 2024 Raid Level : raid1 Array Size : 2147351552 (2047.87 GiB 2198.89 GB) Used Dev Size : 2147351552 (2047.87 GiB 2198.89 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Sep 6 14:49:25 2024 State : active Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Consistency Policy : bitmap Name : artesca-node:artesca (local to host artesca-node) UUID : fb6a67d5:5588a6f9:44de82f9:b97deb65 Events : 109828 Number Major Minor RaidDevice State 3 259 3 0 active sync /dev/nvme0n3 - 0 0 1 removed 1 253 240 - faulty /dev/nvme0n4[artesca-os@<NODE_NAME> ~]$sudo mdadm --manage --remove /dev/md124 /dev/nvme0n4
Replace the faulty physical disk.
Add the new physical disk.
[artesca-os@<NODE_NAME> ~]$sudo mdadm --manage --add /dev/md124 /dev/nvme0n5
Check the RAID being recovered.
[artesca-os@<NODE_NAME> ~]$cat /proc/mdstatPersonalities : [raid1] md124 : active raid1 nvme0n5[1] nvme0n3[0] 501926912 blocks super 1.2 [2/1] [U_] [=>...................] recovery = 5.0% (10539648/209583104) finish=45.9min speed=72272K/sec bitmap: 1/2 pages [4KB], 65536KB chunk
Tips
After modification, MD RAID synchronizes the whole disks to ensure the level of data protection that you want. This initial synchronization can be sped up by increasing the maximum sync speed (useful on high speed NVMe storage):
$cat /proc/sys/dev/raid/speed_limit_max > ./speed_limit.bkp && \ echo 10000000 | sudo tee /proc/sys/dev/raid/speed_limit_max
To reduce the impact on CPU usage during runtime, switch the speed limit back to its original value once the initial sync has been completed:
$cat ./speed_limit.bkp | sudo tee /proc/sys/dev/raid/speed_limit_max && \ rm -f ./speed_limit.bkp
Hardware RAID¶
Detect the RAID controller used on your machine.
[artesca-os@<NODE_NAME> ~]$
lspci | grep -i -e 'raid' -e 'adaptec'
# Example output for Adaptec Smart:
5c:00.0 Serial Attached SCSI controller: Adaptec Smart Storage PQI SAS (rev 01)
# Example output for MegaRAID (SAS-3):
0d:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)
# Example output for MegaRAID (SAS40xx):
10:00.0 RAID bus controller: Broadcom / LSI Fusion-MPT 24GSAS/PCIe SAS40xx (rev 01)
Note
Dell servers also have MegaRAID controller. However, the management tool is different. Use the Dell RAID controller procedure.
Important
The following steps assume that RAID1 mode is being used.
Ensure the RAID controller CLI is installed.
[artesca-os@<NODE_NAME> ~]$sudo dnf install ssacli
Retrieve the RAID controller index (
CONTROLLER_INDEX, depending on specific hardware, will usually be0or1).[artesca-os@<NODE_NAME> ~]$sudo ssacli ctrl all show
HPE Smart Array P816i-a SR Gen10 in Slot 0 (Embedded) (sn: PWXLA0CRHF10J9) # Note: CONTROLLER_INDEX is the value after "Slot" in the output.
Set the
CONTROLLER_INDEXvariable.Replace
<CONTROLLER_INDEX>placeholder with the value ofCtlcolumn in the output from the previous step.[artesca-os@<NODE_NAME> ~]$export CONTROLLER_INDEX=<CONTROLLER_INDEX>
Identify the failed physical drive’s ID.
[artesca-os@<NODE_NAME> ~]$sudo ssacli ctrl slot=$CONTROLLER_INDEX pd all show status
physicaldrive 1I:1:14 (port 1I:box 1:bay 14, 6 TB): Failed *** physicaldrive 2I:1:2 (port 2I:box 1:bay 6, 1 TB): OK physicaldrive 1I:1:49 (port 1I:box 1:bay 49, 6 TB): OK physicaldrive 1I:1:15 (port 1I:box 1:bay 15, 6 TB): OK # Note: The line we are looking for is the one ended by "Failed". # We need to take note of FAILED_DRIVE_ID value: the set of # values separated by colons (":") in the faulty line. # (e.g. 1I:1:14)Set the
FAILED_DRIVE_IDvariable.Replace
<FAILED_DRIVE_ID>placeholder with the value from the previous step.[artesca-os@<NODE_NAME> ~]$export FAILED_DRIVE_ID=<FAILED_DRIVE_ID>
Run the following command for the faulty physical disk’s LED to blink.
[artesca-os@<NODE_NAME> ~]$sudo ssacli controller slot=$CONTROLLER_INDEX pd $FAILED_DRIVE_ID modify led=on
Replace the faulty physical disk.
Run the following commands for the system to detect the new disk and check its availability, only if both disks are failed.
Confirm the new disks are listed (in the
Unassignedsection).[artesca-os@<NODE_NAME> ~]$sudo ssacli ctrl slot=$CONTROLLER_INDEX pd all show
[...] Unassigned physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS HDD, 2 TB, OK) physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS HDD, 2 TB, OK) # Note: The lines we are looking for is the one below "Unassigned". # We need to take note of NEW_DRIVE_IDS value: the set of # values separated by colons (":") in the faulty line.Set the
NEW_DRIVE_IDSvariable.Replace
<NEW_DRIVE_IDS>placeholder with the value from the previous step separated by comma like'1I:1:1,1I:1:2'.[artesca-os@<NODE_NAME> ~]$export NEW_DRIVE_IDS=<NEW_DRIVE_IDS>
Identify the failed logical drive. It is the one with the status NOT OK.
[artesca-os@<NODE_NAME> ~]$sudo ssacli ctrl slot=$CONTROLLER_INDEX ld all show status
logicaldrive 1 (5.46 TB, RAID 1): **NOT OK** logicaldrive 2 (5.46 TB, RAID 0): OK logicaldrive 3 (5.46 TB, RAID 0): OK # Note: The line we are looking for is the one ended by NOT OK. # We need to take note of FAILED_LOGICAL_DRIVE value: # the number that appears next to "logicaldrive" in the faulty line.
Set the
FAILED_LOGICAL_DRIVEvariable to the value of the faulty drive.Replace the
<FAILED_LOGICAL_DRIVE>placeholder with the value of the faulty drive.[artesca-os@<NODE_NAME> ~]$export FAILED_LOGICAL_DRIVE=<FAILED_LOGICAL_DRIVE>
Delete the original logical drive.
[artesca-os@<NODE_NAME> ~]$sudo ssacli ctrl slot=$CONTROLLER_INDEX ld $FAILED_LOGICAL_DRIVE delete
Re-create the new logical drive.
[artesca-os@<NODE_NAME> ~]$sudo ssacli ctrl slot=$CONTROLLER_INDEX create type=ld drives=$NEW_DRIVE_IDS raid=1
Important
The following steps assume that RAID1 mode is being used.
Ensure the RAID controller CLI is installed.
[artesca-os@<NODE_NAME> ~]$sudo dnf install storcli
Retrieve the RAID controller index (
CONTROLLER_INDEX, depending on specific hardware, will usually be0or1).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 show
-------------------------------------------------------------------- Ctl Model Ports PDs DGs DNOpt VDs VNOpt BBU sPR DS EHS ASOs Hlth -------------------------------------------------------------------- 0 SAS3916 16 12 12 0 12 0 N/A On 1&2 Y 4 Opt --------------------------------------------------------------------
Set the
CONTROLLER_INDEXvariable.Replace
<CONTROLLER_INDEX>placeholder with the value ofCtlcolumn in the output from the previous step.[artesca-os@<NODE_NAME> ~]$export CONTROLLER_INDEX=<CONTROLLER_INDEX>
Identify the failed physical drive’s ID.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 /c$CONTROLLER_INDEX /eall /sall show
----------------------------------------------------------------------------- EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type ----------------------------------------------------------------------------- 252:0 50 Offln 0 14.551 TB SAS SSD N N 512B MG08SCA16TE U - *** 252:1 51 Onln 1 14.551 TB SAS SSD N N 512B MG08SCA16TE U - 252:2 52 Onln 2 14.551 TB SAS SSD N N 512B MG08SCA16TE U - 252:3 41 Onln 3 14.551 TB SAS SSD N N 512B MG08SCA16TE U - ----------------------------------------------------------------------------- # Note: The line we are looking for is the one with "Offln" in the "State" column. # We need to take note of two values: # - FAILED_ENCLOSURE_DEVICE_ID: The value of "EID" column. (e.g. 252) # - FAILED_SLOT_NUMBER: The value of "Slt" column. (e.g. 0)
Set the
FAILED_ENCLOSURE_DEVICE_IDandFAILED_SLOT_NUMBERvariables.Replace
<FAILED_ENCLOSURE_DEVICE_ID>and<FAILED_SLOT_NUMBER>placeholders with the values from the previous step.[artesca-os@<NODE_NAME> ~]$export FAILED_ENCLOSURE_DEVICE_ID=<FAILED_ENCLOSURE_DEVICE_ID> FAILED_SLOT_NUMBER=<FAILED_SLOT_NUMBER>
Run the following command for the faulty physical disk’s LED to blink.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 /c$CONTROLLER_INDEX /e$FAILED_ENCLOSURE_DEVICE_ID /s$FAILED_SLOT_NUMBER start locate
Replace the faulty physical disk.
Run the following commands for the system to detect the new disk and check its availability, only if both disks are failed.
Confirm the new disk is listed (with
UGoodstatus, Unconfigured Good).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 /c$CONTROLLER_INDEX /eall /sall show
----------------------------------------------------------------------------- EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type ----------------------------------------------------------------------------- 252:0 50 Offln 0 14.551 TB SAS SSD N N 512B MG08SCA16TE U - 252:1 51 Offln 1 14.551 TB SAS SSD N N 512B MG08SCA16TE U - 252:2 52 Onln 2 14.551 TB SAS HDD N N 512B MG08SCA16TE U - 252:3 41 Onln 3 14.551 TB SAS HDD N N 512B MG08SCA16TE U - 252:10 48 UGood - 14.551 TB SAS SSD N N 512B MG08SCA16TE U - *** 252:11 49 UGood - 14.551 TB SAS SSD N N 512B MG08SCA16TE U - *** # Note: The lines we are looking for are the ones having "UGood" in the "State" column. # We need to take note of two values: # - NEW_ENCLOSURE_DEVICE_ID_[1..2]: The values of "EID" column. # - NEW_SLOT_NUMBER_[1..2]: The values of "Slt" column.
Set the
NEW_ENCLOSURE_DEVICE_IDandNEW_SLOT_NUMBERvariables to the values of the new disk.Replace the
<NEW_ENCLOSURE_DEVICE_ID_[1..2]>and<NEW_SLOT_NUMBER_[1..2]>placeholders with the values of the new disk.[artesca-os@<NODE_NAME> ~]$export NEW_ENCLOSURE_DEVICE_ID_1=<NEW_ENCLOSURE_DEVICE_ID_1> NEW_SLOT_NUMBER_1=<NEW_SLOT_NUMBER_1> NEW_ENCLOSURE_DEVICE_ID_2=<NEW_ENCLOSURE_DEVICE_ID_2> NEW_SLOT_NUMBER_2=<NEW_SLOT_NUMBER_2>
Identify the failed logical drive. It is the one with the status Ofln.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 /c$CONTROLLER_INDEX /vall show
--------------------------------------------------------------- DG/VD TYPE State Access Consist Cache Cac sCC Size Name --------------------------------------------------------------- 2/237 RAID0 Optl RW Yes NRWTD - ON 14.551 TB 1/238 RAID0 Optl RW Yes NRWTD - ON 14.551 TB 0/239 RAID1 Ofln RW No RWTD - ON 14.551 TB *** --------------------------------------------------------------- # Note: The line we are interested in is the one with the status "Ofln". # We need to take note of FAILED_DRIVE_ID value: the number that appears # in VD column in the faulty line.
Set the
FAILED_DRIVE_IDvariable to the value of the faulty drive.Replace the
<FAILED_DRIVE_ID>placeholder with the value of the faulty drive.[artesca-os@<NODE_NAME> ~]$export FAILED_DRIVE_ID=<FAILED_DRIVE_ID>
Delete the original logical drive.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 /c$CONTROLLER_INDEX /v$FAILED_DRIVE_ID delete
Re-create the new logical drive.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 /c$CONTROLLER_INDEX add vd raid1 \ drives="$NEW_ENCLOSURE_DEVICE_ID_1:$NEW_SLOT_NUMBER_1,$NEW_ENCLOSURE_DEVICE_ID_2:$NEW_SLOT_NUMBER_2"
Important
The following steps assume that RAID1 mode is being used.
Ensure the RAID controller CLI is installed.
[artesca-os@<NODE_NAME> ~]$sudo dnf install storcli2
Retrieve the RAID controller index (
CONTROLLER_INDEX, depending on specific hardware, will usually be0or1).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 show
----------------------------------------------------------------------------------------- Ctrl Product Name SASAddress Personality Status PD(s) VD(s) VNOpt EPack ----------------------------------------------------------------------------------------- 0 SAS3916 Adapter 0XXXXXXX RAID Optimal 12 12 0 Optimal -----------------------------------------------------------------------------------------
Set the
CONTROLLER_INDEXvariable.Replace
<CONTROLLER_INDEX>placeholder with the value ofCtlcolumn in the output from the previous step.[artesca-os@<NODE_NAME> ~]$export CONTROLLER_INDEX=<CONTROLLER_INDEX>
Identify the failed physical drive’s ID.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 /c$CONTROLLER_INDEX /eall /sall show
----------------------------------------------------------------------------------------------------- EID:Slt PID State Status DG Size Intf Med SED_Type SeSz Model Sp LU/NS Count Alt-EID ----------------------------------------------------------------------------------------------------- 288:0 275 Conf Offline 0 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - *** 288:1 276 Conf Online 1 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - 288:2 277 Conf Online 2 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - 288:3 278 Conf Online 3 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - ----------------------------------------------------------------------------------------------------- # Note: The line we are looking for is the one with "Offline" in the "Status" column. # We need to take note of two values: # - FAILED_ENCLOSURE_DEVICE_ID: The value of "EID" column. (e.g. 288) # - FAILED_SLOT_NUMBER: The value of "Slt" column. (e.g. 0)
Set the
FAILED_ENCLOSURE_DEVICE_IDandFAILED_SLOT_NUMBERvariables.Replace
<FAILED_ENCLOSURE_DEVICE_ID>and<FAILED_SLOT_NUMBER>placeholders with the values from the previous step.[artesca-os@<NODE_NAME> ~]$export FAILED_ENCLOSURE_DEVICE_ID=<FAILED_ENCLOSURE_DEVICE_ID> FAILED_SLOT_NUMBER=<FAILED_SLOT_NUMBER>
Run the following command for the faulty physical disk’s LED to blink.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 /c$CONTROLLER_INDEX /e$FAILED_ENCLOSURE_DEVICE_ID /s$FAILED_SLOT_NUMBER start locate
Replace the faulty physical disk.
Run the following commands for the system to detect the new disk and check its availability, only if both disks are failed.
Confirm the new disk is listed (with
UGoodstatus, Unconfigured Good).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 /c$CONTROLLER_INDEX /eall /sall show
----------------------------------------------------------------------------------------------------- EID:Slt PID State Status DG Size Intf Med SED_Type SeSz Model Sp LU/NS Count Alt-EID ----------------------------------------------------------------------------------------------------- 288:0 275 Conf Offline 0 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - 288:1 276 Conf Offline 1 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - 288:2 277 Conf Online 2 7.276 TiB SAS HDD - 512B ST8000NM024B U 1 - 288:3 278 Conf Online 3 7.276 TiB SAS HDD - 512B ST8000NM024B U 1 - 288:10 285 UConf Online - 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - *** 288:11 286 UConf Online - 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - *** ----------------------------------------------------------------------------------------------------- # Note: The lines we are looking for are the ones having "UGood" in the "State" column. # We need to take note of two values: # - NEW_ENCLOSURE_DEVICE_ID_[1..2]: The values of "EID" column. # - NEW_SLOT_NUMBER_[1..2]: The values of "Slt" column.
Set the
NEW_ENCLOSURE_DEVICE_IDandNEW_SLOT_NUMBERvariables to the values of the new disk.Replace the
<NEW_ENCLOSURE_DEVICE_ID_[1..2]>and<NEW_SLOT_NUMBER_[1..2]>placeholders with the values of the new disk.[artesca-os@<NODE_NAME> ~]$export NEW_ENCLOSURE_DEVICE_ID_1=<NEW_ENCLOSURE_DEVICE_ID_1> NEW_SLOT_NUMBER_1=<NEW_SLOT_NUMBER_1> NEW_ENCLOSURE_DEVICE_ID_2=<NEW_ENCLOSURE_DEVICE_ID_2> NEW_SLOT_NUMBER_2=<NEW_SLOT_NUMBER_2>
Identify the failed logical drive. It is the one with the status Ofln.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 /c$CONTROLLER_INDEX /vall show
------------------------------------------------------------------ DG/VD TYPE State Access CurrentCache DefaultCache Size Name ------------------------------------------------------------------ 0/1 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 1/2 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 11/12 RAID0 Ofln RW R,WT R,WT 7.276 TiB *** ------------------------------------------------------------------ # Note: The line we are interested in is the one with the status "Ofln". # We need to take note of FAILED_DRIVE_ID value: the number that appears # in VD column in the faulty line.
Set the
FAILED_DRIVE_IDvariable to the value of the faulty drive.Replace the
<FAILED_DRIVE_ID>placeholder with the value of the faulty drive.[artesca-os@<NODE_NAME> ~]$export FAILED_DRIVE_ID=<FAILED_DRIVE_ID>
Delete the original logical drive.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 /c$CONTROLLER_INDEX /v$FAILED_DRIVE_ID delete
Re-create the new logical drive.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 /c$CONTROLLER_INDEX add vd raid1 \ drives="$NEW_ENCLOSURE_DEVICE_ID_1:$NEW_SLOT_NUMBER_1,$NEW_ENCLOSURE_DEVICE_ID_2:$NEW_SLOT_NUMBER_2"
Important
The following steps assume that RAID1 mode is being used.
Ensure the RAID controller CLI is installed.
[artesca-os@<NODE_NAME> ~]$sudo dnf install perccli
Retrieve the RAID controller index (
CONTROLLER_INDEX, depending on specific hardware, will usually be0or1).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 show
-------------------------------------------------------------------- Ctl Model Ports PDs DGs DNOpt VDs VNOpt BBU sPR DS EHS ASOs Hlth -------------------------------------------------------------------- 0 PERC H730 16 12 12 0 12 0 N/A On 1&2 Y 4 Opt --------------------------------------------------------------------
Set the
CONTROLLER_INDEXvariable.Replace
<CONTROLLER_INDEX>placeholder with the value ofCtlcolumn in the output from the previous step.[artesca-os@<NODE_NAME> ~]$export CONTROLLER_INDEX=<CONTROLLER_INDEX>
Identify the failed physical drive’s ID.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 /c$CONTROLLER_INDEX /eall /sall show
----------------------------------------------------------------------------- EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type ----------------------------------------------------------------------------- 252:0 50 Offln 0 14.551 TB SAS SSD N N 512B MG08SCA16TE U - *** 252:1 51 Onln 1 14.551 TB SAS SSD N N 512B MG08SCA16TE U - 252:2 52 Onln 2 14.551 TB SAS SSD N N 512B MG08SCA16TE U - 252:3 41 Onln 3 14.551 TB SAS SSD N N 512B MG08SCA16TE U - ----------------------------------------------------------------------------- # Note: The line we are looking for is the one with "Offln" in the "State" column. # We need to take note of two values: # - FAILED_ENCLOSURE_DEVICE_ID: The value of "EID" column. (e.g. 252) # - FAILED_SLOT_NUMBER: The value of "Slt" column. (e.g. 0)
Set the
FAILED_ENCLOSURE_DEVICE_IDandFAILED_SLOT_NUMBERvariables.Replace
<FAILED_ENCLOSURE_DEVICE_ID>and<FAILED_SLOT_NUMBER>placeholders with the values from the previous step.[artesca-os@<NODE_NAME> ~]$export FAILED_ENCLOSURE_DEVICE_ID=<FAILED_ENCLOSURE_DEVICE_ID> FAILED_SLOT_NUMBER=<FAILED_SLOT_NUMBER>
Run the following command for the faulty physical disk’s LED to blink.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 /c$CONTROLLER_INDEX /e$FAILED_ENCLOSURE_DEVICE_ID /s$FAILED_SLOT_NUMBER start locate
Replace the faulty physical disk.
Run the following commands for the system to detect the new disk and check its availability, only if both disks are failed.
Confirm the new disk is listed (with
UGoodstatus, Unconfigured Good).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 /c$CONTROLLER_INDEX /eall /sall show
----------------------------------------------------------------------------- EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type ----------------------------------------------------------------------------- 252:0 50 Offln 0 14.551 TB SAS SSD N N 512B MG08SCA16TE U - 252:1 51 Offln 1 14.551 TB SAS SSD N N 512B MG08SCA16TE U - 252:2 52 Onln 2 14.551 TB SAS HDD N N 512B MG08SCA16TE U - 252:3 41 Onln 3 14.551 TB SAS HDD N N 512B MG08SCA16TE U - 252:10 48 UGood - 14.551 TB SAS SSD N N 512B MG08SCA16TE U - *** 252:11 49 UGood - 14.551 TB SAS SSD N N 512B MG08SCA16TE U - *** # Note: The lines we are looking for are the ones having "UGood" in the "State" column. # We need to take note of two values: # - NEW_ENCLOSURE_DEVICE_ID_[1..2]: The values of "EID" column. # - NEW_SLOT_NUMBER_[1..2]: The values of "Slt" column.
Set the
NEW_ENCLOSURE_DEVICE_IDandNEW_SLOT_NUMBERvariables to the values of the new disk.Replace the
<NEW_ENCLOSURE_DEVICE_ID_[1..2]>and<NEW_SLOT_NUMBER_[1..2]>placeholders with the values of the new disk.[artesca-os@<NODE_NAME> ~]$export NEW_ENCLOSURE_DEVICE_ID_1=<NEW_ENCLOSURE_DEVICE_ID_1> NEW_SLOT_NUMBER_1=<NEW_SLOT_NUMBER_1> NEW_ENCLOSURE_DEVICE_ID_2=<NEW_ENCLOSURE_DEVICE_ID_2> NEW_SLOT_NUMBER_2=<NEW_SLOT_NUMBER_2>
Identify the failed logical drive. It is the one with the status Ofln.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 /c$CONTROLLER_INDEX /vall show
--------------------------------------------------------------- DG/VD TYPE State Access Consist Cache Cac sCC Size Name --------------------------------------------------------------- 2/237 RAID0 Optl RW Yes NRWTD - ON 14.551 TB 1/238 RAID0 Optl RW Yes NRWTD - ON 14.551 TB 0/239 RAID1 Ofln RW No RWTD - ON 14.551 TB *** --------------------------------------------------------------- # Note: The line we are interested in is the one with the status "Ofln". # We need to take note of FAILED_DRIVE_ID value: the number that appears # in VD column in the faulty line.
Set the
FAILED_DRIVE_IDvariable to the value of the faulty drive.Replace the
<FAILED_DRIVE_ID>placeholder with the value of the faulty drive.[artesca-os@<NODE_NAME> ~]$export FAILED_DRIVE_ID=<FAILED_DRIVE_ID>
Delete the original logical drive.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 /c$CONTROLLER_INDEX /v$FAILED_DRIVE_ID delete
Re-create the new logical drive.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 /c$CONTROLLER_INDEX add vd raid1 \ drives="$NEW_ENCLOSURE_DEVICE_ID_1:$NEW_SLOT_NUMBER_1,$NEW_ENCLOSURE_DEVICE_ID_2:$NEW_SLOT_NUMBER_2"
Important
The following steps assume that RAID1 mode is being used.
Ensure the RAID controller CLI is installed.
[artesca-os@<NODE_NAME> ~]$sudo dnf install perccli2
Retrieve the RAID controller index (
CONTROLLER_INDEX, depending on specific hardware, will usually be0or1).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 show
----------------------------------------------------------------------------------------- Ctrl Product Name SASAddress Personality Status PD(s) VD(s) VNOpt EPack ----------------------------------------------------------------------------------------- 0 PERC H965i Adapter 0XXXXXXXX RAID Optimal 12 12 0 Optimal -----------------------------------------------------------------------------------------
Set the
CONTROLLER_INDEXvariable.Replace
<CONTROLLER_INDEX>placeholder with the value ofCtlcolumn in the output from the previous step.[artesca-os@<NODE_NAME> ~]$export CONTROLLER_INDEX=<CONTROLLER_INDEX>
Identify the failed physical drive’s ID.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 /c$CONTROLLER_INDEX /eall /sall show
----------------------------------------------------------------------------------------------------- EID:Slt PID State Status DG Size Intf Med SED_Type SeSz Model Sp LU/NS Count Alt-EID ----------------------------------------------------------------------------------------------------- 288:0 275 Conf Offline 0 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - *** 288:1 276 Conf Online 1 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - 288:2 277 Conf Online 2 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - 288:3 278 Conf Online 3 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - ----------------------------------------------------------------------------------------------------- # Note: The line we are looking for is the one with "Offline" in the "Status" column. # We need to take note of two values: # - FAILED_ENCLOSURE_DEVICE_ID: The value of "EID" column. (e.g. 288) # - FAILED_SLOT_NUMBER: The value of "Slt" column. (e.g. 0)
Set the
FAILED_ENCLOSURE_DEVICE_IDandFAILED_SLOT_NUMBERvariables.Replace
<FAILED_ENCLOSURE_DEVICE_ID>and<FAILED_SLOT_NUMBER>placeholders with the values from the previous step.[artesca-os@<NODE_NAME> ~]$export FAILED_ENCLOSURE_DEVICE_ID=<FAILED_ENCLOSURE_DEVICE_ID> FAILED_SLOT_NUMBER=<FAILED_SLOT_NUMBER>
Run the following command for the faulty physical disk’s LED to blink.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 /c$CONTROLLER_INDEX /e$FAILED_ENCLOSURE_DEVICE_ID /s$FAILED_SLOT_NUMBER start locate
Replace the faulty physical disk.
Run the following commands for the system to detect the new disk and check its availability, only if both disks are failed.
Confirm the new disk is listed (with
UGoodstatus, Unconfigured Good).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 /c$CONTROLLER_INDEX /eall /sall show
----------------------------------------------------------------------------------------------------- EID:Slt PID State Status DG Size Intf Med SED_Type SeSz Model Sp LU/NS Count Alt-EID ----------------------------------------------------------------------------------------------------- 288:0 275 Conf Offline 0 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - 288:1 276 Conf Offline 1 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - 288:2 277 Conf Online 2 7.276 TiB SAS HDD - 512B ST8000NM024B U 1 - 288:3 278 Conf Online 3 7.276 TiB SAS HDD - 512B ST8000NM024B U 1 - 288:10 285 UConf Online - 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - *** 288:11 286 UConf Online - 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - *** ----------------------------------------------------------------------------------------------------- # Note: The lines we are looking for are the ones having "UGood" in the "State" column. # We need to take note of two values: # - NEW_ENCLOSURE_DEVICE_ID_[1..2]: The values of "EID" column. # - NEW_SLOT_NUMBER_[1..2]: The values of "Slt" column.
Set the
NEW_ENCLOSURE_DEVICE_IDandNEW_SLOT_NUMBERvariables to the values of the new disk.Replace the
<NEW_ENCLOSURE_DEVICE_ID_[1..2]>and<NEW_SLOT_NUMBER_[1..2]>placeholders with the values of the new disk.[artesca-os@<NODE_NAME> ~]$export NEW_ENCLOSURE_DEVICE_ID_1=<NEW_ENCLOSURE_DEVICE_ID_1> NEW_SLOT_NUMBER_1=<NEW_SLOT_NUMBER_1> NEW_ENCLOSURE_DEVICE_ID_2=<NEW_ENCLOSURE_DEVICE_ID_2> NEW_SLOT_NUMBER_2=<NEW_SLOT_NUMBER_2>
Identify the failed logical drive. It is the one with the status Ofln.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 /c$CONTROLLER_INDEX /vall show
------------------------------------------------------------------ DG/VD TYPE State Access CurrentCache DefaultCache Size Name ------------------------------------------------------------------ 0/1 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 1/2 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 11/12 RAID0 Ofln RW R,WT R,WT 7.276 TiB *** ------------------------------------------------------------------ # Note: The line we are interested in is the one with the status "Ofln". # We need to take note of FAILED_DRIVE_ID value: the number that appears # in VD column in the faulty line.
Set the
FAILED_DRIVE_IDvariable to the value of the faulty drive.Replace the
<FAILED_DRIVE_ID>placeholder with the value of the faulty drive.[artesca-os@<NODE_NAME> ~]$export FAILED_DRIVE_ID=<FAILED_DRIVE_ID>
Delete the original logical drive.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 /c$CONTROLLER_INDEX /v$FAILED_DRIVE_ID delete
Re-create the new logical drive.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 /c$CONTROLLER_INDEX add vd raid1 \ drives="$NEW_ENCLOSURE_DEVICE_ID_1:$NEW_SLOT_NUMBER_1,$NEW_ENCLOSURE_DEVICE_ID_2:$NEW_SLOT_NUMBER_2"
No RAID or All Disks Failed¶
Important
This procedure does NOT work for single-node clusters.
Determine the node type: Infrastructure or Workload-Only.
From the faulty node, display the node name.
[artesca-os@<NODE_NAME> ~]$sudo salt-call --local grains.get id --out txt 2>/dev/null | awk -F ': ' '{print $2}'
Important
Copy and keep this value for use in the next steps.
From the bootstrap node:
Set the
NODE_NAMEvariable.Important
Replace
<NODE_NAME>with the value you found in the beginning of this procedure.[artesca-os@bootstrap ~]$export NODE_NAME="<NODE_NAME>"
Check the type of the faulty node.
[artesca-os@bootstrap ~]$if [ "node" == "$(sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get node "$NODE_NAME" --no-headers | awk '{print $3}')" ]; then echo "Workload-Only"; else echo "Infrastructure"; fi
Note
Possible results are:
InfrastructureorWorkload-Only.
Follow the procedure for the type of node you have:
Important
All the commands below must be executed on the faulty node.
Set the
NODE_NAMEvariable.[artesca-os@<NODE_NAME> ~]$export NODE_NAME=$(sudo salt-call --local grains.get id --out txt 2>/dev/null | awk -F ': ' '{print $2}')
Set the
NODE_IPvariable.[artesca-os@<NODE_NAME> ~]$NODE_IP=$(sudo salt-call --local grains.get metalk8s:control_plane_ip --out text 2>/dev/null | awk -F ': ' '{print $2}')
Drain the faulty node.
[artesca-os@<NODE_NAME> ~]$sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ drain --ignore-daemonsets --force \ --delete-emptydir-data $NODE_NAME
Fetch the name of a healthy etcd pod.
[artesca-os@<NODE_NAME> ~]$HEALTHY_ETCD_POD=$(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get pod -n kube-system --selector component=etcd --field-selector="spec.nodeName!=$NODE_NAME,status.phase=Running" --no-headers -o custom-columns=":metadata.name" | head -1)
Get etcd ID of the faulty node’s etcd instance.
[artesca-os@<NODE_NAME> ~]$FAULTY_ID=$(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf exec \ -n kube-system "$HEALTHY_ETCD_POD" -- sh -c " \ ETCDCTL_API=3 etcdctl --endpoints https://127.0.0.1:2379 \ --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ --cacert /etc/kubernetes/pki/etcd/ca.crt \ member list" | grep $NODE_IP | awk -F ', ' '{ print $1 }')
Remove the faulty node from etcd cluster.
[artesca-os@<NODE_NAME> ~]$sudo kubectl --kubeconfig /etc/kubernetes/admin.conf exec \ -n kube-system -ti "$HEALTHY_ETCD_POD" -- sh -c " \ ETCDCTL_API=3 etcdctl --endpoints https://127.0.0.1:2379 \ --cert /etc/kubernetes/pki/etcd/server.crt \ --key /etc/kubernetes/pki/etcd/server.key \ --cacert /etc/kubernetes/pki/etcd/ca.crt \ member remove $FAULTY_ID"
Clean up etcd mountpoint and vgartesca LVs using the following two commands:
[artesca-os@<NODE_NAME> ~]$sudo umount /var/lib/etcd
[artesca-os@<NODE_NAME> ~]$for i in $(sudo dmsetup info | grep vgartesca | cut -f2 -d':' | sed -e 's/ //g'); do sudo dmsetup remove $i; done
Prepare your disk following the Service Disk Preparation section.
Prepare vgartesca VG on the drained node.
Tip
The device path to be employed may vary depending on the system, the type of physical device, and whether RAID is utilized.
For instance, when using a single disk, the device path might appear as
/dev/sda,/dev/nvme0n1or/dev/mmcblk0.In the case of using RAID, the device path could manifest as
/dev/md0or/dev/md/mdartesca.Set the
DEVICEvariable.[artesca-os@<NODE_NAME> ~]$DEVICE="<DEVICE>"
Create an LVM Physical Volume on your device:
[artesca-os@<NODE_NAME> ~]$sudo pvcreate /dev/$DEVICE
Create an LVM Volume Group called vgartesca backed by your device:
[artesca-os@<NODE_NAME> ~]$sudo vgcreate vgartesca /dev/$DEVICE
Create and mount etcd LV.
[artesca-os@<NODE_NAME> ~]$sudo lvcreate --name varlibetcd --size 5G vgartesca && \ sudo mkfs.ext4 /dev/vgartesca/varlibetcd && \ sudo mount /var/lib/etcd
Redeploy etcd.
[artesca-os@<NODE_NAME> ~]$sudo salt-call state.sls metalk8s.kubernetes.etcd pillar='{"metalk8s": {"skip_etcd_healthcheck": True}}'
Export the MetalK8s version you are using as an environment variable:
[artesca-os@bootstrap ~]$export MK8SVERSION=`sudo kubectl \ --kubeconfig=/etc/kubernetes/admin.conf \ get cm -n kube-system artesca-version \ -o jsonpath="{.data.metalk8s}"`
Add the node back into etcd cluster.
Note
This might loop with an error for a few minutes, but the command will succeed.
[artesca-os@<NODE_NAME> ~]$sudo kubectl exec -n kube-system -c salt-master \ --kubeconfig /etc/kubernetes/admin.conf \ $(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ get pods -n kube-system -l app=salt-master -o name) \ -- salt-run state.sls metalk8s.orchestrate.register_etcd pillar='{"orchestrate": {"node_name": "'"$NODE_NAME"'"}}' saltenv=metalk8s-$MK8SVERSION
Move the vgartesca LVs to the new PV.
Retrieve all volumes backed by
vgartesca.[artesca-os@<NODE_NAME> ~]$VGARTESCA_VOLUMES=( $( sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get volumes -o go-template=' {{- range .items }} {{- if and (eq .spec.nodeName "'"$NODE_NAME"'") (eq (or .spec.lvmLogicalVolume.vgName "") "vgartesca") }} {{- .metadata.name }}{{ " " }} {{- end }} {{- end }} ') )
Generate a manifest to re-create these volumes.
[artesca-os@<NODE_NAME> ~]$sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get volumes "${VGARTESCA_VOLUMES[@]:-"my-invalid-volume"}" -o json | jq ' { apiVersion: "v1", kind: "List", items: [ (if .items then .items[] else . end)|{ apiVersion : .apiVersion, kind: .kind, metadata: .metadata|{ annotations: .annotations|del(."kubectl.kubernetes.io/last-applied-configuration"), labels: .labels, name: .name }, spec: .spec } ] }' > ./vgartesca-volumes-manifest.json
Remove all Pods and PVCs bound to these volumes.
[artesca-os@<NODE_NAME> ~]$[ -n "$VGARTESCA_VOLUMES" ] && while read -r PVC NAMESPACE; do [ -z "$PVC" ] && continue POD=$( sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ describe pvc --namespace "$NAMESPACE" "$PVC" | \ awk '/^Used By:/{ print $3 }' ) sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ delete pvc --wait=false --namespace "$NAMESPACE" "$PVC" sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ delete pods --namespace "$NAMESPACE" "$POD" done < <( sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ get pv "${VGARTESCA_VOLUMES[@]}" -o json | \ jq -r '{items: [if .items then .items[] else . end]} | .items[] | "\(.spec.claimRef.name) \(.spec.claimRef.namespace)"' )
Remove all PVs backed by these volumes.
[artesca-os@<NODE_NAME> ~]$[ -n "$VGARTESCA_VOLUMES" ] && \ sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ patch pv --patch '{"metadata": {"finalizers": null}}' \ --type=merge "${VGARTESCA_VOLUMES[@]}" && \ sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ delete pv "${VGARTESCA_VOLUMES[@]}"
Apply the manifest to re-create the volumes.
[artesca-os@<NODE_NAME> ~]$sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ apply -f ./vgartesca-volumes-manifest.json
List all data disks UIDs to clean up.
[artesca-os@<NODE_NAME> ~]$DATA_DISK_UIDS=$(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get volumes -o go-template=' {{- range .items}} {{- if and (eq .spec.nodeName "'"$NODE_NAME"'") (eq (index .spec.template.metadata.labels "xcore.scality.com/volume-type") "data") }} {{- .metadata.uid }}{{ " " }} {{- end }} {{- end }} ')
Delete all contents from the data disks.
[artesca-os@<NODE_NAME> ~]$for disk in ${DATA_DISK_UIDS[@]}; do \ sudo kubectl exec -n kube-system -c salt-master \ --kubeconfig /etc/kubernetes/admin.conf \ $(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get pods -n kube-system -l app=salt-master -o name) \ -- salt "$NODE_NAME" cmd.run " \ echo \"Cleaning disk $disk\" && \ shopt -s dotglob && \ mkdir -p /tmp/mpoints/disk-$disk && \ mount /dev/disk/by-uuid/$disk /tmp/mpoints/disk-$disk && \ rm -rf /tmp/mpoints/disk-$disk/* && \ umount /tmp/mpoints/disk-$disk && \ shopt -u dotglob "; \ done
Uncordon the node.
[artesca-os@<NODE_NAME> ~]$sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ uncordon $NODE_NAME
From the bootstrap node:
Set the
NODE_NAMEvariable.Important
Replace
<NODE_NAME>with the value you found in the beginning of this procedure.[artesca-os@bootstrap ~]$export NODE_NAME="<NODE_NAME>"
Drain the faulty node.
[artesca-os@bootstrap ~]$sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ drain --ignore-daemonsets --force \ --delete-emptydir-data $NODE_NAME
From the faulty node:
Prepare your disk following the Service Disk Preparation section.
Prepare vgartesca VG on the drained node.
Tip
The device path to be employed may vary depending on the system, the type of physical device, and whether RAID is utilized.
For instance, when using a single disk, the device path might appear as
/dev/sda,/dev/nvme0n1or/dev/mmcblk0.In the case of using RAID, the device path could manifest as
/dev/md0or/dev/md/mdartesca.Set the
DEVICEvariable.[artesca-os@<NODE_NAME> ~]$DEVICE="<DEVICE>"
Create an LVM Physical Volume on your device:
[artesca-os@<NODE_NAME> ~]$sudo pvcreate /dev/$DEVICE
Create an LVM Volume Group called vgartesca backed by your device:
[artesca-os@<NODE_NAME> ~]$sudo vgcreate vgartesca /dev/$DEVICE
From the bootstrap node:
Move the vgartesca LVs to the new PV.
Set the
NODE_NAMEvariable.Important
Replace
<NODE_NAME>with the value you found in the beginning of this procedure.[artesca-os@bootstrap ~]$export NODE_NAME="<NODE_NAME>"
Retrieve all volumes backed by
vgartesca.[artesca-os@bootstrap ~]$VGARTESCA_VOLUMES=( $( sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get volumes -o go-template=' {{- range .items }} {{- if and (eq .spec.nodeName "'"$NODE_NAME"'") (eq (or .spec.lvmLogicalVolume.vgName "") "vgartesca") }} {{- .metadata.name }}{{ " " }} {{- end }} {{- end }} ') )
Generate a manifest to re-create these volumes.
[artesca-os@bootstrap ~]$sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf get volumes "${VGARTESCA_VOLUMES[@]:-"my-invalid-volume"}" -o json | jq ' { apiVersion: "v1", kind: "List", items: [ (if .items then .items[] else . end)|{ apiVersion : .apiVersion, kind: .kind, metadata: .metadata|{ annotations: .annotations|del(."kubectl.kubernetes.io/last-applied-configuration"), labels: .labels, name: .name }, spec: .spec } ] }' > ./vgartesca-volumes-manifest.json
Remove all Pods and PVCs bound to these volumes.
[artesca-os@bootstrap ~]$[ -n "$VGARTESCA_VOLUMES" ] && while read -r PVC NAMESPACE; do [ -z "$PVC" ] && continue POD=$( sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ describe pvc --namespace "$NAMESPACE" "$PVC" | \ awk '/^Used By:/{ print $3 }' ) sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ delete pvc --wait=false --namespace "$NAMESPACE" "$PVC" sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ delete pods --namespace "$NAMESPACE" "$POD" done < <( sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ get pv "${VGARTESCA_VOLUMES[@]}" -o json | \ jq -r '{items: [if .items then .items[] else . end]} | .items[] | "\(.spec.claimRef.name) \(.spec.claimRef.namespace)"' )
Remove all PVs backed by these volumes.
[artesca-os@bootstrap ~]$[ -n "$VGARTESCA_VOLUMES" ] && \ sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ patch pv --patch '{"metadata": {"finalizers": null}}' \ --type=merge "${VGARTESCA_VOLUMES[@]}" && \ sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ delete pv "${VGARTESCA_VOLUMES[@]}"
Apply the manifest to re-create the volumes.
[artesca-os@bootstrap ~]$sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ apply -f ./vgartesca-volumes-manifest.json
List all data disks UIDs to clean up.
[artesca-os@bootstrap ~]$DATA_DISK_UIDS=$(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get volumes -o go-template=' {{- range .items}} {{- if and (eq .spec.nodeName "'"$NODE_NAME"'") (eq (index .spec.template.metadata.labels "xcore.scality.com/volume-type") "data") }} {{- .metadata.uid }}{{ " " }} {{- end }} {{- end }} ')
Delete all contents from the data disks.
[artesca-os@bootstrap ~]$for disk in ${DATA_DISK_UIDS[@]}; do \ sudo kubectl exec -n kube-system -c salt-master \ --kubeconfig /etc/kubernetes/admin.conf \ $(sudo kubectl --kubeconfig /etc/kubernetes/admin.conf get pods -n kube-system -l app=salt-master -o name) \ -- salt "$NODE_NAME" cmd.run " \ echo \"Cleaning disk $disk\" && \ shopt -s dotglob && \ mkdir -p /tmp/mpoints/disk-$disk && \ mount /dev/disk/by-uuid/$disk /tmp/mpoints/disk-$disk && \ rm -rf /tmp/mpoints/disk-$disk/* && \ umount /tmp/mpoints/disk-$disk && \ shopt -u dotglob "; \ done
Uncordon the node.
[artesca-os@bootstrap ~]$sudo kubectl --kubeconfig /etc/kubernetes/admin.conf \ uncordon $NODE_NAME
Follow the Full Sync a MongoDB Member procedure.
