Replace an OS Disk¶
Note
In the following commands, replace <NODE_NAME> by the name of the
faulty node.
Software RAID¶
Important
Plese note that devide id, partition number and disk name are
only examples. You need to replace them with the actual values of your
system.
The commands provided in this section are dangerous and can cause data loss.
Find the RAID arrays that must be repaired.
[artesca-os@<NODE_NAME> ~]$md_diag() {\ ROOT_BLK_DEV=$(sudo df "$1" --output=source | tail -n1); \ echo "Root block device: $ROOT_BLK_DEV"; \ ROOT_BLK_TYPE=$(sudo lsblk -no TYPE $ROOT_BLK_DEV); \ echo "Root block device type: $ROOT_BLK_TYPE"; \ if [ "$ROOT_BLK_TYPE" != "lvm" ]; then \ echo "Root block device is not an LVM volume"; \ echo "It differs to recommendations in Artesca Documentation"; \ echo "Please contact Artesca Support"; \ exit 1; \ fi; \ ROOT_VG=$(basename $ROOT_BLK_DEV | cut -d "-" -f 1); \ echo "Root volume group: $ROOT_VG"; \ declare -a ROOT_PVS=$(sudo pvs --reportformat json | jq -r ".report[] | .pv[] | select(.vg_name == \"$ROOT_VG\") | (.pv_name)"); \ echo "Root physical volumes: $(echo $ROOT_PVS | xargs)"; \ ROOT_PVS_TYPES=$(sudo lsblk -nldo TYPE ${ROOT_PVS[@]} | sort -u); \ if [ "$ROOT_PVS_TYPES" != "raid1" ]; then \ echo "Root physical volumes are not RAID1 Arrays"; \ echo "It differs to recommendations in Artesca Documentation"; \ echo "Please contact Artesca Support"; \ exit 1; \ fi; \ echo "Root physical volumes types: $ROOT_PVS_TYPES"; \ declare -a ROOT_DISKS=$(echo "$(for md in $ROOT_PVS; do sudo lsblk -o NAME,TYPE -J | jq -r ".blockdevices[] | select(.type == \"disk\" and .children != null and .children[].children != null and .children[].children[].name == \"$(echo $md | sed 's/\/dev\///g')\") | (.name)"; done)" | sort -u); \ echo "Root disks: $(echo $ROOT_DISKS | xargs)"; \ declare -a ALL_MDS=$(for disk in $ROOT_DISKS; do sudo lsblk -o NAME,TYPE -J /dev/$disk | jq -r '.blockdevices[] | select(.children != null) | .children[] | select(.type == "part" and .children != null) | .children[] | select(.type == "raid1") | (.name)'; done | sort -u); \ echo "All md devices using root disks: $(echo $ALL_MDS | xargs)"; \ declare -a MDS_TO_REPAIR=$(for md in $ALL_MDS; do echo "/dev/$md"; done); \ for md in ${MDS_TO_REPAIR[@]}; do \ MOUNTPOINTS=$(sudo lsblk -o NAME,MOUNTPOINT -J $md | jq -r '.blockdevices[] | if .mountpoint != null then . else .children[] | select(.mountpoint != null) end | (.name + " mounted at " + .mountpoint)'); \ if [ -z "$MOUNTPOINTS" ]; then \ echo "No mountpoints found for $md"; \ else \ echo "Mountpoints using $md: $(echo $MOUNTPOINTS | xargs)"; \ fi; \ DEVICES=$(sudo mdadm -v --detail --scan $md | awk -F= '/^[ ]+devices/ {print $2}'); \ echo "Device or partition using $md: $(echo $DEVICES | xargs)"; \ done; \ echo "RAID arrays to repair: $(echo $MDS_TO_REPAIR | xargs)"; \ }; \ md_diag /etc
Root block device: /dev/mapper/vgos-root Root block device type: lvm Root volume group: vgos Root physical volumes: /dev/md126 Root physical volumes types: raid1 Root disks: nvme0n1 All md devices using root disks: md127 md125 md126 Mountpoints using /dev/md125: md125 mounted at /boot/efi Device or partition using /dev/md125: /dev/nvme0n1p2 Mountpoints using /dev/md126: vgos-root mounted at / vgos-srv_scality_releases mounted at /srv/scality/releases vgos-var_lib_containerd mounted at /var/lib/containerd vgos-var_log mounted at /var/log Device or partition using /dev/md126: /dev/nvme0n1p3 Mountpoints using /dev/md127: md127 mounted at /boot Device or partition using /dev/md127: /dev/nvme0n1p1 RAID arrays to repair: /dev/md127 /dev/md125 /dev/md126
Note
We are interested to the
RAID arrays to repairand values.Repair RAID array:
Identify the failed logical drive (State = degraded).
You have to remove the faulty partition from RAID array find in the previous step.
Note
- The partition can either:
be absent (removed) if it is no longer detected.
be in a faulty state. In this case, you have to manually remove it using the example command below.
For the boot array:
[artesca-os@<NODE_NAME> ~]$sudo mdadm --detail /dev/md127
/dev/md127: Version : 1.2 Creation Time : Wed Aug 28 09:58:29 2024 Raid Level : raid1 Array Size : 1047552 (1023.00 MiB 1072.69 MB) Used Dev Size : 1047552 (1023.00 MiB 1072.69 MB) Raid Devices : 2 Total Devices : 1 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Sep 6 08:43:10 2024 State : clean, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 0 Spare Devices : 0 Consistency Policy : bitmap Name : localhost.localdomain:boot UUID : 86c290fe:c7aee527:02a8d433:6512bd03 Events : 246 Number Major Minor RaidDevice State 3 259 3 0 active sync /dev/nvme0n1p1 - 0 0 1 removed 1 253 240 - faulty /dev/nvme0n2p1[artesca-os@<NODE_NAME> ~]$sudo mdadm --manage --remove /dev/md127 /dev/nvme0n2p1
For the boot_efi array:
[artesca-os@<NODE_NAME> ~]$sudo mdadm --detail /dev/md125
/dev/md125: Version : 1.0 Creation Time : Wed Aug 28 09:58:23 2024 Raid Level : raid1 Array Size : 205760 (200.94 MiB 210.70 MB) Used Dev Size : 205760 (200.94 MiB 210.70 MB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Sep 6 14:39:50 2024 State : clean, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 1 Spare Devices : 0 Consistency Policy : bitmap Name : localhost.localdomain:boot_efi UUID : 6a5d1d42:181eb500:53d807b0:2f41d8bf Events : 199 Number Major Minor RaidDevice State 3 259 3 0 active sync /dev/nvme0n1p2 - 0 0 1 removed 1 253 240 - faulty /dev/nvme0n2p2[artesca-os@<NODE_NAME> ~]$sudo mdadm --manage --remove /dev/md125 /dev/nvme0n2p2
For the LVM Physical Volume array:
[artesca-os@<NODE_NAME> ~]$sudo mdadm --detail /dev/md126
/dev/md126: Version : 1.2 Creation Time : Wed Aug 28 09:58:08 2024 Raid Level : raid1 Array Size : 501926912 (478.67 GiB 513.97 GB) Used Dev Size : 501926912 (478.67 GiB 513.97 GB) Raid Devices : 2 Total Devices : 1 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Fri Sep 6 14:47:07 2024 State : clean, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 0 Spare Devices : 0 Consistency Policy : bitmap Name : localhost.localdomain:pv00 UUID : 0fd95173:fddb3fb0:f560620f:e7402c35 Events : 78877 Number Major Minor RaidDevice State 3 259 3 0 active sync /dev/nvme0n1p3 - 0 0 1 removed 1 253 240 - faulty /dev/nvme0n2p3[artesca-os@<NODE_NAME> ~]$sudo mdadm --manage --remove /dev/md126 /dev/nvme0n2p3
Replace the faulty physical disk.
Add the new physical disk.
Now you have to copy the partiton table from the remaining disk to the new disk.
First of all, you have to backup the partition table of the remaining disk.
[artesca-os@<NODE_NAME> ~]$sudo sgdisk --backup=table /dev/nvme0n1
The operation has completed successfully.
Now you can restore the partition table to the new disk.
[artesca-os@<NODE_NAME> ~]$sudo sgdisk --load-backup=table /dev/nvme0n3
Warning: Partition table header claims that the size of partition table entries is 0 bytes, but this program supports only 128-byte entries. Adjusting accordingly, but partition table may be garbage. Warning: Partition table header claims that the size of partition table entries is 0 bytes, but this program supports only 128-byte entries. Adjusting accordingly, but partition table may be garbage. Creating new GPT entries. The operation has completed successfully.
Finally, generate a new GUID for the new disk.
[artesca-os@<NODE_NAME> ~]$sudo sgdisk -G /dev/nvme0n3
The operation has completed successfully.
Now you have the new partitions created, you are ready to add them to the corresponding RAID array.
First you have to identify the partitions number for every RAID array on the disk still active.
Important
Next steps are dangerous and can cause data loss. You have to be sure that you are working on the correct disk and partition. In the previous step, you have duplicated the partition table from the remaining disk to the new disk, so the partition number should be the same.
To ensure that you are working on the correct partition, you can check the partition affected to each RAID array with the ouput of the first command in this documentation or with the following commands.
For the boot partition:
[artesca-os@<NODE_NAME> ~]$sudo mdadm -v --detail --scan /dev/md127 | awk -F= '/^[ ]+devices/ {print $2}'
/dev/nvme0n1p1
And now you can add the new partition to the RAID arrays.
[artesca-os@<NODE_NAME> ~]$sudo mdadm --manage /dev/md127 --add /dev/nvme0n3p1
mdadm: added /dev/nvme0n3p1
For the EFI partition:
[artesca-os@<NODE_NAME> ~]$sudo mdadm -v --detail --scan /dev/md125 | awk -F= '/^[ ]+devices/ {print $2}'
/dev/nvme0n1p2
And now you can add the new partition to the RAID arrays.
[artesca-os@<NODE_NAME> ~]$sudo mdadm --manage /dev/md125 --add /dev/nvme0n3p2
mdadm: added /dev/nvme0n3p2
For the LVM Physical Volume partition:
[artesca-os@<NODE_NAME> ~]$sudo mdadm -v --detail --scan /dev/md126 | awk -F= '/^[ ]+devices/ {print $2}'
/dev/nvme0n1p3
And now you can add the new partition to the RAID arrays.
[artesca-os@<NODE_NAME> ~]$sudo mdadm --manage /dev/md126 --add /dev/nvme0n3p3
mdadm: added /dev/nvme0n3p3
Check the RAID being recovered.
[artesca-os@<NODE_NAME> ~]$cat /proc/mdstatPersonalities : [raid1] md125 : active raid1 nvme0n3p2[2] nvme0n1p2[3] 205760 blocks super 1.0 [2/1] [UU] bitmap: 1/1 pages [4KB], 65536KB chunk md126 : active raid1 nvme0n3p3[2] nvme0n1p3[3] 501926912 blocks super 1.2 [2/1] [U_] [=>...................] recovery = 5.0% (10539648/209583104) finish=45.9min speed=72272K/sec bitmap: 1/2 pages [4KB], 65536KB chunk md127 : active raid1 nvme0n3p1[2] nvme0n1p1[3] 1047552 blocks super 1.0 [2/1] [UU] bitmap: 1/1 pages [4KB], 65536KB chunk
Tips
After modification, MD RAID synchronizes the whole disks to ensure the level of data protection that you want. This initial synchronization can be sped up by increasing the maximum sync speed (useful on high speed NVMe storage):
$cat /proc/sys/dev/raid/speed_limit_max > ./speed_limit.bkp && \ echo 10000000 | sudo tee /proc/sys/dev/raid/speed_limit_max
To reduce the impact on CPU usage during runtime, switch the speed limit back to its original value once the initial sync has been completed:
$cat ./speed_limit.bkp | sudo tee /proc/sys/dev/raid/speed_limit_max && \ rm -f ./speed_limit.bkp
Hardware RAID¶
Detect the RAID controller used on your machine.
[artesca-os@<NODE_NAME> ~]$
lspci | grep -i -e 'raid' -e 'adaptec'
# Example output for Adaptec Smart:
5c:00.0 Serial Attached SCSI controller: Adaptec Smart Storage PQI SAS (rev 01)
# Example output for MegaRAID (SAS-3):
0d:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)
# Example output for MegaRAID (SAS40xx):
10:00.0 RAID bus controller: Broadcom / LSI Fusion-MPT 24GSAS/PCIe SAS40xx (rev 01)
Note
Dell servers also have MegaRAID controller. However, the management tool is different. Use the Dell RAID controller procedure.
Important
The following steps assume that RAID1 mode is being used.
Ensure the RAID controller CLI is installed.
[artesca-os@<NODE_NAME> ~]$sudo dnf install ssacli
Retrieve the RAID controller index (
CONTROLLER_INDEX, depending on specific hardware, will usually be0or1).[artesca-os@<NODE_NAME> ~]$sudo ssacli ctrl all show
HPE Smart Array P816i-a SR Gen10 in Slot 0 (Embedded) (sn: PWXLA0CRHF10J9) # Note: CONTROLLER_INDEX is the value after "Slot" in the output.
Set the
CONTROLLER_INDEXvariable.Replace
<CONTROLLER_INDEX>placeholder with the value ofCtlcolumn in the output from the previous step.[artesca-os@<NODE_NAME> ~]$export CONTROLLER_INDEX=<CONTROLLER_INDEX>
Identify the failed physical drive’s ID.
[artesca-os@<NODE_NAME> ~]$sudo ssacli ctrl slot=$CONTROLLER_INDEX pd all show status
physicaldrive 1I:1:14 (port 1I:box 1:bay 14, 6 TB): Failed *** physicaldrive 2I:1:2 (port 2I:box 1:bay 6, 1 TB): OK physicaldrive 1I:1:49 (port 1I:box 1:bay 49, 6 TB): OK physicaldrive 1I:1:15 (port 1I:box 1:bay 15, 6 TB): OK # Note: The line we are looking for is the one ended by "Failed". # We need to take note of FAILED_DRIVE_ID value: the set of # values separated by colons (":") in the faulty line. # (e.g. 1I:1:14)Set the
FAILED_DRIVE_IDvariable.Replace
<FAILED_DRIVE_ID>placeholder with the value from the previous step.[artesca-os@<NODE_NAME> ~]$export FAILED_DRIVE_ID=<FAILED_DRIVE_ID>
Run the following command for the faulty physical disk’s LED to blink.
[artesca-os@<NODE_NAME> ~]$sudo ssacli controller slot=$CONTROLLER_INDEX pd $FAILED_DRIVE_ID modify led=on
Replace the faulty physical disk.
Run the following commands for the system to detect the new disk and check its availability, only if both disks are failed.
Confirm the new disks are listed (in the
Unassignedsection).[artesca-os@<NODE_NAME> ~]$sudo ssacli ctrl slot=$CONTROLLER_INDEX pd all show
[...] Unassigned physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS HDD, 2 TB, OK) physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS HDD, 2 TB, OK) # Note: The lines we are looking for is the one below "Unassigned". # We need to take note of NEW_DRIVE_IDS value: the set of # values separated by colons (":") in the faulty line.Set the
NEW_DRIVE_IDSvariable.Replace
<NEW_DRIVE_IDS>placeholder with the value from the previous step separated by comma like'1I:1:1,1I:1:2'.[artesca-os@<NODE_NAME> ~]$export NEW_DRIVE_IDS=<NEW_DRIVE_IDS>
Identify the failed logical drive. It is the one with the status NOT OK.
[artesca-os@<NODE_NAME> ~]$sudo ssacli ctrl slot=$CONTROLLER_INDEX ld all show status
logicaldrive 1 (5.46 TB, RAID 1): **NOT OK** logicaldrive 2 (5.46 TB, RAID 0): OK logicaldrive 3 (5.46 TB, RAID 0): OK # Note: The line we are looking for is the one ended by NOT OK. # We need to take note of FAILED_LOGICAL_DRIVE value: # the number that appears next to "logicaldrive" in the faulty line.
Set the
FAILED_LOGICAL_DRIVEvariable to the value of the faulty drive.Replace the
<FAILED_LOGICAL_DRIVE>placeholder with the value of the faulty drive.[artesca-os@<NODE_NAME> ~]$export FAILED_LOGICAL_DRIVE=<FAILED_LOGICAL_DRIVE>
Delete the original logical drive.
[artesca-os@<NODE_NAME> ~]$sudo ssacli ctrl slot=$CONTROLLER_INDEX ld $FAILED_LOGICAL_DRIVE delete
Re-create the new logical drive.
[artesca-os@<NODE_NAME> ~]$sudo ssacli ctrl slot=$CONTROLLER_INDEX create type=ld drives=$NEW_DRIVE_IDS raid=1
Important
The following steps assume that RAID1 mode is being used.
Ensure the RAID controller CLI is installed.
[artesca-os@<NODE_NAME> ~]$sudo dnf install storcli
Retrieve the RAID controller index (
CONTROLLER_INDEX, depending on specific hardware, will usually be0or1).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 show
-------------------------------------------------------------------- Ctl Model Ports PDs DGs DNOpt VDs VNOpt BBU sPR DS EHS ASOs Hlth -------------------------------------------------------------------- 0 SAS3916 16 12 12 0 12 0 N/A On 1&2 Y 4 Opt --------------------------------------------------------------------
Set the
CONTROLLER_INDEXvariable.Replace
<CONTROLLER_INDEX>placeholder with the value ofCtlcolumn in the output from the previous step.[artesca-os@<NODE_NAME> ~]$export CONTROLLER_INDEX=<CONTROLLER_INDEX>
Identify the failed physical drive’s ID.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 /c$CONTROLLER_INDEX /eall /sall show
----------------------------------------------------------------------------- EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type ----------------------------------------------------------------------------- 252:0 50 Offln 0 14.551 TB SAS SSD N N 512B MG08SCA16TE U - *** 252:1 51 Onln 1 14.551 TB SAS SSD N N 512B MG08SCA16TE U - 252:2 52 Onln 2 14.551 TB SAS SSD N N 512B MG08SCA16TE U - 252:3 41 Onln 3 14.551 TB SAS SSD N N 512B MG08SCA16TE U - ----------------------------------------------------------------------------- # Note: The line we are looking for is the one with "Offln" in the "State" column. # We need to take note of two values: # - FAILED_ENCLOSURE_DEVICE_ID: The value of "EID" column. (e.g. 252) # - FAILED_SLOT_NUMBER: The value of "Slt" column. (e.g. 0)
Set the
FAILED_ENCLOSURE_DEVICE_IDandFAILED_SLOT_NUMBERvariables.Replace
<FAILED_ENCLOSURE_DEVICE_ID>and<FAILED_SLOT_NUMBER>placeholders with the values from the previous step.[artesca-os@<NODE_NAME> ~]$export FAILED_ENCLOSURE_DEVICE_ID=<FAILED_ENCLOSURE_DEVICE_ID> FAILED_SLOT_NUMBER=<FAILED_SLOT_NUMBER>
Run the following command for the faulty physical disk’s LED to blink.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 /c$CONTROLLER_INDEX /e$FAILED_ENCLOSURE_DEVICE_ID /s$FAILED_SLOT_NUMBER start locate
Replace the faulty physical disk.
Run the following commands for the system to detect the new disk and check its availability, only if both disks are failed.
Confirm the new disk is listed (with
UGoodstatus, Unconfigured Good).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 /c$CONTROLLER_INDEX /eall /sall show
----------------------------------------------------------------------------- EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type ----------------------------------------------------------------------------- 252:0 50 Offln 0 14.551 TB SAS SSD N N 512B MG08SCA16TE U - 252:1 51 Offln 1 14.551 TB SAS SSD N N 512B MG08SCA16TE U - 252:2 52 Onln 2 14.551 TB SAS HDD N N 512B MG08SCA16TE U - 252:3 41 Onln 3 14.551 TB SAS HDD N N 512B MG08SCA16TE U - 252:10 48 UGood - 14.551 TB SAS SSD N N 512B MG08SCA16TE U - *** 252:11 49 UGood - 14.551 TB SAS SSD N N 512B MG08SCA16TE U - *** # Note: The lines we are looking for are the ones having "UGood" in the "State" column. # We need to take note of two values: # - NEW_ENCLOSURE_DEVICE_ID_[1..2]: The values of "EID" column. # - NEW_SLOT_NUMBER_[1..2]: The values of "Slt" column.
Set the
NEW_ENCLOSURE_DEVICE_IDandNEW_SLOT_NUMBERvariables to the values of the new disk.Replace the
<NEW_ENCLOSURE_DEVICE_ID_[1..2]>and<NEW_SLOT_NUMBER_[1..2]>placeholders with the values of the new disk.[artesca-os@<NODE_NAME> ~]$export NEW_ENCLOSURE_DEVICE_ID_1=<NEW_ENCLOSURE_DEVICE_ID_1> NEW_SLOT_NUMBER_1=<NEW_SLOT_NUMBER_1> NEW_ENCLOSURE_DEVICE_ID_2=<NEW_ENCLOSURE_DEVICE_ID_2> NEW_SLOT_NUMBER_2=<NEW_SLOT_NUMBER_2>
Identify the failed logical drive. It is the one with the status Ofln.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 /c$CONTROLLER_INDEX /vall show
--------------------------------------------------------------- DG/VD TYPE State Access Consist Cache Cac sCC Size Name --------------------------------------------------------------- 2/237 RAID0 Optl RW Yes NRWTD - ON 14.551 TB 1/238 RAID0 Optl RW Yes NRWTD - ON 14.551 TB 0/239 RAID1 Ofln RW No RWTD - ON 14.551 TB *** --------------------------------------------------------------- # Note: The line we are interested in is the one with the status "Ofln". # We need to take note of FAILED_DRIVE_ID value: the number that appears # in VD column in the faulty line.
Set the
FAILED_DRIVE_IDvariable to the value of the faulty drive.Replace the
<FAILED_DRIVE_ID>placeholder with the value of the faulty drive.[artesca-os@<NODE_NAME> ~]$export FAILED_DRIVE_ID=<FAILED_DRIVE_ID>
Delete the original logical drive.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 /c$CONTROLLER_INDEX /v$FAILED_DRIVE_ID delete
Re-create the new logical drive.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 /c$CONTROLLER_INDEX add vd raid1 \ drives="$NEW_ENCLOSURE_DEVICE_ID_1:$NEW_SLOT_NUMBER_1,$NEW_ENCLOSURE_DEVICE_ID_2:$NEW_SLOT_NUMBER_2"
Important
The following steps assume that RAID1 mode is being used.
Ensure the RAID controller CLI is installed.
[artesca-os@<NODE_NAME> ~]$sudo dnf install storcli2
Retrieve the RAID controller index (
CONTROLLER_INDEX, depending on specific hardware, will usually be0or1).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 show
----------------------------------------------------------------------------------------- Ctrl Product Name SASAddress Personality Status PD(s) VD(s) VNOpt EPack ----------------------------------------------------------------------------------------- 0 SAS3916 Adapter 0XXXXXXX RAID Optimal 12 12 0 Optimal -----------------------------------------------------------------------------------------
Set the
CONTROLLER_INDEXvariable.Replace
<CONTROLLER_INDEX>placeholder with the value ofCtlcolumn in the output from the previous step.[artesca-os@<NODE_NAME> ~]$export CONTROLLER_INDEX=<CONTROLLER_INDEX>
Identify the failed physical drive’s ID.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 /c$CONTROLLER_INDEX /eall /sall show
----------------------------------------------------------------------------------------------------- EID:Slt PID State Status DG Size Intf Med SED_Type SeSz Model Sp LU/NS Count Alt-EID ----------------------------------------------------------------------------------------------------- 288:0 275 Conf Offline 0 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - *** 288:1 276 Conf Online 1 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - 288:2 277 Conf Online 2 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - 288:3 278 Conf Online 3 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - ----------------------------------------------------------------------------------------------------- # Note: The line we are looking for is the one with "Offline" in the "Status" column. # We need to take note of two values: # - FAILED_ENCLOSURE_DEVICE_ID: The value of "EID" column. (e.g. 288) # - FAILED_SLOT_NUMBER: The value of "Slt" column. (e.g. 0)
Set the
FAILED_ENCLOSURE_DEVICE_IDandFAILED_SLOT_NUMBERvariables.Replace
<FAILED_ENCLOSURE_DEVICE_ID>and<FAILED_SLOT_NUMBER>placeholders with the values from the previous step.[artesca-os@<NODE_NAME> ~]$export FAILED_ENCLOSURE_DEVICE_ID=<FAILED_ENCLOSURE_DEVICE_ID> FAILED_SLOT_NUMBER=<FAILED_SLOT_NUMBER>
Run the following command for the faulty physical disk’s LED to blink.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 /c$CONTROLLER_INDEX /e$FAILED_ENCLOSURE_DEVICE_ID /s$FAILED_SLOT_NUMBER start locate
Replace the faulty physical disk.
Run the following commands for the system to detect the new disk and check its availability, only if both disks are failed.
Confirm the new disk is listed (with
UGoodstatus, Unconfigured Good).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 /c$CONTROLLER_INDEX /eall /sall show
----------------------------------------------------------------------------------------------------- EID:Slt PID State Status DG Size Intf Med SED_Type SeSz Model Sp LU/NS Count Alt-EID ----------------------------------------------------------------------------------------------------- 288:0 275 Conf Offline 0 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - 288:1 276 Conf Offline 1 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - 288:2 277 Conf Online 2 7.276 TiB SAS HDD - 512B ST8000NM024B U 1 - 288:3 278 Conf Online 3 7.276 TiB SAS HDD - 512B ST8000NM024B U 1 - 288:10 285 UConf Online - 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - *** 288:11 286 UConf Online - 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - *** ----------------------------------------------------------------------------------------------------- # Note: The lines we are looking for are the ones having "UGood" in the "State" column. # We need to take note of two values: # - NEW_ENCLOSURE_DEVICE_ID_[1..2]: The values of "EID" column. # - NEW_SLOT_NUMBER_[1..2]: The values of "Slt" column.
Set the
NEW_ENCLOSURE_DEVICE_IDandNEW_SLOT_NUMBERvariables to the values of the new disk.Replace the
<NEW_ENCLOSURE_DEVICE_ID_[1..2]>and<NEW_SLOT_NUMBER_[1..2]>placeholders with the values of the new disk.[artesca-os@<NODE_NAME> ~]$export NEW_ENCLOSURE_DEVICE_ID_1=<NEW_ENCLOSURE_DEVICE_ID_1> NEW_SLOT_NUMBER_1=<NEW_SLOT_NUMBER_1> NEW_ENCLOSURE_DEVICE_ID_2=<NEW_ENCLOSURE_DEVICE_ID_2> NEW_SLOT_NUMBER_2=<NEW_SLOT_NUMBER_2>
Identify the failed logical drive. It is the one with the status Ofln.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 /c$CONTROLLER_INDEX /vall show
------------------------------------------------------------------ DG/VD TYPE State Access CurrentCache DefaultCache Size Name ------------------------------------------------------------------ 0/1 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 1/2 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 11/12 RAID0 Ofln RW R,WT R,WT 7.276 TiB *** ------------------------------------------------------------------ # Note: The line we are interested in is the one with the status "Ofln". # We need to take note of FAILED_DRIVE_ID value: the number that appears # in VD column in the faulty line.
Set the
FAILED_DRIVE_IDvariable to the value of the faulty drive.Replace the
<FAILED_DRIVE_ID>placeholder with the value of the faulty drive.[artesca-os@<NODE_NAME> ~]$export FAILED_DRIVE_ID=<FAILED_DRIVE_ID>
Delete the original logical drive.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 /c$CONTROLLER_INDEX /v$FAILED_DRIVE_ID delete
Re-create the new logical drive.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 /c$CONTROLLER_INDEX add vd raid1 \ drives="$NEW_ENCLOSURE_DEVICE_ID_1:$NEW_SLOT_NUMBER_1,$NEW_ENCLOSURE_DEVICE_ID_2:$NEW_SLOT_NUMBER_2"
Important
The following steps assume that RAID1 mode is being used.
Ensure the RAID controller CLI is installed.
[artesca-os@<NODE_NAME> ~]$sudo dnf install perccli
Retrieve the RAID controller index (
CONTROLLER_INDEX, depending on specific hardware, will usually be0or1).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 show
-------------------------------------------------------------------- Ctl Model Ports PDs DGs DNOpt VDs VNOpt BBU sPR DS EHS ASOs Hlth -------------------------------------------------------------------- 0 PERC H730 16 12 12 0 12 0 N/A On 1&2 Y 4 Opt --------------------------------------------------------------------
Set the
CONTROLLER_INDEXvariable.Replace
<CONTROLLER_INDEX>placeholder with the value ofCtlcolumn in the output from the previous step.[artesca-os@<NODE_NAME> ~]$export CONTROLLER_INDEX=<CONTROLLER_INDEX>
Identify the failed physical drive’s ID.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 /c$CONTROLLER_INDEX /eall /sall show
----------------------------------------------------------------------------- EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type ----------------------------------------------------------------------------- 252:0 50 Offln 0 14.551 TB SAS SSD N N 512B MG08SCA16TE U - *** 252:1 51 Onln 1 14.551 TB SAS SSD N N 512B MG08SCA16TE U - 252:2 52 Onln 2 14.551 TB SAS SSD N N 512B MG08SCA16TE U - 252:3 41 Onln 3 14.551 TB SAS SSD N N 512B MG08SCA16TE U - ----------------------------------------------------------------------------- # Note: The line we are looking for is the one with "Offln" in the "State" column. # We need to take note of two values: # - FAILED_ENCLOSURE_DEVICE_ID: The value of "EID" column. (e.g. 252) # - FAILED_SLOT_NUMBER: The value of "Slt" column. (e.g. 0)
Set the
FAILED_ENCLOSURE_DEVICE_IDandFAILED_SLOT_NUMBERvariables.Replace
<FAILED_ENCLOSURE_DEVICE_ID>and<FAILED_SLOT_NUMBER>placeholders with the values from the previous step.[artesca-os@<NODE_NAME> ~]$export FAILED_ENCLOSURE_DEVICE_ID=<FAILED_ENCLOSURE_DEVICE_ID> FAILED_SLOT_NUMBER=<FAILED_SLOT_NUMBER>
Run the following command for the faulty physical disk’s LED to blink.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 /c$CONTROLLER_INDEX /e$FAILED_ENCLOSURE_DEVICE_ID /s$FAILED_SLOT_NUMBER start locate
Replace the faulty physical disk.
Run the following commands for the system to detect the new disk and check its availability, only if both disks are failed.
Confirm the new disk is listed (with
UGoodstatus, Unconfigured Good).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 /c$CONTROLLER_INDEX /eall /sall show
----------------------------------------------------------------------------- EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type ----------------------------------------------------------------------------- 252:0 50 Offln 0 14.551 TB SAS SSD N N 512B MG08SCA16TE U - 252:1 51 Offln 1 14.551 TB SAS SSD N N 512B MG08SCA16TE U - 252:2 52 Onln 2 14.551 TB SAS HDD N N 512B MG08SCA16TE U - 252:3 41 Onln 3 14.551 TB SAS HDD N N 512B MG08SCA16TE U - 252:10 48 UGood - 14.551 TB SAS SSD N N 512B MG08SCA16TE U - *** 252:11 49 UGood - 14.551 TB SAS SSD N N 512B MG08SCA16TE U - *** # Note: The lines we are looking for are the ones having "UGood" in the "State" column. # We need to take note of two values: # - NEW_ENCLOSURE_DEVICE_ID_[1..2]: The values of "EID" column. # - NEW_SLOT_NUMBER_[1..2]: The values of "Slt" column.
Set the
NEW_ENCLOSURE_DEVICE_IDandNEW_SLOT_NUMBERvariables to the values of the new disk.Replace the
<NEW_ENCLOSURE_DEVICE_ID_[1..2]>and<NEW_SLOT_NUMBER_[1..2]>placeholders with the values of the new disk.[artesca-os@<NODE_NAME> ~]$export NEW_ENCLOSURE_DEVICE_ID_1=<NEW_ENCLOSURE_DEVICE_ID_1> NEW_SLOT_NUMBER_1=<NEW_SLOT_NUMBER_1> NEW_ENCLOSURE_DEVICE_ID_2=<NEW_ENCLOSURE_DEVICE_ID_2> NEW_SLOT_NUMBER_2=<NEW_SLOT_NUMBER_2>
Identify the failed logical drive. It is the one with the status Ofln.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 /c$CONTROLLER_INDEX /vall show
--------------------------------------------------------------- DG/VD TYPE State Access Consist Cache Cac sCC Size Name --------------------------------------------------------------- 2/237 RAID0 Optl RW Yes NRWTD - ON 14.551 TB 1/238 RAID0 Optl RW Yes NRWTD - ON 14.551 TB 0/239 RAID1 Ofln RW No RWTD - ON 14.551 TB *** --------------------------------------------------------------- # Note: The line we are interested in is the one with the status "Ofln". # We need to take note of FAILED_DRIVE_ID value: the number that appears # in VD column in the faulty line.
Set the
FAILED_DRIVE_IDvariable to the value of the faulty drive.Replace the
<FAILED_DRIVE_ID>placeholder with the value of the faulty drive.[artesca-os@<NODE_NAME> ~]$export FAILED_DRIVE_ID=<FAILED_DRIVE_ID>
Delete the original logical drive.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 /c$CONTROLLER_INDEX /v$FAILED_DRIVE_ID delete
Re-create the new logical drive.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 /c$CONTROLLER_INDEX add vd raid1 \ drives="$NEW_ENCLOSURE_DEVICE_ID_1:$NEW_SLOT_NUMBER_1,$NEW_ENCLOSURE_DEVICE_ID_2:$NEW_SLOT_NUMBER_2"
Important
The following steps assume that RAID1 mode is being used.
Ensure the RAID controller CLI is installed.
[artesca-os@<NODE_NAME> ~]$sudo dnf install perccli2
Retrieve the RAID controller index (
CONTROLLER_INDEX, depending on specific hardware, will usually be0or1).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 show
----------------------------------------------------------------------------------------- Ctrl Product Name SASAddress Personality Status PD(s) VD(s) VNOpt EPack ----------------------------------------------------------------------------------------- 0 PERC H965i Adapter 0XXXXXXXX RAID Optimal 12 12 0 Optimal -----------------------------------------------------------------------------------------
Set the
CONTROLLER_INDEXvariable.Replace
<CONTROLLER_INDEX>placeholder with the value ofCtlcolumn in the output from the previous step.[artesca-os@<NODE_NAME> ~]$export CONTROLLER_INDEX=<CONTROLLER_INDEX>
Identify the failed physical drive’s ID.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 /c$CONTROLLER_INDEX /eall /sall show
----------------------------------------------------------------------------------------------------- EID:Slt PID State Status DG Size Intf Med SED_Type SeSz Model Sp LU/NS Count Alt-EID ----------------------------------------------------------------------------------------------------- 288:0 275 Conf Offline 0 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - *** 288:1 276 Conf Online 1 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - 288:2 277 Conf Online 2 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - 288:3 278 Conf Online 3 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - ----------------------------------------------------------------------------------------------------- # Note: The line we are looking for is the one with "Offline" in the "Status" column. # We need to take note of two values: # - FAILED_ENCLOSURE_DEVICE_ID: The value of "EID" column. (e.g. 288) # - FAILED_SLOT_NUMBER: The value of "Slt" column. (e.g. 0)
Set the
FAILED_ENCLOSURE_DEVICE_IDandFAILED_SLOT_NUMBERvariables.Replace
<FAILED_ENCLOSURE_DEVICE_ID>and<FAILED_SLOT_NUMBER>placeholders with the values from the previous step.[artesca-os@<NODE_NAME> ~]$export FAILED_ENCLOSURE_DEVICE_ID=<FAILED_ENCLOSURE_DEVICE_ID> FAILED_SLOT_NUMBER=<FAILED_SLOT_NUMBER>
Run the following command for the faulty physical disk’s LED to blink.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 /c$CONTROLLER_INDEX /e$FAILED_ENCLOSURE_DEVICE_ID /s$FAILED_SLOT_NUMBER start locate
Replace the faulty physical disk.
Run the following commands for the system to detect the new disk and check its availability, only if both disks are failed.
Confirm the new disk is listed (with
UGoodstatus, Unconfigured Good).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 /c$CONTROLLER_INDEX /eall /sall show
----------------------------------------------------------------------------------------------------- EID:Slt PID State Status DG Size Intf Med SED_Type SeSz Model Sp LU/NS Count Alt-EID ----------------------------------------------------------------------------------------------------- 288:0 275 Conf Offline 0 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - 288:1 276 Conf Offline 1 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - 288:2 277 Conf Online 2 7.276 TiB SAS HDD - 512B ST8000NM024B U 1 - 288:3 278 Conf Online 3 7.276 TiB SAS HDD - 512B ST8000NM024B U 1 - 288:10 285 UConf Online - 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - *** 288:11 286 UConf Online - 7.276 TiB SAS SSD - 512B ST8000NM024B U 1 - *** ----------------------------------------------------------------------------------------------------- # Note: The lines we are looking for are the ones having "UGood" in the "State" column. # We need to take note of two values: # - NEW_ENCLOSURE_DEVICE_ID_[1..2]: The values of "EID" column. # - NEW_SLOT_NUMBER_[1..2]: The values of "Slt" column.
Set the
NEW_ENCLOSURE_DEVICE_IDandNEW_SLOT_NUMBERvariables to the values of the new disk.Replace the
<NEW_ENCLOSURE_DEVICE_ID_[1..2]>and<NEW_SLOT_NUMBER_[1..2]>placeholders with the values of the new disk.[artesca-os@<NODE_NAME> ~]$export NEW_ENCLOSURE_DEVICE_ID_1=<NEW_ENCLOSURE_DEVICE_ID_1> NEW_SLOT_NUMBER_1=<NEW_SLOT_NUMBER_1> NEW_ENCLOSURE_DEVICE_ID_2=<NEW_ENCLOSURE_DEVICE_ID_2> NEW_SLOT_NUMBER_2=<NEW_SLOT_NUMBER_2>
Identify the failed logical drive. It is the one with the status Ofln.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 /c$CONTROLLER_INDEX /vall show
------------------------------------------------------------------ DG/VD TYPE State Access CurrentCache DefaultCache Size Name ------------------------------------------------------------------ 0/1 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 1/2 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 11/12 RAID0 Ofln RW R,WT R,WT 7.276 TiB *** ------------------------------------------------------------------ # Note: The line we are interested in is the one with the status "Ofln". # We need to take note of FAILED_DRIVE_ID value: the number that appears # in VD column in the faulty line.
Set the
FAILED_DRIVE_IDvariable to the value of the faulty drive.Replace the
<FAILED_DRIVE_ID>placeholder with the value of the faulty drive.[artesca-os@<NODE_NAME> ~]$export FAILED_DRIVE_ID=<FAILED_DRIVE_ID>
Delete the original logical drive.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 /c$CONTROLLER_INDEX /v$FAILED_DRIVE_ID delete
Re-create the new logical drive.
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 /c$CONTROLLER_INDEX add vd raid1 \ drives="$NEW_ENCLOSURE_DEVICE_ID_1:$NEW_SLOT_NUMBER_1,$NEW_ENCLOSURE_DEVICE_ID_2:$NEW_SLOT_NUMBER_2"
