Misconfigured RAID Data Disk Cache Leads to Performance Issues¶
Problem Description¶
Performance issues can arise when the RAID Data Disk cache is misconfigured.
Important
This issue is specific to the RAID Data Disk cache configuration with RAID0, if you are using SSD in JBOD mode, you can ignore this.
Problem Resolution¶
Detect the RAID controller used on your machine.
[artesca-os@<NODE_NAME> ~]$
lspci | grep -i -e 'raid' -e 'adaptec'
# Example output for Adaptec Smart:
5c:00.0 Serial Attached SCSI controller: Adaptec Smart Storage PQI SAS (rev 01)
# Example output for MegaRAID (SAS-3):
0d:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)
# Example output for MegaRAID (SAS40xx):
10:00.0 RAID bus controller: Broadcom / LSI Fusion-MPT 24GSAS/PCIe SAS40xx (rev 01)
Note
Dell servers also have MegaRAID controller. However, the management tool is different. Use the Dell RAID controller procedure.
Cache settings cannot be configured per drive.
Ensure the RAID controller CLI is installed.
[artesca-os@<NODE_NAME> ~]$sudo dnf install storcli
Retrieve the RAID controller index (
CONTROLLER_INDEX, depending on specific hardware, will usually be0or1).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 show
-------------------------------------------------------------------- Ctl Model Ports PDs DGs DNOpt VDs VNOpt BBU sPR DS EHS ASOs Hlth -------------------------------------------------------------------- 0 SAS3916 16 12 12 0 12 0 N/A On 1&2 Y 4 Opt --------------------------------------------------------------------
Set the
CONTROLLER_INDEXvariable.Replace
<CONTROLLER_INDEX>placeholder with the value ofCtlcolumn in the output from the previous step.[artesca-os@<NODE_NAME> ~]$export CONTROLLER_INDEX=<CONTROLLER_INDEX>
Ensure all data drives are correctly set-up with right cache type.
The drives should be listed with the
NRWBDcache type.[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 /c$CONTROLLER_INDEX /vall show
--------------------------------------------------------------- DG/VD TYPE State Access Consist Cache Cac sCC Size Name --------------------------------------------------------------- 11/227 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 10/229 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 9/230 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 8/231 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 7/232 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 6/233 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 5/234 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 4/235 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 3/236 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 2/237 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 1/238 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 0/239 RAID0 Optl RW Yes NRWBD - ON 14.551 TB ---------------------------------------------------------------
If data drive do not have the
NRWBDcache type, you need to change it one by one.Replace the
<VD_ID>placeholder with the value inVDcolum from previous command.[artesca-os@<NODE_NAME> ~]$export VD_ID=<VD_ID>
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli/storcli64 /c$CONTROLLER_INDEX /v$VD_ID set wrcache=wb && \ sudo /opt/MegaRAID/storcli/storcli64 /c$CONTROLLER_INDEX /v$VD_ID set rdcache=NoRA && \ sudo /opt/MegaRAID/storcli/storcli64 /c$CONTROLLER_INDEX /v$VD_ID set iopolicy=Direct
Ensure the RAID controller CLI is installed.
[artesca-os@<NODE_NAME> ~]$sudo dnf install storcli2
Retrieve the RAID controller index (
CONTROLLER_INDEX, depending on specific hardware, will usually be0or1).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 show
----------------------------------------------------------------------------------------- Ctrl Product Name SASAddress Personality Status PD(s) VD(s) VNOpt EPack ----------------------------------------------------------------------------------------- 0 SAS3916 Adapter 0XXXXXXX RAID Optimal 12 12 0 Optimal -----------------------------------------------------------------------------------------
Set the
CONTROLLER_INDEXvariable.Replace
<CONTROLLER_INDEX>placeholder with the value ofCtlcolumn in the output from the previous step.[artesca-os@<NODE_NAME> ~]$export CONTROLLER_INDEX=<CONTROLLER_INDEX>
Ensure all data drives are correctly set-up with right cache type.
The drives should be listed with the
NR,WBcache type.[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 /c$CONTROLLER_INDEX /vall show
------------------------------------------------------------------ DG/VD TYPE State Access CurrentCache DefaultCache Size Name ------------------------------------------------------------------ 0/1 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 1/2 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 2/3 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 3/4 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 4/5 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 5/6 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 6/7 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 7/8 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 8/9 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 9/10 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 10/11 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 11/12 RAID0 Optl RW NR,WB NR,WB 7.276 TiB ------------------------------------------------------------------
If data drive do not have the
NR,WBcache type, you need to change it one by one.Replace the
<VD_ID>placeholder with the value inVDcolum from previous command.[artesca-os@<NODE_NAME> ~]$export VD_ID=<VD_ID>
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/storcli2/storcli2 /c$CONTROLLER_INDEX /v$VD_ID set wrcache=wb && \ sudo /opt/MegaRAID/storcli2/storcli2 /c$CONTROLLER_INDEX /v$VD_ID set rdcache=NoRA
Ensure the RAID controller CLI is installed.
[artesca-os@<NODE_NAME> ~]$sudo dnf install perccli
Retrieve the RAID controller index (
CONTROLLER_INDEX, depending on specific hardware, will usually be0or1).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 show
-------------------------------------------------------------------- Ctl Model Ports PDs DGs DNOpt VDs VNOpt BBU sPR DS EHS ASOs Hlth -------------------------------------------------------------------- 0 PERC H730 16 12 12 0 12 0 N/A On 1&2 Y 4 Opt --------------------------------------------------------------------
Set the
CONTROLLER_INDEXvariable.Replace
<CONTROLLER_INDEX>placeholder with the value ofCtlcolumn in the output from the previous step.[artesca-os@<NODE_NAME> ~]$export CONTROLLER_INDEX=<CONTROLLER_INDEX>
Ensure all data drives are correctly set-up with right cache type.
The drives should be listed with the
NRWBDcache type.[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 /c$CONTROLLER_INDEX /vall show
--------------------------------------------------------------- DG/VD TYPE State Access Consist Cache Cac sCC Size Name --------------------------------------------------------------- 11/227 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 10/229 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 9/230 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 8/231 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 7/232 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 6/233 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 5/234 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 4/235 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 3/236 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 2/237 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 1/238 RAID0 Optl RW Yes NRWBD - ON 14.551 TB 0/239 RAID0 Optl RW Yes NRWBD - ON 14.551 TB ---------------------------------------------------------------
If data drive do not have the
NRWBDcache type, you need to change it one by one.Replace the
<VD_ID>placeholder with the value inVDcolum from previous command.[artesca-os@<NODE_NAME> ~]$export VD_ID=<VD_ID>
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli/perccli64 /c$CONTROLLER_INDEX /v$VD_ID set wrcache=wb && \ sudo /opt/MegaRAID/perccli/perccli64 /c$CONTROLLER_INDEX /v$VD_ID set rdcache=NoRA && \ sudo /opt/MegaRAID/perccli/perccli64 /c$CONTROLLER_INDEX /v$VD_ID set iopolicy=Direct
Ensure the RAID controller CLI is installed.
[artesca-os@<NODE_NAME> ~]$sudo dnf install perccli2
Retrieve the RAID controller index (
CONTROLLER_INDEX, depending on specific hardware, will usually be0or1).[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 show
----------------------------------------------------------------------------------------- Ctrl Product Name SASAddress Personality Status PD(s) VD(s) VNOpt EPack ----------------------------------------------------------------------------------------- 0 PERC H965i Adapter 0XXXXXXXX RAID Optimal 12 12 0 Optimal -----------------------------------------------------------------------------------------
Set the
CONTROLLER_INDEXvariable.Replace
<CONTROLLER_INDEX>placeholder with the value ofCtlcolumn in the output from the previous step.[artesca-os@<NODE_NAME> ~]$export CONTROLLER_INDEX=<CONTROLLER_INDEX>
Ensure all data drives are correctly set-up with right cache type.
The drives should be listed with the
NR,WBcache type.[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 /c$CONTROLLER_INDEX /vall show
------------------------------------------------------------------ DG/VD TYPE State Access CurrentCache DefaultCache Size Name ------------------------------------------------------------------ 0/1 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 1/2 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 2/3 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 3/4 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 4/5 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 5/6 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 6/7 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 7/8 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 8/9 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 9/10 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 10/11 RAID0 Optl RW NR,WB NR,WB 7.276 TiB 11/12 RAID0 Optl RW NR,WB NR,WB 7.276 TiB ------------------------------------------------------------------
If data drive do not have the
NR,WBcache type, you need to change it one by one.Replace the
<VD_ID>placeholder with the value inVDcolum from previous command.[artesca-os@<NODE_NAME> ~]$export VD_ID=<VD_ID>
[artesca-os@<NODE_NAME> ~]$sudo /opt/MegaRAID/perccli2/perccli2 /c$CONTROLLER_INDEX /v$VD_ID set wrcache=wb && \ sudo /opt/MegaRAID/perccli2/perccli2 /c$CONTROLLER_INDEX /v$VD_ID set rdcache=NoRA
