一次IBM 服务器的磁盘故障更换过程

服务器是IBM的X3650 ,5块硬盘RAID 5，1块Hot Spare （但是估计当时装机的人没有配置成hot spare，后面再看详细内容）

故障时的磁盘状态:

[root@serv1 cmdline]# ./arcconf GETCONFIG 1

Could not open log file: UcliEvt.log

Controllers found: 1

----------------------------------------------------------------------

Controller information

----------------------------------------------------------------------

Controller Status : Okay

Channel description : SAS/SATA

Controller Model : IBM ServeRAID 8k

Controller Serial Number : 40703B9

Physical Slot : 0

Installed memory : 256 MB

Copyback : Disabled

Data scrubbing : Enabled
Defunct disk drive count : 1

Logical drives/Offline/Critical : 1/0/1

--------------------------------------------------------

Controller Version Information

--------------------------------------------------------

BIOS : 5.2-0 (15421)

Firmware : 5.2-0 (15421)

Driver : 1.1-5 (2453)

Boot Flash : 5.1-0 (15411)

--------------------------------------------------------

Controller Battery Information

--------------------------------------------------------

Status : Okay

Over temperature : No

Capacity remaining : 100 percent

Time remaining (at current draw) : 4 days, 5 hours, 20 minutes

--------------------------------------------------------

Controller Vital Product Data

--------------------------------------------------------

VPD Assigned# : 39R8875

EC Version# : J85096

Controller FRU# : 25R8076

Battery FRU# : 25R8088

----------------------------------------------------------------------

Logical drive information

----------------------------------------------------------------------

Logical drive number 1

Logical drive name :

RAID level : 5
Status of logical drive : Critical

Size : 1143986 MB

Read-cache mode : Enabled

Write-cache mode : Enabled (write-back)

Write-cache setting : Enabled (write-back) when protected by battery

Partitioned : Yes

Number of segments : 5

Stripe-unit size : 256 KB

Stripe order (Channel,Device) : DDD 0,1 0,2 0,3 0,4

Defunct segments : 0,0

Defunct stripes : No

----------------------------------------------------------------------

Physical Device information

----------------------------------------------------------------------

Device #0

Device is a Hard drive

State : Defunct

Supported : Yes

Transfer Speed : Defunct

Reported Channel,Device : 0,0

Reported Location : Enclosure 0, Slot 0

Reported ESD : 2,0

Vendor : IBM-ESXS

Model : MBA3300RC

Firmware : SA06

Serial number : BJ504MHL

World-wide name : 500000E01F7F8CF1
Size : 0 MB

Write Cache : Unknown

FRU : 43X0817

PFA : Yes

Device #1

Device is a Hard drive

State : Online

Supported : Yes

Transfer Speed : SAS 3.0 Gb/s

Reported Channel,Device : 0,1

Reported Location : Enclosure 0, Slot 1

Reported ESD : 2,0

Vendor : IBM-ESXS

Model : MBA3300RC

Firmware : SA06

Serial number : BJ504M6N

World-wide name : 500000E01F7DC3D1

Size : 286102 MB

Write Cache : Disabled (write-through)

FRU : 43X0817

PFA : No

Device #2

Device is a Hard drive

State : Online

Supported : Yes

Transfer Speed : SAS 3.0 Gb/s

Reported Channel,Device : 0,2

Reported Location : Enclosure 0, Slot 2

Reported ESD : 2,0

Vendor : IBM-ESXS

Model : MBA3300RC

Firmware : SA06

Serial number : BJ504M8J

World-wide name : 500000E01F7DCE11

Size : 286102 MB

Write Cache : Disabled (write-through)

FRU : 43X0817

PFA : No

Device #3

Device is a Hard drive

State : Online

Supported : Yes

Transfer Speed : SAS 3.0 Gb/s

Reported Channel,Device : 0,3

Reported Location : Enclosure 0, Slot 3

Reported ESD : 2,0

Vendor : IBM-ESXS

Model : MBA3300RC

Firmware : SA06

Serial number : BJ504M5C

World-wide name : 500000E01F7DBEE1

Size : 286102 MB

Write Cache : Disabled (write-through)

FRU : 43X0817

PFA : No

Device #4

Device is a Hard drive

State : Online

Supported : Yes

Transfer Speed : SAS 3.0 Gb/s

Reported Channel,Device : 0,4

Reported Location : Enclosure 0, Slot 4

Reported ESD : 2,0

Vendor : IBM-ESXS

Model : MBA3300RC

Firmware : SA06

Serial number : BJ504JWK

World-wide name : 500000E01F6EA341

Size : 286102 MB

Write Cache : Disabled (write-through)

FRU : 43X0817

PFA : Yes
Device #5

Device is a Hard drive

State : Ready

Supported : Yes

Transfer Speed : SAS 3.0 Gb/s

Reported Channel,Device : 0,5

Reported Location : Enclosure 0, Slot 5

Reported ESD : 2,0

Vendor : IBM-ESXS

Model : MBA3300RC

Firmware : SA06

Serial number : BJ504MKL

World-wide name : 500000E01F812561

Size : 286102 MB

Write Cache : Disabled (write-through)

FRU : 43X0817

PFA : No

Device #6

Device is an Enclosure services device

Reported Channel,Device : 2,0

Enclosure ID : 0

Type : SES2

Vendor : IBM-ESXS

Model : VSC7160

Firmware : 1.07

Status of Enclosure services device

Temperature : Normal

从上面的状态可以看到Device #0 状态是Defunct 表示不可用，也就是挂了。预期的Device #5应该是hot spare，在这个时候应该顶替上去先rebuilding然后变为online的。但是估计是当时安装的时候没有配置好，Device #0挂了以后居然没有顶替上去。

现场约了IBM售后工程师后，IBM 工程师到现场看发现服务器上的硬盘灯没有变红或者变黄。也就是说灯坏了。。

再他更换了Device #0 的硬盘后：

Logical drive information

----------------------------------------------------------------------

Logical drive number 1

   Logical drive name                       :

   RAID level                               : 5
   Status of logical drive                  : Critical

   Size                                     : 1143986 MB

   Read-cache mode                          : Enabled

   Write-cache mode                         : Enabled (write-back)

   Write-cache setting                      : Enabled (write-back) when protected by battery

   Partitioned                              : Yes

   Number of se
4000
gments                       : 5

   Stripe-unit size                         : 256 KB

   Stripe order (Channel,Device)            : 0,0 0,1 0,2 0,3 0,4

   Defunct segments                         : No

   Defunct stripes                          : No

      Device #0

         Device is a Hard drive

         State                              : Rebuilding

         Supported                          : Yes

         Transfer Speed                     : SAS 3.0 Gb/s

         Reported Channel,Device            : 0,0

         Reported Location                  : Enclosure 0, Slot 0

         Reported ESD                       : 2,0

         Vendor                             : IBM-ESXS

         Model                              : ST3300655SS

         Firmware                           : BA2D

         Serial number                      : 3LM0LE9Z

         World-wide name                    : 5000C5001C8401B0

         Size                               : 286102 MB

         Write Cache                        : Disabled (write-through)

         FRU                                : 43X0805

         PFA                                : No

把Device #5 设置为hot spare

[root@serv1 cmdline]# ./arcconf SETSTATE 1 DEVICE 0 5 HSP

然后过了3个小时后再看：

[root@serv1 cmdline]# ./arcconf GETCONFIG 1

Could not open log file: UcliEvt.log

Controllers found: 1

----------------------------------------------------------------------

Controller information

----------------------------------------------------------------------

   Controller Status                        : Okay

   Channel description                      : SAS/SATA

   Controller Model                         : IBM ServeRAID 8k    

   Controller Serial Number                 : 40703B9

   Physical Slot                            : 0

   Installed memory                         : 256 MB

   Copyback                                 : Disabled

   Data scrubbing                           : Enabled

   Defunct disk drive count                 : 0

   Logical drives/Offline/Critical          : 1/0/0

   --------------------------------------------------------

   Controller Version Information

   --------------------------------------------------------

   BIOS                                     : 5.2-0 (15421)

   Firmware                                 : 5.2-0 (15421)

   Driver                                   : 1.1-5 (2453)

   Boot Flash                               : 5.1-0 (15411)

   --------------------------------------------------------

   Controller Battery Information

   --------------------------------------------------------

   Status                                   : Okay

   Over temperature                         : No

   Capacity remaining                       : 100 percent

   Time remaining (at current draw)         : 4 days, 5 hours, 20 minutes

   --------------------------------------------------------

   Controller Vital Product Data

   --------------------------------------------------------

   VPD Assigned#                            : 39R8875

   EC Version#                              : J85096

   Controller FRU#                          : 25R8076

   Battery FRU#                             : 25R8088

----------------------------------------------------------------------

Logical drive information

----------------------------------------------------------------------

Logical drive number 1

   Logical drive name                       :

   RAID level                               : 5
   Status of logical drive                  : Okay

   Size                                     : 1143986 MB

   Read-cache mode                          : Enabled

   Write-cache mode                         : Enabled (write-back)

   Write-cache setting                      : Enabled (write-back) when protected by battery

   Partitioned                              : Yes

   Number of segments                       : 5

   Stripe-unit size                         : 256 KB

   Stripe order (Channel,Device)            : 0,0 0,1 0,2 0,3 0,4

   Defunct segments                         : No

   Defunct stripes                          : No

----------------------------------------------------------------------

Physical Device information

----------------------------------------------------------------------

      Device #0

         Device is a Hard drive
         State                              : Online

         Supported                          : Yes

         Transfer Speed                     : SAS 3.0 Gb/s

         Reported Channel,Device            : 0,0

         Reported Location                  : Enclosure 0, Slot 0

         Reported ESD                       : 2,0

         Vendor                             : IBM-ESXS

         Model                              : ST3300655SS

         Firmware                           : BA2D

         Serial number                      : 3LM0LE9Z

         World-wide name                    : 5000C5001C8401B0

         Size                               : 286102 MB

         Write Cache                        : Disabled (write-through)

         FRU                                : 43X0805

         PFA                                : No

      Device #1

         Device is a Hard drive

         State                              : Online

         Supported                          : Yes

         Transfer Speed                     : SAS 3.0 Gb/s

         Reported Channel,Device            : 0,1

         Reported Location                  : Enclosure 0, Slot 1

         Reported ESD                       : 2,0

         Vendor                             : IBM-ESXS

         Model                              : MBA3300RC

         Firmware                           : SA06

         Serial number                      : BJ504M6N

         World-wide name                    : 500000E01F7DC3D1

         Size                               : 286102 MB

         Write Cache                        : Disabled (write-through)

         FRU                                : 43X0817

         PFA                                : No

      Device #2

         Device is a Hard drive

         State                              : Online

         Supported                          : Yes

         Transfer Speed                     : SAS 3.0 Gb/s

         Reported Channel,Device            : 0,2

         Reported Location                  : Enclosure 0, Slot 2

         Reported ESD                       : 2,0

         Vendor                             : IBM-ESXS

         Model                              : MBA3300RC

         Firmware                           : SA06

         Serial number                      : BJ504M8J

         World-wide name                    : 500000E01F7DCE11

         Size                               : 286102 MB

         Write Cache                        : Disabled (write-through)

         FRU                                : 43X0817

         PFA                                : No

      Device #3

         Device is a Hard drive

         State                              : Online

         Supported                          : Yes

         Transfer Speed                     : SAS 3.0 Gb/s

         Reported Channel,Device            : 0,3

         Reported Location                  : Enclosure 0, Slot 3

         Reported ESD                       : 2,0

         Vendor                             : IBM-ESXS

         Model                              : MBA3300RC

         Firmware                           : SA06

         Serial number                      : BJ504M5C

         World-wide name                    : 500000E01F7DBEE1

         Size                               : 286102 MB

         Write Cache                        : Disabled (write-through)

         FRU                                : 43X0817

         PFA                                : No

      Device #4

         Device is a Hard drive

         State                              : Online

         Supported                          : Yes

         Transfer Speed                     : SAS 3.0 Gb/s

         Reported Channel,Device            : 0,4

         Reported Location                  : Enclosure 0, Slot 4

         Reported ESD                       : 2,0

         Vendor                             : IBM-ESXS

         Model                              : MBA3300RC

         Firmware                           : SA06

         Serial number                      : BJ504JWK

         World-wide name                    : 500000E01F6EA341

         Size                               : 286102 MB

         Write Cache                        : Disabled (write-through)

         FRU                                : 43X0817

         PFA                                : Yes

      Device #5

         Device is a Hard drive
         State                              : Hot Spare

         Supported                          : Yes

         Transfer Speed                     : SAS 3.0 Gb/s

         Reported Channel,Device            : 0,5

         Reported Location                  : Enclosure 0, Slot 5

         Reported ESD                       : 2,0

         Vendor                             : IBM-ESXS

         Model                              : MBA3300RC

         Firmware                           : SA06

         Serial number                      : BJ504MKL

         World-wide name                    : 500000E01F812561

         Size                               : 286102 MB

         Write Cache                        : Disabled (write-through)

         FRU                                : 43X0817

         PFA                                : No

      Device #6

         Device is an Enclosure services device

         Reported Channel,Device            : 2,0

         Enclosure ID                       : 0

         Type                               : SES2

         Vendor                             : IBM-ESXS

         Model                              : VSC7160

         Firmware                           : 1.07

         Status of Enclosure services device

            Temperature                     : Normal

Command completed successfully.

希望下次再有硬盘挂的时候hotspare 能顶替上去。<
9e7f
br />