Nefunkční RAID 5

[Marek]

Zdravim,

delsi dobu jsem mel PC s RAID 5 slozenym ze 3 x 2 TB disku vypnuty, ted jsem ho zapnul a nestacim se divit, partition s RAIDem se nepripojila a podle vypisu to vypada ze pri nabihani system 2 disky odmitnul a treti je v tom pripade na nic.

Prikladam vypis:

Kód: [Vybrat]

[    1.427672] md: Scanned 3 and added 3 devices.
[    1.427777] md: autorun ...
[    1.427875] md: considering sdd1 ...
[    1.427981] md:  adding sdd1 ...
[    1.428108] md:  adding sdc1 ...
[    1.428215] md:  adding sdb1 ...
[    1.428900] md: created md127
[    1.429007] md: bind<sdb1>
[    1.429133] md: bind<sdc1>
[    1.429252] md: bind<sdd1>
[    1.429369] md: running: <sdd1><sdc1><sdb1>
[    1.429691] md: kicking non-fresh sdd1 from array!
[    1.429797] md: unbind<sdd1>
[    1.444042] md: export_rdev(sdd1)
[    1.444170] md: kicking non-fresh sdc1 from array!
[    1.444277] md: unbind<sdc1>
[    1.460041] md: export_rdev(sdc1)
[    1.460509] md/raid:md127: device sdb1 operational as raid disk 0
[    1.461626] md/raid:md127: allocated 3260kB
[    1.461827] md/raid:md127: not enough operational devices (2/3 failed)
[    1.461995] RAID conf printout:
[    1.462001]  --- level:5 rd:3 wd:1
[    1.462007]  disk 0, o:1, dev:sdb1
[    1.462648] md/raid:md127: failed to run raid set.
[    1.462753] md: pers->run() failed ...
[    1.462854] md: do_md_run() returned -5
[    1.462953] md: md127 stopped.
[    1.463060] md: unbind<sdb1>
[    1.484070] md: export_rdev(sdb1)
[    1.484413] md: ... autorun DONE.

Kód: [Vybrat]

cat  /proc/mdstat
Personalities : [raid6] [raid5] [raid4] 
md1 : inactive sdd1[2](S) sdb1[0](S) sdc1[1](S)
      5860534784 blocks
       
unused devices: <none>

Nevite prosim nekdo co by bylo mozne zkusit?
Napr jeden odebrat a zkusit to nejak ze 2 zase sestavit a nebo je podle vypisu nahore vse ztraceno?
Docela by to zamrzelo :I

[Reklama]

[Marek]

Tak se mi to timto

Kód: [Vybrat]

mdadm --assemble --force --verbose /dev/md1 /dev/sdb1 /dev/sdc1 /dev/sdd1
mdadm: looking for devices for /dev/md1
mdadm: /dev/sdb1 is identified as a member of /dev/md1, slot 0.
mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 1.
mdadm: /dev/sdd1 is identified as a member of /dev/md1, slot 2.
mdadm: forcing event count in /dev/sdc1(1) from 844492 upto 844498
mdadm: clearing FAULTY flag for device 1 in /dev/md1 for /dev/sdc1
mdadm: Marking array /dev/md1 as 'clean'
mdadm: added /dev/sdc1 to /dev/md1 as 1
mdadm: added /dev/sdd1 to /dev/md1 as 2 (possibly out of date)
mdadm: added /dev/sdb1 to /dev/md1 as 0
mdadm: /dev/md1 has been started with 2 drives (out of 3)

podarilo dostat do:

Kód: [Vybrat]

cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md1 : active raid5 sdb1[0] sdc1[1]
      3907021824 blocks level 5, 64k chunk, algorithm 2 [3/2] [UU_]
      
unused devices: <none>

čekal bych, že se teď rozjede recovery sam a nebo musim sda1 přidat? zkusil jsem to mountnout a jde to

[Trubicoid2]

To je degradovanej stav, jestli data jsou v poradku a na sdd1 tedy neni nic zasadniho, tak ho pridej
Mdadm rika, ze je out-of-date, teda vypadl z pole jako prvni, ale muze se mylit no

Tim pridanim se pusti resync a data na sdd1 se prepisou

Kód: [Vybrat]

mdadm /dev/md1 --add /dev/sdd1

Kdyz by nechtel, muzes mu smazat superblock na sdd1, ale asi nebude potreba

[Marek]

Uff tak za dnesek to dopadlo neajk blbe, po restartu dmesg:

Kód: [Vybrat]

[ 2119.906661] md: md1 stopped.
[ 2119.906684] md: unbind<sdb1>
[ 2119.932041] md: export_rdev(sdb1)
[ 2119.932110] md: unbind<sdd1>
[ 2119.952030] md: export_rdev(sdd1)
[ 2119.952080] md: unbind<sdc1>
[ 2119.972032] md: export_rdev(sdc1)
[ 2125.694126] md: md1 stopped.
[ 2125.771284] md: bind<sdc1>
[ 2125.771781] md: bind<sdd1>
[ 2125.772369] md: bind<sdb1>
[ 2125.775830] md/raid:md1: device sdb1 operational as raid disk 0
[ 2125.775839] md/raid:md1: device sdc1 operational as raid disk 1
[ 2125.776760] md/raid:md1: allocated 3260kB
[ 2125.777609] md/raid:md1: raid level 5 active with 2 out of 3 devices, algorithm 2
[ 2125.777618] RAID conf printout:
[ 2125.777622]  --- level:5 rd:3 wd:2
[ 2125.777628]  disk 0, o:1, dev:sdb1
[ 2125.777633]  disk 1, o:1, dev:sdc1
[ 2125.777703] md1: detected capacity change from 0 to 4000790347776
[ 2125.778484] RAID conf printout:
[ 2125.778493]  --- level:5 rd:3 wd:2
[ 2125.778499]  disk 0, o:1, dev:sdb1
[ 2125.778504]  disk 1, o:1, dev:sdc1
[ 2125.778509]  disk 2, o:1, dev:sdd1
[ 2125.779171] md: recovery of RAID array md1
[ 2125.779183] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[ 2125.779188] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[ 2125.779198] md: using 128k window, over a total of 1953510912k.
[ 2125.814437]  md1: unknown partition table
[ 2136.569661] ata2.00: exception Emask 0x0 SAct 0x3 SErr 0x0 action 0x0
[ 2136.569671] ata2.00: irq_stat 0x40000008
[ 2136.569679] ata2.00: failed command: READ FPDMA QUEUED
[ 2136.569694] ata2.00: cmd 60/00:00:d8:d4:09/04:00:00:00:00/40 tag 0 ncq 524288 in
         res 41/40:00:30:d8:09/00:00:00:00:00/40 Emask 0x409 (media error) <F>
[ 2136.569701] ata2.00: status: { DRDY ERR }
[ 2136.569706] ata2.00: error: { UNC }
[ 2136.578339] ata2.00: configured for UDMA/133
[ 2136.578477] sd 1:0:0:0: [sdb] Unhandled sense code
[ 2136.578484] sd 1:0:0:0: [sdb]  
[ 2136.578489] Result: hostbyte=0x00 driverbyte=0x08
[ 2136.578495] sd 1:0:0:0: [sdb]  
[ 2136.578499] Sense Key : 0x3 [current] [descriptor]
[ 2136.578506] Descriptor sense data with sense descriptors (in hex):
[ 2136.578509]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[ 2136.578527]         00 09 d8 30 
[ 2136.578536] sd 1:0:0:0: [sdb]  
[ 2136.578541] ASC=0x11 ASCQ=0x4
[ 2136.578546] sd 1:0:0:0: [sdb] CDB: 
[ 2136.578550] cdb[0]=0x28: 28 00 00 09 d4 d8 00 04 00 00
[ 2136.578566] end_request: I/O error, dev sdb, sector 645168
[ 2136.578574] md/raid:md1: read error not correctable (sector 643120 on sdb1).
[ 2136.578583] md/raid:md1: Disk failure on sdb1, disabling device.
md/raid:md1: Operation continuing on 1 devices.
[ 2136.578599] md/raid:md1: read error not correctable (sector 643128 on sdb1).
[ 2136.578606] md/raid:md1: read error not correctable (sector 643136 on sdb1).
[ 2136.578612] md/raid:md1: read error not correctable (sector 643144 on sdb1).
[ 2136.578618] md/raid:md1: read error not correctable (sector 643152 on sdb1).
[ 2136.578625] md/raid:md1: read error not correctable (sector 643160 on sdb1).
[ 2136.578631] md/raid:md1: read error not correctable (sector 643168 on sdb1).
[ 2136.578637] md/raid:md1: read error not correctable (sector 643176 on sdb1).
[ 2136.578643] md/raid:md1: read error not correctable (sector 643184 on sdb1).
[ 2136.578650] md/raid:md1: read error not correctable (sector 643192 on sdb1).
[ 2136.578702] ata2: EH complete
[ 2136.616041] md: md1: recovery done.
[ 2136.660747] RAID conf printout:
[ 2136.660756]  --- level:5 rd:3 wd:1
[ 2136.660762]  disk 0, o:0, dev:sdb1
[ 2136.660767]  disk 1, o:1, dev:sdc1
[ 2136.660772]  disk 2, o:1, dev:sdd1
[ 2136.688022] RAID conf printout:
[ 2136.688030]  --- level:5 rd:3 wd:1
[ 2136.688036]  disk 1, o:1, dev:sdc1
[ 2136.688041]  disk 2, o:1, dev:sdd1
[ 2136.688055] RAID conf printout:
[ 2136.688059]  --- level:5 rd:3 wd:1
[ 2136.688063]  disk 1, o:1, dev:sdc1
[ 2136.688068]  disk 2, o:1, dev:sdd1
[ 2136.708023] RAID conf printout:
[ 2136.708030]  --- level:5 rd:3 wd:1
[ 2136.708035]  disk 1, o:1, dev:sdc1
[ 2177.299914] EXT3-fs (md1): error: unable to read superblock

[trubicoid2]

tak sdb chcipa, zkus sestavit pole ze zbyvajicich dvou, sdb vytahni

vono je lepsi ten raid5 kombinovat se smart testama v cronu, trebas jednou mesicne, pak prave vis, kterej disk je nacatej

[Reklama]

[trubicoid2]

vono si to malo lidi uvedomuje, ale takovyto scenar je casty: jeden disk je nacatej, raid se rozpadne, no tak ho sestavim, udela se resync, ale ten zatizi ten nacatej disk tak, ze ho to muze i dodelat 

[Marek]

Diky, akorat z sdd a sdc to nejde sestavit

Kód: [Vybrat]

mdadm: looking for devices for /dev/md1
mdadm: /dev/sdd1 is identified as a member of /dev/md1, slot 3.
mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 1.
mdadm: no uptodate device for slot 0 of /dev/md1
mdadm: no uptodate device for slot 4 of /dev/md1
mdadm: added /dev/sdd1 to /dev/md1 as 3
mdadm: added /dev/sdc1 to /dev/md1 as 1
mdadm: /dev/md1 assembled from 1 drive and 1 spare - not enough to start the array.

Kód: [Vybrat]

Personalities : [raid6] [raid5] [raid4] 
md1 : inactive sdc1[1](S) sdd1[3](S)
      3907023872 blocks
     
unused devices: <none>

[trubicoid2]

ty uz jsi asi zacal delat ten resync

Kód: [Vybrat]

mdadm /dev/md1 --add /dev/sdd1

jak jsem radil prvne, ze?

to asi teda nedobehlo, ne? proto on ho nechce tam dat, porad neni up-to-date

tak dve moznosti:


1: sloz stejne z sdd a sdc a pridej --force, problem je, ze sdd neni up-to-date, teda nejaky fajly muzou byt naboreny

2: sloz z sdc a sdb, problem je, ze sdb je chciply, takze nejaky fajly nepujdou precist a pole se bude furt rozpadat

spis bych zkusil 1

[Marek]

Presne tak pridani sdd1 se za chvili rozpadlo.
z SDD a SDC s --force to prave neslozim, to jsem zkousel v minulem prispevku.
z SDB a SDC to sice slozim, ale asi to nebude uplne OK. CP jsem vypnul a dokud nesezenu nesezen dostatk externich disku abych tam prenesel 3 TB tak ho asi zapinat nebudu a pak bych teda zkusil okpirovat od tech nejdulezitejsich souboru, co se povede no.

[trubicoid2]

ten resync asi bohuzel oznacil sdd1 jako spare az do doby, nez by rsync dobehl

je super strasna metoda, jak to zvratit, a sice musis vytvorit pole znova se stejnyma parametrama (chunk, superblock, .. ) a stejnyma diskama, misto vadnyho sdb1 das missing a nesmis zapomenout --assume-clean (i kdyz s missing mozna ne, ale pro jistotu, jinak se dela hned pri vytvareny resync, coz nechcem)

ja to jednou delal, povedlo se, ale tobe doporucuju to udelat jako uplne posledni moznost ze vsech

udelej teda 2

a ten sdb budes nahrazovat? jeste by slo koupit novej sdb, pomoci ddrescue okopirovat co jde ze sdb vadnyho na sdb novej a pak skladat sdb novej se sdc, naboreny budou jen soubory se sektorama, ktery ddrescue neprecetlo

vyhoda je, ze se pole nebude rozpadat; nevyhoda, ze misto rozpadnuti bude obsah souboru jinej, zato okopirujes vic souboru

ddrescue je specialne delany na vadny disky, akorat dderescue muze bezet dlouho, nekolik dni - i tyden

vono nakonec muzes slozit sdb-sdc, bud ten puvodni, nebo okopirovanej sdb

a pak jeste slozit sdd-sdc tou posledni super strasnou metodou, kdyz das vzdy jen dva disky spolu, tak se nebudou rsyncovat a tedy menit