Hardware RAID Advice

Posted by: Cris

Re: Hardware RAID Advice - 11/04/2007 14:06

Oh yea, big problem there, esp if I use a 2nd hand eBay card.

Well, now I am totally confused. I don't want to spend much more money on this, can anyone recommend a Linux disro I may get on with ???

Cheers

Cris.

Posted by: mlord

Re: Hardware RAID Advice - 11/04/2007 14:31

Quote:
Oh yea, big problem there, esp if I use a 2nd hand eBay card.

Well, now I am totally confused. I don't want to spend much more money on this, can anyone recommend a Linux disro I may get on with ???

Cheers

Cris.

Well, totally sight-unseen, I would recommend Ubuntu Server. The new "Fiesty" is due for final (initial) release later this month, and would probably do an excellent job for you.

(I don't currently have any RAIDs set up around here; the big iron bits I just do manually, because I'm working at a layer or two beneath the RAID s/w. In other words, I'm not much use to you unless you can pop over with your system for an evening.

)

Posted by: RobotCaleb

Re: Hardware RAID Advice - 11/04/2007 14:35

Out of curiosity, what would you do if he showed up with his box?

Posted by: mlord

Re: Hardware RAID Advice - 11/04/2007 14:38

Mmm.. when using the regular (non "server") Ubuntu, the RAID installers are on the "alternate installation" disc images, rather than the default "live" CD images.

Dunno about the "server" edition, but if you go to download a copy, it sounds like the "alternate install" CD is the one you want for RAID.

-ml

Posted by: Cris

Re: Hardware RAID Advice - 11/04/2007 14:42

Quote:
I'm not much use to you unless you can pop over with your system for an evening.

Great! I'll be on the 20:00, see you in 12 hours!

Now do you think I can check in a 4U Rackmount case as hand baggage....

On the more serious note, I will give your suggestion a try and see how I get on.

Cheers

Cris.

Posted by: mlord

Re: Hardware RAID Advice - 11/04/2007 14:46

Quote:
I will give your suggestion a try and see how I get on.

Ack. I'm pulling down the current daily build of "Fiesty Server" to try on one of the racks here later today or tomorrow. If I remember, I'll report back on how it looks for RAID stuff.

Cheers

Posted by: wfaulk

Re: Hardware RAID Advice - 11/04/2007 16:42

Personally, I would suggest running Solaris on it. Two of my biggest problems with Linux are that it regularly completely revamps large subsystems and its documentation is for shit. Both of these problems have a huge impact on RAID.

While Solaris has overhauled its RAID subsystem from time to time (twice in the last ten years, I think), the user interface has remained largely the same. In addition, its documentation is very complete and, I think, understandable.

It may be important to point out that Solaris is free of cost these days, and has been for many years. It is possible that you might have a hardware incompatibility, though. I haven't kept up to date on Solaris x86's hardware compatibility.

Posted by: mlord

Re: Hardware RAID Advice - 11/04/2007 16:57

Quote:
Two of my biggest problems with (???) are that ...

Missing noun, there.

Quote:
... it regularly completely revamps large subsystems

Ah.. MS-Windows. Okay.

Quote:
.. and its documentation is for shit.

Yup, must be MSWin.

It certainly couldn't be Linux on that first point, at least not for RAID stuff -- the interface has been stable for much of the past decade. The second point could apply to almost anything, though.

Cheers

Posted by: wfaulk

Re: Hardware RAID Advice - 11/04/2007 18:07

I will never say that Windows is better than Linux, but that doesn't mean that Linux doesn't change stuff out with little ramification. They might decide to swap out the RAID system tomorrow with a completely new interface. Just because they haven't yet doesn't mean they won't, and their track record doesn't inspire confidence.

That doesn't mean that any of the things they've swapped out or the things they've replaced them with are bad, just that they constantly change, and if a concern is the ability to reproduce, that might be a significant legitimate concern.

Posted by: Neutrino

Re: Hardware RAID Advice - 11/04/2007 18:50

Cris;

One of the questions that I don't believe has been asked yet is why? Why do you want to use raid5? I currently have both the Promise SX6000 and a 3ware RAID 5 controller sitting on the bench. I don't think I'll every go that route again. The cost of harddrives has drop so much that I no longer feel the need to use something as complex as raid 5 for a hardware backup solution. 320gb drives warranted for 5 years are available for $80.00 USD. Why not just go RAID1 or use a large usb drive to backup your data?

Through the course of my usage of the above two controllers I had a few problems. The promise card declared non-existent simultaneous multiple hard drive failures. Not a good thing. Deleting and recreating the array without initialization saved the day but the pucker factor was huge. The 3ware would rebuild the raid1 array I had on it from time to time. I had both a raid5 and a raid1 running on this controller.

At any rate in todays world with the low cost of storage is this really the best solution?

Posted by: mlord

Re: Hardware RAID Advice - 11/04/2007 18:55

Quote:
that doesn't mean that Linux doesn't change stuff out with little ramification.

Huh? Linux is an operating system kernel. It didn't exist in 1991. By 1992 it was a complete OS. By 1999 many of the major subsystems (network, RAID, various filesystems, etc..) had been replaced with better implementations.

For the past decade (give or take a year), *none* of this has had any significant external regressions. Lots of new stuff, for sure, but the old stuff is still there and still works. I still do my company invoices on a WordPerfect binary from the mid-1990s.

We actually have a very firm rule requiring no userspace impact when kernel internals are rearranged.

Quote:

They might decide to swap out the RAID system tomorrow with a completely new interface

Not true. Sure, "they" might *add* another RAID implementation, but the one that's there now will also have to stay put for a very long time -- we require it.

Okay, so now look beyond the kernel itself to the whole system. Redhat is the longest lived example, and they've tried very hard to remain more-or-less "the same" in clients eyes. Which is why I dislike them, of course: their distro is still living in the late 1990s, while newer ones (like Ubuntu) with no legacy-base have been free to build in mid-2000s code.

Cheers

Posted by: Cris

Re: Hardware RAID Advice - 11/04/2007 19:12

Quote:
Why not just go RAID1 or use a large usb drive to backup your data?

All good points! The basic idea was to create something cheap (ie using as many parts as I already have) and something that offers as near to 100% up time of data 24/7 as possible with DIY kit.

I am very very lazy when it comes to backups, I want a NAS to sit on my network to take regular automated backups from my 4 main systems in the house. A USB drive is out of the question. The HD's in this system have been taken from various other ideas for back up involving USB solutions.

As for RAID 5, I guess it looked really good on the box

in effect giving me 50% extra space from 4 drives. I thought it would be easy, as I have found other RAID solutions in the past. I am starting to wonder if I should start looking at the other solutions again, but it has to sit on the network.

FreeNAS has been a bit of a disapointment, it looks really good, says it does everything I want it to, but it's not very stable yet. I think I will go with Mark's suggestion as this was also used in an article I linked to further up in the thread. I also may try clarkconnect as I am confident with that.

One thing I really wanted was a system that would boot from a USB stick. FreeNAS does this really well. Power consumption is quite important for me, and the idea of not having HD's spun up until really needed sounds like a good idea to me. Heat is an ongoing issue with my under stairs server room, which seems to be stable at the moment, but I think 5 HD's spun up will tip that balance

Cheers

Cris.

Posted by: andym

Re: Hardware RAID Advice - 11/04/2007 19:21

Quote:
FreeNAS has been a bit of a disapointment, it looks really good, says it does everything I want it to, but it's not very stable yet.

There's a couple of people I know that use it without any problems. Could there be a hardware problem? Their hardware is a little more antiquated than yours but other than that it seems pretty solid.

Posted by: Cris

Re: Hardware RAID Advice - 11/04/2007 19:30

Quote:
Could there be a hardware problem?

I don't think so.

The lock ups are pretty consistant. For example, once the RAID 5 is complete, it locks when you hit the format button, every time! I have tried different RAM, but when I pulled them out I noticed the originals were Crucial, and that this MD and CPU were my CCTV server for about a year, in this case. Never crashed once, which is pretty amazing for XP I think.

I can bring it over on Sunday if you want to take a look for me

Cheers

Cris.

Posted by: wfaulk

Re: Hardware RAID Advice - 11/04/2007 19:55

Well, my usual example is the NAT/FW code. How many times has that been swapped out over the years? Three? Each incompatible with the previous.

How many attempts at a /dev filesystem have there been? How often did that break existing things? Glibc incompatibilites have abounded. Gcc problems. The list goes on and on.

And I intentionally use "they", specifically because it's not a single entity. And that lack of a single entity is the cause of many of these problems. It's not as if it doesn't create some good things, too. There's a lot to be said for a lack of monopoly (though RedHat had something of a de facto monopoly for many years), but it also results in incompatibilities, reduplication of effort, etc.

And, yes, I recognize that "Linux" only really applies to the kernel. You know what I mean; don't be disingenuous. If there was a better term for what I was talking about, I'd have used it.

Posted by: pim

Re: Hardware RAID Advice - 11/04/2007 20:07

Here's my pro Linux software RAID strory:

All my tunes (flac & mp3) are on a RAID-5 array of 6 250 GB drives. When one of the drives died, I replaced it with a 500 GB drive lying around. The broken drive was under warranty, and a replacement drive arrived two weeks later. I forced the 500 GB drive to failed state and inserted the replacement drive, assuming I could just rebuild the array. Then Murphy hit me: a second drive from my array made horrible noise when I applied power. It was dead. Now two drives were faulty and my array would not start, even after switching back to the 500 GB drive. The 500 GB was non-fresh and would not be considered as part of the array. Luckily mdadm had a --force option that would bring back the array to life. All my data was there and intact. I wonder whether any hardware RAID solution would save me in this situation.

Here's my con hardware RAID story:

I once lost a lot of data on an IBM ServeRAID array, where the RAID controller would give an I/O error on a sector of the virtual disk, where every drive that built the array was just fine. Recreating the array solved the condition, but the data was lost.

Pim

Posted by: pim

Re: Hardware RAID Advice - 11/04/2007 20:09

Quote:
f there was a better term for what I was talking about, I'd have used it.

GNU/Linux ?

Pim

Posted by: LittleBlueThing

Re: Hardware RAID Advice - 11/04/2007 20:23

I set up this page:
http://linux-raid.osdl.org/index.php/Main_Page
It was a while ago and I've not finished <blush>

I got permission to copy and modify the linux-RAID FAQ and I hang out on the linux-raid list a bit too.

I run a 1.2TB RAID5 for my myth box and a 1.2Tb RAID6 for mrs lbt's video editing habit.
Our important stuff is striped on raid0 (and then backed up using snapshots).

I have replaced 4 hard disks showing SMART failures since xmas and have another 3 to return. Currently my RAID6 is 'degraded' (ie has a dead disk) but since it can survive 2 dead disks it would still run fine if another disk died before tomorrow.

Oh, tonight I finally got around to setting up wake-on-lan too.

So I guess if you have any issues.... I've got some hands on experience and I'd be happy to help

Posted by: mlord

Re: Hardware RAID Advice - 11/04/2007 20:41

Quote:
Well, my usual example is the NAT/FW code. How many times has that been swapped out over the years? Three? Each incompatible with the previous.

And very heavy overlap among them. For several years the kernel could still be used with any of the three methods. Gradually, the first one (ipfwadm) was dropped, then the second (ipchains) got dropped quite recently after years of "depreciation". The current userspace interface, "iptables", has been in place for about 9-10 years now. *That's* stability in this world.

Quote:

How many attempts at a /dev filesystem have there been?

The original "static /dev" from 1991 is still supported and works fine. Along the way, we picked up "devfs" as a clearly marked "EXPERIMENTAL" method, and it was carried for many years. Only a month or two ago did it finally get dropped from the bleeding edge kernels. Over the past two years, various new features in the kernel have enabled proper hotplug support, and a userspace "udev" device manager app has grown up to use those features. But again, the original 1991 method still works just fine.

Quote:
How often did that break existing things?

Zero, or near to it, unless one deliberatly switched to a different method.
Installing an entire new system, eg. Ubuntu, of course will give the new methods as default. As it should.

Quote:
Glibc incompatibilites have abounded.

That's a BIG stretch. More correctly, all modern OSs abandoned the original UNIX executable format "a.out" in the early 1990s, in favour of the newer ELF format. This also meant a new set of ELF libraries, separate from the a.out ones. Both can/do coexist. My current bleeding edge Linux kernel *still* can run both kinds of binaries. Very VERY backward compatible.

About 10 years ago, GNU libc (glibc) had a major revision, again producing a new set of library interfaces that are quite different from before. So again, a new set of libs and a new program loader (ld.so.2) got installed in parallel with the old. My old Wordperfect program still uses the old libs, on the latest Kubuntu systems. Fantastic backward compatibility there.

Quote:
Gcc problems.

That's not Linux. And it's not a problem anyway. Just continue to use whatever old version you like. Until very recently, kernel developers were all still using the ancient gcc-2.95 to build kernels. I think it still works today, though we've moved on.

Quote:
The list goes on and on.

It certainly does! Stellar backward compatibility, no major upheavals in established distros. The hot new ones like (K)Ubuntu of course install only the latest stuff by default, but the older stuff is there too.

One can definitely fault a Linux distro on many points, but not this one.

Been there, done that, still doing it.

-ml

Posted by: drakino

Re: Hardware RAID Advice - 12/04/2007 00:31

mdadm --force is a scary thing to use late at night trying to recover the array. For those that have never seen it, --force is buried in the manual under a ton of warnings, then when run, the binary also warns you again in a full screen of text and a second switch you have to use to then really confirm. I had to go through it back when I had one drive fail, and another was on it's way out. It let me get most of the data off the array, with only the files at the bad points being lost. I think it ended up being one CDs worth of MP3s that I had to rerip from that.

In fact, if I remember, thats when I posted here asking about having surface scans done from time to time to catch errors quicker. Double failures are the worst with RAIDs, hardware or software. I got spoiled by the Compaq RAID hardware always checking disks when they could, as it often rooted out failures way before the drive completely pitched over dead.

I've had good experiences with hardware RAID, but the type of solutions I worked with in my past job are likely way outside the price range of home use. One benefit I really remember that usually is only in the realm of software RAID is the portability of the data. A 1995 Smart Array controller could die in some server, and even if a like for like replacement wasn't available, the drives could be plugged into a 2007 Smart array (likely 7th or 8th generation by now) and still be able to read the data. A few newer features like full array expansion and such might not be available, but the data would still be in tact and could be moved off to a newer system. Sadly similar compatibility stories are rare on the hardware side.

Posted by: mlord

Re: Hardware RAID Advice - 13/04/2007 14:39

Quote:
I'll try pulling/reinserting drives next.

Manual intervention is required here. The raid tools, specifically mdadm, are crap. But they do work, so one just has to write the magic into a recipe script and then never worry about it again.

I pulled one of the four SATA drives, which had partitions that were members of the RAID1 (root filesystem) and RAID5 (/usr/local filesystem) arrays. There was a pause, while the low level libata tried to recover the failed drive, and then the system just continued working afterward.

But reinserting the drive is a current known hassle. It gets seen as an entirely different device than the one that was removed. So manual intervention is required to delete the old unit from the RAIDs, and then add the new one.

The drive I pulled was /dev/sdb, and then became /dev/sde on reinsertion.
So I had to do this to get it back into the arrays without a reboot:

mdadm /dev/md0 -f /dev/sdb1 -r /dev/sdb1 -a /dev/sde1
mdadm /dev/md1 -f /dev/sdb2 -r /dev/sdb2 -a /dev/sde2

That could easily be scripted.

But there was another wrinkle: after the drive is pulled, /dev/sdb no longer exists.
Well, sort of. It''s "half there", but udevd nuked the /dev/ entries for it.
This prevents actually issuing the above commands.

So the full sequence on a udev system is this:
---------------------------------
(1) removal:
## yank the drive, wait about 30 seconds for the fault-recovery to give up, then:

mdadm /dev/md0 -f /dev/.static/dev/sdb1 -r /dev/.static/dev/sdb1
mdadm /dev/md1 -f /dev/.static/dev/sdb2 -r /dev/.static/dev/sdb2

## reinsert the drive, wait about 30 seconds again, then:

mdadm /dev/md0 -a /dev/sdb1
mdadm /dev/md1 -a /dev/sdb2
---------------------------------
Note that with this method, the new drive gets the same name as it previously had.
But with USB discs coming and going, there's no guarantees, unfortunately.

Also, if the inserted drive is a new one, then it will need to first be partititioned correctly before doing those last two commands. The simple way, given identical drives, is to do this just before those last two commands:

dd if=/dev/sda of=/dev/sdb bs=512 count=1
hdparm -z /dev/sdb

Definitely in need of a higher level tool to simplify this.
Mind you, a simple bash script (20-50 lines?) could totally automate it all.

Cheers

Posted by: Cris

Re: Hardware RAID Advice - 14/04/2007 05:05

Wow! Thanks Mark!

Well my hardware RAID recovery didn't go quite how I wanted. I didn't loose my data in the event of loosing a drive, which I did with my previous software RAID attempts. But, I couldn't get the thing to rebuild.

I can really see the disadvantage of not having any tools to monitor the array. I think that the drive would begin to recover, but when I re-booted it started over. I couldn't see a way of checking, even from the command line in freenas, what stage the rebuild was at. The only place tht told me that was the RAID's BIOS.

So I am still torn between both solutions. I like both for different reasons. I do have enough spare HD's to run 4 x 250Gb and 3 x 120Gb arrays so maybe I'll run both side by side and directly compare how I cope with recovery etc....

I could of course just drop back to a simple mirror and forget RAID5, but I really want that extra 250Gb.

Cheers

Cris.

Posted by: mlord

Re: Hardware RAID Advice - 14/04/2007 09:44

Quote:

I could of course just drop back to a simple mirror and forget RAID5, but I really want that extra 250Gb.

One other thing I noticed, is that the RAID5 had substantially greater throughput on large sequential reads (like copying big files around -- .flac or .mpeg2 for instance.

On RAID1 (Linux soft) the throughput was the same as for a single drive on reads. For RAID5 it was more like a RAID0 -- 3 drives in parallel. 71Mbytes/sec vs. 210Mbytes/sec.

Cheers

Posted by: mlord

Re: Hardware RAID Advice - 14/04/2007 09:52

Quote:

I could of course just drop back to a simple mirror and forget RAID5, but I really want that extra 250Gb.

Yeah, I know the feeling.

A *lot* of people seem to prefer RAID 1+0 with a four drive setup.
That's two pairs of RAID1, which are then combined into a single virtual drive
with RAID0 striping across the two pairs. Or the other way around (0+1)

Very reliable, very fast -- more so than just two pairs of RAID1 would be.
But still no extra 250GB.

Cheers

Posted by: mlord

Re: Hardware RAID Advice - 14/04/2007 09:57

Quote:
Wow! Thanks Mark!

Well, on a related note: the Feisty bug database is presently littered with RAID roadkill. Lots of people seem to have seen similar installer issues to what I ran into, and my fix for it (intervene in the boot process to get it up, and then just update the packages to repair it all) doesn't work for most of them.

So there are/were still other issues. I think most problems were with PATA (IDE) drives rather than pure SATA configurations -- the kernel currently has two sets of device drivers (backward compatibility overlap again, Bitt) for PATA, and Ubuntu apparently has some kinks to work out in their driver selection logic.

I'll pull another image down later today, and see if things are improving or not -- and if not, then perhaps fix it for them.

Cheers

Posted by: wfaulk

Re: Hardware RAID Advice - 14/04/2007 12:48

Quote:
A *lot* of people seem to prefer RAID 1+0 .... Or the other way around (0+1)

Technically, RAID 1+0 (more commonly referred to as "RAID 10"), is superior to RAID 0+1 due to the number of drives that can be lost and the RAID still remain active, but, in the case of just four drives total (which is the minimum setup for these types of RAID), it degenerates to being the same.

Posted by: mlord

Re: Hardware RAID Advice - 14/04/2007 14:13

Quote:
Quote:
Wow! Thanks Mark!

Well, on a related note: the Feisty bug database is presently littered with RAID roadkill. Lots of people seem to have seen similar installer issues to what I ran into, and my fix for it (intervene in the boot process to get it up, and then just update the packages to repair it all) doesn't work for most of them.

Okay, the latest word on this is that those fixes have now made it into the installer disks as of 14-April-2007, and RAID installs now seem to be working for everyone.

Woo-Hoo!

Posted by: Cris

Re: Hardware RAID Advice - 14/04/2007 14:37

Quote:
But still no extra 250GB

In an ideal world I would just go out any buy 4 x 500Gb SATA drives, a local supplier has Seagate drives at under £70 now. With this solution 1Tb would be more than enough so RAID10 would be the perfect solution.

Although speed isn't a major issue at present, I'm running and ever increasing Gigabit network at home. At the moment it's just the Macbook and one of the servers as my switch only has 2 Gigabit ports. I have noticed that even though the spec of the new NAS box is much higher than my current clarkconnect firewall, the CC box transfers are much quicker. This could be down to the intel gigabit nic I suppose, the one in the NAS is a £15 D-Link (nasty!).

If only I had put a bet on Ascot I could be straight down to pick up my SATA drives

Cheers

Cris.

Posted by: Cris

Re: Hardware RAID Advice - 14/04/2007 14:40

Quote:
Okay, the latest word on this is that those fixes have now made it into the installer disks as of 14-April-2007, and RAID installs now seem to be working for everyone.

With the final release so imminent I may wait for that.

Am I right in assuming I can't boot from RAID5? Or could I do a complex setup of striping and mirroring a 1Gb partition and then RAID5 the rest? Putting a 5th drive in the system isn't something I really want to do at this stage if I can help it.

Cheers

Cris.

Posted by: drakino

Re: Hardware RAID Advice - 14/04/2007 14:57

Oh, one tip if you go software side. Don't RAID the hard disk. Create partitions on the disks and RAID those, making the partition a bit smaller then the hard disk. That way, if one of your Brand X 120gig drives fails, and only a Brand Y 120g can be found later. There is a possibility Brand Y 120 gig will be a smidge smaller then Brand X, and by using a smaller setup of space on the old disk, the new one should still work into the array.

Quote:
I do have enough spare HD's to run 4 x 250Gb and 3 x 120Gb arrays

Another advantage to using partitions would be for this setup. Instead of making two separate arrays with two separate disk groups, you could make 2 arrays across the entire 7 disks. IE, 120gb in use on all 7 disks for a 720gig usable RAID chunk, then with the 130 gigs left on the 250s, turn that into a 390 gig usable array.

I'm currently doing something similar in my server. It has 3 160 drives, 1 200, and 1 500. All 5 are in the 160 gig per disk RAID. The 200 is in there because I couldn't find another 160 to replace the second failure in the array, and the spare 40gigs of space ends up being used as a quick and easy place to store a few backups of /etc and other critical areas of the root drive to allow me to restore to an earlier time. The 500 uses the spare space it has as my "unraid" share. This gets used by my laptop and Mac Mini as a backup space. I'm tempted to go grab another 2 500 gig drives and expand that unprotected space into a proper raid 5, along with expanding the 160 gig per disk raid area I have.

Posted by: mlord

Re: Hardware RAID Advice - 14/04/2007 16:02

Quote:

Am I right in assuming I can't boot from RAID5? Or could I do a complex setup of striping and mirroring a 1Gb partition and then RAID5 the rest? Putting a 5th drive in the system isn't something I really want to do at this stage if I can help it.

I'm not sure on that one. GRUB cannot boot from RAID5 yet; LIlO might be able to. But neither of them would be able to handle failover situations, so, *no*.

The standard way is to have a smallish RAID1 partition for /boot (or the rootfs), and a RAID5 partition for the rest. On each drive. Just like in the article I linked earlier. Plus a few extra GRUBbies so that grub will auto-failover among the available RAID1 partitions. I haven't set that up yet, but will give it a go later. Maybe tomorrow, as The Game will be on in an hour here, and I'd like to watch it (almost) live..

Cheers

Posted by: mlord

Re: Hardware RAID Advice - 14/04/2007 16:07

Quote:

The standard way is to have a smallish RAID1 partition for /boot (or the rootfs), and a RAID5 partition for the rest. On each drive. Just like in the article I linked earlier.

That article also suggested going with a rather complex LVM setup on top of the RAID5, and then a complex partitioning setup on top of that. This also implies having the swap partition there, on the LVM on the RAID5. One plus with this, is that then swap won't go bonkers if a disk fails, thus avoiding another source of system crashes.

EDIT: LVM (Logical Volume Manager) is just a way of grouping things together into a new virtual "block device (disk)", which can then be partitioned etc.. just like a real drive (and more..). The reason you need it here, is that the RAID drivers in Linux don't currently support being partitioned themselves. So by assigning LVM to manage the entire RAID, one can then partition the LVM instead of the lower-level RAID, and it works. Hack. /EDIT.

Personally, I'd partition each drive identically: smallish RAID1 partition, largish RAID5 partition. Put /boot on the RAID1, use LVM on the RAID5. Then partition the LVM with a smallish swap area, a pair of small/medium O/S partitions, and a massive data partition.

With the two O/S partitions, you can experiment with a newer O/S version in the future, and yet still have your original working O/S version intact until all is well with the new one.

Cheers

Posted by: mlord

Re: Hardware RAID Advice - 14/04/2007 16:14

Quote:

With the two O/S partitions, you can experiment with a newer O/S version in the future, and yet still have your original working O/S version intact until all is well with the new one.

Note that an advanced user would likely have just a single O/S partition, and would not immediately allocate the rest of the LVM to the massive data area.

Instead, they would allocate from the LVM to the data area in large chunks, as the need grows. And they'd use the nifty "LVM snapshot" capability to "fork" the O/S partition when experimenting with upgrades.

Very very fancy and useful, but maybe a bit much for starting out.

Cheers

Posted by: andy

Re: Hardware RAID Advice - 14/04/2007 17:41

Quote:
Oh, one tip if you go software side. Don't RAID the hard disk. Create partitions on the disks and RAID those, making the partition a bit smaller then the hard disk. That way, if one of your Brand X 120gig drives fails, and only a Brand Y 120g can be found later. There is a possibility Brand Y 120 gig will be a smidge smaller then Brand X, and by using a smaller setup of space on the old disk, the new one should still work into the array.

The same thing can apply to disks even of the same model number, I had an expensive mistake where my 18GB IBM drives were a few hundreds of kilo bytes smaller than the existing disks with the same model number, making them useless as replacements in my RAID5 array.

Posted by: mlord

Re: Hardware RAID Advice - 15/04/2007 18:47

Mmmm. I stumbled across the www.cooldrives.com site today. Wow, what a selection of interfaces, enclosures, etc. for SATA / IDE stuff.

Cheers

Posted by: Seth

Re: Hardware RAID Advice - 15/04/2007 22:51

Cris,
I too got the same LSI Megaraid card--couple days ago too. Having similar issues. I figured I'd explore recovery scenarios before I have to use them for real in the future. I too had trouble getting the array to rebuild, even in Windows with Power Console Plus. The best it ever did was rebuild to 50% overnight and freeze at 50%. Other times it just failed, or went to about 28% and then started over. I even tried rebuilding just within the WebBIOS utility and that failed.

Oh, yeah, and to clarify, all I did was take one of the working drives, "fail" it, and then attempt to make it rebuild. Same drive that was just working, so that eliminates all those "sizing" issues and other random gotchas. No reason it shouldn't rebuild.

It is SLOOOOOOOOW rebuilding even when making progress. An hour to rebuild about 5% or so--I have brand new 300GB drives. I also tried adjusting the rebuild speed. I can handle it taking a half day to rebuild as long as it actually rebuilds to 100%. But 3 consecutive failures is not acceptable.

Is this the problem you were having?

For me at this point, I'm saying no to MegaRAID. I starting to think mlord is right. Software RAID may be the way to go. I've always been under the impression that a hardware solution was better, or at least easier. But I'm rethinking that opinion now. The i4 card was made before all the gigantic and faster 7200rpm drives, so I'm guessing that has something to do with it. I also don't like the lie in the product datasheet on LSI's site. It says auto-resume rebuild upon reboot. That does NOT happen, and if you search their knowledge base, it actually says that the card restarts the rebuild process FROM THE BEGINNING upon reboot. That is just a simple lie. I suppose the "hot swap" claim is probably also a lie at this point.

I'm curious what others think of this card or other similar cards. Should I just throw it in the garbage?

Posted by: mlord

Re: Hardware RAID Advice - 15/04/2007 23:38

The thing to remember here is, RAID is complicated enough that *nobody* really does it in hardware. It's all software (or flash-based software a.k.a. "firmware").

So.. would you rather trust your data and time to a single factory burn of the software, or rely on something that can-be / is bug-fixed as the need arises, a.k.a. "software RAID". And if the O/S does the (software) RAID, then it can also (in theory) do a superior job of scheduling the I/O to each individual drive (= faster I/O).

The answer may also depend upon one's choice of O/S.

Cheers

Posted by: Seth

Re: Hardware RAID Advice - 16/04/2007 01:48

My problem is that I'm not a Linux god. So without rehashing the debate about backwards compatibility, I'll just say that suppose there are some new updates to the RAID implementation and I end up screwing up my data. Maybe I just don't choose the right compile flags to ensure compatibility or something like that.

What I am saying is that I don't quite trust myself with software ;-).

Also what if I have a catastrophic hardware failure (fried motherboard or something) and have to rebuild everything. I KNOW that it is possible to rebuild everything to get the data off the drives, but maybe I'm not capable of doing it without making a mistake somewhere along the way.

I'm not saying I'm an idiot, just that I've come to understand that I figure things out after stubbing my toe a couple times. I don't want to stub my toe at a critical time in the recovery process.

Not disagreeing with you at all, I just have a few concerns given my ability (or lack of) to fly by the seat of my pants without making any mistakes.

Posted by: Cris

Re: Hardware RAID Advice - 16/04/2007 04:18

Quote:
My problem is that I'm not a Linux god.

Me too!

I have only had the chance to test the MegaRAID i4 once so far. What I did notice is that if you reboot it does start the rebuild right from the start again. ie if you were at 5% and need to reboot your machine for some reason, when it all comes back up the MegaRAID starts again from 0%, but I did find it started again automatically.

I'm really glad I hve given "hardware" RAID a try. I can now really see Mark's point, you do stand more of a chance of getting your data back with software RAID if you know what you are doing.

My problem is I don't know what I am doing, yet

The other problem with the MegaRAID i4 is that it keeps the disc spun up 100% I left the system on over the weekend while I was away from home, nobody as been using the NAS and the drives are a full temp. For this reason alone the i4 is going back on eBay. This can't be good for the type of drive I am using here!

Cheers

Cris.

Posted by: mlord

Re: Hardware RAID Advice - 16/04/2007 11:31

Quote:
suppose there are some new updates to the RAID implementation and I end up screwing up my data. Maybe I just don't choose the right compile flags to ensure compatibility or something like that.

Nope. No need for the FUD. That cannot happen. To screw your data, you have to work much harder than that, and issue commands that do it on purpose, and bypass the warnings that most of them give you.

Quote:
Also what if I have a catastrophic hardware failure (fried motherboard or something) and have to rebuild everything. I KNOW that it is possible to rebuild everything to get the data off the drives, but maybe I'm not capable of doing it without making a mistake somewhere along the way.

Nothing to do with Linux there -- same issues regardless.

Backups are safer than RAID.

Cheers

Posted by: Seth

Re: Hardware RAID Advice - 16/04/2007 15:01

I'm not throwing out FUD, and my comments about rebuilding aren't aimed at Linux. I'm talking about my own comfort level based on what I know and what I don't know. In this case, there is a lot I don't know, and freely admit it.

I use Linux enough to be able to get around. I don't know it forwards and backwards in my sleep. I'm just saying I'm thinking through scenarios, and suppose I have to rebuild the array on a different box. I know it is possible, and I can probably figure out how to do it. But I don't know the process (today), and don't know what risks there might be. I see posts about using the force flag and various "pucker" factors in this thread and hence some of my fears. I figure plugging the array into the new box would require forcing it online.

And the RAID actually is my backup, so if the whole thing goes belly up, I'm actually most likely fine.

Posted by: LittleBlueThing

Re: Hardware RAID Advice - 16/04/2007 15:37

Quote:
Quote:
My problem is that I'm not a Linux god.

I'm really glad I hve given "hardware" RAID a try. I can now really see Mark's point, you do stand more of a chance of getting your data back with software RAID if you know what you are doing.

My problem is I don't know what I am doing, yet

That's the thing about 'community'

The linux-raid list has regular 'help' emails and they almost all get sorted - I think the only caveat is "don't mess if you don't know what you're doing - just ask".

Also, you mentioned wanting to power down the disks in your raid. I'm not sure how well that works 'out of the box' under linux. You may need to do some fiddling.

Posted by: LittleBlueThing

Re: Hardware RAID Advice - 16/04/2007 15:44

Quote:
I'm not throwing out FUD

No - I think Mark was heading it off before it became fear etc ie FUD.

Quote:
I see posts about using the force flag and various "pucker" factors in this thread and hence some of my fears. I figure plugging the array into the new box would require forcing it online.

That's the problem with being fully honest - I did try to emphasise that the --force stuff was to do recovery in situations where you were already royally FUBARed.

Let me put it this way, if and only if your system burns to the ground and is eaten by ants then linux raid *may* need to use the --force command

Better?

Moving to another machine - business as usual.
Disk failure - business as usual.
Switch from RedHat to SUSE to Debian - business as usual.
Upgrade kernel - business as usual.
etc etc

One of my biggest problems with linux RAID way back when was when a disk failed and I didn't notice for months - the thing just kept going.

Posted by: Cris

Re: Hardware RAID Advice - 16/04/2007 16:11

Quote:
Also, you mentioned wanting to power down the disks in your raid. I'm not sure how well that works 'out of the box' under linux. You may need to do some fiddling.

In FreeNAS it was one feature that worked really well, it must cache part/all of the file table (or whatever?) as you could pretty much browse the NAS and the drives spun up when a file was requested, but I never did get to the point of mave many files on it.

The only slight problem was that the WD drive sins up much slower than the seagate driver, that wa causing some retry errors in the log files, but from my point of view it worked pretty well.

I think the Software RAID argument is winning the battle at the moment, like you said if I get stuck all I need to do is ask

Cheers

Cris.

Posted by: mlord

Re: Hardware RAID Advice - 16/04/2007 17:11

Quote:
I'm not throwing out FUD

Just trying to prevent it from being interpreted as such when somebody stumbles over this thread via Google.

You actually sound to me like a pretty competent user of this stuff -- more power to ya!

Cheers

Posted by: Neutrino

Re: Hardware RAID Advice - 17/04/2007 03:07

This has been an interesting thread. As I mentioned at the beginning of it I am no longer interested in going with RAID5. I personally believe that with the huge decrease in the cost of storage that it has lost it’s luster as a viable choice for the home user. I have no desire to put all of my eggs in one basket. I have been down that road where I had a terabyte of data all in a RAID5 configuration and if anything happens, I am hosed.

Considering Linux as a software RAID solution, I wonder how many people in the world have intimate knowledge of Linux on a par with some of the contributors of this forum? I’ll bet not many. This is the reason why software RAID5 is not an option for me. Some of you make it sound so easy just to do whatever needs to be done to fix whatever problems might occur. For me this is an impossibility, heck, I was absolutely overjoyed when I managed to get Ubuntu to run on my Intel box equipped with a 8800 graphics cards. I think this was a “let’s celebrate and go get Sushi, my treat” night.

I would also like to use a NAS so as not to have to have a specific computer running in order to see the data that I want to share across my network. I also don’t want to spend a lot of money finding a solution. I have 6 300GB PATA drives just sitting around that came out of the RAID5. Just for fun I have chosen to play with this –
http://www.newegg.com/Product/Product.aspx?Item=N82E16833124036
along with these to put my drives in - http://www.newegg.com/Product/Product.aspx?Item=N82E16817716034
There are a ton of firmware hacks that will allow you to run multiple drives off of the USB ports or RAID1 if you want. There is also a hardware hack that will let you double the speed of the NAS just by removing a diode.

A raid system is not the answer to total data protection. Corrupted data is just as corrupted regardless of the number of copies you have. To truly protect your data I believe you must do periodic backups to devices that are generally kept offline. I think this is the only way that you can be somewhat sure that your data is safe. Periodic backups, there is no substitute.

I really enjoy this forum; the intellectual power here is truly amazing. You guys rock.

Posted by: Cris

Re: Hardware RAID Advice - 17/04/2007 03:46

Quote:
Periodic backups, there is no substitute.

But what too ???

The task of backing up 700Gb to DVD doesn't really appeal, and thenttrying to find what I want would be almost impossible.

My RAID NAS will be my backup device, all the information will be on other machines as well, it will just be a nice big pool of space to run the automatic backups to.

I think the cost of any other solution, in both money and time, would be to great for me.

Cheers

Cris.

Posted by: Neutrino

Re: Hardware RAID Advice - 17/04/2007 04:33

True, backing up 700GB IS a hassle but how much of this really changes? If you are talking media than I would expect that most of this is static data. This is the case in my situation. 1TB of movies and audio with the occasional movie or itunes download added. After the initial backup of the data I need to only add to it whatever I add to my collect. This can be done periodically. To what, would be off line drives that I only plug in to add data to. I believe that any backup scheme that is constantly online and powered up has the risk failing. Yes, it is true that there is a chance that the drives I keep offline might someday not power up when I need them but the amount of time that they are online is so small that I doubt that will happen. As far as cost is concerned what does a TB cost these days, $250.00 USD? (Based on 3 320GB drives at $80.00 USD each)

Also I didn't realize that you where planning on using your raid as a backup storage for data that is kept online in another location. When I was using RAID5 it was the total repository of all of my media. There was no other backup. At the time I considered RAID5 to be sufficient unto itself. This was a mistake. How are you planning on monitoring the status of your drives? I used the Promise Superswap 1000 drive bays that gave constant condition status via LED’s. Even with this I lost my data once. You think it’s a hassle copying 700GB? How about re-ripping a TB?

I guess I have been burned too many times. What I purpose is probably extreme.

Posted by: mlord

Re: Hardware RAID Advice - 17/04/2007 11:56

Quote:

Considering Linux as a software RAID solution, I wonder how many people in the world have intimate knowledge of Linux on a par with some of the contributors of this forum? I’ll bet not many. This is the reason why software RAID5 is not an option for me. Some of you make it sound so easy just to do whatever needs to be done to fix whatever problems might occur.

Again, the issues with administrating a RAID array are not unique to Linux. RAID5 failures will likely be just as complex to recover from regardless of the underlying O/S, or whether it's "hardware" or "software" RAID.

Just trying to keep things clear here for posterity (aka. "google").

Posted by: Neutrino

Re: Hardware RAID Advice - 17/04/2007 12:23

Quote:
Again, the issues with administrating a RAID array are not unique to Linux. RAID5 failures will likely be just as complex to recover from regardless of the underlying O/S, or whether it's "hardware" or "software" RAID.

Just trying to keep things clear here for posterity (aka. "google").

I completely agree, I did lose a TB of RAID5 data on a Windows box. In fact it sounds to me like recovery might actually be easier under Linux IF you possess the knowledge to do what must be done to recover it.

Posted by: LittleBlueThing

Re: Hardware RAID Advice - 17/04/2007 12:37

Quote:

When I was using RAID5 it was the total repository of all of my media. There was no other backup. At the time I considered RAID5 to be sufficient unto itself. This was a mistake. How are you planning on monitoring the status of your drives? I used the Promise Superswap 1000 drive bays that gave constant condition status via LED’s. Even with this I lost my data once. You think it’s a hassle copying 700GB? How about re-ripping a TB? I guess I have been burned too many times. What I purpose is probably extreme.

There's a lot being said here; some is linux specific but most, I think, is generic.

Clearly if you are experienced in one OS and move to another then there's a learning curve - that said:

Setting up:
Linux RAID is simple and generally just works.
BIOS or windows RAID is probably a touch simpler and generally just works.
Not much in it.

Monitoring:
Linux RAID is simple and generally just works via email. [My machines talk to me if a disk fails.]
BIOS or windows RAID may use email, LEDs, beep or just notify you on reboot (not good for a server) .
Not much in it.

Disaster:
For an average user it is feasible to recover a Linux RAID systems when things go badly wrong (and I'm talking about massive hardware failure here!) but it needs to be dug into very deeply and can become very complex indeed. [However a few hours of hair pulling may be a lot easier than a 1Tb restore!]

For an average user if you have an unexpected hardware failure with windows or BIOS RAID then you are screwed. You would need to spend real cash (many hundreds of $) with a forensic data recovery firm to stand a chance of getting your data back - or just start restoring.

Huge difference.

Another problem area is how people thing about risk, backups and resilience.

In other words : do I bother with RAID or do I just have backups.

It's complex; but in general RAID reduces the likelihood of needing to restore from backups - Neutrino you are 100% spot on when you say it is not sufficient to replace backups. However it does almost eliminate the need to restore from backups for disk failures.

That's worth restating:
* I'll (almost) never need to restore from backup due to disk failure.

Here, I think the problem you will face Neutrino is that you are almost *certain* to lose all your data and have to restore from backup. (I assume you have 1Tb of data on 3x320Gb disks = no resilience). Your system is 3 times more fragile than a single disk system. Assuming drives are 95% likely to die in 6 years (even spread) then your system is 95% likely to die within 2 years - I hope you have an OCD tendency to backup!

FWIW I have multiple data stores.
One is for music and films that I have recorded or ripped. The backup is the original media. When (not if) a drive fails, I just replace it. I doubt I will ever have to re-rip. If I didn't have a resilient setup then I'd have to re-rip (or restore) every time a drive failed.

Another store is a simple striped array.
I hold important data on there and it backs up incrementally onto a separate striped array using snapshots.
Having the backup online all the time means I can monitor the health and replace drives.

I hope this comes across in the spirit it's intended

A few years ago I had a disk failure on the disk with all my photos on it.
I bought a replacement hard disk and did a disk copy from my backup to my new disk.
Sadly I swapped the device names and overwrote my backup with a blank disk. I felt a horrible cold chill and the hairs on the back of my neck stood up as I realised that I'd lost years worth of photos.
Luckily I was a sysadmin at the time and by 'habit', as soon as the disk failed I made a copy of the backup to some free space. It took me 10 seconds to remember that I had the copy - they were not pleasant seconds.
I did however just do a quick 'copy' and the timestamps on all those files show the date and time of the near-disaster.

So the reason I bother with this RAID stuff is because I really worry about the 'normal' people who are going to lose their digital memories and I try to help out

I would like to see Linux RAID get a nicer frontend (for home use) and better disaster recovery etc - it will happen.

Posted by: matthew_k

Re: Hardware RAID Advice - 17/04/2007 13:10

Quote:

So the reason I bother with this RAID stuff is because I really worry about the 'normal' people who are going to lose their digital memories and I try to help out

And I was agreeing with your post until I read that. And it came right after an example of PEBKAC which always shows the true weakness of raid.

It doesn't matter how reliable your raid array is if you press the delete key.

I've said it before here. I've said it at work (though my boss doesn't really take it to heart). I've told family (who give me funny looks when I mention "parity"). Raid is not a backup solution. Raid is a high availability solution.

Reriping from original media is a backup plan, and perfectly viable. You can't do that with digital memories. If it's worth saving, it's worth saving twice.

I'll get off my soap box now. But I won't hesitate to get back on it if someone starts implying raid is a backup solution.

Matthew
PS: Perhaps I'm reading your post wrong, and you mean preserving memories is the reason you help other people with their RAID recovery. I still stand by my rant about backups, even if it doesn't apply to you.

Posted by: LittleBlueThing

Re: Hardware RAID Advice - 17/04/2007 13:29

We agree...

Quote:
Quote:

So the reason I bother with this RAID stuff is because I really worry about the 'normal' people who are going to lose their digital memories and I try to help out

And I was agreeing with your post until I read that.
...
I'll get off my soap box now. But I won't hesitate to get back on it if someone starts implying raid is a backup solution.

Matthew
PS: Perhaps I'm reading your post wrong, and you mean preserving memories is the reason you help other people with their RAID recovery. I still stand by my rant about backups, even if it doesn't apply to you.

You did misunderstand - but it was a long post and they're not always the clearest things to read or write.

For "RAID stuff" substitute "RAID and backup stuff".

RAID is bloody useful but it is not a backup solution.

PS Somehow it has more of an "Oh f*ck, did I really do that" feeling than "Kodak moments"

Posted by: Robotic

Re: Hardware RAID Advice - 18/04/2007 18:56

I've been following this thread's progression with great interest and I can really empathize with Cris and the others. I am also a Linux challenged individual.
With the help of some members on the board I built a NAS out of an old computer of mine. (the thread is around somewhere... ah- here it is.)
It works great (thanks to all!) but this thread has me thinking I need to further my understanding of the device.
For example- I have no idea how to check the health of the unit or recover from a failed drive. I'll certainly have to learn about that soon.

Anyway, I just saw the For Sale posts of Neutrino's and wondered if the following scenario is a possibility under Linux:
Use an IDE expansion PCI card (I'm thinking most often such cards are sold as 'raid controllers', but I haven't investigated this at all, so I may be going in the wrong direction) to allow more drives to be connected to the motherboard. Instead of combining the drives as under 'hardware raid', just leave them as individual drives and let the linux raid use them. You could boot off of one of the motherboard's IDE headers and if *that* drive went south, then reinstalling linux shouldn't be a big deal.

What I'm thinking about is this- in my homebrew NAS, the three drives are partitioned such that the linux install is strewn over all of the units (I'm not even sure exactly how it was set up, so forgive my lack of detail and understanding). It seems that any one failed drive requires that the linux install be recreated before the raid5 array could be rebuilt from the remaining drives.
It just seems that if I want to protect myself against failed drives, then treating each drive as a single partition and single element in the array should make things 'easier'. Sounds less complex to me, but then again I don't know what I'm talking about and that's why I'm asking...
Am I off base?

My kit is soo old that my motherboard won't boot from PCI or USB. That's why I was thinking I could expand the capacity for drives with a card.
Hmmm- maybe if I want to localize my linux install but keep my hardware setup as-is I could boot and run from a purpose-built live CD. ?? again, I'm going off on tangents that I'm not sure are really needed.

Hey, if you got this far I really want to thank you for reading! LOL

Posted by: mlord

Re: Hardware RAID Advice - 18/04/2007 19:02

For a system like that, I would do mostly what you've already figured. Boot from the onboard IDE with a dedicated drive (or compactflash card or DiskOnChip module etc..), and then use the PCI SATA controller to implement a RAID array in software.

The choice would likely fall down to RAID1 (best redundancy, simplest recovery, lowest capacity) and RAID5 (much faster reads, higher capacity, most complex recovery [but not *that* complex!]).

If the onboard IDE is a real drive, then I'd install the O/S there. If the onboard IDE device is a flash thingie, then I'd likely just put the /boot partition there, and everything else onto the RAID array. In fact, the latter is what I often recommend.

Cheers

Posted by: Neutrino

Re: Hardware RAID Advice - 18/04/2007 20:45

Prices are negotiable!

Posted by: Neutrino

Re: Hardware RAID Advice - 18/04/2007 20:55

Last night I was playing around and downloaded FreeNAS. I have an extra PC that is really quite an overkill for this but I used it anyway just to take a look at it. FreeNAS is a cinch to setup. Took maybe 30 minutes start to finish to have a NAS on the network. I only hooked up 1 drive just for testing and it went without a hitch. It took 4 minutes and 39 seconds to transfer a 1GB file across the LAN to the NAS. Network utilization ran from 15 to 65%. Not too bad really. I can see why this can be a popular alternative. I should have my Slug tomorrow. I'll set that up and see how it compares. It will provide a much less energy hungry solution but I have no idea what the speeds are going to be like. With a 266mhz clock I have my doubts.

Posted by: Robotic

Re: Hardware RAID Advice - 18/04/2007 21:24

Quote:
If the onboard IDE is a real drive, then I'd install the O/S there. If the onboard IDE device is a flash thingie, then I'd likely just put the /boot partition there, and everything else onto the RAID array. In fact, the latter is what I often recommend.

Cheers

Thanks, Mark!

What if I install Linux on an old 1GB microdrive through those CF/IDE adaptors like in the CF empeg thread!
(Just kidding- I'd expect performance would suffer. Meanwhile, I've got a couple of useless microdrives hanging around.)

Posted by: matthew_k

Re: Hardware RAID Advice - 18/04/2007 21:34

Quote:
Meanwhile, I've got a couple of useless microdrives hanging around.)

Open them up and frame them... I've never seen the inside of one, but I've always considered them an amazing technical accomplishment.

Matthew

Posted by: drakino

Re: Hardware RAID Advice - 18/04/2007 22:45

Quote:
Use an IDE expansion PCI card (I'm thinking most often such cards are sold as 'raid controllers', but I haven't investigated this at all, so I may be going in the wrong direction) to allow more drives to be connected to the motherboard.

I'm using two different PCI IDE controllers in my server currently, one that only does normal drive access, and one that offers some sort of raid protection. The RAID card has a mini bios setup program to setup the RAID, and if you never go in there and set it up, it just runs as a normal IDE controller. Linux sees both controllers the same way, and just accesses the disks directly.

Posted by: Neutrino

Re: Hardware RAID Advice - 18/04/2007 22:59

I'm using a Promise Ultra/133 2 channel ide / pci card to access more PATA drives. This can be had cheap! One of these days all my PATA drives (7) will have died and I will move on....

Posted by: Neutrino

Re: Hardware RAID Advice - 18/04/2007 23:04

OK, I added a second drive to the FreeNAS sytem and made a RAID1 system. One drive is connected via the secondary channel of IDE1 and the other is through a USB port. Not a conventional construct but it seems to work. Transfer time for a 1GB file has increased to 5:36. Interesting....

Posted by: mlord

Re: Hardware RAID Advice - 19/04/2007 00:52

Quote:
Quote:
Meanwhile, I've got a couple of useless microdrives hanging around.)

Open them up and frame them... I've never seen the inside of one, but I've always considered them an amazing technical accomplishment.

Presenting.. a GeoCaching "Travel Bug" in my honour: the actual item that was inside my camera at the 2003 Empeg Owners' Meet:

I tore the cover off, and later left it as loot in a geocache. Another local cacher retrieved it, and converted it into a "travel bug" named "Mlord's Micro Drive" (TBAF86).

Cheers

Posted by: mlord

Re: Hardware RAID Advice - 19/04/2007 00:56

Quote:

What if I install Linux on an old 1GB microdrive through those CF/IDE adaptors like in the CF empeg thread!
(Just kidding- I'd expect performance would suffer. Meanwhile, I've got a couple of useless microdrives hanging around.)

Heh.. they'd be just fine for the /boot drive, but I didn't find them terribly reliable -- my original died after a small jolt, as did it's replacement.

Still, nifty little things. Would you sell me one, and if so, for how much? (I miss my old ones).
My email address is in my BBS profile.

Cheers!

Posted by: Robotic

Re: Hardware RAID Advice - 19/04/2007 04:45

Quote:
Still, nifty little things. Would you sell me one, and if so, for how much? (I miss my old ones).

You've got mail!

Posted by: LittleBlueThing

Re: Hardware RAID Advice - 19/04/2007 06:19

I forgot to comment on this

Quote:
I'm curious what others think of this card or other similar cards. Should I just throw it in the garbage?

I wouldn't use their RAID capability - but if you have one then I'd use it as a multi-drive controller card.

Actually, you may or may not know that many cheap RAID cards are just simple multi-channel controllers - as this tongue-in-cheek FAQ says...

Posted by: tman

Re: Hardware RAID Advice - 19/04/2007 07:21

Quote:
Actually, you may or may not know that many cheap RAID cards are just simple multi-channel controllers - as this tongue-in-cheek FAQ says...

The Promise PATA "RAID" cards were actually their normal PATA controllers but with a different firmware and a resistor moved to change the chip ID. If you changed the resistor then you could reflash it with the RAID firmware. Not really that useful tho...

Posted by: Roger

Re: Hardware RAID Advice - 19/04/2007 08:12

Quote:
many cheap RAID cards are just simple multi-channel controllers

Yeah, it's often all in the drivers, with maybe some acceleration features on the card.

It's usually better to use the card in JBOD mode (just a bunch of disks), and rely on the OS's software RAID. On Linux, anyway -- I've never tried Windows' software RAID.

Posted by: Neutrino

Re: Hardware RAID Advice - 19/04/2007 21:16

I got "the slug" today. You gotta love newegg. Great service, fast shipping. Anyway, it won hands down on speed against FreeNas on my simple test of writing a 1gb file to it. I was able to write the same file to the slug that I did to the Freenas box in just 3:23. This is a full minute faster. It's as cute as a bug too. Pretty cool for 80 bucks.

Posted by: Folsom

Re: Hardware RAID Advice - 19/04/2007 23:26

I don't suppose you have a Kill-A-Watt or something similar to measure the power drain of the slug, do you? Can you tell me what the power supply is rated to get an upper bound?

I would like to get a NAS but I want low power. I was going to look at a DNS-323, but I would prefer something that took external USB drives rather than internal drives.

Posted by: Neutrino

Re: Hardware RAID Advice - 20/04/2007 04:22

It uses the same 5vdc @ 2amp powersupply as most everything else I've bought from Linksys lately so it must not draw more than 10watts. I would expect half that is more like it. It runs pretty much stone cold.

Posted by: Roger

Re: Hardware RAID Advice - 20/04/2007 04:27

Quote:
along with these to put my drives in - http://www.newegg.com/Product/Product.aspx?Item=N82E16817716034

Hmm. That looks like a rebranded IcyBox (or vice versa).

I've got a couple of those, and I found them a bit flimsy. I've also got a couple of Enermax external enclosures, which are a bit more solid.

Posted by: Seth

Re: Hardware RAID Advice - 25/04/2007 01:53

Quote:
I wouldn't use their RAID capability - but if you have one then I'd use it as a multi-drive controller card.

Problem is that I cannot seem to get that card to present an individual disk as just a single volume. The interface isn't that intuitive, but I've tried just about everything and it seems that I have to create some sort of array between 2 or more disks before I can actually get to the step of creating a volume. If I don't create a volume, then the drive is essentially invisible to the system. Otherwise I DID think of doing that, I just haven't figured it out with that card yet.

Posted by: wfaulk

Re: Hardware RAID Advice - 25/04/2007 02:01

The MegaRAID® i4 IDE RAID Controller Configuration Software Guide has a section on "Using a pre-loaded IDE drive as-is". That's probably what you want.

Posted by: Seth

Re: Hardware RAID Advice - 07/05/2007 19:31

Looks like I was trying the impossible. I had a RAID 5 array defined as the first drive and was trying to make a fourth drive a single drive. According to that I have to do the opposite....

Thanks for pointing that out.

Posted by: mlord

Re: Hardware RAID Advice - 04/06/2007 19:06

Quote:

I can really see the disadvantage of not having any tools to monitor the array. I think that the drive would begin to recover, but when I re-booted it started over. I couldn't see a way of checking, even from the command line in freenas, what stage the rebuild was at. The only place tht told me that was the RAID's BIOS.

cat /proc/mdstat

or even:

watch -n1 cat /proc/mdstat

Posted by: Robotic

Re: Hardware RAID Advice - 20/06/2007 02:39

Quote:
...nifty little things...

Quote:
2 Pack of Hitachi 4GB Microdrive for $24.99 Shipped

on woot today- missed it!

Posted by: mlord

Re: Hardware RAID Advice - 24/08/2007 10:48

Quote:
I found this set of instructions that I'm using as a loose guide for this:

http://www.debian-administration.org/articles/512

Somebody PM'd me this week, asking why that article designated one "drive" as a "hot spare" in a RAID1 array.

I very much prefer such questions to be asked in the forums/threads, like now, rather than in PM -- because this way everyone gets to see the question and contribute to the answers, and it's all nice and searchable afterward.

So, back to the question: why a "hot spare" for RAID1? Well, normally that would not make much sense to most people, as the entire concept of RAID1 is a bunch of "hot spares" kept up-to-date in parallel. But in this case, the RAID1 was just a small partition of each drive, and the bulk of the space was in a RAID5 array. A "hot spare" makes good sense for RAID5, and so that's why they did it that way.

The idea is that when a RAID5 drive fails, the system suddenly has no redundancy, so it becomes imperative to configure/update a new drive immediately --> the hot spare.

And since that article was treating all drives more or less the same, they decided to "hot spare" the RAID1 partition of that same drive for consistency. So that the drive could be removed/replaced/whatever at any time without impacting either of the two RAIDs.

There's also a typo in that article, where in the section dealing with RAID5 they accidently use "RAID1" in one of the instructions:

Now you will want to choose Create MD device again.

* Multidisk device type: RAID5
* Number of active devices for the RAID5 array: 3
* Number of spare devices for the Raid1 array: 1

Cheers

Posted by: andy

Re: Hardware RAID Advice - 24/08/2007 14:26

Surely a hotspare for RAID1 has more uses that just that. If if you have three disk and use all of them in your RAID1 array then the OS has to write to all three disks all the time.

If you make the third disk a hotspare instead then you won't have to take the overhead of writing to the third disk normally, but if one of the other disk fails a rebuild to the hotspare will start without any intervention.

Posted by: LittleBlueThing

Re: Hardware RAID Advice - 24/08/2007 14:35

... and you can have a hot spare shared between multiple arrays.

Posted by: andy

Re: Hardware RAID Advice - 24/08/2007 14:52

Does that work in the same way on Linux software RAID as it does on hardware RAID, i.e. can does the hot spare have to have the right partitions it place or can it be a unpartitioned disk ?

(my RAID1 Debian box doesn't have any hotspares yet)

Posted by: Seth

Re: Hardware RAID Advice - 25/08/2007 23:15

I was the offender..... :0 Sorry about that.

I think the thing that I didn't understand was what the config was actually doing in an operational sense. Andy sort of touched on it here.

The article had three drives in the RAID1 array. Since RAID1 is mirroring, I've never seen 3 drives in a mirrored array. So I was wondering if one drive is actually mirrored to the other 2 in the array.

Then on top of that, the RAID1 array had a spare drive..... So three active, 1 spare. I wasn't sure what that meant it was doing.

So it sounds like this...someone correct me if I am wrong:

- All writes to the boot partition are mirrored to two other drives.
- If any one of the three drives fails, the fourth will automatically join the array so that the first partition is still mirrored to two others.
- If the first drive fails, then the system would boot off drive two which would then be mirrored to drive three and four.

As an aside, the main thing that I didn't like about the article was that it presumed with just 4 drives that you want a hot spare. Well, if I wanted half my total drive capacity, I'd just use RAID1 in the first place. I'm using RAID5 to get more storage space and have it be resilient. I have a spare sitting on my shelf and am not too lazy to plug it in if I need it :-).

Which brings me to another question. Anyone know of any drive monitoring software that can detect drive failures and send an email or an alert of some sort. For my needs, that is sufficient...I don't need a hot spare if I can get notification when there is a problem.

Posted by: Seth

Re: Hardware RAID Advice - 25/08/2007 23:21

Forgot the other question I had. Any recommendations on encrypted file systems?

Posted by: andy

Re: Hardware RAID Advice - 26/08/2007 07:29

Quote:

Which brings me to another question. Anyone know of any drive monitoring software that can detect drive failures and send an email or an alert of some sort.

"aptitude install smartmontools" though you need a relatively recent setup for that to work with RAID (i.e. it doesn't work on Debian Sarge, but it does on Etch).

Posted by: LittleBlueThing

Re: Hardware RAID Advice - 26/08/2007 15:10

and, of course, mdadm runs in monitoring mode and sends emails like these:

Code:

This is an automatically generated mail message from mdadm
running on haze

A DegradedArray event had been detected on md device /dev/md0.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [raid1] [raid6] [raid5] [raid4] 
md1 : active raid5 sdd1[0] sda1[2] sdc1[1]
      390716672 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      
md0 : active raid1 sdc2[1]
      3783232 blocks [2/1] [_U]
      
unused devices: <none>

Posted by: mlord

Re: Hardware RAID Advice - 26/08/2007 18:06

Quote:

The article had three drives in the RAID1 array. Since RAID1 is mirroring, I've never seen 3 drives in a mirrored array. So I was wondering if one drive is actually mirrored to the other 2 in the array.

RAID1 arrays of four drives are actually quite common in the commercial systems I work on here. These people are really *paranoid* about their data, and don't want it ever to be exposed with no hot backup online. With only 2-drive RAID1, when one drive fails they would then be without a live backup until a new drive got "built" back into the RAID1. With three drives, no such problem. With four drives, even safer.

Quote:
So it sounds like this...someone correct me if I am wrong:

- All writes to the boot partition are mirrored to two other drives.
- If any one of the three drives fails, the fourth will automatically join the array so that the first partition is still mirrored to two others.
- If the first drive fails, then the system would boot off drive two which would then be mirrored to drive three and four.

Yup, that's the idea. Many modern BIOSs will automatically do that last step (boot from first available drive) in particular.

-ml

Posted by: andy

Re: Hardware RAID Advice - 26/08/2007 20:40

Quote:

Yup, that's the idea. Many modern BIOSs will automatically do that last step (boot from first available drive) in particular.

Does anyone have any idea on how I can actually test that step in my Linux software RAID setup. I have tested disconnecting the primary drive, but that doesn't seem to be testing quite the same case of the actual failure of one drives to spin up. I can't work out how to simulate that case though.

Posted by: Seth

Re: Hardware RAID Advice - 26/08/2007 21:24

How about just disconnect the power. I know, that isn't quite the same thing, but I've struggled with that same question myself--how to test real life failure scenarios.

Posted by: andy

Re: Hardware RAID Advice - 26/08/2007 22:25

Disconnecting the power to the drive isn't the same thing at all, it is no different to disconnecting the SATA cable, which I have tested.

Posted by: mlord

Re: Hardware RAID Advice - 27/08/2007 00:13

Quote:
Disconnecting the power to the drive isn't the same thing at all, it is no different to disconnecting the SATA cable, which I have tested.

There's no good way to test every possibility. Fortunately, with most drive (it seems), failure to spin-up means the drive will also fail to complete the "IDENTIFY" handshake, and a good BIOS will then assume the drive is not present and try the next.

But there's always a possible pissy spot there, and without a failsafe BIOS (source code generally required to verify that..), one never knows..

Cheers

Posted by: Seth

Re: Hardware RAID Advice - 31/08/2007 04:36

Quote:
Disconnecting the power to the drive isn't the same thing at all

That is why I said it wasn't really the same thing, but how else are you going to test that without hacking a BIOS to control how a drive initializes at boot?

Posted by: andy

Re: Hardware RAID Advice - 31/08/2007 06:23

Quote:
but how else are you going to test that without hacking a BIOS to control how a drive initializes at boot?

I don't know, that was the whole point of asking if it was possible and if so how !

Posted by: tman

Re: Hardware RAID Advice - 31/08/2007 10:06

Quote:
Quote:
but how else are you going to test that without hacking a BIOS to control how a drive initializes at boot?

I don't know, that was the whole point of asking if it was possible and if so how !

I'd say a rubber mallet would work...

Posted by: Roger

Re: Hardware RAID Advice - 31/08/2007 12:17

Quote:
I'd say a rubber mallet would work...

Angle-grinder? Through the platters?

Posted by: Seth

Re: Hardware RAID Advice - 26/10/2007 02:52

Quote:
I don't know, that was the whole point of asking if it was possible and if so how !

I had some unrelated issues on another system here at home which brought this thread back to mind. You were asking about different boot scenarios and how the BIOS will handle them. Without going into the details of my issues here, let me say that I think there is no way to deal with every possible contingency to ensure the system boots properly.

Here are a couple situations I am quite confident will prevent your system from booting despite all the efforts to make everything redundant:
1) Drive spins up and identifies itself to the BIOS but is not functional, or at least not really functional enough to boot from
2) Drive is the master (or is on a SCSI bus) and is malfunctioning to the point of making the channel/bus unuseable. I had this happen where one drive (the second on the bus) went bonkers (not even a real crash), and prevented the other two drives from [effectively] communicating with the controller. And when this happened, half of the boot attempts hung without even giving a "non-system disk" error.
3) Any other type of failure that allows the BIOS to see the presence of the disk, but disk is not accessible.

All of those will likely give you one of those "non-system disk or disk error" messages on boot up. The best failure scenario is where the drive is completely dead or non readable by the BIOS, in which case it most likely will move onto the next device. As I understand it, since drives became auto-detectable by the BIOS years ago, this mechanism actually works by storing the parameters in a sector on the disk, and the BIOS reads that. So if the drive really fails to spin up, the BIOS can't possibly identify it.

It occurred to me that one possible way of testing how your BIOS reacts to this type of scenario where the device exists, has power, but is "not spinning up" is to test something similar but different. If you had two drives on different IDE channels, you could deliberately misconfigure the first to be a slave instead of master on its channel, or something of that nature. If that doesn't work, maybe there is something else along those lines you could do to essentially accomplish the same thing. Then you would see what the BIOS does in that circumstance, i.e. move onto the next drive or just choke.

Just some thoughts I wanted to pass along.

BTW, for those interested, I continue to be more and more a fan of this software RAID approach using a generic Linux distro. I was enamored first by the hardware RAID, then by the specialty NAS OS (with software RAID) approaches, for different reasons. I thought it nice to have a good customized NAS GUI with slick performance graphs and so forth, but in the end none of those options stacked up to just good ol Linux. Far more robust and resilient than anything else I have experimented with. Alas, I have to do some of the customization work and configuration, but in the end I have what I need and it works like I want it to.

Hardware RAID Advice