Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Page 1 of 2 1 2 >
Topic Options
#309701 - 02/05/2008 12:00 I want a new 1.8" hard drive: ZIF vs. IDE ?
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
My Dell X1 notebook could really use a new hard drive -- something larger capacity and faster than the ancient Toshiba MK6006GAH (60GB) that's in there right now.

The existing 1.8" drive uses an IDE connector; most new drives of better capacity use ZIF connectors. I've never had a ZIF drive in my hands, so I don't know if they're compatible or not.

Hugo?

The internals of the notebook include an IDE-to-ZIF conversion cable for the current drive, so I'm guessing that a ZIF drive could probably slide in as a replacement. But what about the ZIF cable?

Do ZIF drives include a short ZIF cable, say about an inch long?

Will an Apple iPod drive (120 - 250GB) work as well as a store-bought Toshiba ZIF drive, or is the firmware weird on those?

???




Attachments
ribbon.jpg

Description: The 1.8" hard drive bay, with ZIF-to-IDE cable.

pins.jpg

Description: The IDE (pins) end of the cable.

zif.jpg

Description: The ZIF end of the cable, inserted into the motherboard ZIF socket.



Top
#309704 - 02/05/2008 12:40 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
tfabris
carpal tunnel

Registered: 20/12/1999
Posts: 31600
Loc: Seattle, WA
Empeg BBS: Where the IDE expert goes to ask questions about IDE. :-)
_________________________
Tony Fabris

Top
#309708 - 02/05/2008 13:28 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
altman
carpal tunnel

Registered: 19/05/1999
Posts: 3457
Loc: Palo Alto, CA
ZIF (45 pin generally) is electrically compatible, but given that it has less pins, a simple converter is out... and ZIF drives tend to come bare, ready to plug into a flex sporting system.

I think you're likely to be limited to a drive of the same connector type. Not sure what the drives are in the iPod Classics, I can take a look today. If your 60GB drive a two platter or a single (ie, how tall is it?)

Hugo

Top
#309709 - 02/05/2008 13:30 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: altman]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: altman
I think you're likely to be limited to a drive of the same connector type. Not sure what the drives are in the iPod Classics, I can take a look today. If your 60GB drive a two platter or a single (ie, how tall is it?)


It's a two-platter drive, I think (as thick as 1.8" drives come), with a big bulge on the platter side of the unit.

And the notebook does appear to have a ZIF connector (50-pins, though), so I do have hopes for this -- just need a short ZIF cable to go between them.

Cheers


Edited by mlord (02/05/2008 13:32)

Top
#309710 - 02/05/2008 13:44 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
tman
carpal tunnel

Registered: 24/12/2001
Posts: 5528

Top
#309713 - 02/05/2008 14:51 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: tman]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada


Very useful, thanks! I've ordered a 1.8" IDE-to-ZIF converter (with cable), just for inventory here, along with a similar 1.8" 1DE-to-CF converter. Those could be useful someday.

But the best candidate appears to be a 160GB iPod Classic from Costco.

Thanks guys!


Edited by mlord (02/05/2008 15:03)

Top
#309714 - 02/05/2008 15:02 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: mlord
But the best candidate appears to be a 160GB iPod Classic from Costco.


So I guess the next question must be: will the iPod firmware accept my old (blank) 60GB drive?

Or do I have to jump through a few hoops to format/populate it first?

And.. anyone want a brand new iPod Classic, with a "refurbished" 60GB drive instead of the original ? smile

-ml

Top
#310172 - 14/05/2008 15:06 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: mlord
I've ordered a 1.8" IDE-to-ZIF converter (with cable), just for inventory here, along with a similar 1.8" 1DE-to-CF converter. Those could be useful someday.


These arrived here today from Hong Kong.

The CF-to-CEATA adapter seems to work -- I've got a Sandisk Extreme-III 4GB CF card in the notebook right now, installing Kubuntu-kde4 onto it.

There seem to be 6 unused pins on the CEATA connectors -- the original Toshiba HD had those "left over" when connected, and the CF-to-CEATA adapter also has those "spare" pins. Gotta look them up someday to see what they were put there for.

Also, the pin-1 labelling inside the notebook, and the pin-1 labelling on the CF-to-CEATA adapter disagree. I had to flip the adapter over to the other side to get it to work. I've added a label to it to help me remember next time around.

Still hunting for a 160GB CEATA drive (or even a 120GB), but so far the "unopenable" 6th generation iPod Classic seems to be the only source.

Pity one has to literally crack the case to open those up; apparently earlier generation iPods could be opened without damage, but not the newest ones.

Cheers

Top
#310174 - 14/05/2008 15:37 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: mlord
The CF-to-CEATA adapter seems to work -- I've got a Sandisk Extreme-III 4GB CF card in the notebook right now, installing Kubuntu-kde4 onto it.


Mmmm.. not bad. I'm typing this from the new Kubuntu installation now. The Sandisk-III card doesn't support UDMA, but it does do multi-word DMA "mode2", which maxes out at a theoretical 16.6666MBytes/sec. In practice, it is achieving 14-5MBytes/sec. Decent. The old Toshiba drive was good for around 18-19Mbytes/sec.

So the adapter does seem to be wired okay for DMA use -- many are not. And now I have a (temporary) silent notebook. Eeery.

Cheers

Top
#310175 - 14/05/2008 16:15 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
tman
carpal tunnel

Registered: 24/12/2001
Posts: 5528
Originally Posted By: mlord
There seem to be 6 unused pins on the CEATA connectors -- the original Toshiba HD had those "left over" when connected, and the CF-to-CEATA adapter also has those "spare" pins. Gotta look them up someday to see what they were put there for.

12 vs 18 pins? If so then its CE-ATA 4x and CE-ATA 8x. The spec allows 4 or 8 data lines.

Top
#310293 - 19/05/2008 01:19 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
altman
carpal tunnel

Registered: 19/05/1999
Posts: 3457
Loc: Palo Alto, CA
CF to CE-ATA converter? These exist? I didn't think they would. This is pretty different from CF-to-flex (just a pinout change) - you need a controller and so on...

Hugo

Top
#310299 - 19/05/2008 11:26 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: altman]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: altman
CF to CE-ATA converter? These exist? I didn't think they would. This is pretty different from CF-to-flex (just a pinout change) - you need a controller and so on...


Wait a sec.. isn't CE-ATA just another name for this connector:

EDIT: Apparently not ... back to the drawing board again.. frown



Attachments
ata.jpg

Description: Standard 1.8" ATA drive connector




Edited by mlord (19/05/2008 11:33)

Top
#310300 - 19/05/2008 11:35 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: mlord
Wait a sec.. isn't CE-ATA just another name for this connector:

EDIT: Apparently not ... back to the drawing board again.. frown


So I guess this leaves me in the hunt for a 5mm (80GB) thick ZIF drive, then. I can tuck the ATA-to-ZIF adapter into the space left over in the 10mm drive cavity.

Cheers

Top
#310302 - 19/05/2008 13:37 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Okay, I've purchased a 100GB used drive through eBay.

-ml

Top
#310608 - 29/05/2008 13:05 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: mlord
Okay, I've purchased a 100GB used drive through eBay.


Mmm.. dead end again. The drive turned out to have a ZIF interface rather than the advertised IDE connector. Returning to seller for full refund (nice seller!).

And now hunting again..

I originally wanted to upgrade for two reasons: the factory drive is much slower than current model drives, and it's too small to hold the ever-increasing RAW files generated by my camera over a 3-4 week holiday.

If it turns out that I cannot get a larger (~100GB) drive for it, then it may be time to go instead for speed alone, and put a fast UDMA CF card in there.

A decent 32GB card seems to run close to $200 right now, and a very fast one for about double that.

But I'd still rather find a larger hard drive that fits. I can fit a 5mm drive + ZIF adapter into it no problem, but there's just no space for an 8mm drive + ZIF adapter. And it takes 8mm non-ZIF drives just fine as well.

???

Top
#310643 - 29/05/2008 16:43 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: mlord
A decent 32GB card seems to run close to $200 right now, and a very fast one for about double that.


Mmm.. I'm going to try this no-name 32GB UDMA CF card for US$150 delivered.

It can go into one of my empegs if it turns out to not really support UDMA. EDIT: ..or if I can find one of these before travelling to Europe.. smile



Cheers


Edited by mlord (29/05/2008 16:48)

Top
#310654 - 29/05/2008 17:44 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
wfaulk
carpal tunnel

Registered: 25/12/2000
Posts: 16706
Loc: Raleigh, NC US
I just ordered a pair of these. US$135.98 delivered for one.

Edit: Nevermind. I'm in the wrong thread. You're going for speed.


Edited by wfaulk (29/05/2008 17:47)
_________________________
Bitt Faulk

Top
#311362 - 14/06/2008 13:05 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: mlord
Mmm.. I'm going to try this no-name 32GB UDMA CF card for US$150 delivered.

It can go into one of my empegs if it turns out to not really support UDMA.


The card arrived yesterday finally, from Germany rather than the advertised Ohio,USA location.

It does indeed do UDMA, up to UDMA/100 ("udma mode-6").
Sequential read speed is very decent, at ~35Mbytes/second.

But.. writing is DOG SLOW. I mean *REALLY* *F'IN* SLOW, at about 3.5Mbytes/second. This is easily the slowest writing CF card I have used since the 8MB card that was included with my first ever digicam.

Too slow for use in my Canon 40D (what a shame, as it could hold more than 1600 RAW files).

And unfortunately just a little too slow for my notebook. Nice and fast on the reads (much better than the existing 20MByte/sec hard disk), but just too slow writing *anything* of any size.

So it is bound for an empeg, where writes are infrequent, and where 3.5Mbyte/sec is still somewhat faster than the ethernet.

All of which brings me back to the original problem:

I want a new 1.8" drive (or decent CF) for my notebook!

Cheers

Top
#311367 - 14/06/2008 14:44 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
Attack
addict

Registered: 01/03/2002
Posts: 599
Loc: Florida
Are you more interested in speed or size? If you want speed have you searched for 1.8" IDE SSD? I see 16-32GB versions available.
_________________________
Chad

Top
#311370 - 14/06/2008 20:56 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: Attack]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: Attack
Are you more interested in speed or size? If you want speed have you searched for 1.8" IDE SSD? I see 16-32GB versions available.

Cost is pretty important here. I might be willing to pay up to $180 for a 32GB or larger SSD. But beyond that is not cost effective for me yet.

Meanwhile, I believe I've worked out a nifty solution, which I can demo at The Party, for anyone interested.

Pretty simple, actually:
  • I have K+Ubuntu 8.04 "Hardy Heron" installed on the CF card now, with all of my fav goodies, including a full empeg/Hijack build system etc..
  • Plus VMware and some VMs for that other O/S, so that SWMBO can access her Italian language s/w (web based, IE-only).
  • At boot time, we now have two choices: boot/run from the CF card, or..
  • Boot from CF, but run with a tmpfs (RAM) overlay on it, so nothing gets written to the drive unless we really want that to happen.

This is more or less what all of the LiveCD's do, except simpler.

And the machine is now *very* fast, silent, and seems to gets a little extra run time from the battery.

Cheers

Top
#311371 - 14/06/2008 21:46 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
andym
carpal tunnel

Registered: 17/01/2002
Posts: 3996
Loc: Manchester UK
Thanks to Mark i managed to do something similar a while ago using Gentoo and unionfs on a 4GB CF card. Very useful, and the card should last for ages as changes are only written to it when the machine is restarted. Not that that's ever happened. The machine is completely solid state.
_________________________
Cheers,

Andy M

Top
#311372 - 14/06/2008 22:20 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: andym]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: andym
Thanks to Mark i managed to do something similar a while ago using Gentoo and unionfs on a 4GB CF card.


Wayy cool, Andy! Good to hear it is still giving great service!

Mine is using aufs ("another unionfs") rather than the older unionfs, but apart from that it's pretty much the same setup.

Cheers!

Top
#311391 - 16/06/2008 14:52 Running (mostly) from RAM (script) [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: mlord
Mine is using aufs ("another unionfs") rather than the older unionfs, but apart from that it's pretty much the same setup.

Continuing in the tradition of using the internet as a huge personal backup device, here is the little script I use to convert my notebook into a RAM-based kiosk at boot time, with read/write access still available under /.rw/ for when I need it.

Save it as /sbin/aufs_init and make it executable (chmod 755 /sbin/aufs_init).

To use it, just add init=/sbin/aufs_init to the kernel command line (via the GRUB menu at boot time, or statically in the /boot/grub/menu.lst file.

It then runs and does its tricks before passing control to the real (usual) /sbin/init program.

I also hacked the system startup script /etc/init.d/checkroot.sh to detect this and not bother attempting filesystem checks when running from RAM.

And just for fun, I also added custom reboot/halt scripts to just do immediate actions rather than running through all of the usual crap, but again only when running from RAM (2-second reboots!).

The result is a *very* fast notebook, with low wear/tear on the el-cheapo solidstate CF card. But it can also be used with a plain hard disk, for situations where you want to loan it or experiment with new software, and not have anything new saved permanently.

Note that this does require a kernel with built-in aufs ("another union filesystem") support, which is not yet in mainline. Ubuntu kernels *do* have it by default, though.

Here's the primary script used to boot/run mostly from RAM:

Code:
#!/bin/bash
#
# /sbin/aufs_init
#
# Generic script to overlay a r/w rootfs with a tmpfs in RAM,
# so that writes go only to RAM rather than the real root device.
# If any step fails, we abort and continue with normal system init.
#
# The real rootfs is still accessible (r/w) as /.rw/ when needed.
#
# Invoke with "init=/sbin/aufs_init" on the kernel command line.
#

PATH="$PATH:/sbin:/usr/sbin"
INIT=/sbin/init
TMPFS=.tmpfs
AUFS=.aufs
OLDROOT=.rw
CONS=/dev/console
ROOTDEV=/dev/root

## We want/need r/w access to the real filesystem:
cd /
mount -n "${ROOTDEV}" / -oremount,rw,noatime,nodiratime,errors=remount-ro

## Create mount points (if not already there) for tmpfs and aufs filesystems:
mkdir -p "${TMPFS}" || exec "${INIT}"
mkdir -p "${AUFS}"  || exec "${INIT}"

## Create/mount the tmpfs RAM-based filesystem:
mount -n tmpfs "${TMPFS}" -t tmpfs || exec "${INIT}"

## Use aufs to overlay the tmpfs on top of the rootfs:
mount -n -t aufs -o"noatime,nodiratime,br:${TMPFS}=rw:/=ro" none ${AUFS} || exec "${INIT}"

# Clean out some leftover temporary files, to save a little space:
cd "${AUFS}"
rm var/log/* &> /dev/null
rm -r tmp/* tmp/.[A-Z] &> /dev/null
rmdir "${TMPFS}" "${AUFS}"

# Create a default /etc/mtab file on the new aufs:
echo "none / aufs rw,noatime,nodiratime 0 0" > "etc/mtab"

## Now switch to the new aufs root filesystem, and continue with normal init:
mkdir -p "${OLDROOT}" || exec "${INIT}"
pivot_root . "${OLDROOT}"
exec chroot . "${INIT}" < "./${CONS}" > "./${CONS}" 2>&1



Edited by mlord (16/06/2008 15:17)

Top
#311394 - 16/06/2008 17:36 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
gbeer
carpal tunnel

Registered: 17/12/2000
Posts: 2665
Loc: Manteca, California
Originally Posted By: mlord

Pretty simple, actually:
  • I have K+Ubuntu 8.04 "Hardy Heron" installed on the CF card now, with all of my fav goodies, including a full empeg/Hijack build system etc..
  • Plus VMware and some VMs for that other O/S, so that SWMBO can access her Italian language s/w (web based, IE-only).
  • At boot time, we now have two choices: boot/run from the CF card, or..
  • Boot from CF, but run with a tmpfs (RAM) overlay on it, so nothing gets written to the drive unless we really want that to happen.

This is more or less what all of the LiveCD's do, except simpler.


Unless you really want it too???

I may be reading too much into this but, Does this mean that at the end of a session you have the choice to commit the current ram overlay onto the CF, that it will become the new boot state?
_________________________
Glenn

Top
#311400 - 16/06/2008 19:18 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: gbeer]
andym
carpal tunnel

Registered: 17/01/2002
Posts: 3996
Loc: Manchester UK
I don't know about Mark's install, but the only things that go into the RAM fs are temporary files which can just be binned anyway. When I need to make changes to system-level stuff (new packages, kernel rebuild and the like) I just type rw and the filesystem remounts read/write (in true empeg style). Once I'm done, I issue a ro which puts the card back into read only mode. My home folder is mounted over NFS.
_________________________
Cheers,

Andy M

Top
#311402 - 16/06/2008 19:27 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: gbeer]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: gbeer
I may be reading too much into this but, Does this mean that at the end of a session you have the choice to commit the current ram overlay onto the CF, that it will become the new boot state?

No, not yet anyway.

When I want to update/write something permanently, I just type rw first, and then perform whatever commands I want from within that shell -- the results are committed directly to the CF/disk from there. Or just copy stuff to anywhwere I like under the /.rw/ tree.

But there are surprisingly few things in that category -- email is all on remote servers, and nothing from the web needs saving usually.

But I am also thinking now about a script to commit (or "snapshot") the current RAM state to disk, permanently. It just means that a script is needed to loop over the .tmpfs filesystem and copy/move stuff from there back to the permanent /.rw/ CF/disk filesystem (and handle deleted stuff as well). On command.

Cheers


Edited by mlord (16/06/2008 19:32)

Top
#311409 - 17/06/2008 00:19 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
gbeer
carpal tunnel

Registered: 17/12/2000
Posts: 2665
Loc: Manteca, California
For a machine like that, Intended to have it's data out on the web somewhere. I can see that.

Me, I'm not so good at keeping things like bookmarks synced across various logins.

I'm finding all this very interesting. I have a little Sony pcg-sr33k that is begging to have one or two CF cards slipped into the place of the hard drive. It only has a 10gb hard drive as it is.

Though, it's a bit short of ram to be giving any away as a drive overlay.
_________________________
Glenn

Top
#311416 - 17/06/2008 11:28 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: gbeer]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: gbeer
For a machine like that, Intended to have it's data out on the web somewhere. I can see that.

No, just the email boxes are out on the web. There's another 20GB of mostly static data in the CF card, and it *can* be rewritten at will, when done deliberately. Just not by default.

Quote:
Me, I'm not so good at keeping things like bookmarks synced across various logins.

That concern/idea is there regardless of running from disk or RAM.

Quote:
I'm finding all this very interesting. I have a little Sony pcg-sr33k that is begging to have one or two CF cards slipped into the place of the hard drive. It only has a 10gb hard drive as it is.

Though, it's a bit short of ram to be giving any away as a drive overlay.

You might be very surprised there. If it can boot/run from a Linux LiveCD, then it has enough RAM for the job. Easy test.

The only RAM difference between using a R/W disk and a mostly R/O disk/CF, is for temporary files. Those will occupy RAM in the latter setup. But they are small, and are not saved across reboots. On my little notebook here, they total 2-20MBytes.

Disable swap, disable browser disk-caches, and hit the road with it!

Cheers

Top
#311418 - 17/06/2008 11:34 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: mlord
The only RAM difference between using a R/W disk and a mostly R/O disk/CF, is for temporary files. Those will occupy RAM in the latter setup. But they are small, and are not saved across reboots. On my little notebook here, they total 2-20MBytes.

Speaking of which, I looked in my notebook here to see exactly what temporary files were being created, (list them all with this: find /.rw/.tmpfs/ -ls).

The biggest culprit turned out to be /lib/modules/*, which for some odd reason always seemed to get a copy of the object file of each loaded kernel module after booting. Weird.

So I looked at the module loader program (/sbin/modprobe), and discovered that it was using O_RDWR when opening the modules for loading, which causes them to be cloned into RAM. So I changed the code to instead use O_RDONLY, and thus saved a decent chunk of RAM.

The rationale for it using O_RDWR looks rather weak -- it wants to lock the file temporarily for something, but that's inherently racey regardless, so what's the point ?

Cheers

Top
#311420 - 17/06/2008 11:41 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: mlord
So I looked at the module loader program (/sbin/modprobe) ...

Speaking of which, here's another code tidbit that I haven't backed up here before:
Mark's simple kernel module loader, useful for when size matters:
Code:
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <syscall.h>

int main (int argc, char *argv[])
{
        int fd;
        struct stat st;
        void *contents;

        fd = open(argv[1], O_RDONLY);
        if (fd != -1 && fstat(fd, &st) != -1) {
                contents = mmap(0, st.st_size, PROT_READ, MAP_SHARED, fd, 0);
                if (contents != MAP_FAILED)
                        if (syscall(SYS_init_module, contents, st.st_size, "") == 0)
                                return 0;
        }
        perror(argv[1]);
        return 1;
}

Top
#312363 - 22/07/2008 11:45 Re: Running (mostly) from RAM (script) [Re: mlord]
andym
carpal tunnel

Registered: 17/01/2002
Posts: 3996
Loc: Manchester UK
Originally Posted By: mlord
Originally Posted By: mlord
Mine is using aufs ("another unionfs") rather than the older unionfs, but apart from that it's pretty much the same setup.

Continuing in the tradition of using the internet as a huge personal backup device, here is the little script I use to convert my notebook into a RAM-based kiosk at boot time, with read/write access still available under /.rw/ for when I need it.


Just tried this script on a little ALIX board with a 4GB CF card, running Ubuntu Server 8.04 LTS. It seems when starting up, amongst various errors from the ksyslogd init script I see this in dmesg:

Code:
[   58.058116] aufs au_opts_parse:957:mount[3486]: unknown option errors=remount-ro
[   58.065444] aufs au_opts_parse:957:mount[3487]: unknown option errors=remount-ro


I've install aufs-tools from apt but other than that it's a stock install.

mount yields the following:

Code:
/dev/sda1 on / type ext3 (rw,relatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
/sys on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,noexec,nosuid,nodev,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
udev on /dev type tmpfs (rw,mode=0755)
devshm on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
lrm on /lib/modules/2.6.24-16-generic/volatile type tmpfs (rw)
securityfs on /sys/kernel/security type securityfs (rw)


but cat /proc/mounts gives:

Code:
rootfs / rootfs rw 0 0
none /.rw/sys sysfs rw,nosuid,nodev,noexec 0 0
none /.rw/proc proc rw,nosuid,nodev,noexec 0 0
udev /.rw/dev tmpfs rw,relatime 0 0
fusectl /.rw/sys/fs/fuse/connections fusectl rw,relatime 0 0
/dev/disk/by-uuid/ac5dd425-ea8d-4173-86cf-fe8029fb0098 /.rw ext3 rw,noatime,nodiratime,relatime,data=ordered 0 0
/dev/disk/by-uuid/ac5dd425-ea8d-4173-86cf-fe8029fb0098 /.rw/dev/.static/dev ext3 rw,relatime,data=ordered 0 0
tmpfs /.rw/.tmpfs tmpfs rw,relatime 0 0
none / aufs rw,noatime,nodiratime,relatime,xino=/.rw/.tmpfs/.aufs.xino,br:/.rw/.tmpfs=rw:/.rw=ro 0 0
proc /proc proc rw,nosuid,nodev,noexec 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec 0 0
tmpfs /var/run tmpfs rw,nosuid,nodev,noexec 0 0
tmpfs /var/lock tmpfs rw,nosuid,nodev,noexec 0 0
tmpfs /lib/modules/2.6.24-16-generic/volatile tmpfs rw,relatime 0 0
none /dev/.static/dev aufs rw,noatime,nodiratime,relatime,xino=/.rw/.tmpfs/.aufs.xino,br:/.rw/.tmpfs=rw:/.rw=ro 0 0
udev /dev tmpfs rw,relatime 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
devpts /dev/pts devpts rw,relatime 0 0
tmpfs /var/run tmpfs rw,nosuid,nodev,noexec 0 0
tmpfs /var/lock tmpfs rw,nosuid,nodev,noexec 0 0
securityfs /sys/kernel/security securityfs rw,relatime 0 0


Any ideas Mark?
_________________________
Cheers,

Andy M

Top
#312365 - 22/07/2008 12:18 Re: Running (mostly) from RAM (script) [Re: andym]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Note that the system looks like it should be working fine, according to the stuff you posted above. But..

Quote:
[ 58.058116] aufs au_opts_parse:957:mount[3486]: unknown option errors=remount-ro

Okay, here are two other scripts that I modified to help the system run smoothly. The first is a hacked copy of the Ubuntu /etc/init.d/checkroot.sh script, from which that error above came.

The second is a replacement for the /etc/init.d/umountroot script to prevent similar complaints on shutdown.

You can use diff to compare these with the originals on your system to see what parts got hacked.

Cheers


Attachments
checkroot.sh (391 downloads)
Description: /etc/init.d/checkroot.sh

umountroot (184 downloads)
Description: /etc/init.d/umountroot




Edited by mlord (22/07/2008 12:24)

Top
#312367 - 22/07/2008 12:22 Re: Running (mostly) from RAM (script) [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
And here are some very fast halt/reboot scripts, which I linked into /etc/rc0.d/K01aufs_sysrq_halt and /etc/rc6.d/K01aufs_sysrq_reboot


Attachments
aufs_sysrq_halt (195 downloads)
Description: /etc/rc0.d/K01aufs_sysrq_halt

aufs_sysrq_reboot (172 downloads)
Description: /etc/rc6.d/K01aufs_sysrq_reboot



Top
#312368 - 22/07/2008 12:26 Re: Running (mostly) from RAM (script) [Re: andym]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Quote:
mount yields the following:
...
but cat /proc/mounts gives:
...
Any ideas Mark?

Nothing looks particularly wrong there to me -- the system should be working with those mounts.

The mount command simply parses /etc/mtab and shows only the mounts listed therein, whereas cat /proc/mounts always shows the real picture of things -- in this case with more detail than is useful. Thus the differences.

Cheers

Top
#312383 - 22/07/2008 17:48 Re: Running (mostly) from RAM (script) [Re: mlord]
andym
carpal tunnel

Registered: 17/01/2002
Posts: 3996
Loc: Manchester UK
Looks like it's done the trick.

Code:
none on / type aufs (rw,noatime,nodiratime)
/dev/sda1 on /.rw type ext3 (relatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
/sys on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,noexec,nosuid,nodev,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
udev on /dev type tmpfs (rw,mode=0755)
devshm on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
lrm on /lib/modules/2.6.24-16-generic/volatile type tmpfs (rw)
securityfs on /sys/kernel/security type securityfs (rw)


...and the bootlog is now clean.

I'm in over the VPN so I can't time the reboots with the new rc scripts. That'll have to wait until tomorrow.

These scripts are seriously useful, have you thought about making them more widely available?
_________________________
Cheers,

Andy M

Top
#312384 - 22/07/2008 18:08 Re: Running (mostly) from RAM (script) [Re: andym]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: andym
These scripts are seriously useful, have you thought about making them more widely available?

Yes, someday. smile

By coincidence, I'm giving a workshop session on "hacking the empeg" at OLS-2008 (Ottawa Linux Symposium) later this week. I basically picked that topic because I didn't have a better one at the time.

But something like this could be my ticket for next year, or perhaps some other conference. smile

The system shutdown is not 100% clean yet, though. It's safe, but it doesn't fully unmount everything as it should. I think another pivot_root (back to the underlying CF/disk) might be needed to correct it. Once I hack at that and get it working 100% correct, *then* I'll have something to give a workshop on.

Cheers


Edited by mlord (22/07/2008 18:12)

Top
#312404 - 24/07/2008 14:30 Re: Running (mostly) from RAM (script) [Re: mlord]
canuckInOR
carpal tunnel

Registered: 13/02/2002
Posts: 3212
Loc: Portland, OR
Originally Posted By: mlord
But something like this could be my ticket for next year, or perhaps some other conference. smile

The Open Source Convention is in Portland during late July...

Top
#312405 - 24/07/2008 14:48 Re: Running (mostly) from RAM (script) [Re: canuckInOR]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: canuckInOR
Originally Posted By: mlord
But something like this could be my ticket for next year, or perhaps some other conference. smile

The Open Source Convention is in Portland during late July...

Yeah, I think it is taking place *this week* already.

Maybe linux.conf.au in Tasmania, though.. smile

Top
#312435 - 25/07/2008 14:55 Re: Running (mostly) from RAM (script) [Re: mlord]
canuckInOR
carpal tunnel

Registered: 13/02/2002
Posts: 3212
Loc: Portland, OR
Originally Posted By: mlord
Originally Posted By: canuckInOR
Originally Posted By: mlord
But something like this could be my ticket for next year, or perhaps some other conference. smile

The Open Source Convention is in Portland during late July...

Yeah, I think it is taking place *this week* already.

It is -- but you did say next year, so I thought I'd throw it out there.
Quote:
Maybe linux.conf.au in Tasmania, though.. smile

Hmm... Oregon... Tasmania... Oregon... Tasmania.

Well, you got me beat. smile

Top
#313241 - 27/08/2008 17:13 aufs_commit.sh [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Here's a new (UPDATED AGAIN!) script for the collection.

This one, will commit changes made during an AUFS session back to the underlying persistent filesystem.

It only looks for added/deleted/modified regular files and directories, and skips changes to /tmp and /var/tmp contents.

Probably not perfect yet, but it does work for me here.
Code:
#!/bin/bash
#
# Script to "commit" aufs tmpfs changes back to the underlying persistent filesystem.
#

SCRIPT="`realpath $0`"
ROOT=/.rw
AUFS=$ROOT/.tmpfs
arg1="$1"

MIRRORDIR=/usr/bin/mirrordir
if [ ! -x $MIRRORDIR ]; then
        echo "$MIRRORDIR: not found; aborting." >&2
        exit 1
fi

COMMIT=""
if [ "$1" = "--commit" ]; then
        COMMIT="$1"
        shift
fi

function doit(){
        echo "$*"
        [ "$COMMIT" = "--commit" ] && $*
}

if [ "$1" = "" ]; then
        cd $AUFS || exit 1
        find . -type f -name .wh.\* | xargs -n1 $SCRIPT $COMMIT WHITEOUT
        doit sync

        cd $AUFS || exit 1
        find . | xargs -n1 $SCRIPT $COMMIT COPY
        doit sync

        [ "$COMMIT" = "--commit" ] || (echo ;echo "$SCRIPT: Dry run complete. Use --commit to do it for real"; echo)
        exit 0
fi

if [ "$1" = "WHITEOUT" ]; then
        cd $AUFS || exit 1
        dir="${2%%/.wh.*}"
        fn="${2##*/.wh.}"
        if [ "$dir" = "." ]; then
                [ "$fn" = ".rw" -o "$fn" = ".aufs" -o "$fn" = ".tmpfs" ] && exit 0
        fi
        if [ "$fn" = ".wh..opq" ]; then
                if [ -d "$ROOT/$dir" ]; then
                        if [ -d "$dir" ]; then
                                doit $MIRRORDIR -v --exclude-regexp '.*/[.]wh[.].*' "$dir" "$ROOT/$dir"
                        else
                                doit rm -rf "$ROOT/$dir"
                        fi
                fi
                exit 0
        fi
        if [ "$dir/.wh.$fn" = "$2" ]; then
                [ -e "$ROOT/$dir/$fn" ] && doit rm -rf "$ROOT/$dir/$fn"
        fi
        exit 0
fi

if [ "$1" = "COPY" ]; then
        cd $AUFS || exit 1
        if [ -d "$2" ]; then
                [ "$2" = "./.wh..wh.plink" ] && exit 0
                cd $AUFS || exit 1
                dir="${2%%/.wh.*}"
                if [ ! -e "$ROOT/$dir" ]; then
                        doit mkdir -p "$ROOT/$dir"
                        doit $MIRRORDIR -v --exclude-regexp '.*/[.]wh[.].*' "$dir" "$ROOT/$dir"
                fi
                exit 0
        fi
        [ "${2:0:6}"  = "./.wh."     ] && exit 0
        [ "${2:0:6}"  = "./tmp/"     ] && exit 0
        [ "${2:0:10}" = "./var/tmp/" ] && exit 0
        dir="${2%%/*}"
        fn="${2##*/}"
        [ "${fn:0:4}" = ".wh." ] && exit 0
        [ ! -e "$ROOT/$dir" ] && doit mkdir -p "$ROOT/$dir"
        if [ -f "$2" -a -f "$ROOT/$2" ]; then
                if diff -q "$2" "$ROOT/$2" >/dev/null ; then
                        [ "$2" -nt "$ROOT/$2" ] && touch -m -r "$2" "$ROOT/$2"
                        exit 0
                fi
        fi
        doit cp -dp "$2" "$ROOT/$2"
        exit 0
fi

echo "$SCRIPT: huh?  $*" >&2

Cheers


Attachments
aufs_commit.sh (181 downloads)
Description: aufs_commit.sh




Edited by mlord (28/08/2008 14:35)

Top
#313242 - 27/08/2008 17:21 Re: aufs_commit.sh [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
So with this new commit script, one can now do things like.. download/install some software, play around with it, and finally either "reboot" to discard everything, or run the script to retain the new software permanently.

Top
#313243 - 27/08/2008 17:25 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: gbeer]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: gbeer
Does this mean that at the end of a session you have the choice to commit the current ram overlay onto the CF, that it will become the new boot state?


With the newly posted aufs_commit.sh script, the answer is now YES! Except that it can be done as often as desired, not only at the end of a session.

Top
#313245 - 27/08/2008 19:00 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: mlord]
tman
carpal tunnel

Registered: 24/12/2001
Posts: 5528
In theory "cheap" SATA SSDs.

Looks pretty neat. I wonder what the price will be?

Top
#313278 - 28/08/2008 14:06 Re: aufs_commit.sh [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
I have now updated the aufs_commit.sh script in the original posting above, to reflect some bugfixes and improvements. Now requires mirrordir to be installed before using (it checks).

-ml

Top
#315589 - 27/10/2008 15:30 Pretec (true) 233X CF cards -- 32GB for US$114 [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Following up again here:

A seller on eBay is now offering Pretec brand 233X CF cards in 32GB capacity for US$114. These cards apparently read at ~35MBbytes/sec and write at ~25MBytes/sec, far faster than the early cheap CF cards.

I'll be replacing the 32GB card in my notebook with a Pretec card, to fix the agonizingly slow write speeds there, and the old slower card (reads at ~35MB/sec, writes at ~1MB/sec) will get moved to one of the empegs here.

Cheers

Top
#315616 - 27/10/2008 23:53 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: tman]
gbeer
carpal tunnel

Registered: 17/12/2000
Posts: 2665
Loc: Manteca, California
Originally Posted By: tman
In theory "cheap" SATA SSDs.

Looks pretty neat. I wonder what the price will be?


If I am reading between the lines correctly. They are getting higher speeds by striping across the chips inside the flash drive?
_________________________
Glenn

Top
#315639 - 29/10/2008 02:31 Re: I want a new 1.8" hard drive: ZIF vs. IDE ? [Re: gbeer]
altman
carpal tunnel

Registered: 19/05/1999
Posts: 3457
Loc: Palo Alto, CA
Originally Posted By: gbeer
Originally Posted By: tman
In theory "cheap" SATA SSDs.

Looks pretty neat. I wonder what the price will be?


If I am reading between the lines correctly. They are getting higher speeds by striping across the chips inside the flash drive?


Pretty much all flash cards do this to some extent. Actual flash chips themselves are dog slow individually (well, below 10MB/sec to a single die, plus there's your flash translation layer overhead given that you can't erase single "sectors", hence there's a lot of logical remapping going on).

I think the faster ones have multiple buses and can have multiple concurrent operations per bus, plus enough buffer memory on the controller to be able to do things fairly optimally. It can get pretty hairy making NANDs go fast in my experience smile

Hugo

Top
#315960 - 06/11/2008 00:15 Re: Pretec (true) 233X CF cards -- 32GB for US$114 [Re: mlord]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Originally Posted By: mlord
I'll be replacing the 32GB card in my notebook with a Pretec card, to fix the agonizingly slow write speeds there, and the old slower card (reads at ~35MB/sec, writes at ~1MB/sec) will get moved to one of the empegs here.


The Pretec 32GB 233X card arrived today, and I've already migrated shrimp to it. Which in itself is a statement, because with the original 32GB card I'd still be waiting for the copy to finish now, instead of typing this post from the new card!

41MBytes/sec sequential reads, 10-20MBytes/sec writes. Good!

I'm happy! smile

Top
Page 1 of 2 1 2 >