#312363 - 22/07/2008 11:45
Re: Running (mostly) from RAM (script)
[Re: mlord]
|
carpal tunnel
Registered: 17/01/2002
Posts: 3996
Loc: Manchester UK
|
Mine is using aufs ("another unionfs") rather than the older unionfs, but apart from that it's pretty much the same setup. Continuing in the tradition of using the internet as a huge personal backup device, here is the little script I use to convert my notebook into a RAM-based kiosk at boot time, with read/write access still available under /.rw/ for when I need it. Just tried this script on a little ALIX board with a 4GB CF card, running Ubuntu Server 8.04 LTS. It seems when starting up, amongst various errors from the ksyslogd init script I see this in dmesg: [ 58.058116] aufs au_opts_parse:957:mount[3486]: unknown option errors=remount-ro
[ 58.065444] aufs au_opts_parse:957:mount[3487]: unknown option errors=remount-ro I've install aufs-tools from apt but other than that it's a stock install. mount yields the following: /dev/sda1 on / type ext3 (rw,relatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
/sys on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,noexec,nosuid,nodev,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
udev on /dev type tmpfs (rw,mode=0755)
devshm on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
lrm on /lib/modules/2.6.24-16-generic/volatile type tmpfs (rw)
securityfs on /sys/kernel/security type securityfs (rw) but cat /proc/mounts gives: rootfs / rootfs rw 0 0
none /.rw/sys sysfs rw,nosuid,nodev,noexec 0 0
none /.rw/proc proc rw,nosuid,nodev,noexec 0 0
udev /.rw/dev tmpfs rw,relatime 0 0
fusectl /.rw/sys/fs/fuse/connections fusectl rw,relatime 0 0
/dev/disk/by-uuid/ac5dd425-ea8d-4173-86cf-fe8029fb0098 /.rw ext3 rw,noatime,nodiratime,relatime,data=ordered 0 0
/dev/disk/by-uuid/ac5dd425-ea8d-4173-86cf-fe8029fb0098 /.rw/dev/.static/dev ext3 rw,relatime,data=ordered 0 0
tmpfs /.rw/.tmpfs tmpfs rw,relatime 0 0
none / aufs rw,noatime,nodiratime,relatime,xino=/.rw/.tmpfs/.aufs.xino,br:/.rw/.tmpfs=rw:/.rw=ro 0 0
proc /proc proc rw,nosuid,nodev,noexec 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec 0 0
tmpfs /var/run tmpfs rw,nosuid,nodev,noexec 0 0
tmpfs /var/lock tmpfs rw,nosuid,nodev,noexec 0 0
tmpfs /lib/modules/2.6.24-16-generic/volatile tmpfs rw,relatime 0 0
none /dev/.static/dev aufs rw,noatime,nodiratime,relatime,xino=/.rw/.tmpfs/.aufs.xino,br:/.rw/.tmpfs=rw:/.rw=ro 0 0
udev /dev tmpfs rw,relatime 0 0
tmpfs /dev/shm tmpfs rw,relatime 0 0
devpts /dev/pts devpts rw,relatime 0 0
tmpfs /var/run tmpfs rw,nosuid,nodev,noexec 0 0
tmpfs /var/lock tmpfs rw,nosuid,nodev,noexec 0 0
securityfs /sys/kernel/security securityfs rw,relatime 0 0 Any ideas Mark?
_________________________
Cheers,
Andy M
|
Top
|
|
|
|
#312365 - 22/07/2008 12:18
Re: Running (mostly) from RAM (script)
[Re: andym]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
Note that the system looks like it should be working fine, according to the stuff you posted above. But.. [ 58.058116] aufs au_opts_parse:957:mount[3486]: unknown option errors=remount-ro Okay, here are two other scripts that I modified to help the system run smoothly. The first is a hacked copy of the Ubuntu /etc/init.d/checkroot.sh script, from which that error above came. The second is a replacement for the /etc/init.d/umountroot script to prevent similar complaints on shutdown. You can use diff to compare these with the originals on your system to see what parts got hacked. Cheers
Attachments
checkroot.sh (391 downloads)Description: /etc/init.d/checkroot.shumountroot (184 downloads)Description: /etc/init.d/umountroot
Edited by mlord (22/07/2008 12:24)
|
Top
|
|
|
|
#312367 - 22/07/2008 12:22
Re: Running (mostly) from RAM (script)
[Re: mlord]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
And here are some very fast halt/reboot scripts, which I linked into /etc/rc0.d/K01aufs_sysrq_halt and /etc/rc6.d/K01aufs_sysrq_reboot
Attachments
aufs_sysrq_halt (195 downloads)Description: /etc/rc0.d/K01aufs_sysrq_haltaufs_sysrq_reboot (171 downloads)Description: /etc/rc6.d/K01aufs_sysrq_reboot
|
Top
|
|
|
|
#312368 - 22/07/2008 12:26
Re: Running (mostly) from RAM (script)
[Re: andym]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
mount yields the following: ... but cat /proc/mounts gives: ... Any ideas Mark? Nothing looks particularly wrong there to me -- the system should be working with those mounts. The mount command simply parses /etc/mtab and shows only the mounts listed therein, whereas cat /proc/mounts always shows the real picture of things -- in this case with more detail than is useful. Thus the differences. Cheers
|
Top
|
|
|
|
#312383 - 22/07/2008 17:48
Re: Running (mostly) from RAM (script)
[Re: mlord]
|
carpal tunnel
Registered: 17/01/2002
Posts: 3996
Loc: Manchester UK
|
Looks like it's done the trick. none on / type aufs (rw,noatime,nodiratime)
/dev/sda1 on /.rw type ext3 (relatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
/sys on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,noexec,nosuid,nodev,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
udev on /dev type tmpfs (rw,mode=0755)
devshm on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
lrm on /lib/modules/2.6.24-16-generic/volatile type tmpfs (rw)
securityfs on /sys/kernel/security type securityfs (rw) ...and the bootlog is now clean. I'm in over the VPN so I can't time the reboots with the new rc scripts. That'll have to wait until tomorrow. These scripts are seriously useful, have you thought about making them more widely available?
_________________________
Cheers,
Andy M
|
Top
|
|
|
|
#312384 - 22/07/2008 18:08
Re: Running (mostly) from RAM (script)
[Re: andym]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
These scripts are seriously useful, have you thought about making them more widely available? Yes, someday. By coincidence, I'm giving a workshop session on "hacking the empeg" at OLS-2008 (Ottawa Linux Symposium) later this week. I basically picked that topic because I didn't have a better one at the time. But something like this could be my ticket for next year, or perhaps some other conference. The system shutdown is not 100% clean yet, though. It's safe, but it doesn't fully unmount everything as it should. I think another pivot_root (back to the underlying CF/disk) might be needed to correct it. Once I hack at that and get it working 100% correct, *then* I'll have something to give a workshop on. Cheers
Edited by mlord (22/07/2008 18:12)
|
Top
|
|
|
|
#312404 - 24/07/2008 14:30
Re: Running (mostly) from RAM (script)
[Re: mlord]
|
carpal tunnel
Registered: 13/02/2002
Posts: 3212
Loc: Portland, OR
|
But something like this could be my ticket for next year, or perhaps some other conference. The Open Source Convention is in Portland during late July...
|
Top
|
|
|
|
#312405 - 24/07/2008 14:48
Re: Running (mostly) from RAM (script)
[Re: canuckInOR]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
But something like this could be my ticket for next year, or perhaps some other conference. The Open Source Convention is in Portland during late July... Yeah, I think it is taking place *this week* already. Maybe linux.conf.au in Tasmania, though..
|
Top
|
|
|
|
#312435 - 25/07/2008 14:55
Re: Running (mostly) from RAM (script)
[Re: mlord]
|
carpal tunnel
Registered: 13/02/2002
Posts: 3212
Loc: Portland, OR
|
But something like this could be my ticket for next year, or perhaps some other conference. The Open Source Convention is in Portland during late July... Yeah, I think it is taking place *this week* already. It is -- but you did say next year, so I thought I'd throw it out there. Maybe linux.conf.au in Tasmania, though.. Hmm... Oregon... Tasmania... Oregon... Tasmania. Well, you got me beat.
|
Top
|
|
|
|
#313241 - 27/08/2008 17:13
aufs_commit.sh
[Re: mlord]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
Here's a new (UPDATED AGAIN!) script for the collection. This one, will commit changes made during an AUFS session back to the underlying persistent filesystem. It only looks for added/deleted/modified regular files and directories, and skips changes to /tmp and /var/tmp contents. Probably not perfect yet, but it does work for me here. #!/bin/bash
#
# Script to "commit" aufs tmpfs changes back to the underlying persistent filesystem.
#
SCRIPT="`realpath $0`"
ROOT=/.rw
AUFS=$ROOT/.tmpfs
arg1="$1"
MIRRORDIR=/usr/bin/mirrordir
if [ ! -x $MIRRORDIR ]; then
echo "$MIRRORDIR: not found; aborting." >&2
exit 1
fi
COMMIT=""
if [ "$1" = "--commit" ]; then
COMMIT="$1"
shift
fi
function doit(){
echo "$*"
[ "$COMMIT" = "--commit" ] && $*
}
if [ "$1" = "" ]; then
cd $AUFS || exit 1
find . -type f -name .wh.\* | xargs -n1 $SCRIPT $COMMIT WHITEOUT
doit sync
cd $AUFS || exit 1
find . | xargs -n1 $SCRIPT $COMMIT COPY
doit sync
[ "$COMMIT" = "--commit" ] || (echo ;echo "$SCRIPT: Dry run complete. Use --commit to do it for real"; echo)
exit 0
fi
if [ "$1" = "WHITEOUT" ]; then
cd $AUFS || exit 1
dir="${2%%/.wh.*}"
fn="${2##*/.wh.}"
if [ "$dir" = "." ]; then
[ "$fn" = ".rw" -o "$fn" = ".aufs" -o "$fn" = ".tmpfs" ] && exit 0
fi
if [ "$fn" = ".wh..opq" ]; then
if [ -d "$ROOT/$dir" ]; then
if [ -d "$dir" ]; then
doit $MIRRORDIR -v --exclude-regexp '.*/[.]wh[.].*' "$dir" "$ROOT/$dir"
else
doit rm -rf "$ROOT/$dir"
fi
fi
exit 0
fi
if [ "$dir/.wh.$fn" = "$2" ]; then
[ -e "$ROOT/$dir/$fn" ] && doit rm -rf "$ROOT/$dir/$fn"
fi
exit 0
fi
if [ "$1" = "COPY" ]; then
cd $AUFS || exit 1
if [ -d "$2" ]; then
[ "$2" = "./.wh..wh.plink" ] && exit 0
cd $AUFS || exit 1
dir="${2%%/.wh.*}"
if [ ! -e "$ROOT/$dir" ]; then
doit mkdir -p "$ROOT/$dir"
doit $MIRRORDIR -v --exclude-regexp '.*/[.]wh[.].*' "$dir" "$ROOT/$dir"
fi
exit 0
fi
[ "${2:0:6}" = "./.wh." ] && exit 0
[ "${2:0:6}" = "./tmp/" ] && exit 0
[ "${2:0:10}" = "./var/tmp/" ] && exit 0
dir="${2%%/*}"
fn="${2##*/}"
[ "${fn:0:4}" = ".wh." ] && exit 0
[ ! -e "$ROOT/$dir" ] && doit mkdir -p "$ROOT/$dir"
if [ -f "$2" -a -f "$ROOT/$2" ]; then
if diff -q "$2" "$ROOT/$2" >/dev/null ; then
[ "$2" -nt "$ROOT/$2" ] && touch -m -r "$2" "$ROOT/$2"
exit 0
fi
fi
doit cp -dp "$2" "$ROOT/$2"
exit 0
fi
echo "$SCRIPT: huh? $*" >&2
Cheers
Attachments
aufs_commit.sh (181 downloads)Description: aufs_commit.sh
Edited by mlord (28/08/2008 14:35)
|
Top
|
|
|
|
#313242 - 27/08/2008 17:21
Re: aufs_commit.sh
[Re: mlord]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
So with this new commit script, one can now do things like.. download/install some software, play around with it, and finally either "reboot" to discard everything, or run the script to retain the new software permanently.
|
Top
|
|
|
|
#313243 - 27/08/2008 17:25
Re: I want a new 1.8" hard drive: ZIF vs. IDE ?
[Re: gbeer]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
Does this mean that at the end of a session you have the choice to commit the current ram overlay onto the CF, that it will become the new boot state? With the newly posted aufs_commit.sh script, the answer is now YES! Except that it can be done as often as desired, not only at the end of a session.
|
Top
|
|
|
|
#313245 - 27/08/2008 19:00
Re: I want a new 1.8" hard drive: ZIF vs. IDE ?
[Re: mlord]
|
carpal tunnel
Registered: 24/12/2001
Posts: 5528
|
In theory "cheap" SATA SSDs. Looks pretty neat. I wonder what the price will be?
|
Top
|
|
|
|
#313278 - 28/08/2008 14:06
Re: aufs_commit.sh
[Re: mlord]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
I have now updated the aufs_commit.sh script in the original posting above, to reflect some bugfixes and improvements. Now requires mirrordir to be installed before using (it checks).
-ml
|
Top
|
|
|
|
#315589 - 27/10/2008 15:30
Pretec (true) 233X CF cards -- 32GB for US$114
[Re: mlord]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
Following up again here:
A seller on eBay is now offering Pretec brand 233X CF cards in 32GB capacity for US$114. These cards apparently read at ~35MBbytes/sec and write at ~25MBytes/sec, far faster than the early cheap CF cards.
I'll be replacing the 32GB card in my notebook with a Pretec card, to fix the agonizingly slow write speeds there, and the old slower card (reads at ~35MB/sec, writes at ~1MB/sec) will get moved to one of the empegs here.
Cheers
|
Top
|
|
|
|
#315616 - 27/10/2008 23:53
Re: I want a new 1.8" hard drive: ZIF vs. IDE ?
[Re: tman]
|
carpal tunnel
Registered: 17/12/2000
Posts: 2665
Loc: Manteca, California
|
If I am reading between the lines correctly. They are getting higher speeds by striping across the chips inside the flash drive?
_________________________
Glenn
|
Top
|
|
|
|
#315639 - 29/10/2008 02:31
Re: I want a new 1.8" hard drive: ZIF vs. IDE ?
[Re: gbeer]
|
carpal tunnel
Registered: 19/05/1999
Posts: 3457
Loc: Palo Alto, CA
|
If I am reading between the lines correctly. They are getting higher speeds by striping across the chips inside the flash drive? Pretty much all flash cards do this to some extent. Actual flash chips themselves are dog slow individually (well, below 10MB/sec to a single die, plus there's your flash translation layer overhead given that you can't erase single "sectors", hence there's a lot of logical remapping going on). I think the faster ones have multiple buses and can have multiple concurrent operations per bus, plus enough buffer memory on the controller to be able to do things fairly optimally. It can get pretty hairy making NANDs go fast in my experience Hugo
|
Top
|
|
|
|
#315960 - 06/11/2008 00:15
Re: Pretec (true) 233X CF cards -- 32GB for US$114
[Re: mlord]
|
carpal tunnel
Registered: 29/08/2000
Posts: 14496
Loc: Canada
|
I'll be replacing the 32GB card in my notebook with a Pretec card, to fix the agonizingly slow write speeds there, and the old slower card (reads at ~35MB/sec, writes at ~1MB/sec) will get moved to one of the empegs here. The Pretec 32GB 233X card arrived today, and I've already migrated shrimp to it. Which in itself is a statement, because with the original 32GB card I'd still be waiting for the copy to finish now, instead of typing this post from the new card! 41MBytes/sec sequential reads, 10-20MBytes/sec writes. Good! I'm happy!
|
Top
|
|
|
|
|
|