Just a simple(?) routing question for the Linux experts out there...

Background
I have a Linux box which connects to the Internet and does NAT based internet sharing for the rest of the network. This box also has a dial in modem attached, and I can dial up with my laptop (or phone) to gain access to my home network and the internet.

I am moving house in the next few days, but my always on connection cannot be moved (the local telco won't install new ISDN lines, and ADSL is a distant speck on the horizon... =( I want to move all my computers to the new place, except for the NAT box.

I have a second linux box for the new place, and it can happily dial into the NAT box, and gain access to the internet, however any machines connected to it cannot.

So the question is, how do I get the new linux box to pass packets to the NAT box?

Some information
NAT box is on 172.20.1.1, The new box is 172.20.2.5, all the machines moving house will have their network addresses changed to the 172.20.2 range)

Hmm. Some gotchas to work out:

You will need to tell the NAT box how to route packets to and from the 172.20.2.0 network. I'm guessing that pppd sets up a host route for machine that dials into it, ie 172.20.2.5, but not a network route.
You will need to ensure that packets from the 172.20.2.0 network get NATed when going out to the internet.
You will need to ensure that the new gateway has routes for the new internal network (172.20.2.0) pointing to the internal ethernet card, and a default pointing to the NAT box.

The NAT box has been told to route packets from the 172.20.1.0 and 172.20.2.0 networks. I am currently testing the routing using two network cards to save on call costs. (The NAT box has eth0, the new box has eth1 attached to the NAT box, and eth0 connected to the new 172.20.2.0 network. I have my OSX laptop connected to the 172.20.2.0 network side and have a ssh session to the new box)

I can Ping the internet and the NAT box from the new box.

If I add a route from the NAT box to the new box, I can ping it back from the NAT box. (I have used "route add -net 172.20.2.0 netmask 255.255.255.0 dev eth0")

I can't ping the NAT box or the internet from anything else on the 172.20.2.0 network from the 172.20.1.0 network)
I can't ping anything else on the 172.20.2.0 from the NAT box (or 172.20.1.1 network)

The new box is showing the default route as (default Gateway 172.20.1.1 Genmask 0.0.0.0 flags UG IFace eth1)

Any ideas? Am I on the right track?

Here's what you want with your current Ethernet config:

NAT box: eth0 (internet IP address), eth1 (172.20.1.1)
New box: eth0 (172.20.1.2), eth1 (172.20.2.1)

The NAT box should have a default route to the ethernet, as normal (I'm ignoring the NAT stuff, but it sounds like you've got that working okay anyway). It should also have a route to 172.20.2/24 pointing at 172.20.1.2 (the local side of the new box). Your new box should have a default route pointing to 172.20.1.1 (the local side of the NAT box). It should also have routing enabled. This is usually referred to as ``IP Forwarding'', which may be configurable in some startup script, or by tweaking some stuff in /proc (I forget exactly what right now). The other machines on the new 172.20.2/24 network should have a default route of 172.20.2.1. All the machines on your 172.20.1/24 network should have a default route of 172.20.1.1, and, in an ideal world, also a route to 172.20.2/24 through 172.20.1.2, but it'll probably work without that, albeit somewhat slower.

Once you've done all of that, make sure everything on your local network can ping each other. If they can all ping each other okay, then your NAT connection for both networks should start working.

When you set up your PPP connection, see if you can get it to emulate that ethernet setup as closely as possible. That is, make sure your PPP server assigns an address on its local network, not set up a new network. It would actually be more efficient the latter way, but it will require even more work to get it set up right and toss most of this ethernet prototyping down the toilet.

Edit: Looking back at what you've done, it appears that most of your problem is likely to do with the route command you issued on the NAT box. Giving it the destination of eth0 told it that the new network is on the same wire as the old network, which is incorrect. You wanted it to route to your new box, which is functioning as a router, so you should have given its IP address as the destination to the route command. In general, you never want to give a device as a route destination. If it actually is running on the same wire, then you should create a subinterface with an address on that network, which should automatically insert that route for you anyway. I've never encountered a situation on a Unix machine where the appropriate destination was not another IP address. And remember that that address must be on a locally connected network; it can't be a hop or two down the line.

Ok, I've got my head round part of this...

As you pointed out, the route command before was wrong...
I've now used "route add -net 172.20.2.0 gw 172.20.1.5 netmask 255.255.255.0 dev eth0"
(Note that I tried this before, but in a fit of stupidity I tried to use the 172.20.2.5 address for the new box, which is where the other route command appeared from!)

The new box is known as 172.20.1.5 (eth1) on the NAT network, and as 172.20.2.5 (eth0) on the new network.

I've also enabled routing by echoing 1 to /proc/sys/net/ipv4/ip_forward (was 0 by default)

The NAT box can ping anything on either network.
The new box can ping anything on either network (probably due to actually being on the other network vie eth1 and the 172.20.1.5 address)

Stuff on the new network can ping the NAT box and get onto the Internet (Primary aim achived)
Stuff on the old network cannot ping the rest of the 172.20.1.0 network (traceroute gives the 172.20.2.5 machine, then nothing past that (just * * *) even though the default route is to the 172.20.1.1 machine. Traceroute to 172.20.1.1 works fine)
Stuff on the NAT network can ping the new box as 172.20.2.5, but cannot ping anything else on the 172.20.2.0 network. (traceroute or ping give "Sendto: Host is down" messages)

Now, if this carries over to the ppp0 interface, then my primary goal is reached, however it would be nicer if both sides of the network could talk to each other properly...

Hmmmm.

Check to make sure that your netmasks on all of the machines are correct. The 172 networks are /16 by default, so if you didn't specify, they will still be that way.

If it's not that, then you still have some sort of routing issue. Post back with the IP addresses w/ netmasks and all routes for all of your boxes. I've lost track of what state everything is in.

Ok on the NAT box...

Dest        Gateway            Genmask        Flags IFace

172.20.2.0  sparc5(172.20.1.5) 255.255.255.0  UG    eth0

172.20.1.0  *                  255.255.255.0  U     eth0

The other routes on the NAT box are for my Inet connection (including the default route out which is on a 212.x.x.x address)

On the New box (sparc5, 172.20.1.5 on eth1 and 172.20.2.5 on eth0)

Dest        Gateway     Genmask        Flags  IFace

172.20.2.0  *           255.255.255.0  U      eth0

172.20.1.0  *           255.255.255.0  U      eth1

default     172.20.1.1  0.0.0.0        UG     eth1

I've also ignored the loopback routes on both machines.

As I said, the default route works for Inet and the NAT box itself, but doesn't work for other machines on the 172.20.1.0 network. Adding a gw for the 172.20.1.0 network on sparc5 stops the 172.20.1.5 address from responding to pings from the 172.20.1.0 side

Right. But what do the routing tables look like on the other machines on the 172.20.1/24 network? I'm guessing that there's some sort of problem there, now, that's cropped up due to the other changes that you mad in the network.

And, just to reiterate, everything's working right now except for the original network that worked right before cannot connect to the internet or to the new network, right? Can they ping each other and/or the interfaces on the two routing boxes that are on their network?

After more testing...

The other machines on 172.20.1.1 had the wrong netmask (I've changed them to 255.255.255.0 now, which gets rid of the host is down message).
The other machines have been set to use either 172.20.1.1 or 172.20.2.5 as the default gateway, depending on which network they are on...

From the Win 9X box (172.20.1.107)

Active Routes:

Network Destination        Netmask          Gateway       Interface

          0.0.0.0          0.0.0.0       172.20.1.1    172.20.1.107

        127.0.0.0        255.0.0.0        127.0.0.1       127.0.0.1

       172.20.1.0    255.255.255.0     172.20.1.107    172.20.1.107

     172.20.1.107  255.255.255.255        127.0.0.1       127.0.0.1

   172.20.255.255  255.255.255.255     172.20.1.107    172.20.1.107

        224.0.0.0        224.0.0.0     172.20.1.107    172.20.1.107

  255.255.255.255  255.255.255.255     172.20.1.107               2

Default Gateway:        172.20.1.1

===================================================================

From the MacOSX box

default 172.20.2.5 UGSc

To recap
Any machine on 172.20.1.0 (original network) and 172.20.2.0 (new network) can ping the NAT box and access the internet
The NAT box can ping any machine.
The sparc5 box can ping any machine (because it has ip addresses on both networks it won't need to route it's own ping traffic)
A normal machine on the 172.20.1.0 network cannot ping a normal machine on the 172.20.2.0 network (the reverse is also true).

A traceroute from the MacosX box (172.20.2.47) to another machine (172.20.1.105) gives

1 172.20.2.5 (172.20.2.5)

2 * * *

A traceroute from the Acorn box (172.20.1.105) to the MacosX box gives

1 172.20.1.1 (172.20.1.1)

2 * * *

You can add the correct route to all of your workstations on the 172.20.1/24 network to point to your new box for the 172.20.2/24 network and that should solve your inter-network problems. In theory, it should still work without that route, but it often doesn't work in reality because it relies on ICMP Redirects, which often don't work right.