Update : tried the IP list, but it doesn't work for me. Eg. I've got an UnRaid server running on 192.168.1.120, but when the WireGuard tunnel is up I still can't access it, also not with this list of IP's in the AllowedIPs section.
When I shutdown the VPN tunnel it immediately works.

Edit : I tried this again, and now it DID work! Go figure! But I don't care, I'm happy! Thanks for all the help!