Can I have multiple interfaces in one subnet?
It may have crossed your mind to have multiple Ethernet interfaces in a single subnet . The reasons might be that you would like to have some kind of manual load balance or restrict access in some way. And the result: It does not work quite right. I have heard many times that this is an invalid and wrong setup and won’t work. The thing is, you can make it work and if you need to have multiple interfaces in the same subnet, read on.
You configured your new and extremely fast NAS head. Plenty of storage, incredible local read and write speed and enough 10Gbit interfaces to satisfy the connected clients and the ones form your neighbor is necessary. You thought about trunking those interfaces but that doesn’t give you the control you’d like to have about the data flow. And, because you are a believer of a single subnet as long as you have less than 240 devices, you added the 4 new 10GbE ports into the same subnet.
Everything seems to be working fine but you expected more performance. While analyzing the traffic on the ports, you realize that the traffic isn’t distributed at all. Moreover, traffic from clients addressing port-3 goes over port-1. Whaaaaat?
There are no associations for an IP address to a physical interface as the IP belongs to a host.
Therefore, if you have 2 or more interfaces in the same subnet, there is no guarantee that the same interface receiving a package will also be the interface used to send a reply.
In other words; The OS does not check which interface it received the data on only which network and it then sends data to that network. It will sometimes pick the correct interface port-1 and sometimes it will send data on port-4 and the client application will never receive the reply because it is using port-2.
We run our server with a quad-port 10GbE NIC in a single class C-subnet at 192.168.30.0. We don’t need any routing through those IP’s as this is intended for production only. The server runs on RedHat 7 (also applicable for version 5 & 6)
port-1 - ens1: 192.168.30.5/24 port-2 - ens2: 192.168.30.6/24 port-3 - ens3: 192.168.30.7/24 port-4 - ens4: 192.168.30.8/24To assure that packets received by port-1 are also going back to port-1, we have to use the internal routing table.
Add 4 new lines (routing tables) to the rt_tables file in /etc/ip2route2
# cat /etc/iproute2/rt_tables101 t1 102 t2 103 t3 104 t4
Now we have to add the routes to the formerly defined routing tables. Create a routing file for each device in /etc/
192.168.30.0/24 dev ens1 src 192.168.30.5 table t1
default via192.168.30.5 dev ens1 table t1 #
192.168.30.0/24 dev ens2 src 192.168.30.6 table t2
default via192.168.30.6 dev ens2 table t2 #
192.168.30.0/24 dev ens3 src 192.168.30.7 table t3
192.168.30.7 dev ens3 table t3 #
192.168.30.0/24 dev ens4 src 192.168.30.8 table t4
default via192.168.30.8 dev ens4 table t1
Last but not least, we need to create the routing rules for each device. Create a new file for each device.
# cat /etc/sysconfig/network-scripts/rule-ens1
table t1 from 192.168.30.5
# cat /etc/sysconfig/network-scripts/rule-ens2
table t2 from 192.168.30.6
# cat /etc/sysconfig/network-scripts/rule-ens3
table t3 from 192.168.30.7
# cat /etc/sysconfig/network-scripts/rule-ens4
table t4 from 192.168.30.8
That’s pretty much it. As said above, we don’t worry about routing in that setup Do a network restart, an
If you want to teach your server to respond to ARP[/zt_highlight] replies, add following lines to your /etc/sysctl.conf file
net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
As having many interfaces in the same network might for some people not be best practice, it serves a purpose for others. This setup above demonstrates that you can control very well which port has to answer when addressed directly.