Can I have multiple interfaces in one subnet?

Topic

It may have crossed your mind to have multiple Ethernet interfaces in a single subnet . The reasons might be that you would like to have some kind of manual load balance or restrict access in some way. And the result: It does not work quite right. I have heard many times that this is an invalid and wrong setup and won’t work. The thing is, you can make it work and if you need to have multiple interfaces in the same subnet, read on.

Situation

You configured your new and extremely fast NAS head. Plenty of storage, incredible local read and write speed and enough 10Gbit interfaces to satisfy the connected clients and the ones form your neighbor is necessary. You thought about trunking those interfaces but that doesn’t give you the control you’d like to have about the data flow. And, because you are a believer of a single subnet as long as you have less than 240 devices, you added the 4 new 10GbE ports into the same subnet.
Everything seems to be working fine but you expected more performance. While analyzing the traffic on the ports, you realize that the traffic isn’t distributed at all. Moreover, traffic from clients addressing port-3 goes over port-1. Whaaaaat?

Reason

There are no associations for an IP address to a physical interface as the IP belongs to a host.

Therefore, if you have 2 or more interfaces in the same subnet, there is no guarantee that the same interface receiving a package will also be the interface used to send a reply.

In other words; The OS does not check which interface it received the data on only which network and it then sends data to that network. It will sometimes pick the correct interface port-1 and sometimes it will send data on port-4 and the client application will never receive the reply because it is using port-2.

Solution

We run our server with a quad-port 10GbE NIC in a single class C-subnet at 192.168.30.0. We don’t need any routing through those IP’s as this is intended for production only. The server runs on RedHat 7 (also applicable for version 5 & 6)

port-1 - ens1:    192.168.30.5/24
port-2 - ens2:    192.168.30.6/24 
port-3 - ens3:    192.168.30.7/24 
port-4 - ens4:    192.168.30.8/24 

To assure that packets received by port-1 are also going back to port-1, we have to use the internal routing table.
Add 4 new lines (routing tables) to the rt_tables file in /etc/ip2route2

# cat /etc/iproute2/rt_tables
 101 t1
 102 t2
 103 t3
 104 t4

Now we have to add the routes to the formerly defined routing tables. Create a routing file for each device in /etc/sysconfig/network-scripts/ with the name route-devicename.

#cat /etc/sysconfig/network-scripts/route-ens1
# route-ens1 
192.168.30.0/24 dev ens1 src 192.168.30.5 table t1 
default via 192.168.30.5 dev ens1 table t1

#cat /etc/sysconfig/network-scripts/route-ens2
# route-ens2 
192.168.30.0/24 dev ens2 src 192.168.30.6 table t2
default via 192.168.30.6 dev ens2 table t2
 
#cat /etc/sysconfig/network-scripts/route-ens3
# route-ens3
192.168.30.0/24 dev ens3 src 192.168.30.7 table t3 
default via 192.168.30.7 dev ens3 table t3
 
#cat /etc/sysconfig/network-scripts/route-ens4
# route-ens4 
192.168.30.0/24 dev ens4 src 192.168.30.8 table t4 
default via 192.168.30.8 dev ens4 table t1 

Last but not least, we need to create the routing rules for each device. Create a new file for each device.

# cat /etc/sysconfig/network-scripts/rule-ens1
# rule-ens1 
table t1 from 192.168.30.5
 
# cat /etc/sysconfig/network-scripts/rule-ens2 
# rule-ens2 
table t2 from 192.168.30.6
  
# cat /etc/sysconfig/network-scripts/rule-ens3 
# rule-ens3 
table t3 from 192.168.30.7
  
# cat /etc/sysconfig/network-scripts/rule-ens4 
# rule-ens4 
table t4 from 192.168.30.8

That’s pretty much it. As said above, we don’t worry about routing in that setup Do a network restart, an ifdown & ifup on the interfaces or reboot the server and that’s all.

If you want to teach your server to respond to ARP[/zt_highlight] replies, add following lines to your /etc/sysctl.conf file

 
# vi /etc/sysctl.conf 
net.ipv4.conf.all.arp_filter = 1 
net.ipv4.conf.default.arp_filter = 1  
net.ipv4.conf.all.arp_announce = 2 
net.ipv4.conf.default.arp_announce = 2 

As having many interfaces in the same network might for some people not be best practice, it serves a purpose for others. This setup above demonstrates that you can control very well which port has to answer when addressed directly.