Some time ago I had to do some serious testing that involved 4 VMs – each equipped with 4 NICs and multitude of VLANs (LVS clusters with pacemaker & co.). When things did not work, i.e. there was no connectivity between tagged interfaces I thought the problem is related to VirtualBox – but that sounded rather odd. Not-so-quick investigation showed that the problem is located within the e1000 driver.
To make it even more confusing – it’s not all kernel/driver versions that are affected. The problem I am describing occurs on CentOS 6.2 (2.6.32-200.el6.x86_64) and 6.3 (2.6.32-279.2.1.el6.x86_64). For VirtualBox users the solution is easy – just switch your NIC to PCnet or Paravirtualized – then tagged interfaces work out of the box. In case you have physical cards that rely on e1000 – and you happen to be running CentOS – a sane option would be to try Intel’s driver. Trying to fix the issue with the following command ethtool -K eth<num> rxvlan off tx off unfortunately did not help me (CentOS comes with ethtool 2.6.33-pre1 – I tried with 2.6.39 and 3.4.2 – both report that operation is not supported/flags cannot be changed).
Since it’s already confusing – e1000 + tagged interfaces work out of the box on Ubuntu 12.04 LTS (server edition, kernel 3.2.0-23-generic, x86_64). To conclude – I did not detect any issues related to VirtualBox. I played with tagged interfaces in different scenarios – bridged adapter, internal network etc. – it just works. It was only the e1000 driver misbehaving every now and then.
A small side note – if you happen to be configuring machines with tagged interfaces – aka “more advanced routing”, please consider reading more about Reverse Path Filtering (rp_filter in proc). It is used to prevent spoofed traffic entering your machine (you can detect this problem by logging martian packets – sysctl -w net.ipv4.conf.all.log_martians=1). It’s on by default, but for more complex configurations it might prevent you from from getting the job done