VMware Network Traffic Routing
March 16, 2012VMware has lots of ways to setup networking on their ESXi hosts. In order to set this up in the best way for your needs, it’s important to understand how the traffic will be routed between VMs, virtual switches, physical switches and physical network adapters.
Before looking at an example, we should review some networking 101. Machines on the same vlan on the same switch can communicate with one another (assuming there is no firewall type devices in the way). Machines on different vlans on the same switch cannot communicate unless the traffic passes through a router.
Now that we’ve had a short refresher course on networking, lets look at how VMware uses virtual switches to pass traffic between VMs and the physical network.
VMs on the same vlan on the same host.
The diagram below shows two virtual machines on vlan 10, which are connected to the same virtual switch. If the top VM wants to send data to the bottom vlan it simply sends a frame to the connected virtual switch and that switch forwards the frame to the bottom VM. Nothing else needs to occur, in fact you don’t even need to have a physical NIC attached as an uplink port and no physical equipment is necessary to do this forwarding.
VMs on different Vlans on the same host
Now lets look at what happens when the top VM is on a seperate vlan from the bottom vlan. This process starts out the same, with the top VM sending a frame to the virtual switch. Since the virtual switch doesn’t see the destination VM on the same subnet it will forward the frame to it’s uplink (physical NIC) and out to the physical network. Once at the physical network we will hit our router which will then be able to route it back to the virtual switch on the new vlan. This type of routing is sometimes referred to as a “router on a stick”. Once the virtual switch gets this new frame, it can find the bottom VM and forward the frame. A reply would then travel this entire distance in the opposite direction.
As you can see this isn’t necessarily the best use of bandwidth because now you’re limited by the physical NIC’s adapter speed.
VMs on different Switches on the same host.
If we look at the next example, we have two separate virtual switches on the same host. This example really is the same as our previous example because a vlan basically separates the traffic just like having two completely separate switches. It doesn’t matter if the VMs are on the same VLAN or not, they can’t communicate without getting to the physical network. The difference would be that once the frame is in the physical network, it doesn’t need a router to forward on a packet, it can just have a switch forward the frame.
VMs on different hosts on same vlan.
Clearly since the two VMs below are on different hosts, they are going to require the frames to be sent to the physical network to get between hosts. This is true even if you’re using distributed switches. The diagrams that you see depicting distributed switches make it look like the switches are somehow attached to each other, but they are just trying to show you that you’ve created one distributed switch instead of individual vSwitches.
This document was created using the official VMware icon and diagram library.
Copyright © 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and
international copyright and intellectual property laws. VMware products are
covered by one or more patents listed at http://www.vmware.com/go/patents.
VMware does not endorse or make any representations about third party information
included in this document, nor does the inclusion of any VMware icon or diagram
in this document imply such an endorsement.
Hi,
In the 3rd scenario ‘VMs on different Switches on the same host’, you said “it doesn’t need a router to forward on a packet, it can just have a switch forward the frame.”
Care to explain why? or even give me some keywords I can feed to Google to read more about it ?
VMware newbie here, still learning, looooooooove your blog 🙂
Thanks for expressing your interest in the blog. I hope the information is useful.
If I know the scenario you’re writing about, you’re wondering why you don’t need a router to forward frames between VMs that are on different switches within the same host.
Machines (physical or virtual) should be able to communicate with each other if they are on the same subnet. I should have specified more clearly that the in my example the subnet and vlan are configured together. Examples would be 192.168.1.0 is vlan1, and 192.168.2.0 is vlan2 etc. So assuming that your machines are on the same subnetvlan there is no need to use a router to forward the traffic. Routers forward packets between networks. Since the VMs would be on the same network, but different switches, they should be able to communicate without a router.
I hope this makes sense.
Thanks for the quick reply. Yes, that’s what I wanted 🙂
What about the last scenario, a router or switch is needed?
A router wouldn’t be needed, but a physical switch would be necessary.
Thanks again.
Thank you,
The information you shared is very informative.
Wifi Router
With regards to “VMs on different Vlans on the same host” – what is the best practice to prevent the traffic from having to traverse the physical LAN connection to get routed between the IP Subnets?? I’ve read that there’s Virtual routers available to install in the ESX environment, for example the Cisco NEXUS1000V. Are there any other alternatives or do VMWare ship anything with ESX that can do it??
To my knowledge the Nexus 1KV does not do layer 3 routing (routing between subnets). I’ve used vyatta to do this in the lab on occasions and I’m sure there are others that do. VMware has a product called NSX that could be used to route between subnets (kind of). It would encapsulate the packets first but might do what you want.
For the most part, the routing is just done on the physical network and hair pinned if needed.
Hi in the third scenario you told you have two virtual switches on single host , but you can’t create multiple virtual standard switches on single host right ? You can only create multiple portgroups in a single virtual standard switch. Please correct me if I am wrong.
You can have multiple standard switches on the same host.
Could you please explain the above same if ESXi hosts are Cisco UCS blades.