Layer 2 Network Overlays for Mobility, a Network-centric View

Marco Di Benedetto (Profile)
Monday, February 4th 2013

Designing Physical Networks

Believe it or not, some people make a living designing network connectivity. Like any set of pipes, network ‘pipes’ and the resources they provide are not infinite. If you’ve ever tried to create a full-scale replica of Niagara Falls in your backyard, you must have realized you were not getting enough water through your home water pipes. Someone designed the water supply system to your neighborhood and your house and has defined water budgets to match your plausible water use cases.

Along the same lines, network designers assign budgets of layer 2 bandwidth to each rack according to reasonable usage assumptions. These “rack-centric” network budgets are typically described in terms of the number of top-of-rack (ToR) switch uplinks. If the total bandwidth consumed by the rack does not exceed the budget, the network behaves well. If the rack pushes more bandwidth than the allocated budget, the network becomes congested.

At a very high level, a physical rack can be seen as a container with a certain amount of CPU and memory (the “compute budget”), and a specific amount of network capacity in and out of the rack (the “network budget”). Every time a virtual machine (VM) is provisioned, it consumes some compute budget and some network budget.

In the diagram, VM1 is in the green subnet and VM2 is in the red subnet, while the blue subnet is empty. The top-of-rack switches are the default gateways for the three physical subnets, and the two VMs live in their “physical home.” In this example, with or without layer 2 overlays (VXLAN, NVGRE, etc.), the two VMs consume a proportional amount of compute and network budgets. The network budget component can be visualized as follows: if VM1 wants to talk to VM2, their respective default gateways need to be involved, and some capacity of the uplinks of ToR1 and ToR2 are consumed.