NSX Design Notes (3/10) – Logical Routing

1) Logical Routing Components

NSX Logical routing components: DLR and ESG

  1. ESG Router – Centralized routing
    1. Communication between logical networks and external L3 physical infrastructure.
    2. Optimized for north-south traffic. Though DLR can be used for east-west traffic, it is not optimized for the traffic.

  2. Distributed Logical Router (DLR)
    1. Communication between logical switches
    2. Optimized for east-west traffic

    3. DLR Control Plane
      1. Provided by DLR Control VM and NSX Controller
      2. Supports dynamic protocols – OSPF & BGP
      3. Exchanges routing updates with NSX edge (next layer3 hop device )
      4. DLR Control VM runs on Active-Standby mode
      5. DLR Control VM has two addresses – Protocol address ( Routing Information exchange) and Forwarding address ( Data path)
    4. DLR Data Plane
      1. DLR kernel modules (VIBs) have Routing Information Base (RIB) pushed through controller cluster
      2. DLR kernel modules have LIFs. LIFs are connected to Logical Switches
      3. Each LIF has a unique IP address (default gateway for L2 logical segment) and vMAC address.
      4. VMAC and IP address of LIF will be same in all ESX servers. So even after vMotion the default gateway and MAC remains the same.
  3. Interaction between Logical Routing Components

    1. Deploy DLR Instance / Control VM
    2. Controller pushes the new DLR configuration, including LIFs and their associated IP and vMAC addresses, to ESXi hosts
    3. DLR peers with NSX edge and exchanges routing information.
      DLR -> ESG: DLR can be configured to redistribute IP prefixes into OSPF for all logical networks. It then pushes routes to NSX edge. Next hop IP address for all logical networks would be DLR Forwarding address.
      ESG->DLR:

      The NSX Edge pushes the prefixes to reach IP networks in the external network to the control VM. A single default route is sent by the NSX Edge, since it represents the single point of exit toward the physical network infrastructure
    4. The DLR control VM pushes the IP routes learned from the NSX Edge to the controller cluster.
    5. The controller cluster distributes routes learned from the DLR control VM across the hypervisors.
    6. The DLR routing kernel modules on the hosts handle the data path traffic for communication to the external network via the NSX Edge
  4. Routed communication between VMs in different logical segments

    VM1 in VXLAN A sends a packet to VM2 in VXLAN B.
    1. Default gateway: Packet goes to Local DLR in Source ESX Host (ESXA), because default gateway for each VXLAN is a LIF in DLR.(LIFA)
    2. Routing Lookup :
      1. Is Destination Network a VXLAN? : Yes. Find Destination LIF.
      2. Destination VM MAC Address? : First it looks up in destination LIF ARP Table, if it does not find in the table then it sends ARP request.
        Note: Every host maintains an ARP table for every connected LIF.
        Note: DLR does not communicate with controller to figure out MAC address.
      3. Destination ESX Host? : On Which ESX Host the destination MAC is connected. Looks up MAC table to Get VTEP address for the destination MAC address.
    3. Encapsulate packet and send to destination host (ESXB) VTEP
    4. Destination host decapsulates packet
    5. Send to destination VM(VM2)

    Note: When VM2 replies to VM1 the routing between logical segments would be performed on ESX-2

  5. Communication from external networks to logical networks
    Ingress Traffic


    1. Physical network -> NSX Edge. External device wants to communicate to VM1. The packet is delivered to NSX edge.
    2. NSX Edge Routing Lookup: Does routing lookup and finds out that IP prefix was learned from DLR. Next hop is DLR Forwarding Address.
    3. NSX Edge -> DLR: Packet routed to DLR Instance.
    4. DLR Transit Network -> VXLAN Segment: The destination network is directly connected to DLR so the packet is routed from transit network to destination VXLAN segment. After L2 lookup, packet is encapsulated and sent to destination host VTEP.
    5. ESX Host -> VM. Destination host decapsulates and sends traffic to destination VM.

    Egress Traffic: VM1 replies to external destination

    1. VM -> Local DLR: Packet is delivered to VM1 default gateway interface located on local DLR
    2. Routing Lookup: Routing lookup is performed and finds the next hop is NSX edge interface in transit network.
    3. L2 Lookup (DLR->ESX VTEP): L2 Lookup is performed to determine how to reach the NSX edge interface. Packet is encapsulated and sent to ESX host where NSX edge is running.
    4. DLR -> NSX Edge: ESX host decapsulates the packet and send to destination NSX edge.
    5. NSX Edge -> L3 Router: NSX edge performs a routing lookup then sends the packet to next L3 hop on physical network.

2) Routing Capabilities

  1. Concepts
    1. Localizing routing and forwarding in hypervisor reduces the oversubscription on uplinks from hosts to access switch. Eliminates hair pinning.
    2. Localization of forwarding inside the hypervisor allows higher speed transfer of data
    3. Forwarding between VXLAN segments in hypervisor independent of underlying physical topology.
  2. Both DLR and ESG support OSPF and BGP.
  3. NSX routing domain connects as an edge network. (It does not act as transport network)

  4. NSX does not dictate the routing protocol choice in itself, however the design requirements and connectivity options may restrict the choice to a protocol.
  5. Routing protocol best practices
    1. Use summarization. Advertise reachability of NSX domain via summarized networks.
    2. Use default route to reach any destination outside NSX routing domain.
    3. The non-default neighbor hello/hold timers can be used to achieve faster convergence in ECMP based topologies.
    4. Use routed (or SVI) links between pair of physical routers (both top-of-rack and aggregation), to heal the routing protocol connectivity in case of loss of all uplinks from a router/switch (see Figure 116). This is highly recommended in OSPF but not limited to only that protocol.
    5. Use single protocol (OSPF/BGP) for both DLR to NSX Edge and NSX Edge to Physical network connectivity.
    6. If Core-Aggregation is running non-standard protocol or non-standard OSPF configuration then run eBGP for NSX Edge to Physical network connectivity.
NSX Design Notes (2/10) – Logical Switching
NSX Design Notes (4/10) - Firewall and Security Services
No tags for this post.

Leave a Comment