Month: April 2018


  • P Routers – Provider routers
    • MPLS Core
  • PE Routers – Provider Edge routers
    • MPLS – IP Edge
  • CE Routers – Customer Edge routers
    • IP Edge

Traceroute (R6 -> R7)

Layer 3 setup:






[DC] NX-OS – Overlay Transport Virtualization

What is OTV:

  • Layer 2 VPN over IPv4
  • Used over the DCI to extend VLANs between datacenter sites

OTV was designed for Layer 2 DCI

  • Optimizes ARP Flooding over DCI
  • Does not extend STP domain
  • Can overlay multiple VLANs without complicated design
  • Allows multiple edge routers without complicated design

OTV benefits

  • Provides a flexible overlay VPN on top of without restrictions for the IP nework
  • L2 transports leveraging the transport IP network capabilities
  • Provides a virtual multi-access L2 network that supports efficient transport of unicast, multicast and broadcast traffic

OTV Control Plane

  • Uses IS-IS to advertise MAC addresses between AEDs
    • “Mac in IP” Routing
  • Encapsulated as Control Group Multicast
    • Implies that DCI Must support ASM Multicast
    • Can be encapsulated as Unicast with OTV Adjacency Server

OTV Data Plane

  • Uses both Unicast and Multicast Transport
  • Multicast Control Group
    • Multicast or Broadcast Control Plane Protocols
    • eg. ARP, OSPF, EIGRP etc
  • Unicast Data
    • Normal Unicast is encapsulated as Unicast between AEDs
  • Multicast Data Group
    • Multicast data flows are encapsulated as SSM Multicast
    • Implies AED use IGMPv3 for (S,G) joins
  • OTV Adjacency Server can remove requirement for Multicast completely
    • Will result in Head End Replication when more than 2 DC’s connected over the DCI

OTV DCI Optimizations

  • Other DCI options bridge all traffic over DCI
    • eg. STP, ARP, Broadcast storms etc
  • OTV reducdes unnecessary flooding by:
    • Proxy ARP/ICMPv6 ND Cache on AED
    • Assumption is that hosts are bi-directional (not silent)
    • Inital ARPs are flooded, then cache is used
    • Terminating the STP Domain on AED.

OTV Configuration:


License needed:


[DC] Storage Networking & FibreChannel

LAN and SAN Separation

  • Security  Ensures protection from hacking
  • Bandwidth – SAN needs more bandwidth than LAN
  • Flow Control – SAN is lossless and LAN is lossy
    • Ethernet Flow control ( LAN ):
      • Source transmits packets untill receiver buffer overflow, then sends a “Pause” frame
      • Lost packets are retransmitted
    • Fibre Channel ( SAN ):
      • Credit based mechanism – Receiver has control
      • Source does not send a frame until the receiver telsl the source it can receive a frame by sending “Ready” signal Back
  • Performance – SAN provides more performance than LAN enviorments

LAN vs SAN flow control

  • Flow control is how data is controlled in a network
  • Ethernet Flow control ( LAN )
    • Source transmits packets until receiver buffers overflow, then sends a “Pause” frame
    • Lost packets are retransmitted
  • Fibre Channel ( SAN )
    • Credit based mechanism – Receiver has control
    • Source does not send a frame until the receiver tells the source it can receive a frame by sending “Ready” signal back.
    • “Lossless Fabric”


  • San Topologies
    • Point-to-Point
      • Initiator (server) and Target (Storage) directly connected
    • Arbitraded Loop (FC-AL) (Legacy)
      • Logical ring topology, similar to token ring
      • Implies connection is required on the ring
    • Switched Fabric ( FC-SW ) ( Standard)
      • Logical equivalent to a switched ethernet LAN
      • Switches manage the fabric allowing any-to-any communication
      • Support more than 16 million device addresses
  • FibreChannel Port types
    • N_port – Node Port
    • NL_port – Node Loop Port
    • F_port – Fabric Port
    • FL_port – Fabric Loop Port
    • E_port – Expansion Port ( ISL )
    • TE_port – Trunking Expansion Port
  • FC Addressing is analogous to IP over Ethernet
    • IP addresses are logical and manually assigned
    • Ethernet MAC Addresses are physical and burned in
    • FC World Wide Names ( WWNs )  / MAC / Zoning

      • 8 byte address burned in by manufacturer
      • Word Wide Node Name
      • World Wide Port Name
    • FC Identifier ( FCID )  / IP / Routing

      • 3 byte logical address assigned by fabric
      • FCID is subdevided into three fields:
        • Domain ID
          • Each switch gets a domainID
        • Area ID
          • Group of ports on a switch have an Area ID
        • Port ID
          • End station connected to switch gets a Port ID
  • FibreChannel Nameserver ( FCNS)
    • analogous to ARP cache
    • Used to resolve WWN ( pysical address ) to FCID ( logical address )
    • Like FSPF, FCNS requires no configuration
  • FibreChannel Logins
    • Ethernet networks are connectionless
    • Fibre Channel networks are connection oriented
      • All end stations must first register with the control plane of the fabric before sending any traffic.
    • Fabric Registration has three parts
      • Fabric Login ( FLOGI)
      • Port Login ( PLOGI)
      • Process Login ( PLRI )
    • sh flogi database
    • sh fcns database
  • VSANs
    • Logical seperation of SAN traffic
  • Zoning
    • like an ACL in the IP world



[DC] NX-OS – Fabricpath


Cisco FabricPath is a Cisco NX-OS software innovation combining the plug-and-play simplicity of Ethernet with the reliability and scalability of Layer 3 routing.

Using FabricPath, you can build highly scalable Layer 2 multipath networks without the Spanning Tree Protocol. Such networks are particularly suitable for large virtualization deployments, private clouds, and high-performance computing (HPC) environments.


Datacenter Design V ( TRILL, Fabric Path )

  • Classic Ethernet ( CE )
    • Regular internet with regular flooding, regular STP, etc.
  • Leaf switch
    • Connects CE domain to FP domain
  • Spine switch
    • FP backbone switch all ports in the FP domain only
  • FP Core Ports
    • Links on leaf up to Spine, or Spine to Spine
    • i.e. the switchport mode fabricpath links
  • CE Edge Ports
    • Links of leaf connecting to regular CE domain (to servers / switches)
    • i.e. NOT the switchport mode fabricpath links

Activating the fabricpath feature set.

For the activation is the “ENHANCED_LAYER2.PK” license needed, or the grace-period of 120 days:


vlan 100
  mode fabricpath
  name test

interface Ethernet2/1
  switchport mode fabricpath
  no shutdown

interface Ethernet2/2
  switchport mode fabricpath
  no shutdown

N7K3# sh run int e2/9
interface Ethernet2/9
  switchport access vlan 100
  no shutdown
N7K3# sh fabricpath isis

Fabricpath IS-IS domain : default
  System ID : 0026.c734.4f2f  IS-Type : L1 Fabric-Control SVI: Unknown
  SAP : 432  Queue Handle : 15
  Maximum LSP MTU: 1492
  Graceful Restart enabled. State: Inactive
  Last graceful restart status : none
  Graceful Restart holding time:60
  Metric-style : advertise(wide), accept(wide)
  Start-Mode: Complete [Start-type configuration]
  Area address(es) :
  Process is up and running
  CIB ID: 1
  Interfaces supported by Fabricpath IS-IS :
  Level 1
  Authentication type and keychain not configured
  Authentication check specified
  LSP Lifetime: 1200
  L1 LSP GEN interval- Max:8000 Initial:50      Second:50
  L1 SPF Interval- Max:8000     Initial:50      Second:50
  MT-0 Ref-Bw: 400000
        Max-Path: 16
  Address family Swid unicast :
    Number of interface : 6
    Distance : 115
  L1 Next SPF: Inactive

N7K3# sh fabricpath switch-id
                        FABRICPATH SWITCH-ID TABLE
Legend: '*' - this system
        '[E]' - local Emulated Switch-id
        '[A]' - local Anycast Switch-id
Total Switch-ids: 4
    1           0026.c751.bd2f    Primary     Confirmed Yes     No
    2           0026.c71f.a62f    Primary     Confirmed Yes     No
*   3           0026.c734.4f2f    Primary     Confirmed Yes     No
    4           0026.c7cb.4b2f    Primary     Confirmed Yes     No
N7K3# sh cdp nei
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater,
                  V - VoIP-Phone, D - Remotely-Managed-Device,
                  s - Supports-STP-Dispute

Device-ID          Local Intrfce  Hldtme Capability  Platform      Port ID
N7k1(TBC751BD00B)   Eth2/1         147    R S I s   N7K-C7018     Eth2/5
N7k1(TBC751BD00B)   Eth2/2         148    R S I s   N7K-C7018     Eth2/6
N7K2(TBC71FA600B)   Eth2/5         170    R S I s   N7K-C7018     Eth2/5
N7K2(TBC71FA600B)   Eth2/6         170    R S I s   N7K-C7018     Eth2/6
R1                  Eth2/9         134    R S I     3725          Fas0/0

Total entries displayed: 5
N7K3# sh fab
fabric       fabricpath
N7K3# sh fabri
fabric       fabricpath
N7K3# sh fabricpath route
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

0/3/0, number of next-hops: 0
        via ---- , [60/0], 0 day/s 03:03:28, local
1/1/0, number of next-hops: 2
        via Eth2/1, [115/400], 0 day/s 03:01:13, isis_fabricpath-default
        via Eth2/2, [115/400], 0 day/s 03:01:13, isis_fabricpath-default
1/2/0, number of next-hops: 2
        via Eth2/5, [115/400], 0 day/s 03:00:59, isis_fabricpath-default
        via Eth2/6, [115/400], 0 day/s 03:00:59, isis_fabricpath-default
1/4/0, number of next-hops: 4
        via Eth2/1, [115/800], 0 day/s 03:00:59, isis_fabricpath-default
        via Eth2/2, [115/800], 0 day/s 03:00:59, isis_fabricpath-default
        via Eth2/5, [115/800], 0 day/s 03:00:59, isis_fabricpath-default
        via Eth2/6, [115/800], 0 day/s 03:00:59, isis_fabricpath-default


  • VDC
  • VPC
  • Fabricpath
  • Fabric Extenders (FEX)
  • OTV

VDC ( Virtual Device Context )

A VDC can be used to virtualize the device itself, presenting the physical switch as multiple logical devices. Within that VDC it can contain its own unique and independent set of VLANs and VRFs. Each VDC can have assigned to it physical ports, thus allowing for the hardware data plane to be virtualized as well. Within each VDC, a separate management domain can manage the VDC itself, thus allowing the management plane itself to also be virtualized.

Create a new VDC:

N7k1(config)# vdc N5K1
N7k1# switchto vdc N5K1

Show allocated interfaces:

switch# show vdc membership

vdc_id: 0 vdc_name: switch interfaces:

        Ethernet2/1           Ethernet2/2           Ethernet2/3
        Ethernet2/4           Ethernet2/5           Ethernet2/6
        Ethernet2/7           Ethernet2/8           Ethernet2/9
        Ethernet2/10          Ethernet2/11          Ethernet2/12
        Ethernet2/13          Ethernet2/14          Ethernet2/15
        Ethernet2/16          Ethernet2/17          Ethernet2/18
        Ethernet2/19          Ethernet2/20          Ethernet2/21
        Ethernet2/22          Ethernet2/23          Ethernet2/24
        Ethernet2/25          Ethernet2/26          Ethernet2/27
        Ethernet2/28          Ethernet2/29          Ethernet2/30
        Ethernet2/31          Ethernet2/32          Ethernet2/33
        Ethernet2/34          Ethernet2/35          Ethernet2/36
        Ethernet2/37          Ethernet2/38          Ethernet2/39
        Ethernet2/40          Ethernet2/41          Ethernet2/42
        Ethernet2/43          Ethernet2/44          Ethernet2/45

vdc_id: 1 vdc_name: N5K1


Allocate interfaces:

N7k1(config)#vdc N5K1
N7k1(config-vdc)#allocate interface e2/1 - 12

VPC ( Virtual Port Channel )

LAB IV ( vPC – virtual Port-channels )


[DC] Nexus Models

 Nexus 7000/7700Nexus 5500/5600Nexus 2000 ( FEX ) 
1/10/40/100Gbps1/10/40Gbps1/10/40Gbps Fabric Extender
Layer2 and Layer3 LAN switchingLayer2 and Layer3 LAN switchingNo local switching (Traffic is done by parent)
FCoE SAN SwitchingFCoE SAN Switching
No native FC portsNative FC Ports
Highly redundant

LAB VII: BGP communities

Building a case study from the ARCH FLG book; BGP communities.

The idea is to use BGP communities to influence the routing between Autonomous Systems with the following goals in mind:

  • Configure communities to tag the routes per building on each AS.
  • Configure communities as no-export so the routes of AS65001.building2 and AS65002.building2 are not exported through AS65000.
    • The routes will be tagged on R6 and R9 with community 65000:99 and processed on the AS boundry.
    • The routes of AS65001.building1 and AS65002.building1 are allowed to be exported.
  • Configure communities so that R7 and R8 can set their local preference on the AS65000 side.
    • The routes will be tagged on R7 will be tagged with 65000:200 resulting in a local-preference of 200.
    • The routes will be tagged on R8 will be tagged with 65000:300 resulting in a local-preference of 300.
AS65000Building 1 ( Router 1 )
AS65000Building 2 ( Router 2 ) uplink to AS65001
AS65000Building 3 ( Router 3 ) uplink to AS65002
AS65000Building 3 ( Router 4 ) uplink to AS65002
AS65001Building 1 ( Router 5 )
AS65001Building 2 ( Router 6 )
Community 65000:99 is used for no-export
AS65002Building 1 ( Router 7 )
65000:200 is used for local preference 200 in AS65000
AS65002Building 1 ( Router 8 )
65000:300 is used for local preference 300 in AS65000
AS65002Building 3 ( Router 9 )
Community 65000:99 is used for no-export




Starting CCNA: Datacenter

Next up, Datacenter!


  • Data Center Physical Infrastructure
  • Basic Data Center Networking Concepts
  • Advanced Data Center Networking Concepts
  • Basic Data Center Storage
  • Advanced Data Center Storage


  • Unified Computing
  • Network Virtualization
  • Cisco Data Center Networking Technologies
  • Automation and Orchestration
  • Application Centric Infrastructure