Category: Datacenter

[DC] Cloud Computing

Basic cloud computing

  • Essential Characteristics
    • Broad network access
    • Rapid elasticity
    • Measured Service
    • On-demand Self-service
    • Resource pooling
  • Service Models
    • SaaS – Software as a Service
    • PaaS – Platform as a Service
    • IaaS – Infrastructure as a Service
  • Deployment Models
    • Public
      • Provisitioned for open use by the general public
    • Private
      • Cloud for the exclusive use by a single organization
      • Managed by IT or thirdparty
      • on-premise or off-premise
    • Hybrid
      • Two or more cloud infrastructures combined
    • Community
      • Multiple organisations combined

What is an API

  • A precise specification written by providers of a service
  • You must follow the specification when using that service
  • An API decribes what functionality is available, how ti must be used and what formats it will accept as input or return as output



ACI > Contructs

Tenant VDC
Context VRF
Bridge domain Subnet / SVI
EPG Broadcast domain / VLAN
Contract ACL
L2 External EPG 802.1Q trunk
L3 External EPG L3 Routed link


  • Open and Secure
  • Apps and Infrastructure
  • Physical and Virtual
  • On-Site and Cloud

Bringing up the Fabric:

  • Physical requirements
    • Power
    • Cabling + mgmt0
    • Rack and Stack
  • Power on/Connect to APICs
    • How many APICs
    • Fabric Name
    • Admin Password
    • Setup Fabric Network ( IP & VLAN)
  • Log into the APIC (HTTP out of band)
    • NTP
    • Route Reflectors
    • MGMT IP Fabric
    • Leaf and Spine Name/#

Fabric Discovery

  • Zero touch fabric, the controller does everything
  • APIC uses LLDP to get information about the leaf switches it’s connected to
  • First the leaf is dicovered and will be named (101)
  • Then the Spine is connected and named  (201)
  • Then the leafs are discovered (103,104)


[DC] Datacenter Interconnects (DCI, OTV)

Distributed Data center Goals

  • Ensure business continuity
  • Distributed applications
  • Seamless workload mobility
  • Maximize compute resources

Challenges in traditional Layer 2 VPN:

  • Flooding Behavior
    • Unknown unicast for mac propagation
    • Unicast Flooding reaches all sites
  • Pseudo-wire Maintenance
    • Full mesh of Pseudo-wire is complex
    • Head-End replication is a common problem
  • Multi-Homing
    • Requires additional protocols and extends STP
    • Malfunctions impact multipe sites


[DC] Nexus features config / commands

VDC Configuration

  • Show license usage
  • Show vdc
  • Show vdc membership
  • vdc DCC01
  • allocate resource command
  • limit-resource command
  • show run vdc
  • switchto vdc DCC01

FEX Configuration

  • Enable feature FEX
  • configure fex 100
  • interface e1/25
  • switchport mode fex-fabric
  • fex associate 100
  • show fex

VPC Configuration

  • feature vpc
  • vpc domain 100
  • peer-keepalive destination source vrf management
  • sh vpc
  • int po10
    • vpc peer-link
  • int e 1/25
    • channel-group 10 mode active
  • int po10
    • vpc 10


[DC] Unified Computing Systems ( UCS )

UCS Physical Infrastructure

  •  Fabric Interconnect  ( 6248UP )
    • 32x Fixed unified ports: 1/10 GE or 1/2/4/8 FC
    • Expansion Module
    • Run in an Active / Active state for the dataplane
    • Run in a clustered Active/Passive state for the management
    • Connected to the UCS Chassis
    • Managed via UCSM or Cli (NX-OS)

  •  UCS Chassis
    • 6U Chassis, 32″Deep
    • Passive backplane
    • 8x Half width blades
    • 4x Full width blades
    • Everything is managed by the Fabric Interconnects.

Connecting the Fabric Interconnects to the LAN and SAN:



[DC] FC / FCoE

FCoE is short for Fibre Channel over Ethernet.

Fibre Channel over Ethernet (FCoE) solves the problem of organizations having to run parallel network infrastructures for their local area networks (LANs) and their storage area networks (SANs). As a result, they have to operate separate switches, host bus adapters (HBAs), network interface cards (NICs) and cables for each of these networks. Even utilizing a virtualization solution like VMware can actually increase the number of network adapters required to carry traffic out of the servers.


  • FIP – FCOE Initialization Protocol
  • FLOGI – Fabric login
  • FcF – FibreChannel Forwarder
  • FSPF – FibreChannel Shortest Path First
FC PortNameDescrption
N_PortNode PortEnd Device
F_PortFabric PortSwitch Port
L_PortLoop PortLoop Topo, End Device
NL_PortNode Loop PortN Port voor arbitrated loop ToPo
FL_PortFabric Loop PortAllows loops to connect to Fabric
E_PortExpansion PortSwitch to Switch connectivity ( ISL )
G_PortGeneric PortAllows auto config on the switch
B_PortBridge PortFC WAN Gateway Port
U_PortUniversal PortAUTO E, F, or FL Port



[DC] Unified Fabric and FCoE

Unified Fabric

  • Traditional DCs
    • LAN and SAN fabric isolation
    • Server has 2 adapters  HBA for SAN and NIC for LAN
    • Kept completely seperate end-to-end
  • Unified Fabric DCs
    • Server 10G ethernet Converged Network Adapters ( CNAs ) for both LAN and SAN ( FCoE )
    • the LAN and SAN traffic is Unified on the same wire providing I/O consolidation



  • FCoE is a protocol for transporting native FC frames over an 10G ethernet transport link.
  • Full FC frame is encapsulated onto a Jumbo Ethernet Frame.
  • FCoE requires lossless delivery.
  • FCoE requires a seperate FCoE VLAN from a normal VLAN traffic

FCoE Terminology

  • FIP – FCoE Initialisation Protocol
  • FCF – FCoE Forwarder ( switch )
    • accesswitch connected to the initiator
  • Enode ( server)
    • CNA running FCOE
  • Virtual Fibre Channel ( VFC ) Interface
  • Virtual Port Types
    • VN_Port
      • virtual node port
    • VF_Port
      • virtual facbric port
    • VE_Port
      • virtual extension port, switch to switch, multihop

Data Center Bridging

  • Data Center Briding ( DCB ) is a set of IEEE standards for Unified Fabrics
  • Priority Flow Control ( PFC )  ( 802.1Qbb )
    • Lossless delivery for selected CoS
  • Enhanced Traffic Selection ( ETS ) ( 8021.Qaz )
    • Bandwith Management and priority selection
  • These protocols combined: Datacenter Bridging Exchange


[DC] NX-OS – Overlay Transport Virtualization

What is OTV:

  • Layer 2 VPN over IPv4
  • Used over the DCI to extend VLANs between datacenter sites

OTV was designed for Layer 2 DCI

  • Optimizes ARP Flooding over DCI
  • Does not extend STP domain
  • Can overlay multiple VLANs without complicated design
  • Allows multiple edge routers without complicated design

OTV benefits

  • Provides a flexible overlay VPN on top of without restrictions for the IP nework
  • L2 transports leveraging the transport IP network capabilities
  • Provides a virtual multi-access L2 network that supports efficient transport of unicast, multicast and broadcast traffic

OTV Control Plane

  • Uses IS-IS to advertise MAC addresses between AEDs
    • “Mac in IP” Routing
  • Encapsulated as Control Group Multicast
    • Implies that DCI Must support ASM Multicast
    • Can be encapsulated as Unicast with OTV Adjacency Server

OTV Data Plane

  • Uses both Unicast and Multicast Transport
  • Multicast Control Group
    • Multicast or Broadcast Control Plane Protocols
    • eg. ARP, OSPF, EIGRP etc
  • Unicast Data
    • Normal Unicast is encapsulated as Unicast between AEDs
  • Multicast Data Group
    • Multicast data flows are encapsulated as SSM Multicast
    • Implies AED use IGMPv3 for (S,G) joins
  • OTV Adjacency Server can remove requirement for Multicast completely
    • Will result in Head End Replication when more than 2 DC’s connected over the DCI

OTV DCI Optimizations

  • Other DCI options bridge all traffic over DCI
    • eg. STP, ARP, Broadcast storms etc
  • OTV reducdes unnecessary flooding by:
    • Proxy ARP/ICMPv6 ND Cache on AED
    • Assumption is that hosts are bi-directional (not silent)
    • Inital ARPs are flooded, then cache is used
    • Terminating the STP Domain on AED.

OTV Configuration:


License needed:


[DC] Storage Networking & FibreChannel

LAN and SAN Separation

  • Security  Ensures protection from hacking
  • Bandwidth – SAN needs more bandwidth than LAN
  • Flow Control – SAN is lossless and LAN is lossy
    • Ethernet Flow control ( LAN ):
      • Source transmits packets untill receiver buffer overflow, then sends a “Pause” frame
      • Lost packets are retransmitted
    • Fibre Channel ( SAN ):
      • Credit based mechanism – Receiver has control
      • Source does not send a frame until the receiver telsl the source it can receive a frame by sending “Ready” signal Back
  • Performance – SAN provides more performance than LAN enviorments

LAN vs SAN flow control

  • Flow control is how data is controlled in a network
  • Ethernet Flow control ( LAN )
    • Source transmits packets until receiver buffers overflow, then sends a “Pause” frame
    • Lost packets are retransmitted
  • Fibre Channel ( SAN )
    • Credit based mechanism – Receiver has control
    • Source does not send a frame until the receiver tells the source it can receive a frame by sending “Ready” signal back.
    • “Lossless Fabric”


  • San Topologies
    • Point-to-Point
      • Initiator (server) and Target (Storage) directly connected
    • Arbitraded Loop (FC-AL) (Legacy)
      • Logical ring topology, similar to token ring
      • Implies connection is required on the ring
    • Switched Fabric ( FC-SW ) ( Standard)
      • Logical equivalent to a switched ethernet LAN
      • Switches manage the fabric allowing any-to-any communication
      • Support more than 16 million device addresses
  • FibreChannel Port types
    • N_port – Node Port
    • NL_port – Node Loop Port
    • F_port – Fabric Port
    • FL_port – Fabric Loop Port
    • E_port – Expansion Port ( ISL )
    • TE_port – Trunking Expansion Port
  • FC Addressing is analogous to IP over Ethernet
    • IP addresses are logical and manually assigned
    • Ethernet MAC Addresses are physical and burned in
    • FC World Wide Names ( WWNs )  / MAC / Zoning

      • 8 byte address burned in by manufacturer
      • Word Wide Node Name
      • World Wide Port Name
    • FC Identifier ( FCID )  / IP / Routing

      • 3 byte logical address assigned by fabric
      • FCID is subdevided into three fields:
        • Domain ID
          • Each switch gets a domainID
        • Area ID
          • Group of ports on a switch have an Area ID
        • Port ID
          • End station connected to switch gets a Port ID
  • FibreChannel Nameserver ( FCNS)
    • analogous to ARP cache
    • Used to resolve WWN ( pysical address ) to FCID ( logical address )
    • Like FSPF, FCNS requires no configuration
  • FibreChannel Logins
    • Ethernet networks are connectionless
    • Fibre Channel networks are connection oriented
      • All end stations must first register with the control plane of the fabric before sending any traffic.
    • Fabric Registration has three parts
      • Fabric Login ( FLOGI)
      • Port Login ( PLOGI)
      • Process Login ( PLRI )
    • sh flogi database
    • sh fcns database
  • VSANs
    • Logical seperation of SAN traffic
  • Zoning
    • like an ACL in the IP world



[DC] NX-OS – Fabricpath


Cisco FabricPath is a Cisco NX-OS software innovation combining the plug-and-play simplicity of Ethernet with the reliability and scalability of Layer 3 routing.

Using FabricPath, you can build highly scalable Layer 2 multipath networks without the Spanning Tree Protocol. Such networks are particularly suitable for large virtualization deployments, private clouds, and high-performance computing (HPC) environments.


Datacenter Design V ( TRILL, Fabric Path )

  • Classic Ethernet ( CE )
    • Regular internet with regular flooding, regular STP, etc.
  • Leaf switch
    • Connects CE domain to FP domain
  • Spine switch
    • FP backbone switch all ports in the FP domain only
  • FP Core Ports
    • Links on leaf up to Spine, or Spine to Spine
    • i.e. the switchport mode fabricpath links
  • CE Edge Ports
    • Links of leaf connecting to regular CE domain (to servers / switches)
    • i.e. NOT the switchport mode fabricpath links

Activating the fabricpath feature set.

For the activation is the “ENHANCED_LAYER2.PK” license needed, or the grace-period of 120 days:


vlan 100
  mode fabricpath
  name test

interface Ethernet2/1
  switchport mode fabricpath
  no shutdown

interface Ethernet2/2
  switchport mode fabricpath
  no shutdown

N7K3# sh run int e2/9
interface Ethernet2/9
  switchport access vlan 100
  no shutdown
N7K3# sh fabricpath isis

Fabricpath IS-IS domain : default
  System ID : 0026.c734.4f2f  IS-Type : L1 Fabric-Control SVI: Unknown
  SAP : 432  Queue Handle : 15
  Maximum LSP MTU: 1492
  Graceful Restart enabled. State: Inactive
  Last graceful restart status : none
  Graceful Restart holding time:60
  Metric-style : advertise(wide), accept(wide)
  Start-Mode: Complete [Start-type configuration]
  Area address(es) :
  Process is up and running
  CIB ID: 1
  Interfaces supported by Fabricpath IS-IS :
  Level 1
  Authentication type and keychain not configured
  Authentication check specified
  LSP Lifetime: 1200
  L1 LSP GEN interval- Max:8000 Initial:50      Second:50
  L1 SPF Interval- Max:8000     Initial:50      Second:50
  MT-0 Ref-Bw: 400000
        Max-Path: 16
  Address family Swid unicast :
    Number of interface : 6
    Distance : 115
  L1 Next SPF: Inactive

N7K3# sh fabricpath switch-id
                        FABRICPATH SWITCH-ID TABLE
Legend: '*' - this system
        '[E]' - local Emulated Switch-id
        '[A]' - local Anycast Switch-id
Total Switch-ids: 4
    1           0026.c751.bd2f    Primary     Confirmed Yes     No
    2           0026.c71f.a62f    Primary     Confirmed Yes     No
*   3           0026.c734.4f2f    Primary     Confirmed Yes     No
    4           0026.c7cb.4b2f    Primary     Confirmed Yes     No
N7K3# sh cdp nei
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater,
                  V - VoIP-Phone, D - Remotely-Managed-Device,
                  s - Supports-STP-Dispute

Device-ID          Local Intrfce  Hldtme Capability  Platform      Port ID
N7k1(TBC751BD00B)   Eth2/1         147    R S I s   N7K-C7018     Eth2/5
N7k1(TBC751BD00B)   Eth2/2         148    R S I s   N7K-C7018     Eth2/6
N7K2(TBC71FA600B)   Eth2/5         170    R S I s   N7K-C7018     Eth2/5
N7K2(TBC71FA600B)   Eth2/6         170    R S I s   N7K-C7018     Eth2/6
R1                  Eth2/9         134    R S I     3725          Fas0/0

Total entries displayed: 5
N7K3# sh fab
fabric       fabricpath
N7K3# sh fabri
fabric       fabricpath
N7K3# sh fabricpath route
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

0/3/0, number of next-hops: 0
        via ---- , [60/0], 0 day/s 03:03:28, local
1/1/0, number of next-hops: 2
        via Eth2/1, [115/400], 0 day/s 03:01:13, isis_fabricpath-default
        via Eth2/2, [115/400], 0 day/s 03:01:13, isis_fabricpath-default
1/2/0, number of next-hops: 2
        via Eth2/5, [115/400], 0 day/s 03:00:59, isis_fabricpath-default
        via Eth2/6, [115/400], 0 day/s 03:00:59, isis_fabricpath-default
1/4/0, number of next-hops: 4
        via Eth2/1, [115/800], 0 day/s 03:00:59, isis_fabricpath-default
        via Eth2/2, [115/800], 0 day/s 03:00:59, isis_fabricpath-default
        via Eth2/5, [115/800], 0 day/s 03:00:59, isis_fabricpath-default
        via Eth2/6, [115/800], 0 day/s 03:00:59, isis_fabricpath-default