Cisco Data Center: Unified Fabric and FCoE

TIP: In FCoE world there are no HBAs, but there are CNAs instead. All the Storage concepts stay the same.
TIP: If you want to use the FCoE, you first need to enable the feature using the command “feature fcoe”. Also, have in mind that Max Length of the FCoE is 2240 bytes, and Jumbo Frames are required.

FCoE is a great model because it uses the Ethernet, and therefore enables the convergence of SAN and LAN (Unified Fabric). FCoE is an encapsulation of the FC frames into Ethernet. The only difference between FC and FCoE is the L1 and L2 TRANSPORT.

In the simplest FCoE topology, a server with CNA (converged network adapter; a card that can send both Ethernet and FCoE traffic over the same gigabit Ethernet uplink) is connected to an FCoE-enabled switch, which has a direct connection into the legacy FC network.
The idea is to make Ethernet lossless, because Storage requires the lossless flow. FCoE is fully defined in FC-BB-5 standard, and it is defined by the following 3 criteria:
-        PFC (Lossless Ethernet), 802.1Qbb. It enables Flow Control using the Priority Bit using the Pause Frames (802.1p). The link is devided into the 8 “lanes” using 3 bits, and only one lane is dedicated to the FCoE. This enables FCoE to create the lossless environment. When too many frames are sent – the PAUSE is sent on the lane basis, not on the entire link.
-        ETS (Priority Grouping), 802.1Qaz. This is a way to guarantee the BW, because it can assign BWs percentages to groups.
-        DCBX (Data Center Bridging Excange, used for the Configuration Verification), 802.1Qaz
It requires Jumbo Frames and 10Gbps Links. FCoE requires that the FC IDs be mapped to the Ethernet MAD Addresses.
FIP (FCoE Initialization Protocol) is a Control Plane on FCoE, and it will be running between the FCoE Initiators and FCoE Switch, known as the FCF (FCoE Forwarder). FLOGI happens inside of FIP within the FCoE.

The port roles in FCoE are the following:
-        ENode is the FCoE Node Port. ENode gets a Fabric Provided MAC Address (FPMA) for FCoE, which is ultimately used for the Data Plane as a SAN-MAC (apart from the FCID, that it also gets assigned). It still uses the MAC address for the LAN traffic, and it´s 6 bytes (3 bytes of FCID and 3 bytes of FC-MAP). This FC-MAP is a mechanism that separates the VSANs, so that one VSAN can never go through a switch it´s not meant to (The Switch descarts MACs that are not part of a particular fabric).
-        Virtual Fibre Channel (VFC) Interface, the logical representation of the FC interface to the Ethernet world. The FIP runs between the ENode and the VFC. They form a p2p logical link between them.
-        Virtual Port Types:
  • VN_Port, like the N port in the FC
  • VF_Port, like the F port in the FC
  • VE_Port, like the E port in the FC (normally these are actually TE ports, running Trunking)

FCF (Fiber Channel Forwarders) are like a switching devices for FCoE, do Forwarding of the Information and the Encapsulation and Decapsulation of the FC.  These are the combination of the FCoE Termination Functions and Fiber Channel stack on Ethernet Switches, asa Dual Stack Switches.
End Nodes represent Virtual FC Interfaces, called VN ports, or the virtual N Ports. The Virtual Fabric ports are the VF ports on the switch.

Configure FCoE

You need to be sure you need the release notes for the exact CNA card, and the exact release of your NX-OS to be sure what is supported.
Step 1: First of all you need to create a VSAN:
(config)# vsan database
(config-vlan)# vsan 1010
(config-vlan)# vsan 1010 name VSAN_1010

TIP: A link going from the Switch to the CNA has to be a Trunk Link, and the FCoE VLAN has to be on the allowed list. The FCoE VLAN must not be a native VLAN, and don’t use the VLAN 1. If you want the CNA to have an Access VLAN, that has to be a NATIVE VLAN.
Step 2: Map a VLAN to VSAN:
(config)# vlan 1010
(config-vlan)# fcoe vsan 1010

If you want FLOGI not to fail on the FCoE ports directly connected to the CNAs, you need to make it a EDGE port, because the FLOGI timeout will expire cause it´s shorter then STP timer:
(config-if)# spanning-tree portfast edge trunk

Step 3: Configure the vFC and bind the VSAN to the physical interface where the CNA is connected:
(config)# interface vfc1010
(config-if)# bind interface gi1/1
(config-if)# switchport trunk allowed vsan 1010
(config-if)# no shut

Step 4: vFC (virtual Fiber Channel) Interface should be associated to only one VSAN. You need to bind these:
(config)# vsan database
(config-vlan)# vsan VSAN_1010 interface vfc1010

Step 5: Configure the Ethernet Link (facing the Server) as a Trunk:
(config)# interface Gi1/1
(config-if)# shut
(config-if)# spanning-tree portfast edge trunk
(config-if)# switchport trunk native vlan 10
(config-if)# switchport trunk allowed vlan 10,1010 


All Fibre Channel and FCoE control and data traffic is automatically classified into the FCoE system class, which provides no-drop service. There are two protocols here:
  • FCoE is in charge of the Data Plane
  • FIP (FC Initialization Protocol) handles the Logins and the Control Plane.
  • This class is created automatically when the system starts up (the class is named class-fcoe in the CLI). 

You cannot delete this class, and you can only modify the IEEE 802.1p CoS value to associate with this class. This class is identified by qos-group 1.  In the N5k Switch you will see the following in the default running configuration:
# class-map type qos class-fcoe
# class-map type queuing class-fcoe
# match qos-group 1

Setting up an FCoE connection on the host or storage requires one or more supported converged network adapters (CNAs) connected to a supported FCoE switch. The CNA is a consolidation point and effectively serves as both an FC HBA and an Ethernet adapter
The CNA is presented to the host and target as both an FCoE Initiator HBA and a 10-Gb Ethernet adapter. The FCoE Initiator HBA portion of the CNA handles the FCoE traffic when traffic is sent and received as FC frames mapped into Ethernet packets (FC over Ethernet). The Ethernet adapter portion of the CNA handles the standard Ethernet IP traffic, such as iSCSI, CIFS, NFS, and HTTP, for the host. Both the FCoE and standard Ethernet portions of the CNA communicate over the same Ethernet port, which connects to the FCoE switch.
The FCoE target adapter is also sometimes called a "unified target adapter" or UTA. Like the CNA, the UTA supports both FCoE and regular Ethernet traffic. FCoE frames have the MAC Addresses (hop-by-hop) and the FC addresses (end-to-end).

You should configure jumbo frames (MTU = 9000) for the Ethernet adapter portion of the CNA. You cannot change the MTU for the FCoE portion of the adapter.

SAN Port-Channels like Port-Channels, but to aggregate SAN PCs using the PCP (Port Channeling Protocol).
(config)# interface san-port-channel-1
(config-pc)# switchport mode E
(config-pc)# channel mode active
(config-pc)# switchport trunk allowed vsan 1
(config-pc)# switchport speed 4000


Most Popular Posts

SDN Wars: Cisco ACI vs VMware NSX

Cisco ACI Guide for Humans, Part 1: Physical Connectivity

VMware NSX Home Lab