Nexus 9000 Standalone mode eNXOS - New Feature Walkthrough

I have just got two new Nexus 9396PX in my lab, and let's quickly walkthrough some unique features and capabilities on N9K eNXOS.

Comparing with Nexus 7K and 5K which have 2 separate images (kickstart and system), N9K eNXOS has only a single image file.

Although there is VDC command available, only single VDC is supported.

You can enable Linux bash shell access by issuing command "feature bash-shell"

You can also access to Python shell to do some programming and automation to the switch using python scripts.

Moreover, you can access to Broadcom shell for low level troubleshooting

There is a built in tcpdump-like sniffer, this example is simply sniff the traffic of my OOB mgmt port.

N9K standalone offers RESTful NXAPI as the northbound API, you can enable it by using command "feature nxapi"

then access the page http://<your 9K address> from your browser

Then you will see this Developer sandbox, which allows you to test the API calls and let you know the syntax and format:

For example you can submit a "show version" command and it will show the corresponding JSON request format and response, it saves you time to look at documentation to understand the request and response format.

And you can even submit bash shell command via NXAPI, it is really powerful.

Cisco vNAM and ERSPAN config on N7K

NAM is a tool which helps you to collect network information via SPAN, ERSPAN and Netflow.  There is a virtual version vNAM which allows you to install it on VM.  I have a setup in my NAM using vNAM 6.0(2).  This is particular useful after 1040 Sensor announced EoS, and it can provide data to Prime Collaboration Assurance for better reporting capability on voice service quality.  I have a Nexus 7000 in my lab, ERSPAN is configured to send traffic to send my vNAM.  This is my ERSPAN configuration on 7K and related show command.

monitor erspan origin ip-address 10.1.90.254 global

monitor session 10 type erspan-source
  erspan-id 90
  vrf default
  destination ip 10.1.90.71
  source vlan 90-91,93 both
  no shut

N7K# sh monitor session 10
   session 10
---------------
type              : erspan-source
state             : up
erspan-id         : 90
vrf-name          : default
acl-name          : acl-name not specified
ip-ttl            : 255
ip-dscp           : 0
destination-ip    : 10.1.90.71
origin-ip         : 10.1.90.254 (global)
source intf       :
    rx            :
    tx            :
    both          :
source VLANs      :
    rx            : 90-91,93
    tx            : 90-91,93
    both          : 90-91,93
source exception  :
filter VLANs      : filter not specified


Feature       Enabled   Value   Modules Supported       Modules Not-Supported
-----------------------------------------------------------------------------
MTU-Trunc     No
rate-limit-rx No
rate-limit-tx No
Sampling      No
MCBE          No
ERSPAN-ACL    -           -     1  4  7  8              5  10
ERSPAN-V2     Yes       -       1  4  7  8              5  10
RB span       No


Legend:
  MCBE  = Multicast Best Effort
  L3-TX = L3 Multicast Egress SPAN
  ExSP-X = Exception Span for type X (L3, FP, or misc)

Extensible Network Controller - XNC

XNC is Cisco distribution of OpenDayLight, with some features add on like network slicing, topology independent forwarding etc.  The original OpenDayLight will talk to the network element using Openflow as the southbound API.  For XNC, OnePK support is added and you can use it as a southbound protocol to talk to the network elements.  This blog post will give you a preliminary overview on how it works and how to add a "REAL" device to XNC / OpenDaylight.  On Internet right now there are a lot of tutorial on how OpenDayLight works with Mininet, but not much about how OpenDayLight / XNC works with a "REAL" network device that's why I want to give it a try and test it out.

To make Cisco switches Openflow capable, you need to install the "Cisco Plug-in for Openflow" to the Nexus switch.  In my testing environment I have deployed the Cisco Plug-in for Openflow 1.1.1 .ova file on Nexus 5596 virtual service container.  To understand how it is installed and activated, you can follow this guide:

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sdn/configuration/openflow-agent-nxos/cg-virtual-service-container.html

After installed and activated, you should see the Cisco Plugin for Openflow is running on your Nexus.  You need at least NX-OS 7.0.1.N1.1 on N5K to support this plugin.



Then you can start with some basic Openflow configuration on your Nexus box.  Detail configuration you can refer to the following guide:

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sdn/configuration/openflow-agent-nxos/cg-nxos-openflow.html#task_19259681156D48A9B43E37CE4CF650E6

This is my Openflow configuration.  My XNC IP address is 10.1.90.51.

 

One caveat is the default tcp port for Nexus and XNC is 6653/tcp on NX-OS.  If you don't specify the port in your configuration, it will default to 6653.tcp, which is WRONG!  I have spent days to find that out.  The correct port is 6633/tcp.  Needs to be careful about that.



Then you will able to see the state "ACTIVE" once you connect the XNC controller.
For the XNC controller, I have downloaded the version 1.5 from CCO and you can use it once you unzip it in a Linux box.  You need to make sure you have Java 1.7 or above before running it.  To run it, simply run the "runxnc.sh" and the controller process will run in background.

You can see from below screen capture that my N5K is discovered by the XNC controller:



I have also installed OpenDaylight in my environment, in the same machine.  Since both of them are listened to 8080/tcp for the web elements, I cannot turn them on at the same time.  So I have stopped my XNC and run ODL and it is how it looks.  Pretty similar to XNC but with less features.




You can also add on module to XNC such as TIF manager.  That is yet another zip file, once you unzip it, the .jar file will be placed in the plugin folder and it will show up on the web GUI once you restart the controller process.

OTV - Selective Unicast Flooding

Normally in OTV, unknown unicast frames are not flooded between OTV sites and MAC addresses are not learned across the overlay interface.  Any unknown unicast messages that reach the OTV edge device are blocked to prevent layer 2 errors spreading to remote sites.  It is assumed the end points are not silent or unidirectional.  If there is any silent hosts or Microsoft unicast mode NLB is used in your data center, your host will be "disappeared" from the view of other OTV sites.  In 6.2(2) and later, a new feature called selective unicast flooding is introduced and you can issue the command on the OTV VDCs at the site where the server / silent host exists, in result the specified destination MAC address is flooded to all other edge devices in the OTV overlay network with that unknown unicast traffic.

 otv flood mac 0011.2233.4455 vlan 66

RBAC - san-admin on Nexus 5500

Starting from NX-OS 5.2(1)N1(1), Nexus 5500 has a new predefined role "san-admin" which allows you to provide clear demarcation on what SAN and LAN team can do on the Nexus 5500.  You can use this for your DCNM for SAN too.  I have tried it on my DCNM 6.2(3) in lab and it works fine.  User with this right can do most of the things on DCNM for SAN but not DCNM for LAN.  Here is the detail on what a predefined san-admin role can do:

POC-N5K# sh role name san-admin

Role: san-admin

  Description: Predefined system role for san administrators. This role

  cannot be modified.

  vsan policy: permit(default)

  Vlan policy: permit(default)

  Interface policy: permit(default)

  Vrf policy: permit(default)

  -------------------------------------------------------------------

  Rule    Perm    Type        Scope               Entity                  

  -------------------------------------------------------------------

  27      permit  read        

  26      permit  read-write  feature             fcdomain                

  25      permit  read-write  feature             rdl                     

  24      permit  read-write  feature             trunk                   

  23      permit  read-write  feature             fcmgmt                  

  22      permit  read-write  feature             fcfe                    

  21      permit  read-write  feature             port-track              

  20      permit  read-write  feature             fcoe                    

  19      permit  read-write  feature             port-security           

  18      permit  read-write  feature             copy                    

  17      permit  read-write  feature             rmon                    

  16      permit  read-write  feature             rscn                    

  15      permit  read-write  feature             fspf                    

  14      permit  read-write  feature             fdmi                    

  13      permit  read-write  feature             fcsp                    

  12      permit  read-write  feature             fcns                    

  11      permit  read-write  feature             span                    

  10      permit  read-write  feature             zone                    

  9       permit  read-write  feature             wwnm                    

  8       permit  read-write  feature             vsan                    

  7       permit  read-write  feature             vsanIfvsan              

  6       permit  read-write  feature             fabric-binding          

  5       permit  read-write  feature             interface               

  4       permit  read-write  feature             trapRegEntry            

  3       permit  read-write  feature             snmpTargetAddrEntry     

  2       permit  read-write  feature             snmpTargetParamsEntry   

  1       permit  read-write  feature             snmp                

10GBase-LRM SFP+ support on Nexus

A quick note to bear in mind is 10GBase-LRM SFP+ is not supported on Nexus 5500 and Nexus 2000.  It only supports on Nexus 7000 in the Nexus family.

For detail compatibility information please check out here:

http://www.cisco.com/en/US/docs/interfaces_modules/transceiver_modules/compatibility/matrix/OL_6974.html

Comparing N5K and N7K - a very high level key features comparison

5K
- A-FEX
- VM-FEX
- FC
- Unified Ports
- EvPC

7K
- OTV
- LISP
- MPLS
- VDC

Both of them support FabricPath (N7K F1 and F2 cards), vPC, L3, FEX, etc

ASR 1000 OTV supports on IOS XE 3.5

Recently OTV is introduced to the ASR 1000 platform and is supported starting from IOS XE 3.5.  Basically the OTV support on ASR1K and N7K is similar, there are only a few difference that needs to note in this release:

- Support OTV with GETVPN on ASR1K
- No Adjacency server support, multicast WAN is required
- Support fragmentation
- Support of one Joint interface and one access interface per box

Nexus 7000 - M1 modules and F2 modules quick comparison

A fetaure comparison for quick reference:

M1	F2
Max 8 x 10GE line rate ports	Max 48 1/10GE line rate ports
Full L2 and L3 feature	Full L2 and L3 feature
Large FIB, ACL, QoS Tables	Small FIB, ACL, QoS Tables
MPLS	MPLS NOT supported
LISP	LISP NOT supported
FEX Support	FEX Support
FabricPath NOT Supported	FabricPath
FCoE NOT supported	FCoE (will support in future release)

Nexus 1000v Setup and Configuration

My environment:  Nexus1000v.4.0.4.SV1.3b, VMWare ESXi 4.1, vCenter 4.1

Nexus 1000v consists of VSM (control plane) and VEM (data plane).  VSM can run as a VM or in a standalone appliance.  In my setup I run it as a VM on one of my ESXi hosts.  The Nexus1000v software comes with both VSM and VEM within a zip file.

1. To install the VSM, locate the .ova file in the VSM folder.  In your vSphere client, click File > “Deploy OVF template”

2. During the deployment, you need to provide the following information:

• Control VLAN ID
• Packet VLAN ID
• Domain ID
• Management IP address

3. You can use the same vlan for control, packet and management, and do not place data traffic on this VLAN.  The wizard is quite straight forward so the details will not be covered here.

4. Next step is to create Nexus 1000v plug-in.  Browse the page http://vsm-ip-address and download the cisco_nexus1000v_extension.xml file

5. In vSphere Client, Plug-Ins menu, choose “Manage Plug-Ins”.  Click “Register Plug-In” and import the XML file downloaded from the VSM page.

6. Then connect your VSM to the vCenter

conf t

svs connection VC

protocol vmware-vim

remote ip address 1.2.3.4

vmware dvs datacenter-name panda-DC

connect

7. Then you can create system port-profile and data port-profile. 

port-profile type ethernet system-uplink
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan all
  no shutdown
  system vlan 2010
  state enabled

port-profile type vethernet cciev-server
  vmware port-group
  switchport mode access
  switchport access vlan 100
  no shutdown
  state enabled

8. Next step is to install VEM, the easiest way is to do it via VMWare Update Manager.  In vSphere client > Update Manager > Patch Repository, click “Import Patch”, then choose the VEM module in the Nexus 1000v VEM folder.

9. Then create a new baseline.  Under Baseline and Groups, click create baselines.  Choose VEM under extensions. 

10.  You can now apply the baseline to the host.  In vSphere Client > Hosts and Clusters, choose the host you want to install VEM.  Click the Update Manager tab which is usually the last tab.  Click “Attach” to attach the baseline you’ve just created, then click Remediate to apply.

11.  Now the final step is to add the host to the Nexus 1000v switch.  Under Inventory > Networking, choose the Nexus 1000v switch and right click, click “Add host”.  Apply the system port-profile to the vmnic and apply appropriate data port profile to the VMs.

12.  After adding, you can type the command “show interface virtual” and “show module” on VSM to verify your work.

Nexus 5000 - SFP validation failed on 1G port

My environment: Nexus 5010 with NX-OS 4.2(1)N2(1)

Nexus 5000 supports 1G connection starting from 4.0(1a)N1(1). I have tried to plug in the GE SFP to N5K and it shows the "SFP validation failed" message" in the "show interface" output. The trick is:

1. Unplug your GE SFP

2. go to your interface configuration, and type "speed 1000". You CAN'T change the port speed after you plug in your SFP.

3. Plug in your GE SFP again. And it works!

More information:

http://www.cisco.com/en/US/partner/docs/switches/datacenter/nexus5000/sw/command/reference/rel_4_1/ethernet-cmd-ref.html#wp1619365

"If the interface and transceiver speed is mismatched, the SFP validation failed message is displayed when you enter the show interface ethernet slot/port command. For example, if you insert a 1-Gigabit SFP transceiver into a port without configuring the speed 1000 command, you will get this error. By default, all ports on a Cisco Nexus 5000 Series switch are 10 Gigabits. "

Learning Nexus 7000 QoS by Example

My environment: Nexus 7010, NX-OS 4.2

Just got a chance to play with the Nexus 7000 NX-OS QoS, the “mls qos” syntax has gone and now all the commands are based on MQC.

 

In the following example, I’m going to configure 2 ingress ports, one with high priority traffic and one with normal traffic, and at egress port high priority traffic will be assigned to PQ to dequeue first.

 

Classification

N7K# conf t

class-map type queuing match-any 1p3q4t-out-pq1

  match cos 5

It can only be done via the main VDC.  It can’t be done on child VDC.  And you can ONLY match cos in the class-map.

Marking

Now let’s mark the ingress traffic, I will do this at the port assigned to the vdc “test”.

N7K# switchto vdc test

N7K-test#

policy-map type queuing highpriority-in-policy

   class type queuing 2q4t-in-q-default

     set cos 5

Scheduling and Queuing

Finally, I want to assign the high priority traffic (cos=5) to PQ

N7K-test#

policy-map type queuing highpriority-out-policy

  class type queuing 1p3q4t-out-pq1

    priority level 1

Last step is to assign service policy to the interface:

N7K-test#

int e3/1

description - High Priority – Ingress -

service-policy type queuing input highpriority-in-policy

!

int e3/2

description – Low Priority – Ingress -

!

int e3/3

description – Egress Port -

service-policy type queuing output highpriority-out-policy

!

Nexus: Hands on with NX-OS, Part#1

A good article from Mark Lewis on Cisco Subnet about NX-OS:

http://www.networkworld.com/community/node/26877

Some notable difference between NX-OS and IOS are some of the L3 features like OSPF, HSRP, etc.

NX-OS Stateful Process Restart

Demo on NX-OS Stateful Process Restart from Techwise TV:

NX-OS Virtual Device Context

A Demonstration on NX-OS Virtual Device Context (VDC) from TechWise TV:

Data Center of the future with Cisco Nexus 7000

New Monster Switch - Cisco Nexus 7000

Some key points for N7K:

• Data Center Class Switching Platform
• Today 1.4Tbps, Scales beyond 15Tbps
• New OS - NX-OS, a combination of IOS and SAN-OS
• Cisco Trusted Sec enabled - link layer encryption at line rate and RBAC
• Virtualization - Virtual Device Contexts
• Future FCoE support
• Unified I/O - save CapEx and OpEx - less HBA cards

Interesting Facts with N7K:

• Download entire Wikipedia in 10ms
• Download and transfer the entire searchable Internet in 7.5 mins
• Send a photo to every single person on earth in 1/2 hr.
• Transfer 5 mil concurrent Telepresence sessions