I was troubleshooting an MTU related issue for NFS connectivity in a Flexpod (Cisco UCS, Cisco Nexus, and Netapp storage with VMware vSphere, running the Nexus 1000v). Regular-sized frames were making it through, but not jumbo frames. I ensured the endpoints were set up correctly, then moved inwards….in my experience, the problem is usually there.
The original design basically included the use of CoS tag 2 for all NFS traffic, so that it could be honored throughout the network, and given jumbo frames treatment.
This was easy enough to do with a Nexus 1000v port profile for all vSphere VMKernel interfaces:
1KVVSM# show run port-profile NFS port-profile type vethernet NFS vmware port-group switchport mode access switchport access vlan 260 service-policy input TAG_COS_2 no shutdown max-ports 96 state enabled
I looked on the 1000v uplink port profile, I looked everywhere in UCS, and the Neuxus 5Ks. I had this CoS tag going to the “Silver” class in UCS and the Nexus 5K pair:
I was at a loss. Then I realized that I hadn’t done anything to classify the traffic coming from the storage array, which was also directly connected to the Nexus 5Ks. So just to get things working, I went ahead and just enabled jumbo frames on the default class (unclassified traffic). I’d have to figure out how to classify that traffic later - probably with a per-port policy-map. (ick)
However, jumbo frames still weren’t working! I enabled jumbo frames on the default class, or as I like to call it, the “screw-it-we’ll-deal-with-it-later” class.
Now I was really frustrated. So, I got out one of my favorite MTU troubleshooting commands, which is:
show queuing interface
and after verifying that all of my QoS groups that I expected to have jumbo frames had the correct MTU set, and they did.
That’s when I noticed something peculiar. Under QoS group 4, which I had not intentionally done anything with in my configuration, I noticed some packets were coming in.
DCB5596LANA# show queuing int Eth1/35 (some output omitted) RX Queuing qos-group 4 q-size: 22720, HW MTU: 1500 (1500 configured) drop-type: drop, xon: 0, xoff: 22720 Statistics: Pkts received over the port : 1651 Ucast pkts sent to the cross-bar : 1651 Mcast pkts sent to the cross-bar : 0 Ucast pkts received from the cross-bar : 0 Pkts sent to the port : 0 Pkts discarded on ingress : 0 Per-priority-pause status : Rx (Inactive), Tx (Inactive)
Obviously this isn’t always the case, but it just so happens that anything with a CoS tag of 4 makes it into this QoS group. That’s how I have my policy maps configured:
class-map type qos match-all class-gold match cos 4 class-map type qos match-all class-bronze match cos 1 class-map type qos match-all class-silver match cos 2 class-map type qos match-all class-platinum match cos 5 policy-map type qos system-level-qos class class-platinum set qos-group 5 class class-gold set qos-group 4 class class-silver set qos-group 3 class class-bronze set qos-group 2 class class-fcoe set qos-group 1
This means that the traffic leaving the Netapp filers (non-FCoE anyways) is automatically marked with a CoS value of 4. This is good news for me, because I ultimately don’t really care what CoS values are used, as long as it’s something. Any value will allow me to place that traffic into a class other than the default class, which is one of my own personal recommendations for this kind of configuration. No way to shoot yourself in the foot later that way.
The fix was simply to use this to my advantage. Rather than use a CoS tag of 2, which was more or less arbitrary, I just moved the NFS class to use CoS 4. It would be nice in the future if Netapp allows this value to be configurable, but until then I can make this work.
So there you have it. If you’re using a Netapp filer for IP storage, make use of the fact that these markings are taking place.