Cisco Nexus 9300 Limitations
The Cisco Nexus 9300 series have a few limitations, which differs a bit by model.
Limitations
The following limitations did we encounter.
Q-in-Q tunneling.
Say we have 4 VPC port-channels on 2x Nexus 93180YC-FX3.
Po1 = Uplink 9216 MTU
Po2 = Cisco ASR-A 9216 MTU
Po3 = Cisco ASR-B 9216 MTU
Po4 = Fortigate 1500 MTU
We are using S-VLAN 100 and C-VLAN 10
Nothing can ping each other in the C-VLAN, but we see 1 device learning ARP.
Now we put S-VLAN 200 on Po2 and Po3, and everything works again.
We change it back to S-VLAN 100 and it is broken again.
Now we remove S-VLAN 100 from Po4 (Fortigate MTU 1500). And everything works.
Now w add S-VLAN 100 to Po4 and up the MTU to 9216. And it works.
So it seems that all ports that need to do Q-in-Q are required to have MTU 9216, even tough they don't send that big packets (ARP and PING surely should work on MTU 1500).
Q-in-Q Termination
The Nexus 9300 does not support Q-in-Q termination towards L3 capable interfaces. It does support Selective Q-in-Q and Q-in-Q tunneling, but not the termination of the interface.
For example on the Cisco ASR1001-X (and family members), you can do this like
interface Po1.5400 encapsulation dot1q 54 second-dot1q 100 ip address 10.10.10.1 255.255.255.0
Unfortunately this is not possible on the Nexus 9300 series.
Reserved VLANs
The Nexus 9300 series reserves the vlans 3968 till 4095 by default. In normal cases this would not cause a lot of issues, because we can plan what vlans to use. However it does also impose a problem on vlan mapping. Neither the original or translated vlan can be in the 3968-4095 range.
We normally would do something like this
NEXUS9300(config-if)# switchport vlan mapping 1480 1906
But if we fill in a vlan higher then 3967 as original vlan we it will not let us enter a translated vlan
NEXUS9300(config-if)# switchport vlan mapping 3970 1908
^
% Invalid command at '^' marker.
Cisco 93180YC-FX and SFP-10G-T-X
The Cisco 93180YC-FX series supports the SFP-10G-T-X copper SFP+ since NX-OS release 9.3(5). However there are some limitations.
The SFP-10G-T-X module cannot operate with other ports next to it active. What the exact reason for this is, is unknown to us, but we suspect it has to do with heat development as these modules get very hot, even when unused.
The switch will put this in the log if you use a unsupported port:
%ETHPORT-3-IF_XCVR_ERROR: Interface Ethernet1/21, doesn't support SFP-10G-T-X %ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet1/21 is down (Error disabled. Reason:Unsupported media-type or configuration)
The documentation on cisco.com states:
When you connect a SFP-10G-T-X device into a port, all the neighboring ports of this device must be either empty, or be connected to passive copper links only.
The following table should indicate which ports can be used, but remember the neighbouring ports can only contain passive copper links. NO optics!
| Device Name | Port Map |
|---|---|
| Cisco Nexus N9K-C93180YC-EX, N9K-C93180YC-FX, N9K-C93180YC-FX3 and N9K-C93180YC-FX3S | PI/PE: 1, 4-5, 8-9, 12-13, 16, 37, 40-41, 44-45, 48 |
Deployment Scheme for SFP-10G-T-X Transceivers[1]
| Active Port deploying the SFP+ 10GBASE-T transceiver, with max power consumption up to 2.5W.
Once configured with “media-type 10g-tx” in NX-OS or “Link Level Policy -> Physical Media Type -> SFP 10G TX” in ACI, these ports can deploy SFP-10G-T-X. Without such configuration, they behave like normal ports. | |
| Port Shutdown or Active with Passive Copper Cables only (Max. power consumption up to 0.1W).
Once 10g-tx is configured on yellow ports, ports to the left, right, top and bottom of the yellow port are referenced as blue ports. These adjacent ports will then support only low power Passive Copper DAC cable, or these can be left empty to conserve power. If 10g-tx configuration is removed from adjacent yellow ports, the blue ports will revert to behaving like normal ports. | |
| Active Ports 17 - 36 deploying any Cisco 1/10/25G optics (SFP, SFP+, SFP28) EXCLUDING SFP+ 10GBASE-T, with max power consumption up to 1.5W and Active Ports 49 - 54 deploying any Cisco QSFP, QSFP+, QSFP28 with max power consumption up to 3.5W. These ports are not part of any scheme and can deploy all regular Cisco optics and behave like normal ports. |
When inserting the module in one of the supported slots the switch will warn you about it:
%ETHPORT-3-IF_XCVR_ERROR: Interface Ethernet1/44, must configure [media-type 10g-tx] to support SFP-10G-T-X
So the media-type also has to be changed on the port (this is NOT documented).
