Question New to 10GbE Fiber - Can't connect machine to switch

[NVergunst]

So I am having a problem trying to get my windows machine to talk to my switch connected via LC-LC fiber connection. Since I am new at this, I am not sure what the steps for debug even are. I believe I have ruled out hardware problems except for the switch expansion card itself.

The switch is a Brocade FastIron CX 648S-PoE configured as a single node, not stacked. It has an FCX-2XG 2-port 10G module which has 2 XFP connectors on it. I run a XFP to LC adapter and plug the fiber in. That connects to 5m of cable to an LC to SFP+ adapter. That plugs into an Intel X520-2 NIC on a Windows Server 2016 machine.

Using the same card in the same WS2016 machine to an SFP+ adapter on the switch I can auto-negotiate and get 1Gbps speeds. All works, but it is the same speed as the copper line connected too. The SFP+ ports on the switch are limited to 1Gbps, so they don't help. So I have new hardware, this FCX-2XG that was a working used pull (or so they say). The switch tests it as being OK. When I have it connected to the WS2016 machine the Windows box autonegotiates a 10GbE speed and starts sending packets. No response back (0 received packets).

When I look on the switch, it says the status is Up which is good, but the Actual Speed/Mode section says "None" and that's consistent with what I am seeing. It appears to me that the switch just isn't auto-negotiating. Unfortunately I can't really test this as I don't have other compatible hardware at 10GbE. This is my first go. That's why I thought it was something silly. So I bought a new cable just in case. No difference, still connects through the other port at 1Gbps. I checked the laser coming out to make sure it was crossing over A-B and B-A. For giggles I swapped it, and the WS2016 box said the network cable was unplugged. Swapped it back, it auto-negotiated to 10GbE and no received packets.

At this point I am thinking the switch and/or the expansion card in the switch is the problem. But I don't see why it wouldn't work and yet pass all tests when the hardware is supposedly known working good. Any ideas from people that work with this stuff all the time? Something simple I may be missing?

 

[bill001g]

The ports on your switch are SFP not SFP+ which is why they are limited to 1gbit.

XFP is a strange device I have only seen them on fiber channel boxes and on telco equipment to support oc192. Part of the issue is they can run these other protocols so you need to be sure your switch somehow knows you want to run 10g ethernet over it rather than these other protocols.

First thing to check is optics. LC is only the physical connection. The actual light colors..ie frequencies must match. 2 common ones are 850nm and 1300nm. XFP are kinda strange they can also support transmission and reception on a single fiber stand using 2 different lasers on difference frequencies to run transmit and receive.

Verify the SFP and XFP are actually using the same light transmission. Note you can have issue if the optics are designed for long distance ie 10kilometers and you use a short cable to connect them. In the long run it will damage the optics.

So after all this you can still have issues. You need to disable any form of negotiation of port aggregation. I know cisco ones have issues with negotiation of "trunk" ..ie port tagging. It could be the port is getting stuck in one these negotiation that don't matter because you are hooking it to a server rather than another switch.

 

[NVergunst]

First off thankyou very much for your response.

The ports on your switch are SFP not SFP+ which is why they are limited to 1gbit.

Yes, understood. I was using it more to suggest that the fiber cable itself seems to work and I was trying intermediary troubleshooting without success. 1Gb speeds are great, but I am saturating the line so I really need 3Gb+, so hopefully the 10Gb will work.

XFP is a strange device I have only seen them on fiber channel boxes and on telco equipment to support oc192. Part of the issue is they can run these other protocols so you need to be sure your switch somehow knows you want to run 10g ethernet over it rather than these other protocols.

That seems like a likely candidate. Unfortunately I am not sure how to tell it to do normal ethernet and not something weird. I haven't seen it on the GUI settings anywhere, and when I telnet into the box and check the list of options, I don't see anything obvious that would say "become a scsi node" or anything like that. I do love these weird old Brocade switches, but I am the first to admit I really barely understand them at all. Most of my network has been on Cisco hardware but these were the first switches I got that had redundant high-power power supplies so every one of the 48 PoE ports could actually be used. Too many of my Cisco devices had 48 PoE ports but less than 300W of total power which didn't make sense.

First thing to check is optics. LC is only the physical connection. The actual light colors..ie frequencies must match. 2 common ones are 850nm and 1300nm. XFP are kinda strange they can also support transmission and reception on a single fiber stand using 2 different lasers on difference frequencies to run transmit and receive.

Verify the SFP and XFP are actually using the same light transmission. Note you can have issue if the optics are designed for long distance ie 10kilometers and you use a short cable to connect them. In the long run it will damage the optics.

Great suggestion. That's what I tried doing earlier as well, and I did re-check it.

So the PC has the X520-2 card in it with 2 SFP+ ports. They should be able to take 810nm and 1310nm SFP+ adapters.

I am using 10GTek SFP+ adapters in the Intel PCI card that say: 10 Gigabit SFP+ LC Multi-mode Transceiver, 10GBASE-SR Module for Intel E10GSFPSR (850nm, DDM, 300m). And in the description it says the "SR" version is using MMF.

This goes through some LC-LC cable (OM1 (62.5/125) Fiber Optic Cables are pre-terminated with LC to LC Connectors) to the Brocade switch. In the Brocade switch are the XFP to LC (direct) adapters which are brand new and say: Brocade 10G XFP SR Optic Transceiver Module (MMF, 850nm, 300m, LC, DOM)

So as I understand it, everything should be 850nm and using MMF. I am not sure if I am correct in that all of these should play nicely with eachother, but on paper I don't see why not...

So after all this you can still have issues. You need to disable any form of negotiation of port aggregation. I know cisco ones have issues with negotiation of "trunk" ..ie port tagging. It could be the port is getting stuck in one these negotiation that don't matter because you are hooking it to a server rather than another switch.

I have disabled everything I can except for speed autonegotiation because there is no option on the XFP ports. Either it is autonegotiated on, or disabled as far as I know. I can't select a forced speed like with the other copper lines.

 

[bill001g]

I wish I could be more help the only brocade command I remember was show media. I suspect you have the correct XFP. The ones that run fiber channel are 16g.

The server guys considered storage networks there area so I did little with the brocade boxes. They pretty much always ran the ports in fiber channel mode and used other switch brands for the network facing side of the servers.

Maybe a silly question but you did cross the fiber pair when you hooked it up.

850 nm you can actually see in a dark room. You do not want to stare into too long but the tip of the fiber will glow slightly red.

 

[NVergunst]

I wish I could be more help the only brocade command I remember was show media. I suspect you have the correct XFP. The ones that run fiber channel are 16g.

The server guys considered storage networks there area so I did little with the brocade boxes. They pretty much always ran the ports in fiber channel mode and used other switch brands for the network facing side of the servers.

Maybe a silly question but you did cross the fiber pair when you hooked it up.

850 nm you can actually see in a dark room. You do not want to stare into too long but the tip of the fiber will glow slightly red.

A second mind generally helps, even if it is just to talk through things, so thankyou.

As far as I know this is the only XFP module that fits this box, so I hope it's the only one lol. This particular box actually has the 48 normal 1G PoE ports plus 4 SFP 1G ports (they take over the normal 1/1/1-1/1/4 ports so not at the same time, still 48 total), plus x2 16Gb ports on the back that are the stack ports but I think you can do some commands to force them to be uplink ports, then the 2 front XFP ports I am having trouble with.

I am thinking if I can find another switch that is compatible with my current FCX stacks which I think limits me to FCX and ICX, and maybe VDX (but I have heard horror stories about VCX...), then I will just buy one that stacks through the 16Gb connection and gives me 10Gb SFP+ ports on it somewhere. Might be easier. I don't know why I can't get it to work, but they are so out of support (and I am not the original customer) that it's unlikely I will get any official help. :/

 

[bill001g]

I keep this one bookmarked because people ask for PoE powered switches every so often. For $139 you get that plus 24 gigbit and 2 10g sfp+ports. Cost has come way down. I am tempted every time I look at this switch but I really don't need any more switches.

https://mikrotik.com/product/CSS326-24G-2SplusRM#fndtn-specifications