Hello,

We are using pairs of Cisco SFP-10G-BXU-kompatibles SFP+ 10GBASE-BX10-U SFP modules for a fast fiber link between the Kasli master and the Kasli satellites.

We also want to use a fast fiber link between the host computer and the Kasli master but it seems that the Kasli (or the computer?) doesn't allow a connection with the fiber SFP modules.

We have no problems when using a copper SFP module, so I don't think it's the network configuration.

Has anyone experienced these problems?

We also flashed the Cisco SFP to make it generic, e.g., compatible with Intel SFP network PCIe cards.

Best,
Bodo

    bodokaiser We also flashed the Cisco SFP to make it generic, e.g., compatible with Intel SFP network PCIe cards.

    With most 10G SFPs such flashing only replaces the content of an EEPROM to be read by the host. ARTIQ does not use this EEPROM. Check what your network card does. Cheap media converters also ignore the EEPROM as far as I know.

    On the Ethernet port, Kasli uses 1000BASE-X and does not support 10Gbps modes. What is your setup on the other end of the fiber? Note that 10Gbps transceivers are backward compatible with 1Gbps links and you do not need to replace those.

    On the Ethernet port, Kasli uses 1000BASE-X and does not support 10Gbps modes. What is your setup on the other end of the fiber? Note that 10Gbps transceivers are backward compatible with 1Gbps links and you do not need to replace those.

    We have a Intel 82599ES card with Dual-Port SFP+.

    We used nmtui to set the IP of enp1s0f0 to 192.168.0.100/24 and activate the link (setting this with ip addr is not permanent when doing reboots).
    Otherwise, we haven't changed anything.

    The relevant output of lspci -vv (with the working copper ethernet SFP module):

    00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-V (rev 10)
            DeviceName: Onboard - Ethernet
            Subsystem: Fujitsu Technology Solutions Device 1248
            Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
            Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
            Latency: 0
            Interrupt: pin A routed to IRQ 125
            Region 0: Memory at 8d900000 (32-bit, non-prefetchable) [size=128K]
            Capabilities: [c8] Power Management version 3
                    Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                    Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
            Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
                    Address: 00000000fee00278  Data: 0000
            Kernel driver in use: e1000e
            Kernel modules: e1000e
    
    01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
            Subsystem: Intel Corporation Ethernet Server Adapter X520-2
            Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
            Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
            Latency: 0, Cache Line Size: 64 bytes
            Interrupt: pin A routed to IRQ 16
            Region 0: Memory at 6000020000 (64-bit, prefetchable) [size=128K]
            Region 2: I/O ports at 3020 [disabled] [size=32]
            Region 4: Memory at 6000044000 (64-bit, prefetchable) [size=16K]
            Capabilities: [40] Power Management version 3
                    Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
                    Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
            Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                    Address: 0000000000000000  Data: 0000
                    Masking: 00000000  Pending: 00000000
            Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
                    Vector table: BAR=4 offset=00000000
                    PBA: BAR=4 offset=00002000
            Capabilities: [a0] Express (v2) Endpoint, MSI 00
                    DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                    DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                            MaxPayload 256 bytes, MaxReadReq 512 bytes
                    DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                    LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <1us
                            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                    LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                    LnkSta: Speed 5GT/s (ok), Width x2 (downgraded)
                            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                    DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                             FRS- TPHComp- ExtTPHComp-
                             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                    DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                             AtomicOpsCtl: ReqEn-
                    LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                             Compliance De-emphasis: -6dB
                    LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                             Retimer- 2Retimers- CrosslinkRes: unsupported
            Capabilities: [e0] Vital Product Data
                    Unknown small resource type 06, will not decode more.
            Capabilities: [100 v1] Advanced Error Reporting
                    UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                    UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                    UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                    CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                    CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                    AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                    HeaderLog: 00000000 00000000 00000000 00000000
            Capabilities: [140 v1] Device Serial Number 00-00-00-ff-ff-00-00-00
            Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                    ARICap: MFVC- ACS-, Next Function: 1
                    ARICtl: MFVC- ACS-, Function Group: 0
            Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
                    IOVCap: Migration-, Interrupt Message Number: 000
                    IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
                    IOVSta: Migration-
                    Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
                    VF offset: 128, stride: 2, Device ID: 10ed
                    Supported Page Size: 00000553, System Page Size: 00000001
                    Region 0: Memory at 000000008da00000 (64-bit, non-prefetchable)
                    Region 3: Memory at 000000008db00000 (64-bit, non-prefetchable)
                    VF Migration: offset: 00000000, BIR: 0
            Kernel driver in use: ixgbe
            Kernel modules: ixgbe
    
    01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
            Subsystem: Intel Corporation Ethernet Server Adapter X520-2
            Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
            Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
            Latency: 0, Cache Line Size: 64 bytes
            Interrupt: pin B routed to IRQ 17
            Region 0: Memory at 6000000000 (64-bit, prefetchable) [size=128K]
            Region 2: I/O ports at 3000 [disabled] [size=32]
            Region 4: Memory at 6000040000 (64-bit, prefetchable) [size=16K]
            Capabilities: [40] Power Management version 3
                    Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
                    Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
            Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                    Address: 0000000000000000  Data: 0000
                    Masking: 00000000  Pending: 00000000
            Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
                    Vector table: BAR=4 offset=00000000
                    PBA: BAR=4 offset=00002000
            Capabilities: [a0] Express (v2) Endpoint, MSI 00
                    DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                    DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                            MaxPayload 256 bytes, MaxReadReq 512 bytes
                    DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                    LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <1us
                            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                    LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                    LnkSta: Speed 5GT/s (ok), Width x2 (downgraded)
                            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                    DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                             10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                             FRS- TPHComp- ExtTPHComp-
                             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                    DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                             AtomicOpsCtl: ReqEn-
                    LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                             Retimer- 2Retimers- CrosslinkRes: unsupported
            Capabilities: [e0] Vital Product Data
                    Unknown small resource type 06, will not decode more.
            Capabilities: [100 v1] Advanced Error Reporting
                    UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                    UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                    UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                    CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                    CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                    AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                    HeaderLog: 00000000 00000000 00000000 00000000
            Capabilities: [140 v1] Device Serial Number 00-00-00-ff-ff-00-00-00
            Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                    ARICap: MFVC- ACS-, Next Function: 0
                    ARICtl: MFVC- ACS-, Function Group: 0
            Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
                    IOVCap: Migration-, Interrupt Message Number: 000
                    IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
                    IOVSta: Migration-
                    Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 01
                    VF offset: 128, stride: 2, Device ID: 10ed
                    Supported Page Size: 00000553, System Page Size: 00000001
                    Region 0: Memory at 000000008dc00000 (64-bit, non-prefetchable)
                    Region 3: Memory at 000000008dd00000 (64-bit, non-prefetchable)
                    VF Migration: offset: 00000000, BIR: 0
            Kernel driver in use: ixgbe
            Kernel modules: ixgbe

    and lshw (with the working copper ethernet SFP module):

            *-pci:0
                 description: PCI bridge
                 product: Cannon Lake PCH PCI Express Root Port #5
                 vendor: Intel Corporation
                 physical id: 1c
                 bus info: pci@0000:00:1c.0
                 version: f0
                 width: 32 bits
                 clock: 33MHz
                 capabilities: pci pciexpress msi pm normal_decode bus_master cap_list
                 configuration: driver=pcieport
                 resources: irq:122 ioport:3000(size=4096) memory:8da00000-8ddfffff ioport:6000000000(size=1048576)
               *-network:0
                    description: Ethernet interface
                    product: 82599ES 10-Gigabit SFI/SFP+ Network Connection
                    vendor: Intel Corporation
                    physical id: 0
                    bus info: pci@0000:01:00.0
                    logical name: enp1s0f0
                    version: 01
                    serial: 00:e0:ed:37:be:a6
                    size: 1Gbit/s
                    capacity: 1Gbit/s
                    width: 64 bits
                    clock: 33MHz
                    capabilities: pm msi msix pciexpress vpd bus_master cap_list ethernet physical tp 1000bt-fd autonegotiation
                    configuration: autonegotiation=on broadcast=yes connector=RJ45 driver=ixgbe driverversion=5.14.0-427.24.1.el9_4.x86_64 duplex=full firmware=0x00011bab ip=192.168.0.100 latency=0 link=yes maxlength=100m module=SFP-GB-GE-T multicast=yes port=twisted pair speed=1Gbit/s
                    resources: iomemory:600-5ff iomemory:600-5ff irq:16 memory:6000020000-600003ffff ioport:3020(size=32) memory:6000044000-6000047fff memory:8da00000-8dafffff memory:8db00000-8dbfffff
               *-network:1
                    description: Ethernet interface
                    product: 82599ES 10-Gigabit SFI/SFP+ Network Connection
                    vendor: Intel Corporation
                    physical id: 0.1
                    bus info: pci@0000:01:00.1
                    logical name: enp1s0f1
                    version: 01
                    serial: 00:e0:ed:37:be:a7
                    capacity: 1Gbit/s
                    width: 64 bits
                    clock: 33MHz
                    capabilities: pm msi msix pciexpress vpd bus_master cap_list ethernet physical tp 1000bt-fd autonegotiation
                    configuration: autonegotiation=on broadcast=yes driver=ixgbe driverversion=5.14.0-427.24.1.el9_4.x86_64 firmware=0x00011bab latency=0 link=no multicast=yes port=twisted pair
                    resources: iomemory:600-5ff iomemory:600-5ff irq:17 memory:6000000000-600001ffff ioport:3000(size=32) memory:6000040000-6000043fff memory:8dc00000-8dcfffff memory:8dd00000-8ddfffff

    Do you think the card is not compatible or it needs further configuration?