Interface Guide
Introduction
LINX LANs are a shared Layer 2 network, so a number of rules must be followed to ensure only permitted traffic is sent on your LAN Port. This is to stop any untoward behaviour or issues on the LAN. If you are sending permitted traffic on the LAN we reserve the right to shutdown your peering port if this is not rectified, as per the MoU.
The full MoU can be found on the main website here - Memorandum of Understanding - LINX
Technical requirements
These can also be found in 'Appendix1' of the MoU linked above.
Physical
Physically we recommend all ports are set to auto-negotiation for speed and duplex settings.
MAC layer
Frames forwarded to LINX ports of the following ether-types only:
0x0800 - IPv4
0x0806 - ARP
0x86dd - IPv6
We restrict ports to a single MAC address, with LAG ports treated as a single interface from this point of view. This MAC address is set during the provisioning process to bring your port live and if you need the MAC changed please contact the NOC at [email protected]. We can allow 2 MACs for brief periods to allow a smooth transition to a new MAC if needed.
Broadcast or Multicast packets should not be sent on your peering interface apart from the below examples:
Broadcast ARP packets
Multicast IPv6 Neighbour Solicitation packets
and link-local protocols should not be forwarded except for:
ARP
IPv6 Neighbour solicitations and advertisements
Link-local protocols include, but are not limited to:
IRDP
ICMP redirects
IEEE802 Spanning Tree
Vendor proprietary discovery protocols (e.g. CDP, EDP)
Interior routing protocol broadcasts (e.g. OSPF, ISIS, IGRP, EIGRP)
BOOTP/DHCP PIM-SM PIM-DM DVMRP
IP layer
You should only ever use IP addresses and net masks assigned by LINX on your peering port. IPv6 addresses (link & global scope) should be explicitly configured and not auto-configured and IPv6 site-local addresses should not be used.
Routing
BGP should be configured as BGP4, and AS numbers used in BGP4 sessions should not be from private ranges reserved for private use.
We encourage good engineer practice and members are encouraged to aggregate routes. IP address space assigned to the LINX peering LANs should never be advertised to other networks without explicit permission from LINX.
All routes advertised across the LINX network should point to the router advertising it, and all routes advertised in a peering session across LINX should be registered in the RIPE DB or other public routing registry.
For more detailed routing requirements see section 4 of ‘Appendix 1’ of the MoU linked above.
Vendor specific guides
Please see below for some vendor specific interface examples
Cisco
There are a number of specifics that should be disabled on your Cisco interface, these are:
no ip redirects
no ip proxy-arp
no cdp enable
no mop enable
ipv6 nd ra suppress all
And on switch interfaces you may need:
no keepaliveSo an example Cisco router interface config would be:
interface GigabitEthernet0/0
description LINX Interface
ip address 195.66.224.n 255.255.254.0
no ip redirects
no ip proxy-arp
no mop enable
no cdp enable
ipv6 nd ra suppress allJuniper
If anyone is still using JunOS versions up to 5.3R4 (this was end of support in 2003 so we hope not) there is a bug that will cause a Juniper router to emit IGMP packets on all its interfaces, even when IGMP is disabled. The only way to stop your router from transmitting IGMP is to configure outgoing packet filters on your LINX interface(s), or upgrade your JunOS version.
Unicast BGP configuration
To ensure you only exchange unicast routes in the unicast ISP peering LAN, explicitly add the following statement to all neighbors, groups and prefix-limits:
set family inet unicastYou need to be thorough with family inet unitcast. If even one of the neighbors, groups or prefix-limits is defined with a family inet “any”, you'll enable multicast and turn on MBGP.
IPv4 ARP Cache Timeout
Juniper’s default ARP cache timeout is 20 minutes. To reduce the amount of unnecessary broadcast traffic, we recommend setting the ARP cache timeout on Juniper routers to 4 hours:
set system arp aging-timer 240You can configure a hold timer on 10GE interfaces. A sensible value for this would be 1200ms:
set interfaces xe-0/0/0 hold-time up 1200 down 1200If this is being added to aggregate interfaces this needs to be added to all physical interfaces, and the logical ae interface.
Interface Configuration Guides
Here are some basic pointers that will help to connect your device to any of the LINX LANs.
The LINX MoU rules restrict the type of traffic and the number of source MAC addresses that members are allowed to present to any of the LINX exchanges.
This article explain how to configure your interface towards the LINX LANs to only send allowed traffic to the exchanges.
Introduction
LINX operates a shared Layer 2 (L2) Ethernet infrastructure. It is required by the technical specifications laid out in the LINX MoU that all members abide by the same rules. This is so the stability of the LAN is maintained as a member responsibility.
1.1 Definition of Terms
In this document we refer to terms like 'L2 device', 'L2/L3 hybrid', etc. Here are our definitions:
L2 Device is a device that functions as a Layer 2 (Ethernet) bridge (a.k.a. 'switch', 'bridge', 'hub', etc).
L3 Device is a device that functions as a L3 (IP) router only. This means it does not bridge any Ethernet frames between its interfaces. Such a device is typically called a 'router'.
L2/L3 Hybrid is a device that functions both as a L2 bridge and a L3 router. This means it can both bridge Ethernet frames between its interfaces as well as route IP traffic and participate in IP routing protocols. Foundry/Brocade, Force10 and Extreme are common examples of this type of device.
2. The LINX LAN Topologies
2.1 London LON1
The LINX London LON1 network consists of two separate high-performance Ethernet switching platforms accessible from 16 data centre locations. The two vendors used in LON1 are Juniper Networks and Nokia, both employing EVPN over VPLS.
2.2 London LON2 and LINX Manchester
The LINX London LON2 and LINX Manchester networks employ EVPN over VXLAN and features leaf-spine topology. They use IP Infusion's OcNOS™ network operating system and switch hardware from Edgecore Networks. LINX's disaggregated LON2 platform came into operation in the summer of 2018 with LINX being the first IXP anywhere in the World to implement such a system.
2.3 LINX NoVA
The LINX NoVA network consists of a Juniper Networks Ethernet switching platform, accessible from five data centre locations, all employing VPLS.
2.4 LINX Wales and LINX Scotland
The LINX Wales and LINX Scotland networks both consist of a L2 Extreme Networks switching platform. The platform for each site are accessible from two data centre locations in their respective regions.
3. Configuration Recommendations
3.1 IPv4 ARP / IPv6 Neighbour Timeout
All equipment vendors implement their own maximum age limits for IPv4 ARP and IPv6 neighbour caches.
LINX recommends setting the ARP cache timeout on your device to at least two hours, preferably four (240 minutes).
Note that a low ARP timeout can lead to excessive ARP traffic, especially if the values are lower than the BGP KEEPALIVE interval timers. Long timeouts can lead to longer downtime if you change equipment (since your peers still have the old MAC address in their ARP cache).
3.2 Peering LAN Prefix
Each of the LINX LANs have a different peering LAN prefix range. Each range is supposed to be globally routable.
Members must ensure that:
None of the LINX LAN prefix ranges are configured as networks to announce in your router's BGP configuration.
None of the LINX LAN prefix ranges are redistributed, supernetted, or made a more specific outside of your AS.
Each of the Peering LANs can be viewed as a link-local address range and you may decide to not even redistribute it internally. In that case you may want to set a static route for management access so you can troubleshoot peering, etc.
3.2.1 LINX LAN Prefix Ranges
The following is list of LAN prefix ranges used for members to connect to. Members are allotted an IP address from that range when a port, or an aggregation of ports, are ordered and successfully provisioned.
LINX London LON1 - 195.66.224.0/21
LINX London LON2 - 195.66.236.0/22
LINX Manchester - 195.66.244.0/23
LINX NoVa - 206.55.196.0/23
LINX Wales - 195.66.254.0/24
LINX Scotland - 195.66.246.0/24
3.3 BGP Routing
As stated in the LINX MoU, only unicast routes over your BGP sessions in all LINX peering LANs are allowed.
The exchange of multicast routes is not permitted.
3.4 General 10GE Specifics
In order to avoid BGP instability, you should configure your router to ignore events like short link flaps that are typically <20ms. Most vendors implement specific commands to ensure that BGP ignores such events. If your router platform does not support such a feature, we advise you to configure the equivalent of:
no bgp fast-external-falloverwhich should ignore link flaps and wait for the BGP hold timers to expire before resetting sessions.
4. Allowed Traffic Types and Configurations
4.1 MoU Technical Specifications
The technical specifications of the LINX MOU states that only the following three ether types are allowed to be present on any of the LINX exchanges:
0x0800 - IPv4
0x0806 - ARP
0x86dd - IPv6
Further rules apply,
Only one MAC address allowed on any one port or aggregated number of ports, i.e. all frames sent towards the LINX LANs should have only one unique MAC address.
The only non-unicast traffic allowed is:
Broadcast ARP
Multicast ICMPv6 Neighbour Discovery (ND) packets. (NOTE: this does not include Router Advertisement (ND-RA) packets!)
LINX member equipment should only reply to ARP queries for IP addresses of their own directly connected LINX interfaces.
Proxy ARP is not allowed.
Discovery protocols are not allowed.
Traffic for link-local protocols is not allowed, except for ARP and IPv6 ND.
IP packets addressed to LINX peering LAN's directed broadcast address shall not be automatically forwarded to LINX ports.
The LINX platform is designed to carry Ethernet frames with a payload of up to 1500 bytes. MTU settings must be configured accordingly.
4.2 Connecting your Device to the LINX LAN
4.2.1 Connecting a L2 Device
LINX does not encourage members to connect their device to any of its LANs via a L2 device such as a switch. If members do so the following rules must be adhered to help keep the exchange stable, and in good health.
You must make sure that all legitimate traffic to and from your L3 router's interface goes to and from the LINX port.
Spanning-Tree-Protocol (STP) must be disabled on your link towards LINX.
4.2.2 Connecting a L3 Device
LINX recommends and encourages this method as the preferred way of connecting to the LINX LANs directly to a L3 device (router). Only one MAC address will be presented to the exchange and STP will not leak onto the exchange which increases the stability and ongoing health of the exchange.
4.2.3 Connecting a L2/L3 Hybrid
When connecting a L2/L3 Hybrid device to a LINX LAN, members need to keep the following factors in mind:
You must configure your LINX port as a 'router only' port.
Spanning-Tree-Protocol (STP)must be disabled on your link towards LINX.
Consider placing your 'untagged' connected LINX interface in a separate non-default port-based VLAN without STP and with no other ports configured in this VLAN. This will prevent traffic being bridged from other ports onto your LINX port.
5. Vendor Configuration Hints
5.1 Cisco
Cisco default seems to be to enable as many protocols and features as possible so any Cisco device will work straight out of the box in many scenarios.
This means that a lot of unnecessary work and configuration will be required to disable these protocols and features which could be harmful to the stability and health of an exchange. Typical protocols and features that may need to be disabled are:
DHCP
BOOTP
TFTP
CDP
DEC MOP
IP Redirects
IP Directed Broadcasts
Proxy-ARP
Pv6 Router Advertisements
For Cisco switches and hybrid devices the following will also need to be disabled.
VTP
STP
CDP
! Disable DHCP server/relay agent
no service dhcp
! Disable BOOTP - older IOS versions
no ip bootp server
! Do not download configs with TFTP
no service config
! Disable CDP globally
no cdp run5.1.1 Interface Configuration
!Disable IP redirects
no ip redirects
! Disable proxy-arp on LINX interface
no ip proxy-arp
! Discable CDP on LINX interface
no cdp enable
! Disable Directed broadcasts
no ip directed-broadcast
! Disable the DEC
no mop enable
! For (Fast)Ethernet: no auto-negotiation on your connection.
! no negotiation auto
duplex full
! Disable L2 keepalives
no keepalive5.2 Extreme Networks
5.2.1 L2 Interface Configuration
The configuration below shows how to configure an intermediate L2 switch. Port 1 is connected to the LINX switch.
create vlan "LINX"
configure vlan "LINX" tag 1700 # VLAN-ID=0x6a4 Global Tag 9
configure vlan "LINX" add port 1 untagged
#Set 1GE ports for 1GE to LINX
configure port 1 auto off speed 1000 duplex full
#Disable Extreme Discovery Protocol
disable edp port 1
#Disable IGMP Snooping
disable igmp snooping
disable igmp snooping with-proxy5.2.2 L3 Interface Configuration
The configuration below shows configuration information for a L3-only device. As in the previous example, port 1 is connected to LINX and is configured in the 'LINX' VLAN (untagged).
#VLAN LINX and IP Addressing
create vlan "LINX"
configure vlan "LINX" tag 1200
configure vlan "LINX" protocol "IP"
configure vlan "LINX" ipaddress 195.66.x.y 255.255.25X.Y
configure vlan "LINX" add port 1 untagged
#
# Configure IP Route Config
#
configure iproute add blackhole default
disable icmpforwarding vlan "LINX"
disable igmp vlan "LINX"
#
#Add description for display string
configure port 1 display-string "LINX Port"
#Disable Extreme Discovery Protocol
disable edp port 1
#
#Discable ipforwading and others
enable ipforwarding vlan "LINX"
disable ipforwarding broadcast vlan "amsix"
disable ipforwarding fast-direct-broadcast vlan "LINX"
disable ipforwarding ignore-broadcast vlan "LINX"
disable ipforwarding lpm-routing vlan "LINX"
disable isq vlan "LINX"
disable irdp vlan "LINX"
disable icmp unreachable vlan "LINX"
disable icmp redirects vlan "LINX"
disable icmp port-unreachables vlan "LINX"
disable icmp time-exceeded vlan "LINX"
disable icmp parameter-problem vlan "LINX"
disable icmp timestamp vlan "LINX"
disable icmp address-mask vlan "LINX"
disable subvlan-proxy-arp "LINX"
configure ip-mtu 1500 vlan "LINX"5.2.3 Aggregated Ports
We would appreciate feedback from people running Extreme equipment on how they configure their LINX facing side.
5.2.3.1 - LACP
The format to configure LACP in EXOS CLI is as follows.
enable sharing <MasterPort> grouping <PortList> { algorithm [ address-based { L2 | L3 | L3_L4 | custom } | port-based }]} lacpAn example is as follows for LACP (port 1 being the master port and the grouping of ports 1, 2,3 and 5).
enable sharing 1 grouping 1-3,5 algorithm L3_L4 lacpTo change the LACP parameters,
configure sharing <MasterPort> lacp activity-mode [ active | passive ]
configure sharing <MasterPort> lacp timeout [ long | short ]
configure sharing <MasterPort> lacp system-priority <System_Priority>
configure lacp member-port <Port> priority <Port_Priority>5.2.3.2 - Static
To configure a static LAG without LACP the following CLI syntax applies.
configure sharing <MasterPort> [ add | delete ] ports <Ports>An example is as follows with port 1 being the master port and the grouping of ports 1, 2,3 and 5.
configure sharing 1 add ports 1-3,5 5.3 Foundry / Brocade
The configuration provides an example of how to configure a Foundry BigIron device. Depending on the type of software installed on the device the role will need to be set as either a router or switch and may need to be a combination.
! Define single-port VLAN for the LINX-LON1 port
vlan number name "LINX-LON1" by port
#Disable Spanning-Tree
no spanning-tree
untagged ethernet 1/1
! Configure the LINX-LON1 interface
interface ethernet 1/1
port-name "LINX-LON1"
! Set role as a router
route-only
no spanning-tree
! No IPv6 ND-RA (Router Advertisements)
ipv6 nd suppress-ra
! Disable discovery protocol
no vlan-dynamic-discovery
! Configure LINX-LON1 IP address
ip address 195.66.228.212. 255.255.248.0
! No redirects
no ip redirect
no ipv6 redirect
! For fast-ethernet: no autoconfig.
speed-duplex auto
enable5.3.1 Aggregated Ports
5.3.1.1 - LACP
! Interface e1/1 for first in aggregate
interface ethernet 1/1
! Configure key for e1/1 to join aggregate
link-aggregate configure key 10000
! LACP for link aggregation configuration and negotiation
link-aggregate active
!
! Interface e1/2 for first in aggregate
interface ethernet 1/2
! Configure key for e1/2 to join aggregate
link-aggregate configure key 10000
! LACP for link aggregation configuration and negotiation
link-aggregate active5.4. Juniper
Juniper has very useful documentation and hints on how to configure your router. Generally, when configuring a Juniper router there is not much to disable. The configuration examples detailed here are mostly taken from MX devices unless otherwise stated.
The configured MTU on your Juniper device should be 1514, which includes Ethernet headers but not the FCS, or 1518 when tagged.
5.4.1 Enabling Packet Mode for a SRX
If your device is switch such as an SRX there may be some config to apply to allow it to work in packet-mode. The following set commands will need to be applied to a SRX for it to work in packet-mode.
delete security
set security forwarding-options family mpls mode packet-based
set security forwarding-options family iso mode packet-based
set security forwarding-options family inet6 mode packet-basedOnce the above has been applied and committed the SRX device will need to be rebooted.
5.4.2 Configuring single interface
set interfaces ge-0/0/0 description "LINX LON2 Peering"
set interfaces ge-0/0/0 hold-time up 0
set interfaces ge-0/0/0 hold-time down 2000
set interfaces ge-0/0/0 unit 0 family inet address 195.66.238.35/22
set interfaces ge-0/0/0 unit 0 family inet6 address 2001:7f8:4:1::1553:2/645.4.2 Aggregated Ports
5.4.2.1 LACP
set chassis aggregated-devices ethernet device count 1
set interfaces xe-1/2/0 description "LINX-LON1-port1"
set interfaces xe-1/2/0 gigether-options 802.3ad ae200
set interfaces xe-2/2/0 description "LINX-LON1-port2"
set interfaces xe-2/2/0 gigether-options 802.3ad ae200
set interfaces ae200 aggregated-ether-options lacp active
set interfaces ae200 unit 0 description "LINX-LON1"
set interfaces ae200 unit 0 family inet address 195.66.228.212/21
set interfaces ae200 unit 0 family inet6 address 2001:7f8:4::693:1/64
set interfaces ae200 unit 0 filter input LINX-LON1-in
set interfaces ae200 unit 0 filter output LINX-LON1-out5.4.2.2 Static
set chassis aggregated-devices ethernet device count 1
set interfaces xe-1/2/0 description "LINX-LON1-port1"
set interfaces xe-1/2/0 gigether-options 802.3ad ae200
set interfaces xe-2/2/0 description "LINX-LON1-port2"
set interfaces xe-2/2/0 gigether-options 802.3ad ae200
set interfaces ae200 unit 0 description "LINX-LON1"
set interfaces ae200 unit 0 family inet address 195.66.228.212/21
set interfaces ae200 unit 0 family inet6 address 2001:7f8:4::693:1/64
set interfaces ae200 unit 0 filter input LINX-LON1-in
set interfaces ae200 unit 0 filter output LINX-LON1-out5.5 Arista
Arista support documentation on how to configure your device is located on their website. Below are some examples of how to configure a single and aggregated interface for an Arista device.
5.5.1 Physical Interface Configuration
Below is an example if a physical interface on a Arista device is being used to peer with.
Th default ARP raging timeout on a Arista is 4 hours. If you need to change this then please do so.
For connections with 10G or 100G to ignore short link flaps then configure the link-debounce setting in your interface configuration.
interface Ethernet1/1
!! Set description on interface
description LINX-LON1
!! Disable switchport
no switchport
!! Configure IPv4 address
ip address 195.66.228.212/21
!! Configure IPv4 address
ipv6 address 2001:7f8:4::693:1/64
!! Disable ND and RA
ipv6 nd ra disabled
!! Disable LLDP discovery protocol
no lldp transmit
!! Set ARP Aging timeout
arp aging timeout 10800
!!For 10G & 100G Interfaces
link-debounce time 1200
!! Enable interface
no shutdown5.5.2 VLAN Interface Configuration
Below is an example if a VLAN Interface on a Arista device is being used to peer with with a switched port where Spanning-Tree-Protocol needs to be disabled . Some aspects like ARP ageing may need to be set.
interface vlan 200
!! Set description on interface
description LINX-LON1
!! Disable spanning-tree
no spanning-tree
!! Configure IPv4 address
ip address 195.66.228.212/21
!! Configure IPv4 address
ipv6 address 2001:7f8:4::693:1/64
!! Disable ND and RA
ipv6 nd ra disabled
!! Disable LLDP discovery protocol
no lldp transmit
!! Set ARP Aging timeout
arp aging timeout 10800
!! Enable interface
no shutdown5.5.3 Aggregated Ports
5.5.3.1 LACP
interface Ethernet1/1
description LINX-LON1-port 1
channel-group 1 mode active
!!For 10G & 100G Interfaces
link-debounce time 1200
interface Ethernet1/2
description LINX-LON1-port 2
channel-group 1 mode active
!!For 10G & 100G Interfaces
link-debounce time 1200
interface Port-Channel1
description LINX-LON1
ip address 195.66.228.212/21
!! Configure IPv4 address
ipv6 address 2001:7f8:4::693:1/64
!! Disable ND and RA
ipv6 nd ra disabled
!! Disable LLDP discovery protocol
no lldp transmit
!! Set ARP Aging timeout
arp aging timeout 108005.5.3.2 Static
interface Ethernet2/1
description LINX-LON1-port 1
channel-group 200 mode
!!For 10G & 100G Interfaces
link-debounce time 1200
interface Ethernet2/2
description LINX-LON1-port 2
channel-group 200 mode
!!For 10G & 100G Interfaces
link-debounce time 1200
interface Port-Channel200
description LINX-LON1
ip address 195.66.228.212/21
!! Configure IPv4 address
ipv6 address 2001:7f8:4::693:1/64
!! Disable ND and RA
ipv6 nd ra disabled
!! Disable LLDP discovery protocol
no lldp transmit
!! Set ARP Aging timeout
arp aging timeout 108005.6 Nokia
Classic CLI Configuration
A:ALA-A> config# info
#------------------------------------------
# Router Configuration
#------------------------------------------
router
interface "system"
address 10.10.10.103/32
exit
iinterface "LINX-LON1"
address 195.66.228.212/21
port 1/1/1
no shutdown
exit
autonomous-system 12341
#------------------------------------------
A:ALA-A> config#