Overview
Normally on a peering exchange, all connected parties will establish bilateral peering relationships with each other member port connected to the exchange. As the number of connected parties increases, it becomes increasingly more difficult to manage peering relationships with members of the exchange. A typical peering exchange full-mesh eBGP configuration might look something similar to the diagram on the left hand side.
![[ IXP full mesh peering relationships ]](images/route-server-peering-fullmesh.png "IXP full mesh peering relationships")
|
![[ IXP route server peering relationships ]](images/route-server-peering-rsonly.png "IXP route server peering relationships")
|
|
| IXP full mesh peering relationships | IXP route server peering relationships |
The full-mesh BGP session relationship scenario requires that each BGP speaker configure and manage BGP sessions to every other BGP speaker on the exchange. In this example, a full-mesh setup requires 7 BGP sessions per member router, and this increases every time a new member connects to the exchange.
However, by using a route server for all peering relationships, the number of BGP sessions per router stays at two: one for each route server. Clearly this is a more sustainable way of maintaining IXP peering relationships with a large number of participants.
Should I use this service?
The BCIX route server cluster is aimed at:
- small to medium sized members of the exchange who don’t have the time or resources to aggressively manage their peering relationships
- larger members of the exchange who have an open peering policy, but where it may not be worth their while managing peering relationships with smaller members of the exchange.
As a rule of thumb: If you do not have any good reason not to use the route server cluster, you should probably use it.
The service is designed to be reliable. It operates on two physical servers, each located in a different data centre. The service is available for both IPv4 and IPv6. Each server runs a separate routing daemon per VLAN and per L3 protocol. Should a single BGP server die for some unlikely reason, no other BGP server is likely to be affected. If one of the physical servers becomes unavailable, the second server will continue to provide BGP connectivity.
BCIX has also implemented inbound prefix filtering on its route-server cluster. This uses both RPKI and internet routing registry data from the various IRR databases to allow connected members to announce only the address prefixes which they have registered publicly. If your prefix has a valid RPKI ROA, it will pass. If the RPKI ROA check result is unknown (you have not set up a ROA), we fall back to the IRRDB test.
BCIX uses BIRD2 (Bird Internet Routing Demon) running on Centos 7 for our route server cluster. BIRD is widely used at internet exchanges for route server clusters, and has been found to be reliable in production.
How do I use the service?
If enabled, the route servers are set up to accept BGP connections from your router. Once this has been done, you will need to configure a BGP peering session to the correct internet address. The IP addresses of the route servers are listed as follows:
| Route Server | IPv4 Address | IPv6 Address |
|---|---|---|
| Route Server #1 | 193.178.185.5 | 2001:7f8:19:1::3ff6:5 |
| Route Server #2 | 193.178.185.6 | 2001:7f8:19:1::3ff6:6 |
Maximum Prefix Recommendations
We recommend setting the max-prefix limit of the BGP sessions to our route servers to
| IPv4 | IPv6 |
|---|---|
| 160000 | 100000 |
Example Configs
Cisco IOS
router bgp 99999 no bgp enforce-first-as ! ! Route Server #1 ! neighbor 193.178.185.5 remote-as 16374 neighbor 193.178.185.5 description BCIX Route Server 1 neighbor 193.178.185.5 password PASSWORD ! address-family ipv4 neighbor 193.178.185.5 activate neighbor 193.178.185.5 route-map BCIX-IN-v4 in neighbor 193.178.185.5 route-map BCIX-OUT-v4 out neighbor 193.178.185.5 maximum-prefix 160000 exit-address-family ! neighbor 2001:7f8:19:1::3ff6:5 remote-as 16374 neighbor 2001:7f8:19:1::3ff6:5 description BCIX Route Server 1 neighbor 2001:7f8:19:1::3ff6:5 password PASSWORD ! address-family ipv4 no neighbor 2001:7f8:19:1::3ff6:5 activate exit-address-family ! address-family ipv6 neighbor 2001:7f8:19:1::3ff6:5 activate neighbor 2001:7f8:19:1::3ff6:5 route-map BCIX-IN-v6 in neighbor 2001:7f8:19:1::3ff6:5 route-map BCIX-OUT-v6 out neighbor 2001:7f8:19:1::3ff6:5 maximum-prefix 100000 exit-address-family ! ! Route Server #2 ! neighbor 193.178.185.6 remote-as 16374 neighbor 193.178.185.6 description BCIX Route Server 2 neighbor 193.178.185.6 password PASSWORD ! address-family ipv4 neighbor 193.178.185.6 activate neighbor 193.178.185.6 route-map BCIX-IN-v4 in neighbor 193.178.185.6 route-map BCIX-OUT-v4 out neighbor 193.178.185.6 maximum-prefix 160000 exit-address-family ! neighbor 2001:7f8:19:1::3ff6:6 remote-as 16374 neighbor 2001:7f8:19:1::3ff6:6 description BCIX Route Server 2 neighbor 2001:7f8:19:1::3ff6:6 password PASSWORD ! address-family ipv4 no neighbor 2001:7f8:19:1::3ff6:6 activate exit-address-family ! address-family ipv6 neighbor 2001:7f8:19:1::3ff6:6 activate neighbor 2001:7f8:19:1::3ff6:6 route-map BCIX-IN-v6 in neighbor 2001:7f8:19:1::3ff6:6 route-map BCIX-OUT-v6 out neighbor 2001:7f8:19:1::3ff6:6 maximum-prefix 100000 exit-address-family ! !
Cisco IOS XR
router bgp 99999 bgp enforce-first-as disable ! ! Route Server #1 ! neighbor 193.178.185.5 remote-as 16374 password encrypted PASSWORD description BCIX Route Server 1 address-family ipv4 unicast maximum-prefix 160000 75 route-policy BCIX-IN-v4 in route-policy BCIX-OUT-v4 out ! ! neighbor 2001:7f8:19:1::3ff6:5 remote-as 16374 password encrypted PASSWORD description BCIX Route Server 1 address-family ipv6 unicast maximum-prefix 100000 75 route-policy BCIX-IN-v6 in route-policy BCIX-OUT-v6 out ! ! ! Route Server #2 ! neighbor 193.178.185.6 remote-as 16374 password encrypted PASSWORD description BCIX Route Server 2 address-family ipv4 unicast maximum-prefix 160000 75 route-policy BCIX-IN-v4 in route-policy BCIX-OUT-v4 out ! ! neighbor 2001:7f8:19:1::3ff6:6 remote-as 16374 password encrypted PASSWORD description BCIX Route Server 2 address-family ipv6 unicast maximum-prefix 100000 75 route-policy BCIX-IN-v6 in route-policy BCIX-OUT-v6 out ! !
Juniper JunOS
protocols {
bgp {
group AS16374-BCIX-RS1 {
type external;
description "BCIX Route Server 1";
remove-private;
peer-as 16374;
neighbor 193.178.185.5 {
import BCIX-IN-v4;
family inet {
unicast {
prefix-limit {
maximum 160000;
teardown 75 idle-timeout 60;
}
}
}
authentication-key PASSWORD;
export BCIX-OUT-v4;
}
neighbor 2001:7f8:19:1::3ff6:5 {
import BCIX-IN-v6;
family inet6 {
unicast {
prefix-limit {
maximum 100000;
teardown 75 idle-timeout 60;
}
}
}
authentication-key PASSWORD;
export BCIX-OUT-v6;
}
}
group AS16374-BCIX-RS2 {
type external;
description "BCIX Route Server 2";
remove-private;
peer-as 16374;
neighbor 193.178.185.6 {
import BCIX-IN-v4;
family inet {
unicast {
prefix-limit {
maximum 160000;
teardown 75 idle-timeout 60;
}
}
}
authentication-key PASSWORD;
export BCIX-OUT-v4;
}
neighbor 2001:7f8:19:1::3ff6:6 {
import BCIX-IN-v6;
family inet6 {
unicast {
prefix-limit {
maximum 100000;
teardown 75 idle-timeout 60;
}
}
}
authentication-key PASSWORD;
export BCIX-OUT-v6;
}
}
}
}
You should by all means use filtering to control incoming and outgoing prefix announcements to allow only prefixes which you intend to announce and receive.
Note that the route server system depends on information in either RPKI or the various IRR databases (typically the RIPE IRR database for
BCIX
members). If you have not published either a valid ROAs or correct route: and route6: objects in this database, your prefix announcements will be ignored by the route server and your peers will not route their traffic to you via the exchange. This is checked by
BCIX
Operations during provisioning and guidance and assistance is provided as required.
AS-SET
If you do IRRDB based filtering, have look at our AS-SET object AS16374:AS-BCIX-RS in the RIPE DB.
Community based prefix filtering
The BCIX route server system also provides well known communities to allow members to control the distribution of their prefixes. These communities are defined as follows:
| Description | Community |
|---|---|
| Prevent announcement of a prefix to a peer | 0:peer-as |
| Announce a route to a certain peer | 16374:peer-as |
| Prevent announcement of a prefix to all peers | 0:16374 |
| Announce a route to all peers | 16374:16374 |
So, for example, to instruct the route server to distribute a particular prefix only to
AS64111 and AS64222, the prefix should be tagged with communities: 0:16374, 16374:64111
and 16374:64222.
Alternatively, to announce a prefix to all BCIX members, excluding AS64333, the prefix
should be tagged with community 0:64333.
The BCIX route server cluster supports BGP large community prefix distribution control, using the following policy:
| Description | Community |
|---|---|
| Prevent announcement of a prefix to a peer | 16374:0:peer-as |
| Announce a route to a certain peer | 16374:1:peer-as |
| Prevent announcement of a prefix to all peers | 16374:0:0 |
| Announce a route to all peers | 16374:1:0 |
The BCIX route server cluster also supports BGP large community AS path prepending control, using the following policy:
| Description | Community |
|---|---|
| Prepend to peer AS once | 16374:101:peer-as |
| Prepend to peer AS twice | 16374:102:peer-as |
| Prepend to peer AS three times | 16374:103:peer-as |
Graceful BGP Shutdown
The BCIX route servers support the well-known Graceful BGP Session Shutdown
Community (RFC
8326). By adding the BGP community 65535:0 to your routes you
may indicate that the path shall only be used with lowest priority, i.e. in
case there is no other path available.
The route servers will forward the community transparently so that all peers can handle it accordingly.
You should add this community before starting any maintenance to instruct your peers to use other routes, before shutting down your router or connectivity. If your equipment does not yet handle receiving the graceful shutdown community by default, you should implement this by configuration as described in the BGP filter guide.
Looking Glass
We operate an Alice Looking Glass for you to check our view of your announcements.
If a route servers is filtering a prefix, it will show in the looking glass along with a reason why the prefix is filtered.
Filtering Policy
Our Route Server filtering policy is defined in the source code for IXP Manager on github. This is a summary of what it does.
- Drop small prefixes – longer than /24 for IPv4 and longer than /48 for IPv6.
- Drop all well-known martians and bogons.
- Ensure that there is at least 1 ASN and less than 64 ASNs in the AS path.
- Ensure that the peer AS is the same as the first AS in the AS path.
- Drop any prefix where the next-hop IP address is not the same as the peer IP address. This prevents prefix hijacking.
- Drop any prefix with a transit network ASN in the AS path.
- Ensure that origin AS is in set of ASNs from the client’s IRRDB AS-SET.
- If the prefix is evaluated as RPKI valid, accept.
- If the prefix is evaluated as RPKI invalid, drop.
- If the prefix is evaluated as RPKI unknown, revert to standard IRRDB prefix filtering.
BGP ADD-PATH capability
The route servers support the BGP ADD-PATH capability (RFC 7911) in send mode. Neighbors negotiating it will receive all the paths available for a route. Using the additional information, peers may implement their own routing policies more effectively.