Understanding AWS Direct Connect BGP Communities

Let's start with a little memory refresher… AWS Direct Connect is a service that enables you to establish a dedicated network connection from your on-premises data centers to AWS. A key component of this service is the use of Border Gateway Protocol (BGP) communities, which play a vital role in network routing and traffic management. I had many conversation lately with multiple customers who wanted to limit their scope of advertisements when it comes to Direct Connect with Public Virtual Interface (Public VIF) and/or how to perform routing engineering having multiple Direct Connect locations, so here I am writing a short quick guide about these functionalities. We are going to talk about BGP Communities, how to use them, how to manipulate routing to/from AWS/on-prem.

Here the Lab diagram for today topic:

Image not found

What is a Virtual Interface (VIF)?

First of all, just because I mentioned the Virtual Interfaces, I think it would be good to provide the reader with a brief intro about what they are and how they differ. AWS Direct Connect offers three types of Virtual Interfaces (VIFs): Public VIF, Private VIF, and Transit VIF, each serving distinct use cases and offering different capabilities.

Public VIF: A Public Virtual Interface allows access to AWS public services such as Amazon S3, Amazon DynamoDB, and other AWS services with public endpoints. When you use a Public VIF, you advertise your public IP prefixes to AWS and receive public IP prefixes for AWS services. This setup is typically used when you need to reach AWS services directly without routing through the internet, providing a more secure and constant/lower-latency connection to public AWS endpoints.

Private VIF: A Private Virtual Interface connects directly to your Virtual Private Cloud (VPC) resources over a private IP space. This type of VIF is ideal for accessing EC2 instances, RDS databases, and other resources within your VPC. Private VIFs enable private, secure communication between your on-premises data center and your AWS environment, bypassing the public internet entirely, which is crucial for sensitive or high-availability applications.

Transit VIF: A Transit Virtual Interface connects to an AWS Transit Gateway, which allows you to interconnect multiple VPCs and on-premises networks via a single interface. This setup is particularly useful for complex network architectures where multiple VPCs need to communicate with each other or with on-premises networks through a “centralised gateway”. The Transit VIF simplifies network management and reduces the number of required interfaces, promoting a hub-and-spoke model for routing. Additionally providing, through the transit gateway, network segmentation capabilities.

Each VIF type is designed to optimize different networking needs, from direct access to public AWS services, private VPC connectivity, to complex multi-VPC and on-premises network integration. I will not dive deep into more details as this post is not intended to go through Direct Connect specifics. Let’s start talking about BGP Communities…

Introduction to BGP Communities

BGP communities are tags that you can assign to your IP prefixes. These tags help in controlling routing policies and making routing decisions. In general BGP, these communities influence how routes are propagated, preferred, or filtered across different networks. They provide a mechanism for grouping prefixes to apply specific routing policies easily.

BGP Communities in AWS Direct Connect

In the context of AWS Direct Connect, BGP communities are used to manage the traffic flow between your on-premises network and AWS over multiple Direct Connect locations. Additionally, AWS provides specific community tags that help in controlling the scope and preference of routes advertised/learned from or to on-prem and AWS.

Manipulating Traffic in Multiple DX Locations (Private VIF/Transit VIF)

When you have multiple Direct Connect (DX) locations, you can use BGP community tags to influence routing preferences. AWS supports various BGP community tags:

7224:7100: LOW preference
7224:7200: MEDIUM preference
7224:7300: HIGH preference

By assigning these tags to your prefixes, you can control which DX connection AWS prefers for incoming and outgoing traffic.

Traffic Management from AWS to On-Premis

Using Private VIF or Transit VIF

For traffic from AWS to your on-premises network using Private Virtual Interfaces (Private VIF) or Transit Virtual Interfaces (Transit VIF), you can manipulate routing paths using BGP attributes such as BGP Communities, AS_PATH, and MED (Multi-Exit Discriminator). Here’s how you can achieve primary vs. secondary path manipulation:

BGP Communities and Local Preference: Set a higher preference (e.g., 7224:7300) on the primary connection and a lower one (e.g., 7224:7100) on the secondary connection to prefer the primary path.
AS_PATH Prepending: Increase the AS_PATH length on the secondary connection to make it less preferred compared to the primary connection.
MED Values: Set lower MED values on the primary path if local preference and AS_PATH are the same.

Usually, you would want to use BGP Communities when you have multiple locations and AS_Path Prepending within the same location in case you have multiple links.

These techniques ensure that AWS prefers your primary Direct Connect location for most traffic while falling back to secondary connections if the primary one fails.

Using Public VIF - Inbound Routing Policies

When dealing with public virtual interfaces (Public VIF), AWS provides specific BGP community tags for controlling the scope of prefix advertisements:

7224:9100: Local AWS Region
7224:9200: All AWS Regions on the same continent
7224:9300: Global (all public AWS Regions)

By applying these tags to your advertised prefixes, you can control the propagation scope within the AWS network. Additionally, you can filter prefixes received from AWS to ensure only the necessary routes are accepted. AWS also tags all its advertised routes with the NO_EXPORT community, meaning these routes should not be advertised outside your network.

Using Public VIF - Outbound Routing Policies

AWS Direct Connect allows you to specify which prefixes are advertised from AWS to your network by using BGP communities and local preference.

7224:8100: For routes originating from the same AWS Region as the Direct Connect location
7224:8200: For routes from the same continent
NO_TAG: For routes originating from other continents

Additionally, to manipulate the routing of outbound traffic, you can use these communities in conjunction with local preference settings. For example, assigning a higher local preference value to prefixes from a preferred Direct Connect location ensures that traffic flows through that location by default. This is valid if you have a single device on-prem. If you want to manipulate the routing preferences in case of multiple locations, AWS Public VIF supports “longest prefix match” preferring the advertisement with the most specific prefix for example:

192.168.0.0/24 - Preferred
192.168.0.0/16 - NOT Preferred

NO_EXPORT BGP Community

The NO_EXPORT BGP community tag indicates that the tagged route should not be advertised beyond the local AS (Autonomous System) boundary. This is useful in preventing route leaks and ensuring that specific routes are contained within your network.

Practical Example Implementation (Private + Transit VIF)

Consider a scenario where you have two Direct Connect locations (DX1 and DX2) and you want to set up an active/passive configuration:

Configure BGP Communities:

For DX1 (primary): Apply 7224:7300 to set a high preference.
For DX2 (secondary): Apply 7224:7100 to set a low preference.

AS_PATH Prepending:

On DX2, prepend the AS_PATH to make it less preferred.

MED Configuration:

Set a lower MED value on DX1 to ensure it’s preferred when all other attributes are equal.

Route Filters:

On your on-premises BGP routers, filter received AWS routes to ensure only the necessary prefixes are accepted, applying the NO_EXPORT tag as needed

CISCO config example (single service config):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
		! Define the BGP neighbor and basic BGP configuration
		router bgp 65000
		 bgp log-neighbor-changes
		
		! Configure first AWS Direct Connect link (primary)
		 neighbor 192.0.2.1 remote-as 7224
		 neighbor 192.0.2.1 description AWS Direct Connect Primary
		 neighbor 192.0.2.1 password 7 <Your_BGP_Password>
		 neighbor 192.0.2.1 route-map SET_HIGH_PREFERENCE out
		 neighbor 192.0.2.1 send-community both
		
		! Configure second AWS Direct Connect link (secondary)
		 neighbor 198.51.100.1 remote-as 7224
		 neighbor 198.51.100.1 description AWS Direct Connect Secondary
		 neighbor 198.51.100.1 password 7 <Your_BGP_Password>
		 neighbor 198.51.100.1 route-map SET_LOW_PREFERENCE out
		 neighbor 198.51.100.1 send-community both
		
		! Define address family for IPv4
		 address-family ipv4
		  neighbor 192.0.2.1 activate
		  neighbor 198.51.100.1 activate
		 exit-address-family
				
		! Define prefix list for advertised routes
		ip prefix-list BGP-ADVERTISED-PREFIXES seq 5 permit 192.168.0.0/16
			
		! Define route-map to set high preference for primary link
		route-map SET_HIGH_PREFERENCE permit 10
		 match ip address prefix-list BGP-ADVERTISED-PREFIXES
		 set community 7224:7300
         
		! Define route-map to set low preference for secondary link
		route-map SET_LOW_PREFERENCE permit 10
		 match ip address prefix-list BGP-ADVERTISED-PREFIXES
		 set community 7224:7100
		
		! Apply route-maps to BGP neighbors
		router bgp 65000
		 address-family ipv4
		  neighbor 192.0.2.1 route-map SET_HIGH_PREFERENCE ou
		  neighbor 198.51.100.1 route-map SET_LOW_PREFERENCE out
		 exit-address-famil

Explanation

BGP Neighbor Configuration:

192.0.2.1 is configured as the primary AWS Direct Connect link.
198.51.100.1 is configured as the secondary AWS Direct Connect link.

Route-Maps:

SET_HIGH_PREFERENCE: This route-map sets the community 7224:7300 for routes advertised to the primary link, giving it higher precedence.
SET_LOW_PREFERENCE: This route-map sets the community 7224:7100 for routes advertised to the secondary link, giving it lower precedence.

Prefix List:

BGP-ADVERTISED-PREFIXES: This prefix list specifies which routes are advertised to AWS. Here it permits the prefix 192.168.0.0/16.

Address Family Activation:

Both neighbors are activated under the IPv4 address family.

Applying Route-Maps:

The route-maps SET_HIGH_PREFERENCE and SET_LOW_PREFERENCE are applied to the respective BGP neighbors to set the desired preferences for the primary and secondary links.

This simple configuration ensures that the primary link (192.0.2.1) is preferred over the secondary link (198.51.100.1) by assigning a higher preference community to it. Adjust IP addresses and AS numbers as per your actual configuration needs. Please note that this is strictly a test environment so apply proper judgement when configuring production environments.

Conclusion

BGP communities in AWS Direct Connect provide a powerful tool for controlling and optimizing your network traffic. By understanding and effectively using these communities, you can achieve greater control over routing decisions, ensuring optimal performance and reliability for your connections to AWS. As usual, thank you for your time, hope this was useful, see you soon

Any opinions in this post are those of the individual author and may not reflect the opinions of AWS.

Select your cookie preferences

Site Terms, Privacy, and more.

Understanding AWS Direct Connect BGP Communities

1 Comment