General description
An example of implementing a fault-tolerant network with ESR redundancy and last-mile PE router redundancy is shown in Fig. 1. Access points placed at the client are included in PoE switches, which allow traffic to be forwarded up to L3 switches/routers in the operator's access network.
- The entire access network, to enable AP, is combined in a VRF AP.
- Exit to the Internet is done through the VRF NAT, in which the NAT is located.
- Control network through VRF core allows to organize the connectivity between ESR and SoftWLC.
- ESRs are included in different PE routers (last mile) to provide physical redundancy of elements.
- ESRs are interconnected to exchange service information about the list and status of GRE tunnels.
Thus, if any last-mile PE router or ESR goes down, or if physical or logical connectivity is lost, the system continues to provide a range of services with minimal interruption of communication.
Fig. 1 Use case.
Modes of GRE tunnel creation for AP in Wireless-Controller
In the L3 enable scheme, when a AP raises two GRE tunnels (mgmt and data) using the data obtained from the 43rd option, the ESR side also needs to raise the tunnels for the APs. In order not to manually set up tunnels for each AP, ESR has implemented a mode of operation that allows to perform this procedure automatically. За данный функционал возложен на wireless-controller. The wireless-controller is responsible for this functionality. To get access to the wireless-controller settings you need to install the "WIFI - Wi-Fi controller" license on ESR. Using this functionality allows you to automate the management of tunnels raised on the ESR towards the AP. For this purpose it is enough to configure once the default profiles for tunnels raising and wireless-controller functionality in accordance with the pre-selected addressing plan and vlan. The automatically created GRE tunnels on ESR will be further referred to as "SoftGRE".
There are two modes of tunnel creation:
- Creation using local ESR configuration profile – this mode is called "Local configuration profile of SoftGRE tunnel". Tunnels will be raised for each AP according to the local ESR configuration.
Creation of data tunnels using radius exchange with PCRF, which receives information about which data tunnels should be raised according to AP position in EMS tree and its SSID settings – this mode is called "Dynamic configuration profile of SoftGRE tunnel". Tunnels received from PCRF will be raised for each AP.
The number of tunnels per ESR is limited, it is preferable to use the dynamic "Dynamic configuration profile of SoftGRE tunnel" as only the required tunnels will be raised.
WiFi license is required to access the wireless-controller ESR functionality. You can check if you have a WiFi license by using the show license command:
esr-1000# show licence Licence information ------------------- Name: eltex Version: 1.0 Type: ESR-1000 S/N: NP00000033 MAC: A8:F9:4B:AB:B3:80 Features: WIFI - Wi-Fi controller
Algorithm of creating tunnels on ESR with local configuration profile of SoftGRE tunnel
This mode requires a configured customized local profile on ESR:
tunnel softgre 1 mode management local address <IP address> default-profile enable exit tunnel softgre 1.1 bridge-group <№> enable exit tunnel softgre 2 mode data local address <IP address> default-profile enable exit tunnel softgre 2.10 bridge-group <№> enable exit tunnel softgre 2.11 bridge-group <№> enable exit wireless-controller enable exit
Fig. 2 shows the diagram of tunneling rise on ESR when AP is connected.
Fig. 2. Diagram of tunnels rise on ESR when connecting AP, using local configuration profile pf SoftGRE tunnel.
Step 1. AP gets the primary address with 43 option containing GRE termination addresses (11 – mgmt, 12 – data), which are on ESR.
Step 2. AP raises 2 GRE tunnels from its primary address to GRE termination addresses. Inside the mgmt GRE tunnel, it sends a DHCP-discover for the management address in the 1st VLAN.
Using default profile settings, ESR receiving such a packet (GRE, with DHCP-discover encapsulated inside) raises a counter SoftGRE control tunnel (mgmt) with a sub-tunnel with the 1st VLAN, which is terminated in a bridge acting as a gateway for AP control addresses.
The decapsulated DHCP-discover is sent to the DHCP server via relay, and in response a DHCP-offer is sent, which ESR packs into SoftGRE and sends to the AP.
The AP sends a DHCP-request, which is also sent to the DHCP server via relay.
Step 3. After the DHCP-ack comes from the DHCP server and is sent to the AP, ESR will raise data SoftGRE tunnels for this AP using the local profile settings for data tunnels.
The advantage of this method is the simplicity of ESR configuration and simple logic of tunnel raising, which does not depend on SoftWLC settings. The disadvantage of this method is that for each connecting AP, all data sub-tunnels specified in the profile settings are raised, even if no traffic is transferred from the AP side. This leads to uneconomical consumption of sub-tunnel capacity on ESR.
Algorithm of creating tunnels on ESR with dynamic configuration profile of SoftGRE tunnel
In this mode, data tunnels for APs on ESR side are raised by ELTEX-PCRF (hereinafter referred to as PCRF) commands, which receives necessary information from SoftWLC complex database depending on SSID settings, their bindings and AP position in the object tree. Message exchange is performed via RADIUS protocol, UDP ports 3799, 31812, 31813 are used (if PCRF default settings are used). Access to the server with ELTEX-PCRF is required for correct operation. Below is an example of the minimum required settings:
object-group network SoftWLC ip address <address of the server with ELTEX-PCRF of SoftWLC complex> exit radius-server host <address of the server with ELTEX-PCRF> key ascii-text <password> auth-port 31812 acct-port 31813 exit aaa radius-profile PCRF radius-server host <address of the server with ELTEX-PCRF> exit das-server COA key ascii-text <password to be used when accessing ESR from ELTEX-PCRF side> port 3799 clients object-group SoftWLC exit aaa das-profile COA das-server COA exit tunnel softgre 1 mode management local address <IP address> default-profile enable exit tunnel softgre 1.1 bridge-group <№> enable exit tunnel softgre 2 mode data local address <IP address> default-profile enable exit wireless-controller data-tunnel configuration radius aaa das-profile COA aaa radius-profile PCRF enable exit bridge 10 location <имя> enable exit
In case of using the dynamic configuration profile of SoftGRE tunnels, there are 7 possible scripts in which the tunnels will be formed.
They can be categorized into two groups – ESR-initiated data tunnel creation (when connecting a AP) and EMS-initiated data tunnel creation. The scripts provide both positive results if all the parameters for data tunnel creation are passed correctly and negative results if there are incorrect or missing settings:
ESR-initiated data tunnel creation:
1. AP is building tunnels on the ESR, is not initialized and is sandboxed or connecting for the first time. No data tunnel will be generated in this case.
2. AP is tunneling on ESR, initialized and is in the domain tree in EMS, SSID settings and bindings to the domain are correct. Data tunnel and corresponding sub tunnels will be generated.
3. AP builds tunnels on ESR, initialized and in the EMS tree, SSID parameters contain incorrect settings. A data tunnel or some data sab tunnels will not be generated.
EMS-initiated data tunnel creation:
4. The data-tunnel create command from EMS is executed, AP with the specified tunnel address is not found on the ESR. No data tunnel will be generated.
5. The data-tunnel create command from EMS is executed, AP with the specified tunnel address is found on the ESR. The data tunnel and corresponding sub-tunnels will be generated.
6. The data-tunnel create command from EMS is executed, AP is found in the EMS tree, the SSID parameters contain incorrect settings. No data tunnel or some data sub tunnels will be generated.
7. The ESR that receives the data-tunnel create command is in the VRRP BACKUP state. The device forms all tunnels according to the information transmitted from the ESR that is in the VRRP MASTER state.
All the scripts of data tunnel creation will be discussed in detail below.
ESR-initiated data tunnel creation:
When executing these scenarios, the initiator of interaction with PCRF is the ESR to which the AP is connected and it needs to get all the data to build the data tunnel and its sub-tunnels. It sends a radius request to the PCRF of type ip-shaper, which contains the tunnel address of the AP and the NAS-IP of the ESR. Then PCRF searches for the AP by tunnel IP, determines its domain and searches for SSID bindings to this domain and domains above in the tree (the search goes from the AP upwards until the ESR to which this AP is connected is found), after which it generates a response containing parameters:
- office:name – the domain to which the SSID is assigned, located close to the TD
- vlan:id – number of vlan, which is configured in SSID parameters (parameter "VLAN-ID")
- name – Bridge, Location configured in SSID parameters (parameter "Bridge, Location")
- sc – sign of Switch Community(sc), configured in SSID parameters. (If the option is enabled, the sc=1 parameter will be added to the packet, if the option is disabled, the packet will not contain the sc parameter).
- gre-location – sign of data sub-tunnel allowing to uniquely associate it with a certain AP, has the form NODE. <Object index>.<Vlan number> (e.g NODE.8498.10). The parameter "Object index" is unique for each TD and is set during adding TD to the object tree (both during initialization and manual addition). Gre-location is necessary for correct operation of BRAS, it acts as an L2 interface.
Parameters 2-5 are transmitted for each found SSID binding, there can be several of them within one radius PCRF response for one AP (but not more than 20 pcs).
The "Bridge Location" parameter must coincide with the "location" setting on one of the ESR bridges, because it determines which ESR bridge the data sub-tunnel will be built in.
The "vlan:id" parameter is necessary, because without it, it is impossible to form a data sub-tunnel with the corresponding sub-interface number.
Script 1 - AP is not initialized and is sandboxed or connecting for the first time
AP is not initialized and is in the sandbox or connecting for the first time - this situation usually occurs for newly connected APs or if the AP has been removed from the EMS tree.
Fig. 3 shows a diagram of the logic of interaction with PCRF:
Fig. 3.
Step 1. AP кeceives a primary address with 43 options containing GRE termination addresses (11 – mgmt, 12 – data) that are on the ESR.
Step 2. AP raises 2 GRE tunnels from its primary address to GRE termination addresses. Inside the mgmt GRE tunnel, it sends a DHCP-discover to get the management address in the 1st VLAN.
Using default profile settings, ESR receiving such a packet (GRE, with DHCP-discover encapsulated inside) raises a counter SoftGRE control tunnel (mgmt) with a sub-tunnel with the 1st VLAN, which is terminated in a bridge acting as a gateway for AP control addresses.
The decapsulated DHCP-discover is relayed to the DHCP server, which sends a DHCP-offer, which the ESR packs into SoftGRE and sends to the AP.
The AP sends a DHCP-request, which will be sent to the DHCP server using relay.
Step 3. After the DHCP-ack arrives from the DHCP server and is sent to the AP:
a) ESR will make a radius request of type ip-shaper to PCRF containing the tunnel (primary) address of the AP and the NAS-IP of ESR;
b) PCRF will access the SotWLC complex database to get the necessary data;
c) After that, AP will be searched by its tunnel (primary) address – the address will not be found;
d) PCRF, having determined that there is no data about its tunnel address for this AP in the database, will send a radius access-reject response to the ESR;
Step 4: The data tunnel for the AP will not be raised on the ESR.
As logic of work shows, in this case the data-tunnel for AP is not formed on ESR. Information about it is not transmitted to EMS.
Script 2 - AP is initialized and is in the domain tree in EMS, SSID settings and bindings to the domain are correct
AP is initialized and is in the domain tree in EMS, SSID settings and bindings to the domain are correct - this situation corresponds to normal operation of AP after it was initialized earlier and its reconnection occurs after communication is restored or tunnels are destroyed on ESR.
Fig. 4 shows a the diagram of the logic of data tunnels raising:
Fig. 4. Raising data tunnels using PCRF interaction.
Step 1. AP receives a primary address with 43 options containing GRE termination addresses (11 - mgmt, 12 - data) that are on the ESR.
Step 2. The AP raises 2 GRE tunnels from its primary address to GRE termination addresses. Inside the mgmt GRE tunnel, it sends a DHCP-discover for the management address in the 1st VLAN.
Using default profile settings, ESR receiving such a packet (GRE, with DHCP-discover encapsulated inside) raises a counter SoftGRE control tunnel (mgmt) with a sub-tunnel with the 1st VLAN, which is terminated in a bridge acting as a gateway for AP control addresses.
The decapsulated DHCP-discover is relayed to the DHCP server, which sends a DHCP-offer, which the ESR packs into SoftGRE and sends to the AP.
The AP sends a DHCP-request, which will be sent to the DHCP server using relay.
Step 3. After the DHCP-ack arrives from the DHCP server and is sent to the AP:
a) ESR will make a radius request of type ip-shaper to PCRF containing the tunnel (primary) address of the AP and the NAS-IP of ESR;
b) PCRF will access the SotWLC complex database to get the necessary data;
c) After that, AP will be searched by its tunnel (primary) address and all the necessary data for building a data tunnel will be obtained;
d) Having received the necessary data, PCRF will send an access-accept response to radius, containing all the necessary parameters to form a data tunnel and sub-tunnels to ESR;
e) ESR, using the received data, forms a data tunnel and sub-tunnels.
Step 4: The data tunnel for the AP will not be raised on the ESR.
As logic of work shows, in this case the data-tunnel and sub-tunnels for AP is formed on ESR. Information about it is not transmitted to EMS.
Script 3 - AP builds tunnels on ESR, initialized and in the EMS tree, SSID parameters contain incorrect settings.
AP is nitialized and in the EMS tree, SSID parameters contain incorrect settings.
A misconfiguration is defined as a mismatch between the value of "Bridge Location" parameter and any such value configured on the ESR bridge by the location command.
If "VLAN-ID" parameter is not specified - parameters for such a data-sub tunnel are not transmitted to ESR and it is not formed. This is not considered an error, because such SSID setting is allowed.
If SSID binding is performed not to the domain but to the AP mac-address - parameters for such a data-sab tunnel are not passed to the ESR and sub tunnel is not generated. This is not considered an error, because this method of SSID binding is allowed.
Fig. 5 shows a diagram of the raising data tunnels when the "Bridge, Location" parameter is incorrect.
Fig. 5.
As a result of specifying an incorrect "Bridge, Location" data sub-tunnel with these parameters will not be able to be formed on ESR. In the ESR log, if APTD ERROR log is enabled (enabled by default), a message will appear:
%APTD-E-ERROR: <aptd_validate_access_data> AP <Tunnel IP AP>: SSID '<Bridge Location setup value>' not found in locations!
As a result data the sub-tunnel will not be formed.
Starting from ESR software version 1.11.0, if the response to the ip-shaper request contains parameters for raising several data sub-tunnels - then in case of an error in the "Bridge Location" setting, only this sub-tunnel will not be formed. On earlier versions, in case of an error, all data sub tunnels will not be formed.
EMS-initiated data tunnel creation:
Since in case of ESR-initiated creation of a data tunnel it may not always be formed, e.g. for a newly connected AP - there are scenarios when the command to create a data tunnel on the ESR is initiated by the EMS. This occurs in the following cases:
1. initialization of a new AP.
2. Re-initialization of the AP.
3. Moving the AP to another node;
4. Executing the "Create data-tunnel on ESR" command in EMS for one or a group of APs.
When creating a data tunnel at the initiative of EMS, PCRF executes a radius CoA-request ("data-tunnel create") on UDP port 3799 of ESR. In this case it is always necessary to know beforehand to which ESR or which ESR group this command should be sent. In order for an ESR to be found, it must be added to the EMS object tree. ESR search is performed according to the following logic (Fig. 6):
Fig. 6.
step 1: The AP for which the data-tunnel create command should be executed is located in the Eltex node;
step 2: The Eltex node is checked for ESR - not found, go to the next node up the tree - Novosibirsk;
step 3: The Novosibirks node is checked for ESR - not found, move to the next node up the tree, Novosibirskaya_oblast;
step 4: The Novosibirskaya_oblast node is checked for ESR - not found, move to the next up the tree node MRF_Sibir;
step 5: Check MRF_Sibir node for ESR - found! Use it to send a command to create a data tunnel.
If several ESRs have been found in the node that are suitable for data tunnel creation, the command will be sent to all of them. The CoA-request radius packet of the data-tunnel create type contains the tunnel address of the AP for which the data tunnel should be created and the NAS-IP address of the ESR.
As it can be seen from the above logic - to be able to execute this command AP must be in the EMS tree, you cannot execute it for an uninitialized AP that is in the sandbox. It also follows from this logic that this command can be sent to an ESR to which the AP is not connected.
When executing the data-tunnel create command from EMS, the availability of AP is always first checked by SNMP-ping, and then its tunnel address is read from AP by snmp command and changed if it does not match. If AP has not responded to SNMP-ping - no further actions are performed, you can see a message in the task log in EMS:
FATAL System error of SNMP request processing: TIMEOUT/Timeout: 300 ms
If no ESR was found as a result of searching for the ESR to which the data-tunnel create command should be sent, the command is not executed. This is not considered an error, because ESR with local SoftGRE tunnel configuration profile can be used.
Script 4 - The data-tunnel create command from EMS is executed, AP with the specified tunnel address is not found on ESR.
Fig. 7 shows the algorithm of actions when the data-tunnel create command is executed at the initiative of EMS. When the command is processed on the ESR, the AP with the specified tunnel address is not found.
Fig. 7.
In this script, the data tunnel will not be formed on the ESR. In EMS, the task log will contain a message:
Send 'data-tunnel create' command to ESR NAS ip <IP address of ESR> - 'Data-tunnel create' command error (ESR: <name of ESR>): Data tunnel for AP <Tunnel IP AP> is not formed on <IP address of ESR>
Script 5 - The data-tunnel create command from EMS is being executed, AP with the specified tunnel address is found on ESR.
Fig. 8, shows a diagram of the logic when AP with the specified tunnel address is found on ESR, SSID settings and bindings in EMS are correct.
Fig. 8.
As can be seen from the diagram in Fig. 8, the main difference between this script and script 2 above is the presence of CoA commands for data tunnel creation. As a result ESR, after receiving CoA request Data tunnel create performs the same actions as in variant 2 and upon their completion sends CoA-ACK Created message, which indicates successful creation of data tunnel on ESR. The EMS task log will contain a message:
Send 'data-tunnel create'command to ESR NAS ip <IP address of ESR> - Data tunnel for AP <Tunnel IP AP> is formed <IP address of ESR>
Script 6 - AP is in the EMS tree, SSID parameters contain incorrect settings.
Fig. 9 shows a diagram of the logic when AP with the specified tunnel address is found on ESR, SSID settings in EMS are incorrect. The concept of incorrect settings is defined in the description of script 3.
Fig. 9.
The EMS task log will contain a message:
Send'data-tunnel create' command to ESR NAS ip <IP address of ESR> -'Data-tunnel create' command error (ESR: <name of ESR>): Error creating an interface to AP <Tunnel IP AP> to <IP address of ESR>
Script 7 - ESR that receives the data-tunnel create command is in the VRRP BACKUP state
In a VRRP redundant scheme, two ESRs are used - a primary and a backup ESR. One of the ESRs is always in the VRRP BACKUP state. The diagram for such an ESR is shown in Figure 10.
Fig. 10.
ESR that is in the VRRP BACKUP state always send "State is backup" response and performs no more actions. The EMS task log will contain a message:
Send 'data-tunnel create' command to ESR NAS ip <IP address of ESR> - ESR with NAS IP '<IP адрес ESR>' is in VRRP backup status
Such a message is not an error.
CoA commands of PCRF domain update and update location
In addition to AP operations, EMS may perform operations on SSIDs that may result in the need to reconfigure data tunnels. When changing SSID settings that may affect the configuration of data tunnels on the ESR, namely:
1) "Bridge Location";
2) "Switch Community";
3) "VAP status";
4) "VAP traffic mode (only for GRE)";
5) "VLAN-ID";
6) Create/delete SSID binding to a domain;
The "domain:command=update" and "data-tunnel:command=update-location" commands are executed. When these commands are executed:
1) The domains to which the SSID is bound are defined;
2) Identify all APs that are in domains (and down the tree) to which SSID bindings are made;
3) Sending to all ESRs that are defined for these SSID bindings according to the logic shown in Fig. 6, the changed parameters with the command "domain:command=update" (this command does not contain the tunnel addresses of APs for which the changes should be made and the gre-location parameter, by which the sub-tunnel date is bound to a particular AP);
4) PCRF waiting for a response to the command "domain:command=update", if the response is successful, "data-tunnel:command=update-location" is sent.
5) The command "data-tunnel:command=update-location" is executed, which contains changed SSID parameters, for each found AP by separate CoA command (each packet contains information about vlan:id and its corresponding gre-location parameter). The ESR to which to send this command is determined according to the logic shown in Fig. 6;
6) ESR performs the configuration change of the sab-tunnels date according to the new data.
Splitting the command into two parts is done for the following reasons:
1) Parameter changes may affect too many APs and if one tries to send this information in one packet - its size may exceed the radius packet size limit of 4096 bytes (RFC2865);
2) SSID can be bound to different domains which APs are served by different ESRs;
3) Since the number of APs may be too large, sending the parameter update information for each AP sequentially with the update location command avoids the high CPU utilization of the ESR by the need to reconfigure a large number of data tunnels;
4) The "data-tunnel:command=update-location" command should be executed only after the "domain:command=update" command is completed, because after its completion all necessary tunnels for which it is necessary to set new parameters will be raised.
In case of success/failure of this command execution no information about it will appear in EMS.
Adding ESR to EMS
As follows from the mechanism of raising tunnel data by PCRF commands, ESR must be added to the EMS object tree. It is recommended to place ESR in the object tree in such a way that all APs connecting to it are located in nodes which domains form branches from the domain of its location. To add an ESR it is necessary to stand on the required object in the object tree and press the "+" (Add) button:
Fig. 11.
In the window (fig. 11), type:
- Object name - random name of ESR.
- Type - choose the type of ESR.
- IP address - ip address of ESR configuration.
Push "Add".
Then follow to "RADIUS" → "AP management" and entering the ESR management address in the "Filter" field find it and correct the radius key to the one used in the ESR configuration (in the current example it is "testing123"):
Fig. 12.
Push "Accept".
In the "Access" field correct the name read/write SNMP community to the one used in ESR configuration.
When adding ESR-100/200 "ESR mode" field in the "Access " window will be "StationCE".
In this case it is necessary to change the field value to "Station", otherwise such ESR will not be used to build data tunnels for AP.
ESR configuration
General description of the principles and steps of configuration
Next we will consider ESR configuration in L3 wireless-controller mode, in the mode of raising data tunnels by PCRF commands, with redundancy using VRRF and redundancy of the last-mile router to which ESR is connected. The configuration is discussed using an example with real addressing. It is assumed that the SoftWLC complex is already installed and configured.
It is recommended to follow the configuration order:
- Identify subnets, addresses and AS private number to be used in the configuration (an example of required addresses is given in Table 1 below).
- Configure the equipment to be paired with ESR.
- Configure the ESR; it is recommended to disable the firewall on all L3 interfaces of the ESR for easy debugging.
- Configure SoftWLC complex for interaction with ESR.
- Configure and enable the firewall on ESR.
- Check the scheme operability and redundancy operation.
Addressing description
Select the addresses to be configured according to Table 1, where ESR 1 is the primary router and ESR 2 is the backup router.
| AS 64603 | ESR 1 | ESR 2 | |||
| destination | interface | IP address | VRRP IP | interface | IP address |
|---|---|---|---|---|---|
| стык с VRF AP | gi1/0/1.206 | 100.64.0.34/30 | n/a | gi1/0/1.207 | 100.64.0.38/30 |
| стык с VRF core | gi1/0/1.208 | 100.64.0.42/30 | n/a | gi1/0/1.209 | 100.64.0.46/30 |
| стык с VRF NAT | gi1/0/1.210 | 100.64.0.50/30 | n/a | gi1/0/1.211 | 100.64.0.54/30 |
| Termination of GRE traffic from AP | bridge 1 / 101 | 192.168.200.51/28 | 192.168.200.49/32 192.168.200.50/32 | bridge 1 / 101 | 192.168.200.52/28 |
| Termination of AP configuration subnet | bridge 3 / 3 | 198.18.128.2/21 | 198.18.128.1/32 | bridge 3 / 3 | 198.18.128.3/21 |
| junction address with ESR-1 | bridge 9 / 9 | 100.64.0.57/30 | n/a | bridge 9 / 9 | 100.64.0.58/30 |
Termination of AP clients subnet | bridge 10 / 10 | 198.18.136.2/22 | 198.18.140.1/32 | bridge 10 / 10 | 198.18.140.3/22 |
Table 1.
Fig. 13 shows the scheme of communication at the network level.
Fig. 13.
To connect ESRs (located in AS 64603) to PEs (located in AS 65001) an interface is used, where 3 sub-interfaces are configured to organize BGP connections in 3 corresponding VRFs:
- VRF AP - where all APs are enabled. All traffic exchange between APs and ESRs goes through it.
- VRF core - is designed for interaction with SotfWLC complex and management, it transfers all the management and monitoring traffic between ESR and AP, as well as through it receive addresses by DHCP client devices.
- VRF NAT - designed to access the Internet using the operator's NAT.
Through VRF AP ESR receives routes for subnets 10.10.60.0/24 and 192.168.250.0/24, in which APs receive primary addresses. The subnet 192.168.200.48/28, which contains GRE termination addresses, is announced to the same VRF from the ESR. The APs will build GRE tunnels to the ESR at 192.168.200.49 (management) and 192.168.200.50 (client traffic).
Through VRF core to ESR a route is received for subnet 100.123.0.0/24, where SoftWLC complex servers (EMS, DHCP) are located. The same VRF from ESR announces subnets: 198.18.128.0/21 (AP management) and 198.18.136.0/22 (addresses of AP clients, for the possibility of obtaining and renewing addresses).
A default route of 0.0.0.0.0/0 is taken over the VRF NAT to the ESR. In the same VRF with ESR is announced subnet 198.18.136.0/22 - for passing the traffic of clients to the Internet network.
Both ESRs are in the same autonomous system (AS 64603), and exchange routing information over iBGP. The interface for communication is the Bridge 9 interface. It is recommended to perform physical connection via Port Channel, thus providing redundancy of this channel and increasing the bandwidth of the connection.
With this connection scheme, asymmetric traffic flow is possible, which is undesirable. A detailed description of the causes of this phenomenon and the settings required to prevent it are described in the section "Preventing traffic passing through the jumper in VRRP+BGP redundancy scheme". It is recommended to read after studying this manual on "Configuring ESR in wireless-controller mode with last-mile router redundancy".
ESR configuration
Perform the necessary initial settings. These settings are performed using the console connection.
Since ESR is delivered with factory configuration - reset it to an empty configuration:
esr1000# copy system:default-config system:candidate-config Entire candidate configuration will be reset to default, all settings will be lost upon commit. Do you really want to continue? (y/N): y |******************************************| 100% (52B) Default configuration loaded successfully.
Then perform the initial configuration of ESR 1 (it will be named "Alfa") and ESR 2 (named "Beta"):
After connecting the ESR to the network and verifying access from the mating equipment (PE router), you can disconnect the console and proceed to configuration using telnet or ssh connection. Note that before BGP configuration ESRs will be available only from the addresses of the junction interfaces of the mating equipment.
Сonfigure GRE termination interfaces, AP control subnets and client subnets, as well as junction interfaces between ESRs:
Perform BGP configuration:
Next, configure the wireless-controller for use in a scheme with configuration of data tunnels by PCRF commands:
The settings are identical except for source-address in the radius-server settings, because each ESR should send packets from its own management address. Peer-address, where we specify the address of the neighboring router and nas-ip-address, where the ESR management address is also specified.
To finish the configuration, we will configure SNMP on ESR, the configuration will be the same for each router:
ESR firewall configuration
In the above configuration, the firewall is disabled for all IP interfaces. This is done in order to simplify diagnostics during the initial configuration of the equipment. Figure 14 shows the scheme of assigning security zones to the interfaces (assigning security zones is done above, when configuring the interfaces).
Fig. 14.
The following permissions must be configured:
- From the trusted zone, allow all traffic to the self and trusted zones from the subnet side of the SoftWLC complex.
- Allow BGP traffic to the self zone for all junction interfaces.
- From the gre zone allow all GRE traffic to ESR.
- Allow DHCP packets (and DNS if necessary) from user zone to trusted and self zones.
- From user zone to untrusted zone allow all traffic.
- In the user neighbor direction, allow DNS traffic and any traffic that is not directed to private subnets (in order to provide access to the Internet through the jumper and prevent users from accessing the addresses of internal subnets).
- For the neighbor self direction allow BGP protocol operation.
- In the neighbor untrusted direction allow all traffic.
Firewall configuration looks the same for ESR 1 and 2:
After configuring the firewall, be sure to enable it on all interfaces on which it was previously disabled with the no ip firewall disable command.
Description of the traffic passing diagram
Fig. 15 shows the diagram of traffic passing through ESR in normal operation.
Рис. 15.
ESR management and monitoring is performed at the address specified in the Bridge 3 interface configuration, all traffic is routed through interface gi1/0/1.208 (for ESR 1 Alfa) and gi1/0/1.209 (for ESR 2 Beta) to the VRF core.
Traffic coming from the APs is routed through interface gi1/0/1.206 to Bridge1. The APs establish GRE tunnels on the addresses of this interface:
- MNG GRE – at address 192.168.200.49 which transmits AP management traffic on VLAN ID 1;
- DATA GRE – at address 192.168.200.50 which transmits traffic of clients with APs in VLAN IDs set in the configuration of these SSIDs.
Each Sub GRE interface, is enabled according to the configuration in Bridge:
- Bridge 3 – L3 interface that integrates AP management. The address of this interface serves as the Default Gateway for the AP. DHCP-relay works on this interface, so that the APs receive an address from the DHCP server located in the VRF core.
- Bridge 10 – L3 interface that integrates SSID1 client traffic coming from the APs. The address of this interface serves as the Default Gateway for Wi-Fi users. DHCP-relay works on this interface, so that client devices get the address from the DHCP server located in the VRF core.
User traffic is routed through interface gi1/0/1.210 to the VRF NAT, which passes data to the Internet.
- Bridge 9 – L3 interface for organizing a channel to the backup ESR. The connection to the backup ESR is made through interface gi1/0/2. (It is recommended to use an aggregated interface, which is a combination of several physical ports into one logical Port Channel (via LACP protocol).
Scheme of ESR redundancy VRRP implementation
VRRP redundancy scheme is shown in Fig. 16. VRRP protocol is configured on each Bridge interface of ESR, which participates in GRE tunnel termination, all interfaces are united into one logical group (vrrp group 1). At ESR 1 Alfa and ESR 2 Beta, real IP addresses (Real IP) from the respective pool are set on the respective bridges and Virtual IP (VIP) is specified. These bridges are interconnected via vlan, which are connected via a jumper between ports gi1/0/2. ESR 1 Alfa is set to priority 200 and ESR 2 Beta is set to priority 20. When ESR 1 Alfa fails or when there is a loss of connectivity in vlan (101, 3, 10) - the master is switched to ESR 2 Beta, resulting in VIPs being set on its interfaces.
On the interfaces bridge 3 and bridge 10 the VIP address is used, which is the gateway for all devices on this subnet. Thus, when switching master, for all connected client devices, the MAC and IP address of the gateway will not change.
To raise GRE tunnels, 2 VIP addresses of bridge 1 interface will be used.
Using the bridge 9 junction interface, tunnel state synchronization is performed. ESR 2 Beta, which is in VRRP backup state, receives information about tunnel state from ESR 1 Alfa, which performs periodic distribution of this information.
Thus, when switching the master to ESR, AP will not need to raise new SoftGRE tunnels, as the IP address of ESR for them will remain the same.
Fig. 16.
Possible reasons of network failure
Below we will consider possible variants of user traffic passing during normal operation and network failure.
Network operational state
Figure 17 shows the operational state of the network. Traffic from the connected APs from the VRF AP passes through PE 1 master and gets to ESR 1. Then it is routed: AP management traffic to VRF core, AP user traffic to VRF NAT. PE 2 backup and ESR 2 do not participate in traffic transfer.
Failure of PE 1 master
Figure 18 shows the traffic flow diagram when PE 1 master fails.
Fig. 18.
When PE 1 master fails, it will stop announcing routes received from ESR 1. In this case, there will be no VRRP master change and ESR 1 will remain in the VRRP Master state. Traffic from the AP from the VRF AP will go through PE 2 backup, be routed to ESR 2 and get to ESR 1 through the jumper on the gi1/0/2 interfaces. User and AP management traffic from ESR 1 will be routed through the jumper between the gi1/0/2 ports through the bridge 9 junction interface and passed through P2 backup to VRF core and VRF NAT.
Failure of ESR 1
Figure 19 shows the traffic flow diagram when ESR 1 fails.
If ESR 1 fails, it stops sending VRRP announcements. When ESR 2 detects this, it enters the VRRP MASTER state, assigning VIP addresses to the corresponding interfaces. As a result, traffic from AP from VRF AP through PE 2 backup gets to ESR 2. Then it is routed: AP management traffic to VRF core, AP user traffic to VRF NAT. PE 1 master does not participate in traffic processing.
Recovery of ESR 1 after a failure
The behavior of the VRRP protocol after ESR 1 is recovered to service depends on the settings. In general, ESR 1, as having the highest priority should hijack VRRP master. However, this is unacceptable because immediately after booting ESR 1 will have no information about the state of the tunnels. Therefore, the VRRP interfaces can be configured with:
1) vrrp preempt disable - this setting prohibits VRRP master hijacking if there is a VRRP Master with a lower priority. In this case, ESR 2 will remain in the VRRP master state until rebooted or until ESR 1's mastery capture is otherwise ensured.
2) vrrp preempt delay <number of seconds> - this setting prohibits VRRP mastery hijacking if there is a VRRP Master with a lower priority for the specified period of time. In this case, ESR 2 will remain in the VRRP master state during this period. The recommended value is 420 seconds - this is enough time to synchronize the state of the tunnels on the VRRP Backup, which will be ESR 1 after booting.
Figure 20 below shows the traffic flow diagram for the situation when ESR 1 is recovered and ESR 2 is in the VRRP Master state.
Fig. 20.
Traffic from the AP from the VRF AP will come through PE 1 master to ESR 1, from it through the jumper to ESR 2. Then AP control traffic and user traffic will be passed through PE 2 backup to the corresponding VRFs.
This asymmetric traffic flow is very undesirable, so there is a way to configure BGP announcements and use VRRP state tracking, which will prevent this traffic flow scheme, more details are described above.
Appendix
Full configuration of ESR 1 Alfa and ESR 2 Beta
Configuration is for ESR 1.11.0 firmware version