Bandwidth Control (QoS)

1. QoS Overview

Bandwidth control is also known as Quality of Service (QoS) control. mbox appliances (HSG/CMG/HSA) support extremely granular QoS control options, depending on the actual business requirements.

There're several mechanisms to "control" bandwidth:

    • Set maximum allowable bandwidth, this is often called "policing" or rate-limiting

    • Guarantee minimum bandwidth for a particular network or application. This is part of our traffic-shaping feature

    • Queue traffic based on different categories of traffic, and dequeue/release packets in order or priorities. This is often known as class-based queuing.

    • Prioritize certain critical classes of traffic or application. In our mbox configuration, we simply need to put "prioritized" traffic on the highest queue (lowest class no.) in our traffic-shaping configuration.

There're several ways to identify/mark interesting traffic for control:

    • network-based, controls bandwidth (min and max) for a network or VLAN

    • application-based, controls bandwidth (min and max) for a particular application

    • host-based, controls bandwidth (min and max) on a per host basis (it's usually referred as rate-limiting)

    • user-based, controls bandwidth (min and max) for a particular user. A “user” can be identified by username or MAC address.

NOTE: HSG & HSA support additional bandwidth control features such as dynamic and adaptive bandwidth control.

We can refer to below topology to explain overall QoS concept. In the diagram, we have a backhaul of 1Gbps, and there are multiple internal VLANs. Each VLAN is typically mapped to different SSID for wireless networks.

    • Each VLAN is allocated with a bandwidth (we can set a min and max). The min guarantees a bandwidth, max sets how much this VLAN can burst when there’s available bandwidth in the whole pipe (eg. when other VLANs are not fully utilizing their min allocated bandwidth)

    • For VLAN30, as an example, we use application-based QoS to further split and prioritize available bandwidth for different applications.

    • For VLAN40, as an example, we use user-based QoS to offer differentiated services/experiences for different groups of users.

2. Class-based QoS

Class-based QoS basically allocates different types of traffic into different queues/classes, where you can also control the min/max bandwidth for each queue, and dequeues/releases packets out according to their priority (class no.).

It involves two simple steps:

    1. Traffic marking

    2. Traffic control (classification, prioritization, shaping)

2.1 Traffic marking

Traffic/packet marking is a process to “color” the packets so that we can identify the targeted/"interesting" traffic which we want to control. Marking can be based on source/destination network/subnets, or protocol & port numbers (application).

Traffic marking is achieved using firewall-set command with the “mark” option, eg.

firewall-set <rule-no> mark <marking-value> access src/dst/tcp/udp/dport/sport

NOTE:

    • the <rule-no> determinants the sequence of how firewall engines process each packets, matching a top-down process, once a packet is matched by a rule, it will not be processed further.

    • the <marking-value> basically sets a mark values for specific packets for next processing. note this packet marking does not alter any packet header, meaning the marking does not follow a packet when it goes out, it’s “internal” within mbox for other process to refer (eg. classification, traffic shaping etc).

    • the <src/dst/tcp/udp/dport/sport> sets matching criterion for the packets which we want to control. If we set a subnet/mask, we’re allocating bandwidth for a network; if we want to set bandwidth for applications, we use a combination of protocol and port numbers.

For example, if we want to allocate bandwidth for entire network 172.16.1.0/24, configure below rules to match the packets:

!firewall-set 10 mark 100 access src 172.16.1.0/24 remark “upload packets”firewall-set 11 mark 100 access dst 172.16.1.0/24 remark “download packets”!

If we want to do QoS for http application only, configure below rules to match the packets:

!firewall-set 20 mark 100 access tcp dport 80 remark “user http requests”firewall-set 21 mark 100 access tcp sport 80 remark “server http replies"!

2.2 Traffic control (classification, prioritization, shaping)

Classification basically arranges the colored packets into respective classes. A class identifies a network (VLAN), or type/group of application, depending on the marking criterion. All traffic with the same marking values belongs to the same class, and they’re assigned to their respective queue with desired bandwidth.

Each class has a class no (queue no), which sets the priority to dequeue the traffic when they leave mbox exit interface. The lower class no. gets higher chance to go out (therefore higher priority).

mbox traffic shaping suppresses traffic when the link or traffic class gets congested. Unlike traffic policing, shaping method queues excess packets in a buffer (till the buffer is full) before dropping packets. This is a less “drastic” and smoother mechanism to managed traffic, compared with policing which simply drops excess packets. And mbox hardware usually comes with abundant memory/RAM to support bigger buffer, making it ideal solution for applications that are sensitive to packet drops.

We use "traffic-shape" command to apply QoS control to mbox exit interface.

!traffic-shape <min-link-bandwidth> <max-link-bandwidth> class <priority1> <min-class-bandwidth> <max-class-bandwidth> match fwmark <marking-value> class <priority2> <min-class-bandwidth> <max-class-bandwidth> match fwmark <marking-value> class <priority3> <min-class-bandwidth> <max-class-bandwidth> match fwmark <marking-value>!

NOTES:

    • traffic-shape only controls the traffic exiting the interface (egress traffic).

      • To control both upload & download, we need to apply duplicated rules (mirrored config) to both inbound and outbound interfaces.

      • The <min-link-bandwidth> & <max-link-bandwidth> is the total available bandwidth for the applied interface (eg. subscribed link speed). Then we use class to further split different bandwidth for different groups of traffic.

      • If you are shaping traffic for different VLANs, the traffic-shape rule must be applied on the physical interface which the VLAN is tagged, then use class to identify VLAN traffic. Do not configure traffic-shape on VLAN interface.

      • If you are shaping traffic for a hotspot instance, the shaping rule must be configured with the hotspot instance (hotspot-shape)

    • Different classes can have different bandwidth allocation and prioritization.

      • <min-class-bandwidth> is the guaranteed bandwidth for this class. <max-class-bandwidth> is the maximum/ceiling bandwidth. NOTE: each class bandwidth setting applies to all the hosts traffic within this class.

      • To accurately control the traffic, the sum of all classes <min-class-bandwidth> should not exceed <min-link-bandwidth>.

      • Traffic within a class can only burst to the <max-class-bandwidth>, even if there's spare bandwidth within the entire link. If we set the min and max to be the same, we’re effectively capping the bandwidth for a class. This is similar to policing/rate-limiting but in a smoother approach; For critical classes (which we want them to burst as much as possible when there's bandwidth available), we typically set <max-class-bandwidth> to be the same as <max-link-bandwidth> so that they can burst up to fully utilize the link when there are spare (unused) bandwidth from other classes.

      • When a class actual bandwidth usage is less than <min-class-bandwidth>, the remaining bandwidth will be given to other needy classes which are allowed to burst so that we can use up all available bandwidth. If multiple classes are bursting to grab the same spare bandwidth, they are “prioritized” based on the class priority (eg. <priority1>, <priority2>, <priority3>)

Configuration prerequisites:

    • Determine the subscribed bandwidth for the link (WAN link speed).

    • Determine which interface to apply traffic control (typically the WAN/outbound interface and LAN interface).

    • Understand business requirements (eg. define traffic flows and assign them to respective class and apply bandwidth for each class).

EXAMPLE CONFIGURATION (per VLAN QoS)

    1. Identify "interesting" trafic (as per step #1.1)

    2. Apply traffic-shaping on the target interface/link

    3. Set class priority and class bandwidth

Example scenario:

    • 1Gbps subscribed Internet link

    • VLAN103 network: 10.0.103.0/24

    • VLAN104 network: 10.0.104.0/24

    • Assign VLAN103 network total bandwidth = 30Mbps

    • Assign VLAN104 network total bandwidth = 40Mbps

step 1: Traffic marking, identify the interesting traffic

!firewall-set 1031 mark 103 access src 10.0.103.0/24 remark VLAN103-ULfirewall-set 1032 mark 103 access dst 10.0.103.0/24 remark VLAN103-DLfirewall-set 1041 mark 104 access src 10.0.104.0/24 remark VLAN104-ULfirewall-set 1042 mark 104 access dst 10.0.104.0/24 remark VLAN104-DL!

step 2: Traffic control, on both inbound and outbound interface

!interface eth0description “Interface to WAN”enableip address 172.16.0.6/24traffic-shape 1000000000 1000000000 class 103 30000000 30000000 match fwmark 103 remark “VLAN103 upload” class 104 40000000 40000000 match fwmark 104 remark “VLAN104 upload”!interface eth1description “Interface to LAN”enabletraffic-shape 1000000000 1000000000 class 103 30000000 40000000 match fwmark 103 remark “VLAN103 download” class 104 40000000 40000000 match fwmark 104 remark “VLAN104 download”!interface vlan 1 103description "Wired Network vlan103"enableip address 10.0.103.1/24!interface vlan 1 104description "Wired Network vlan104"enableip address 10.0.104.1/24!

NOTE:

    1. For download control, “traffic-shape” is applied on the physical LAN/eth1 interface, not each VLAN interface. VLAN103 and VLAN104 are attached to eth1 physical interface. If we want to perform shaping for multiple VLANs, use “firewall-set” to mark packets from each VLAN subnets, and under class bandwidth setting, set bandwidth for each VLAN based on the marking values.

    2. While It is OK to use the same marking values for both inbound and outbound traffic for configuration simplicity, do note that each class control takes place at exit interface (egress traffic). For example, for VLAN103 traffic, although we’re using the same marking value (103) for both upload and download in VLAN103, but they underlying processing is not same:

      • on interface eth0, class 103 matches the packets marked by firewall-set 1031

      • on interface eth1, class 103 matches the packets marked by firewall-set 1032

Troubleshooting/Verification commands:

========================================

1. check if packets are marked

mbox# show firewall set-listChain PREROUTING (policy ACCEPT 15M packets, 14G bytes)pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 383K packets, 94M bytes)pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 15M packets, 14G bytes)pkts bytes target prot opt in out source destination 417K 684M MARK all -- * * 10.0.103.0/24 0.0.0.0/0 /* set-list 1031 */ MARK set 0x67348K 819M MARK all -- * * 0.0.0.0/0 10.0.103.0/24 /* set-list 1032 */ MARK set 0x67454K 695M MARK all -- * * 10.0.104.0/24 0.0.0.0/0 /* set-list 1041 */ MARK set 0x68329K 693M MARK all -- * * 0.0.0.0/0 10.0.104.0/24 /* set-list 1042 */ MARK set 0x68

2. check if packets are matched and shaped.

mbox# show interface traffic-shape

3. Host-based QoS (rate-limiting)

In our earlier section, we explained class-based QoS. Note that per class bandwidth refers to total available bandwidth for that class, shared by all user within the network. Host-based control basically limits (or rate limiting) maximum bandwidth per IP host based on the configured policies.

Because some “bursty” users can exhaust the entire bandwidth and cause congestion to the class, especially during virus/storm outbreak, it is useful to be able to cap (or rate limit) per user or per host bandwidth so that all users or applications within the network can fairly share the available upstream pipe. Rate limiting is particularly important in public Wi-Fi networks, where attacker could use the free network (with huge backhaul) to launch DDoS attack against another victim networks.

To control per host bandwidth, we use firewall-limit command. It is like a typical firewall rule but setting a bandwidth to this connection. This also means we can define very granular options to match a very specific host connection only.

!firewall-limit <rule_no> <max_bandwidth, Kbps> <direction> <src/dst> <Proto/Port or IP or subnet> !

NOTE:

    • firewall-limit is also a type of firewall-access but it supersedes all other access list. It’s also processed “top-down” based on the <rule_no>.

    • There’s an implicit “permit” for each firewall-limit rule, eg. all traffic matching a limit rule will be permitted to pass through at the allowed <max_bandwidth> rate. Use “show firewall limit-list” to verify detail policy of each configured rate-limit setting.

    • If the rule defines src only, we typically are trying to limit upload; likewise if it defines dst only, we are limiting download.

    • If we src/dst specifies a host IP only, this rule is just rate-limiting the particular host; if we define src/dst as subnet/mask, we are applying the same rate-limit to all hosts within the subnets.

    • We can also limit per session bandwidth by combining src/dst/protocol (tcp/udp) and port numbers (eg, http/80 etc).

For example, if we configure below rules, it means “each host within subnet 172.16.1.0/24 will be rate-limited to 2Mbps upload and download, for all applications, going through all interfaces.”

!firewall-limit 10 2048 all src 172.16.1.0/24 remark ULfirewall-limit 11 2048 all dst 172.16.1.0/24 remark DL!

If we configure below rules, it means “each host within subnet 172.16.1.0/24 will be rate-limited to 2Mbps upload and download, for http traffic only, going through all interfaces.”

!firewall-limit 10 2048 all tcp src 172.16.1.0/24 dport 80 remark UL-httpfirewall-limit 11 2048 all tcp sport 80 dst 172.16.1.0/24 remark DL-http!

CONFIGURATION EXAMPLE

It is possible to combine class-based QoS and Host-based QoS together. For example, we can set a total bandwidth for a network, then further rate limit per host bandwidth within that network.

In this example, we are trying to achieve below objectives:

    • subscribed backhaul bandwidth is 100Mbps

    • network (LAN-eth1) is allocated total 50Mbps

    • within LAN-eth1, per host is capped at 2Mbps upload/download


!interface eth0description "Link to WAN/Internet"enableip address dhcptraffic-shape 100000 100000 class 1 50000 50000 match fwmark 110!interface eth1 description "connection to LAN-eth1" enable ip address 172.16.1.1/24 dhcp-server dns 8.8.8.8 8.8.4.4 router 172.16.1.1 range 172.16.1.5 172.16.1.100 enabletraffic-shape 50000 50000 class 1 50000 50000 match fwmark 110!ip name-server 8.8.8.8 8.8.4.4!firewall-set 10 mark 110 access src 172.16.1.0/24 remark Network-ULfirewall-set 11 mark 110 access dst 172.16.1.0/24 remark Network-DL!firewall-limit 10 2048 all src 172.16.1.0/24 remark "host UL"firewall-limit 11 2048 all dst 172.16.1.0/24 remark "host DL"!firewall-snat 10 overload outbound eth0!

4. User-based QoS

Per user bandwidth control/allocation provides the granularity to assign bandwidth based on user’s identity. This feature is only available when hotspot service is enable (HSG and HSA). And this requires HSG/HSA to work with local or external RADIUS server, where we can configure different user profiles, and attach the desirable access profile to specific users to map to our access policies, eg. user1 is assigned with 2Mbps upload/download, user2 can be assigned with 4Mbps upload/download.

NOTES:

    • user accounts must be locally created within HSG RADIUS (or cloud RADIUS for HSA)

    • user authorization profiles (access policies) must be created within RADIUS server. Each access profile sets the min/max bandwidth per user.

    • each user must be attached with their respective access profile.

    • different user can have different access profile, therefore having different access privileges even within the same network

Configuration prerequisites:

    • HotSpot service must be enabled on the user LAN, configured through CLI

    • Configure user accounts, profiles and assignments from RADIUS portal

Please refer to this video demo on how it works.

5. Layer 2 QoS (CoS)

CMG and HSG support QoS at layer 2 by setting class of service (CoS) for Ethernet frames. CoS (as in IEEE P802.1P) is a 3-bit field called the Priority Code Point (PCP) within an Ethernet frame header when using VLAN tagged frames as defined by IEEE 802.1Q. It specifies a priority value of between 0 and 7 inclusive that can be used by QoS disciplines to differentiate traffic.

In mbox, when configuring a VLAN interface, there're two command options we can use

!interface vlan 0 10 enable set cos <default-outbound-priority> <mapped-priority> match cost <default-in-bound-priority> <mapped-priority>!

NOTE:

  • set cos <outbound-priority> <mapped-priority>. This command rewrite priority for frames going out from the vlan interface, so that upstream switch can provide differential treatment (QoS) for the "colored" frames based on CoS value. By default, all frames have 0 (lowest priority, treat with best efforts). Set a target priority value <mapped-priority> for upstream switch to use.

  • match cost <inbound-priority> <mapped-priority>. This command does the reverse, for frames coming into mbox. By default, it rewrites all frames to 0, irregardless the incoming frame priority. Use this command to preserve or change to other values.

NOTE:

  1. "firewall-set" command marks IP packet headers at layer 3/4 for local or upstream router to perform QoS control

  2. "set" command sets priority bit on Ethernet frames at layer 2 for upstream switch to perform QoS control

Some ISPs require to set CoS for CPE WAN interface when trunking is configured. Below is an example of Cisco CoS config and mbox CoS config.

Cisco IOS config:

!policy-map COS class class-default set cos 1!interface FastEthernet0.1103 encapsulation dot1Q 1103 ip address dhcp service-policy output COS!

mbox config:

!interface vlan 0 1103 enable set cos 0 1 ip address dhcp!