Bandwidth Control (QoS)

Traffic/packet marking is a process to “color” the packets so that we can identify the targeted/"interesting" traffic which we want to control. Marking can be based on source/destination network/subnets, or protocol & port numbers (application). 

Traffic marking is achieved using firewall-set command with the “mark” option, eg.

firewall-set <rule-no> mark <marking-value> access src/dst/tcp/udp/dport/sport

NOTE:

• • the <rule-no> determinants the sequence of how firewall engines process each packets, matching a top-down process, once a packet is matched by a rule, it will not be processed further.

  • the <marking-value> basically sets a mark values for specific packets for next processing. note this packet marking does not alter any packet header, meaning the marking does not follow a packet when it goes out, it’s “internal” within mbox for other process to refer (eg. classification, traffic shaping etc).

  • the <src/dst/tcp/udp/dport/sport> sets matching criterion for the packets which we want to control. If we set a subnet/mask, we’re allocating bandwidth for a network; if we want to set bandwidth for applications, we use a combination of protocol and port numbers.

For example, if we want to allocate bandwidth for entire network 172.16.1.0/24, configure below rules to match the packets:

If we want to do QoS for http application only, configure below rules to match the packets:

Classification basically arranges the colored packets into respective classes. A class identifies a network (VLAN), or type/group of application, depending on the marking criterion. All traffic with the same marking values belongs to the same class, and they’re assigned to their respective queue with desired bandwidth.

Each class has a class no (queue no), which sets the priority to dequeue the traffic when they leave mbox exit interface. The lower class no. gets higher chance to go out (therefore higher priority).

mbox traffic shaping suppresses traffic when the link or traffic class gets congested. Unlike traffic policing, shaping method queues excess packets in a buffer (till the buffer is full) before dropping packets. This is a less “drastic” and smoother mechanism to managed traffic, compared with policing which simply drops excess packets. And mbox hardware usually comes with abundant memory/RAM to support bigger buffer, making it ideal solution for applications that are sensitive to packet drops.

We use "traffic-shape" command to apply QoS control to mbox exit interface.

NOTES:

• • traffic-shape only controls the traffic exiting the interface (egress traffic).

    • To control both upload & download, we need to apply duplicated rules (mirrored config) to both inbound and outbound interfaces.

    • The <min-link-bandwidth> & <max-link-bandwidth> is the total available bandwidth for the applied interface (eg. subscribed link speed). Then we use class to further split different bandwidth for different groups of traffic.

    • If you are shaping traffic for different VLANs, the traffic-shape rule must be applied on the physical interface which the VLAN is tagged, then use class to identify VLAN traffic. Do not configure traffic-shape on VLAN interface.

    • If you are shaping traffic for a hotspot instance, the shaping rule must be configured with the hotspot instance (hotspot-shape)

  • Different classes can have different bandwidth allocation and prioritization.

    • <min-class-bandwidth> is the guaranteed bandwidth for this class. <max-class-bandwidth> is the maximum/ceiling bandwidth. NOTE: each class bandwidth setting applies to all the hosts traffic within this class.

    • To accurately control the traffic, the sum of all classes <min-class-bandwidth> should not exceed <min-link-bandwidth>.

    • Traffic within a class can only burst to the <max-class-bandwidth>, even if there's spare bandwidth within the entire link. If we set the min and max to be the same, we’re effectively capping the bandwidth for a class. This is similar to policing/rate-limiting but in a smoother approach; For critical classes (which we want them to burst as much as possible when there's bandwidth available), we typically set <max-class-bandwidth> to be the same as <max-link-bandwidth> so that they can burst up to fully utilize the link when there are spare (unused) bandwidth from other classes.

    • When a class actual bandwidth usage is less than <min-class-bandwidth>, the remaining bandwidth will be given to other needy classes which are allowed to burst so that we can use up all available bandwidth. If multiple classes are bursting to grab the same spare bandwidth, they are “prioritized” based on the class priority (eg. <priority1>, <priority2>, <priority3>) 

Configuration prerequisites:

• • Determine the subscribed bandwidth for the link (WAN link speed).

  • Determine which interface to apply traffic control (typically the WAN/outbound interface and LAN interface).

  • Understand business requirements (eg. define traffic flows and assign them to respective class and apply bandwidth for each class). 

EXAMPLE CONFIGURATION (per VLAN QoS)

• 1. Identify "interesting" trafic (as per step #1.1)

  2. Apply traffic-shaping on the target interface/link

  3. Set class priority and class bandwidth

Example scenario:

• • 1Gbps subscribed Internet link

  • VLAN103 network: 10.0.103.0/24

  • VLAN104 network: 10.0.104.0/24

  • Assign VLAN103 network total bandwidth = 30Mbps

  • Assign VLAN104 network total bandwidth = 40Mbps 

step 1: Traffic marking, identify the interesting traffic

step 2: Traffic control, on both inbound and outbound interface

NOTE:

• 1. For download control, “traffic-shape” is applied on the physical LAN/eth1 interface, not each VLAN interface. VLAN103 and VLAN104 are attached to eth1 physical interface. If we want to perform shaping for multiple VLANs, use “firewall-set” to mark packets from each VLAN subnets, and under class bandwidth setting, set bandwidth for each VLAN based on the marking values.

  2. While It is OK to use the same marking values for both inbound and outbound traffic for configuration simplicity, do note that each class control takes place at exit interface (egress traffic). For example, for VLAN103 traffic, although we’re using the same marking value (103) for both upload and download in VLAN103, but they underlying processing is not same:

• • • on interface eth0, class 103 matches the packets marked by firewall-set 1031

    • on interface eth1, class 103 matches the packets marked by firewall-set 1032 

Troubleshooting/Verification commands:

========================================  

1. check if packets are marked 

2. check if packets are matched and shaped.

In our earlier section, we explained class-based QoS. Note that per class bandwidth refers to total available bandwidth for that class, shared by all user within the network. Host-based control basically limits (or rate limiting) maximum bandwidth per IP host based on the configured policies. 

Because some “bursty” users can exhaust the entire bandwidth and cause congestion to the class, especially during virus/storm outbreak, it is useful to be able to cap (or rate limit) per user or per host bandwidth so that all users or applications within the network can fairly share the available upstream pipe. Rate limiting is particularly important in public Wi-Fi networks, where attacker could use the free network (with huge backhaul) to launch DDoS attack against another victim networks. 

To control per host bandwidth, we use firewall-limit command. It is like a typical firewall rule but setting a bandwidth to this connection. This also means we can define very granular options to match a very specific host connection only.

NOTE:

• • firewall-limit is also a type of firewall-access but it supersedes all other access list. It’s also processed “top-down” based on the <rule_no>.

  • There’s an implicit “permit” for each firewall-limit rule, eg. all traffic matching a limit rule will be permitted to pass through at the allowed <max_bandwidth> rate. Use “show firewall limit-list” to verify detail policy of each configured rate-limit setting.

  • If the rule defines src only, we typically are trying to limit upload; likewise if it defines dst only, we are limiting download.

  • If we src/dst specifies a host IP only, this rule is just rate-limiting the particular host; if we define src/dst as subnet/mask, we are applying the same rate-limit to all hosts within the subnets.

  • We can also limit per session bandwidth by combining src/dst/protocol (tcp/udp) and port numbers (eg, http/80 etc). 

For example, if we configure below rules, it means “each host within subnet 172.16.1.0/24 will be rate-limited to 2Mbps upload and download, for all applications, going through all interfaces.”

If we configure below rules, it means “each host within subnet 172.16.1.0/24 will be rate-limited to 2Mbps upload and download, for http traffic only, going through all interfaces.”

CONFIGURATION EXAMPLE

It is possible to combine class-based QoS and Host-based QoS together. For example, we can set a total bandwidth for a network, then further rate limit per host bandwidth within that network. 

In this example, we are trying to achieve below objectives:

• • subscribed backhaul bandwidth is 100Mbps

  • network (LAN-eth1) is allocated total 50Mbps

  • within LAN-eth1, per host is capped at 2Mbps upload/download

Per user bandwidth control/allocation provides the granularity to assign bandwidth based on user’s identity. This feature is only available when hotspot service is enable (HSG and HSA). And this requires HSG/HSA to work with local or external RADIUS server, where we can configure different user profiles, and attach the desirable access profile to specific users to map to our access policies, eg. user1 is assigned with 2Mbps upload/download, user2 can be assigned with 4Mbps upload/download.

NOTES:

• • user accounts must be locally created within HSG RADIUS (or cloud RADIUS for HSA)

  • user authorization profiles (access policies) must be created within RADIUS server. Each access profile sets the min/max bandwidth per user.

  • each user must be attached with their respective access profile.

  • different user can have different access profile, therefore having different access privileges even within the same network

Configuration prerequisites:

• • HotSpot service must be enabled on the user LAN, configured through CLI

  • Configure user accounts, profiles and assignments from RADIUS portal

Please refer to this video demo on how it works.

CMG and HSG support QoS at layer 2 by setting class of service (CoS) for Ethernet frames. CoS (as in IEEE P802.1P) is a 3-bit field called the Priority Code Point (PCP) within an Ethernet frame header when using VLAN tagged frames as defined by IEEE 802.1Q. It specifies a priority value of between 0 and 7 inclusive that can be used by QoS disciplines to differentiate traffic.

In mbox, when configuring a VLAN interface, there're two command options we can use

NOTE:

•  set cos <outbound-priority> <mapped-priority>. This command rewrite priority for frames going out from the vlan interface, so that upstream switch can provide differential treatment (QoS) for the "colored" frames based on CoS value. By default, all frames have 0 (lowest priority, treat with best efforts). Set a target priority value <mapped-priority> for upstream switch to use.

•  match cost <inbound-priority> <mapped-priority>. This command does the reverse, for frames coming into mbox. By default, it rewrites all frames to 0, irregardless the incoming frame priority. Use this command to preserve or change to other values.

NOTE:

1. "firewall-set" command marks IP packet headers at layer 3/4 for local or upstream router to perform QoS control

2. "set" command sets priority bit on Ethernet frames at layer 2 for upstream switch to perform QoS control

Some ISPs require to set CoS for CPE WAN interface when trunking is configured. Below is an example of Cisco CoS config and mbox CoS config.

Cisco IOS config:

mbox config: