Per network/application QoS

For network or application based traffic control, mbox uses the Hierarchy Token Bucket (HTB) control algorithm as a faster (high performance) replacement for the traditional class-based queueing (CBQ). HTB provides intuitive, hierarchical and extremely granular traffic control to manage traffic over a physical link and offers many options to managed quality of services (QoS) for different purposes.

It's extremely useful for networks where you're using high-speed physical interface (eg. 1Gbps fiber) but your upstream bandwidth is limited to subscribed bandwidth so that you can configure traffic-shaping to match your subscribed bandwidth and effectively control traffic according to your objectives (different bandwidth for different classes, identified by different networks/VLANs or application)

Generally, traffic control involves four major steps:

    • Traffic/packet marking
    • Traffic classification
    • Traffic shaping
    • Traffic prioritization


Traffic marking is a process to “color” the packets that matches the configured condition, based on source/destination IP/subnets, or protocol & port numbers (application).

Traffic marking is achieved using firewall-set command with the “mark” option, eg.

firewall-set <rule-no> mark <marking-value> access src/dst/tcp/udp/dport/sport


    • the <rule-no> determinants the sequence of how firewall engines process each packets, matching a top-down process, once a packet is matched by a rule, it will not be processed further.
    • the <marking-value> basically sets a mark values for specific packets for next processing. note this packet marking does not set a value on any packet header, meaning of the marking does not follow a packet when it goes out, it’s “internal” within mbox for other process to refer (eg. classification, traffic shaping etc).
    • the <src/dst/tcp/udp/dport/sport> sets matching criterion for the packets which we want to control. If we set a subnet, we’re allocating bandwidth for a network; if we want to set bandwidth for applications, we use a combination of protocol and port numbers.

For example, if we want to allocate bandwidth for entire network, configure below rules to match the packets:

!firewall-set 10 mark 100 access src remark “upload packets”firewall-set 11 mark 100 access dst remark “download packets”!

If we want to allocate bandwidth for http application, configure below rules to match the packets:

!firewall-set 20 mark 100 access tcp dport 80 remark “user http requests”firewall-set 21 mark 100 access tcp sport 80 remark “server http replies"!


Traffic shaping is process of setting min and max bandwidth for each class. It is VERY IMPORTANT to note that the allocated bandwidth per class shared by all the hosts within each class. To further control per user or per host bandwidth, use firewall-limit or client-bandwidth (for hotspot users), refer to per host bandwidth control or per user bandwidth control.

A class identifies a network (either physical LAN links, VLAN or hotspot instance) or type/group of application, depending on the matching criterion. All traffic with the same marking values belongs to the same class, and they’re assigned to their respective que with desired bandwidth. Some vendor products call a class as service tier.

mbox uses traffic shaping techniques to suppress traffic when the link or traffic class gets congested. Unlike traffic policing, Shaping method queues excess packets in a buffer (till the buffer is full) before dropping packets. This is a less “drastic” and smoother mechanism to managed traffic, compared with policing which simply drops excess packets. And mbox hardware usually comes with abundant memory/RAM to support bigger buffer, making it ideal solution for networks with applications which are sensitive to packet drops.

!traffic-shape <min-link-bandwidth> <max-link-bandwidth> class <priority1> <min-class-bandwidth> <max-class-bandwidth> match fwmark <marking-value> class <priority2> <min-class-bandwidth> <max-class-bandwidth> match fwmark <marking-value> class <priority3> <min-class-bandwidth> <max-class-bandwidth> match fwmark <marking-value> …..!


    • Each shaping rule ( traffic-shape) applied to an interface only controls the traffic exiting the interface (in outbound direction).
      • To control both upload & download, we need to apply duplicated rules (mirrored config) to both inbound and outbound interfaces.
      • The <min-link-bandwidth> & <max-link-bandwidth> is the total available bandwidth for the applied interface (eg. subscribed link speed”). Then we can use class to further split different bandwidth for different group of traffic.
      • If you are shaping bandwidth for a different VLANs, the shaping rule must be applied on the physical interface of the VLAN, then use class to identify traffic and allocate bandwidth for each VLAN.
      • If you are shaping traffic for a hotspot instance, the shaping rule must be configured with the hotspot instance (hotspot-shape)
    • We can have multiple class of traffic with different bandwidth allocation and prioritization.
      • Traffic with a class will be given/guaranteed with the min configured bandwidth (<min-class-bandwidth>) when it needs to have, guaranteed bandwidth.
      • Traffic within a class can only burst to the maximum rate configured (<max-class-bandwidth> even if there's spare bandwidth within the entire link. For bursty traffic, we typically set the min and max to be the same, we’re effectively capping the bandwidth for a class. eg. <min-link-bandwidth> = <max-link-bandwidth>. This is similar to traditional "policing" but in a smoother approach; For critical classes (which we want them to burst as much as possible when there's bandwidth available), we typically set <max-class-bandwidth> to be the same as <max-link-bandwidth> so that they can burst up to fully utilize the link when there are spare (unused) bandwidth from other classes.
      • To effectively manage the traffic, the sum of all classes <min-class-bandwidth> should not exceed <min-link-bandwidth>.
      • When actual traffic within a class consumes less than <min-class-bandwidth>, the remaining bandwidth becomes available to be shared by other needing classes which are configured to burst so that they can use up this available bandwidth. If multiple classes of traffic are bursting to grab the same spare bandwidth, they are “prioritized” based on the class priority (eg. <priority1>, <priority2>, <priority3>)

Configuration prerequisites:

    • Determine which interface to apply traffic control (typically the WAN/outbound interface and LAN interface or hotspot instance) and what's the subscribed bandwidth for the link
    • Understand business requirements (eg. define traffic flows and assign them to respective class and apply bandwidth for each class).


    1. Define class and assign bandwidth for each class. Need to match traffic flows to the respective class, two ways to do it.
      • Use firewall marking capabilities (firewall-set xxx mark). This is very power for very complex flow matching when mBox firewall is turned on.
      • Use native classifier instead of marking. This is good when we don't want to turn on mbox firewall feature, eg. when it's running as a router only, but this method can’t be used if mbox is doing NAT.
    2. Configure traffic-shaping on the target interface/link and set the subscribed bandwidth

Below scenario uses firewall marking to identify interesting traffic, allocate them to each class and assign bandwidth to respective class. Brief scenario:

    • 1Gbps subscribed Internet link
    • VLAN103 network:
    • VLAN104 network:
    • Assign VLAN103 network total bandwidth = 30Mbps
    • Assign VLAN104 network total bandwidth = 40Mbps

step 1: Identify the interesting traffic (mark the packets!)

!firewall-set 1031 mark 103 access src remark "outbound from vlan103"firewall-set 1032 mark 103 access dst remark "inbound into vlan103"firewall-set 1041 mark 104 access src remark "outbound from vlan104"firewall-set 1042 mark 104 access dst remark "inbound into vlan104"!

step 2: Apply the traffic control rules, on both inbound and outbound interface

!interface eth0description “Interface to WAN”enableip address 1000000000 1000000000 class 103 30000000 30000000 match fwmark 103 remark “VLAN103 upload” class 104 40000000 40000000 match fwmark 104 remark “VLAN104 upload”!interface eth1description “Interface to LAN”enabletraffic-shape 1000000000 1000000000 class 103 30000000 40000000 match fwmark 103 remark “VLAN103 download” class 104 40000000 40000000 match fwmark 104 remark “VLAN104 download”!interface vlan 1 103description "Wired Network vlan103"enableip address!interface vlan 1 104description "Wired Network vlan104"enableip address!


    1. For inbond shaping, “traffic-shape” is applied on the physical eth1 interface, not each VLAN interface (VLAN103 and VLAN104 are attached to eth1 physical interface). If we want to perform shaping for multiple VLANs, use “firewall-set” to mark packets from each VLAN subnets, and under class bandwidth setting, set bandwidth for each VLAN based on the marking values.
    2. While It is OK to set the same marking for both inbound and outbound traffic for configuration simplicity, do note that each class matching takes place at exit interface on the outbound direction. For example, for VLAN103 traffic, although we’re using the same marking value (103) for both upload and download in VLAN103, but they underlying processing is not same:
      • on interface eth0, class 103 matches the packets marked by firewall-set 1031
      • on interface eth1, class 103 matches the packets marked by firewall-set 1032

EXAMPLE CONFIGURATION STEPS (hotspot instance bandwidth shaping)

In this example,

    1. We want to split the total bandwidth 10Mbps to network (LAN-eth1 and LAN-eth2). LAN-eth2 has captive portal (hotspot service) enabled.
    2. We use firewall-set to mark packets for these two networks, and use class to assign bandwidth to each network based on packet markings.
    3. Assign network link eth1: 5Mbps. For physical link (LAN-eth1)
    4. Assign network link eth2: 5Mbps. For interface with hotspot instance running (LAN-eth2)

!hostname mbox!interface eth 0description "Link to WAN/Internet"enableip address dhcptraffic-shape 10000000 10000000 class 1 50000000 5000000 match fwmark 110 class 2 50000000 5000000 match fwmark 120!interface eth 1description "connection to LAN-eth1"enableip address 50000000 50000000 class 1 5000000 5000000 match fwmark 110!interface eth 2description "connection to LAN-eth2 with Captive Portal enabled"enabletraffic-shape 50000000 50000000 class 2 5000000 5000000 match fwmark 120!ip name-server!!DHCP pool for local LANip dhcp-server!firewall-set 11 mark 110 access src remark "outbound from LAN-eth1"firewall-set 12 mark 110 access dst remark "inbound into LAN-eth1"firewall-set 21 mark 120 access src remark "outbound from LAN-eth2"firewall-set 22 mark 120 access dst remark "inbound into LAN-eth2"!firewall-access 10 permit outbound eth1!firewall-snat 10 overload outbound eth1!security hotspot eth2hotspot-wan eth1hotspot-server ports 5000 5001client-network localhost testing123hotspot-online-portal!

Troubleshooting/Verification commands:


1. check if packets are marked

mbox# show firewall set-listChain PREROUTING (policy ACCEPT 15M packets, 14G bytes)pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 383K packets, 94M bytes)pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 15M packets, 14G bytes)pkts bytes target prot opt in out source destination 417K 684M MARK all -- * * /* set-list 1031 */ MARK set 0x67348K 819M MARK all -- * * /* set-list 1032 */ MARK set 0x67454K 695M MARK all -- * * /* set-list 1041 */ MARK set 0x68329K 693M MARK all -- * * /* set-list 1042 */ MARK set 0x68

2. check if packets are matched and shaped.

mbox# show interface traffic-class eth0class htb 1:104 parent 1:999 prio 0 quantum 200000 rate 40000Kbit ceil 90000Kbit burst 1586b/1 mpu 0b overhead 0b cburst 1586b/1 mpu 0b overhead 0b level 0Sent 1130297933 bytes 1156630 pkt (dropped 71, overlimits 0 requeues 0)rate 0bit 0pps backlog 0b 0p requeues 0lended: 750476 borrowed: 0 giants: 0tokens: 2143 ctokens: 2143 class htb 1:103 parent 1:999 prio 0 quantum 200000 rate 30000Kbit ceil 60000Kbit burst 1590b/1 mpu 0b overhead 0b cburst 1590b/1 mpu 0b overhead 0b level 0Sent 2297169367 bytes 2782631 pkt (dropped 0, overlimits 0 requeues 0)rate 0bit 0pps backlog 0b 0p requeues 0lended: 2113609 borrowed: 0 giants: 0tokens: 3215 ctokens: 3215