hotspot troubleshooting (on-premise)

For mbox hotspot gateway to work successfully, there are a number of prerequisites and components involved, so it’s very important to understand what they are, and during troubleshooting process, we do step-by-step isolation to find out the root cause of problem.

For field installation, It is imperative that at the end of our configuration, we must verify if hotspot service is working properly on mbox itself, before trying to test on the client/user end.

Common hotspot configuration mistakes are:

  1. No firewall-snat configured. Usually when HSG is used as a gateway, it needs to hide/PAT internal private user IP address to its interface public IP.
  2. No radius-server configured. Most of the time we use HSG local built-in UAM/RADIUS server, so we need to configure and enable local RADIUS server. (security radius-server).
  3. Wrong hotspot instance combination.
    • Wrong hotspot LAN interface, in "security hotspot <lanif>", make sure the lanif is indeed the target vlan/LAN user interface
    • No WAN interface configured, "hotspot-wan wanif" (default to eth0 if not explicitly configured)
  4. Landing page problem. This is usually due to wrong template used or mistakes in landing page customization.

Below are the tests and troubleshooting steps for mbox hotspot service:

    1. check mbox is properly connected to Internet, eg. WAN interface IP, default gateway
    2. check local DNS resolution is working, eg. name-server
    3. check local mbox local firewall rules (firewall-dnat and firewall-access rules)
    4. check upstream connectivity path is clear (in case there's firewall in front of mbox)
    5. check RADIUS working status
    6. check hotspot instance status
    7. check portal name and configured hotspot-portal URL
    8. debug/tcpdump client dhcp request and access requests
    9. show mysql running logs and URL access logs

TESTING & TROUBLESHOOTING STEPS

1. check mbox is properly connected to Internet

mbox must have good Internet connection with below prerequisites met:

Verify WAN interface has IP address (either through DHCP or static assignment, depending on ISP config).


HSG-DEMO# show ip interface brief Interface IP_Address NetMask Broadcast MAC_Address --------------------------------------------------------------------------------eth0 172.16.99.2 255.255.255.0 172.16.99.255 00:60:E0:6B:DA:22eth1 192.168.8.1 255.255.255.0 192.168.8.255 00:60:E0:6B:DA:23eth2 192.168.100.1 255.255.255.0 192.168.100.255 00:60:E0:6B:DA:24lo 127.0.0.1 255.0.0.0 0.0.0.0 00:00:00:00:00:00tun0 192.168.80.1 255.255.255.0 0.0.0.0 00:00:00:00:00:00vlan10 192.168.50.1 255.255.255.0 192.168.50.255 00:60:E0:6B:DA:23HSG-DEMO#

verify default route is available

HSG-DEMO# show ip routeCodes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel, > - selected route, * - FIB route
S>* 0.0.0.0/0 [1/0] via 172.16.99.1, eth0C>* 2.1.2.1/32 is directly connected, loS>* 10.10.0.0/24 [1/0] via 192.168.50.160, vlan10

2 verify DNS is correctly configured

If we don't explicitly configure client name-serve IP address, mbox will default use google name-server IP addresses to clients (8.8.8.8).

mbox#mbox# configurembox(config)# ip name-server 203.211.152.66 210.193.2.66mbox(config)# endmbox# ping www.yahoo.comPING fd-fp3.wg1.b.yahoo.com (106.10.138.240) 56(84) bytes of data.64 bytes from 106.10.138.240: icmp_req=1 ttl=52 time=5.41 ms64 bytes from 106.10.138.240: icmp_req=2 ttl=52 time=5.27 ms64 bytes from 106.10.138.240: icmp_req=3 ttl=52 time=5.37 ms64 bytes from 106.10.138.240: icmp_req=4 ttl=52 time=5.33 ms64 bytes from 106.10.138.240: icmp_req=5 ttl=52 time=5.23 ms--- fd-fp3.wg1.b.yahoo.com ping statistics ---5 packets transmitted, 5 received, 0% packet loss, time 4005msrtt min/avg/max/mdev = 5.233/5.326/5.415/0.104 msmbox#mbox#mbox# ping www.yahoo.comPING fd-fp3.wg1.b.yahoo.com (106.10.138.240) 56(84) bytes of data.64 bytes from 106.10.138.240: icmp_req=1 ttl=52 time=3.88 ms64 bytes from 106.10.138.240: icmp_req=2 ttl=52 time=3.22 ms64 bytes from 106.10.138.240: icmp_req=3 ttl=52 time=2.83 ms64 bytes from 106.10.138.240: icmp_req=4 ttl=52 time=4.01 ms^C--- fd-fp3.wg1.b.yahoo.com ping statistics ---4 packets transmitted, 4 received, 0% packet loss, time 3004msrtt min/avg/max/mdev = 2.832/3.489/4.018/0.486 msmbox#HSG-DEMO# ping splash.ransnet.comPING splash.ransnet.com (2.1.2.1) 56(84) bytes of data.64 bytes from 2.1.2.1: icmp_req=1 ttl=64 time=0.096 ms64 bytes from 2.1.2.1: icmp_req=2 ttl=64 time=0.062 ms64 bytes from 2.1.2.1: icmp_req=3 ttl=64 time=0.053 ms64 bytes from 2.1.2.1: icmp_req=4 ttl=64 time=0.061 ms64 bytes from 2.1.2.1: icmp_req=5 ttl=64 time=0.053 ms
--- splash.ransnet.com ping statistics ---5 packets transmitted, 5 received, 0% packet loss, time 4074msrtt min/avg/max/mdev = 0.053/0.065/0.096/0.015 msHSG-DEMO#

NOTE:

    • if no default gateway available, we can’t go anywhere. Check:
      • If WAN is using DHCP to grab IP (either from ISP connection or upstream firewall), default gateway should be auto assigned, otherwise check with ISP or upstream firewall
      • if WAN is using static IP, we must manually configure default gateway (eg. ip default-gateway 192.168.100.1)
    • if can ping out to 8.8.8.8, but can’t ping out via DNS name (www.yahoo.com), that means name-server is not working. check our dns name-server setting (eg. ip name-server 203.211.152.66 210.193.2.66)

3 check mbox local firewall rules

check client source address translations (firewall-dnat). In most cases, users/clients are given private IP addresses, so when they browse Internet, their private source addresses must be hidden/translated to a public address. There are two potential scenarios:

  • mbox is deployed as the all-in-one gateway (router, firewall, hotspot gateway), so mbox is required to hide Internal client source IP addresses by mapping them to mbox external WAN interface address. This is done using firewall-dnat rules.
  • if there are other firewall or router in front of mbox, and it’s preferred by customer (for whatever reason) that mbox not perform address translations (eg. firewall needs to “see” client real IP addresses). Then there’s a need to make sure the front router or firewall has performed source address translation (PAT) and also they need to add static route for client’s subnet pointing back to mbox WAN IP as next hop.

Very firewall-dnat/PAT when mbox is deployed as all-in-one gateway.

mbox#mbox# show running-config include firewall-snatfirewall-snat 1 overload outbound eth0 <--eth0 is the WAN interface mbox# show firewall snat-listChain POSTROUTING (policy ACCEPT 189K packets, 16M bytes)pkts bytes target prot opt in out source destination 29M 2150M MASQUERADE all -- * eth0 0.0.0.0/0 0.0.0.0/0 mbox#

4 check upstream connectivity path is clear

Sometimes, it’s not always possible to access customers’ firewall or router, use below simple command on mbox to check if upstream router/firewall is properly configured.

HSG-DEMO# show security hotspot Authentication service: running---------------------------------HotSpot service: running---------------------------------LAN TUN Server IP Client-Net Client-DHCP DHCP Issued Clients -------------------------------------------------------------------------------------------------------------vlan80 tun0 192.168.80.1 192.168.80.0/255.255.255.0 / 0 0 HSG-DEMO# HSG-DEMO# traceroute 8.8.8.8 source 192.168.80.1traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets 1 172.16.99.1 0.528 ms 0.497 ms 0.510 ms ......<intentionally hidden> 7 203.211.159.108 5.168 ms 203.211.159.90 4.699 ms 203.211.159.108 4.626 ms 8 203.211.158.77 4.768 ms 203.211.158.79 3.885 ms 203.211.158.77 5.299 ms 9 210.193.4.254 3.724 ms 3.534 ms 3.789 ms10 108.170.254.225 5.205 ms 108.170.240.225 5.294 ms 108.170.254.225 5.520 ms11 209.85.242.121 4.734 ms 209.85.143.7 3.823 ms 108.170.237.227 3.713 ms12 8.8.8.8 3.603 ms 3.515 ms 3.402 msHSG-DEMO#

NOTE:

    • the traceroute source (192.168.80.1) is the hotspot server IP address configured under mbox, eg. getting from “show security hotspot”
    • if above traceroute fails, that means the required configs on upstream router/firewall are not done properly. Check upstream firewall/router to make sure they have configured source address translation and added static route for user/client subnets

In case if there’s firewall in front of mbox, Outbound DNS query from mbox must be permitted, and outbound http and https from mbox to any common websites must be permitted. Check from mbox:


mbox#mbox# telnet www.google.com 80Trying 74.125.130.147...Connected to www.google.com.Escape character is '^]'.Connection closed by foreign host.mbox#mbox# telnet www.google.com 443Trying 74.125.130.104...Connected to www.google.com.Escape character is '^]'.Connection closed by foreign host.mbox#

NOTE:

    • if any of above step fails, check upstream firewall configuration and make sure the required accesses are permitted

5. check mbox RADIUS service

In most cases, mbox is also configured as a RADIUS server for it’s own access-controller service (details). Double check RADIUS configuration: make sure mbox itself is added as a local client for radius-server; make sure test authentication work and verify with logs.


mbox# show running-config……!security radius-serverclient 127.0.0.1 key testing123 name mbox-HSGdata-lifetime 30start!……mbox# show security radius-clientNAS Name NAS IP -------------------------------------------mbox-HSG 127.0.0.1 OKmbox#mbox# show security radius-serverRADIUS server is running... OK mbox#mbox# test authentication radius-server localhost radius-key testing123 username demouser password demouserSending Access-Request of id 150 to 127.0.0.1 port 1812 User-Name = "demouser" User-Password = "demouser" NAS-IP-Address = 127.0.0.1 NAS-Port = 1812 Message-Authenticator = 0x00000000000000000000000000000000rad_recv: Access-Accept packet from host 127.0.0.1 port 1812, id=150, length=20mbox#mbox# show security radius-logSat Jan 24 23:19:59 2015 : Auth: Login OK: [demouser] (from client JTC-HSG port 1812)^Cmbox#

NOTE:

    • if any of above fails, mbox will not be able to authenticate user clients therefore user login will fail.

6. check mbox hotspot services

Follow the hotspot configuration guide very closely, make sure at least all the base commands are configured and verify if hotspot access controller is running.

NOTE: one of the most common mistake is configuring wrong LAN and WAN interface. Each hotspot instance must match a correct pair of LAN and WAN interface. So do make sure "security hotspot <lanif>" and "hotspot-wan <wanif>" are correctly configured.

Use “?” to view what are the minimum/base commands required to activate a hotspot service.

mbox#mbox# configurembox(config)# security hotspot vlan10(config-hotspot-eth1)# ? ! Comments allowed-domain (optional) default allowed domains, comma seperated. allowed-url (optional) default allowed URL or networks, comma seperated. client-bandwidth (optional) Per client maximum bandwidth. ......... HSG-DEMO# show running-config begin "security hotspot"security hotspot vlan80 client-dhcp-dns 203.211.152.66 210.193.2.66 client-dhcp-helper 172.16.30.30,172.16.40.5 client-static 192.168.80.2 255.255.255.0 bypass-domain .facebook.com,.facebook.net,.akamaihd.net,fb.me,.fbcdn.net,.fbsbx.com,.twitter.com,.twimg.com,.linkedin.com,.static.licdn.com,.weibo.cn,.ransnet.com,.weibo.com bypass-dst portal.ransnet.com,splash.ransnet.com bypass-mac radius redirect-url http://mboxstatus.net radius-server localhost testing123 hotspot-portal https://splash.ransnet.com/pid/ransnet/login.php start!

NOTE:

    • The original IP addresses for each hotspot interface (lanif) will be removed, and a virtual interface (tun0, tun1...) will be created and the respective tunnel interface will take over original lanif interface IP address.

HSG-DEMO# show security hotspot Authentication service: running---------------------------------HotSpot service: running---------------------------------LAN TUN Server IP Client-Net Client-DHCP DHCP Issued Clients -------------------------------------------------------------------------------------------------------------vlan80 tun0 192.168.80.1 192.168.80.0/255.255.255.0 / 0 0
HSG-DEMO# show ip interface brief Interface IP_Address NetMask Broadcast MAC_Address --------------------------------------------------------------------------------eth0 172.16.99.2 255.255.255.0 172.16.99.255 00:60:E0:6B:DA:22eth1 192.168.8.1 255.255.255.0 192.168.8.255 00:60:E0:6B:DA:23eth2 192.168.100.1 255.255.255.0 192.168.100.255 00:60:E0:6B:DA:24lo 127.0.0.1 255.0.0.0 0.0.0.0 00:00:00:00:00:00tun0 192.168.80.1 255.255.255.0 0.0.0.0 00:00:00:00:00:00vlan80 NON-IP NON-IP NON-IP 00:60:E0:6B:DA:23HSG-DEMO#
HSG-DEMO# show security hotspot access-list
##### Access-list summary for all hotspot instances ##### pkts bytes target prot opt in out source destination12815 2202K HSFWD_vlan80 all -- tun0 eth0 192.168.80.0/24 0.0.0.0/0 /* vlan80_ACL */
##### HSFWD_vlan80 Summary ###########Chain HSFWD_vlan80 (1 references) pkts bytes target prot opt in out source destination 12815 2202K HSACL_vlan80 all -- * * 0.0.0.0/0 0.0.0.0/0 12815 2202K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
##### HSACL_vlan80 detail ACL ########Chain HSACL_vlan80 (1 references) pkts bytes target prot opt in out source destination
HSG-DEMO#

7. hotspot portal URL matches with portal name on mbox GUI

The portal URL must match exactly the port configured on HSG. and make sure the user device is able to resolve to the portal URL also (as in step #2)

HSG-DEMO# show running-config begin "security hotspot"security hotspot vlan80 client-dhcp-dns 203.211.152.66 210.193.2.66 client-dhcp-helper 172.16.30.30,172.16.40.5 client-static 192.168.80.2 255.255.255.0 bypass-domain .facebook.com,.facebook.net,.akamaihd.net,fb.me,.fbcdn.net,.fbsbx.com,.twitter.com,.twimg.com,.linkedin.com,.static.licdn.com,.weibo.cn,.ransnet.com,.weibo.com bypass-dst portal.ransnet.com,splash.ransnet.com bypass-mac radius redirect-url http://mboxstatus.net radius-server localhost testing123 hotspot-portal https://splash.ransnet.com/pid/vlan10/login.php start!

Then check the portal portal (make sure the right template is used and the settings match customer requirements etc).

8. Use TCPDUMP

If all previous tests are successful, we are ready to test from client device. But many times, we still run into embarrassment that users complains it’s not working. Well, there can be many many reasons why clients can not access Internet successfully, eg. user device problem, wireless signal, LAN switch, etc etc, especially in a very large environment, many problems can happen in the WLAN/LAN side.

At minimum, to isolate if the problem is due to mbox or not, we can use tcpdump to check if user DHCP request is able to reach to mbox via mbox LAN interface, where clients' traffic is coming from.


HSG-DEMO# tcpdump interface vlan80 port 67tcpdump: WARNING: vlan80: no IPv4 address assignedtcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on vlan80, link-type EN10MB (Ethernet), capture size 65535 bytes10:52:36.388276 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from dc:72:9b:c5:1d:b3, length 31610:52:37.391311 IP 192.168.80.1.67 > 192.168.80.10.68: BOOTP/DHCP, Reply, length 30010:52:37.414554 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from dc:72:9b:c5:1d:b3, length 32810:52:37.424069 IP 192.168.80.1.67 > 192.168.80.10.68: BOOTP/DHCP, Reply, length 300

Then from RADIUS log, check if authentication requests is coming in.


HSG-DEMO# show security radius-log Tue Apr 23 10:52:38 2019 : Info: rlm_sql_mysql: Starting connect to MySQL server for #16Tue Apr 23 10:52:38 2019 : Info: rlm_sql (sql): Attempting to connect rlm_sql_mysql #15Tue Apr 23 10:52:38 2019 : Info: rlm_sql_mysql: Starting connect to MySQL server for #15Tue Apr 23 10:52:38 2019 : Info: rlm_sql (sql): Connected new DB handle, #16Tue Apr 23 10:52:38 2019 : Info: rlm_sql (sql): Connected new DB handle, #15Tue Apr 23 10:53:39 2019 : Auth: Login OK: [Demouser] (from client Local port 1 cli DC-72-9B-C5-1D-B3)

Once user is logged, "show security hotspot clients" to very the final status


HSG-DEMO# show security hotspot clients Client-MAC Client-IP Username %/MaxUp %/MaxDown Idle/Max Duration/Max------[vlan80 ]--------------------------------------------------------------------------------------------------DC-72-9B-C5-1D-B3 192.168.80.10 Demouser 0%/0 0%/0 8/0 72/0
HSG-DEMO#

NOTE:

    • where vlan10 is the user LAN interface, which is used in hotspot configuration, eg. "security hotspot vlan10" (if there’s only one network behind, it should be a physical interface, eg. eth1)
    • if we do not see any DHCP requests at all, investigate mbox connection to LAN switch and also internal wireless controller configuration, make sure it’s relaying DHCP requests to mbox.
    • if we see some “Requests”, observe for a few seconds or even longer, but do not see any “Reply”. That means mbox is not responding to DHCP requests, check if hotspot service is running (go back to previous hotspot service troubleshooting steps)
    • if we see both some requests and some reply, that means mbox DHCP is working fine; however, some users still complain not getting IP addresses. With tcpdump we can filter down to specific client mac address (eg. tcpdump interface vlan80 port 67 filter 60:f1:89:51:35:fe), if we do not see any requests from this client mac, that means client is not even connected to the WLAN network, or wireless controller is too busy (not relaying requests fast enough to mbox).