Internet Protocol Security (IPsec) is a very complex protocol suite for secure Internet Protocol (IP) communications, by building secure tunnels/sessions to authenticate and encrypt each IP packet passing between two hosts or gateways.

IPsec includes protocols for establishing mutual authentication between agents at the beginning of the session and negotiation of cryptographic keys to be used during the session. An IPSec tunnel consists of 3 Security Associations (SA). An SA defines a bundle of algorithms and parameters (such as keys) that is being used to encrypt and authenticate a particular flow in one direction. Therefore, in normal bi-directional traffic, the flows are secured by a pair of security associations.

There're a lot of details we can google about IPSec. In summary, to establish an IPSec tunnel, we configure three sets of policies:

  • IKE policy (as in IPSec Phase I). This phase basically authenticates peers first then negotiate a bundle of encryption/hashing algorithms to generate session key, which is used by Phase II to encrypt packets. One peer (host or gateway) can be configured with multiple set/combination of policies, and when two peers are initiating a tunnel, they will agree on a matched policy. If no matching policy between two peers, tunnel will not come up. Each IKE policy consists of a combination of either one of these parameters. Supported parameters are listed below:

    • authentication: pre-shared key or RSA-sig

    • encryption: 3DES, AES, AES-192, AES-256

    • hashing: MD5, MD5-128, SHA, SHA-256, SHA-512

    • version: IKEv1 or IKEv2

  • ESP policy (as in IPSec Phase II). IPSec can use either Authentication Header (AH) or Encapsulating Security Payload (ESP) to secure data, but AH is rarely used, so mbox will only support ESP. Under ESP, there're also two modes to "carry" the packets: transport mode or tunnel mode. Again, since transport mode is rarely used, mbox will only support tunnel mode. Diagram below illustrates a typical ESP packet. In our configuration, we usually call ESP-policy, which consists of a combination of either one of these options:

    • encryption: 3DES, AES, AES-192, AES-256

    • hashing: MD5, MD5-128, SHA, SHA-256, SHA-512

  • IPSec peer. After we've defined IKE and ESP policies (both sides should be the same), we define peering information to each other, particularly the local and remote private networks (some vendors call "interesting" traffic), and also choose the pre-defined IKE and ESP policies (sometime if we have multiple remote peers with different IKE/ESP policies, we need to chose the one that matches each other).


we must configure firewall rules to bypass NAT for VPN traffic

!firewall-snat 10 exempt outbound eth0 src dst remark "bypass NAT for VPN tunnel"firewall-snat 11 exempt outbound eth0 src dst remark "bypass NAT for VPN tunnel"firewall-snat 12 overload outbound eth0 remark "PAT all sources for Internet access"!

this is a sample policy of using IKE version 1, 3DES encryption, MD5 hashing with DH2 key group, with pre-shared key for peer authentication

!ipsec ike-policy 10 authentication psk policy 3DES MD5 2 keylife 3600 version 1!

we can configure multiple policies on one gateway, and map different policies to different remote peers, if needed

!ipsec ike-policy 5 authentication psk policy AES SHA 2 keylife 28800 version 1!

we can also put DH group into IPSec policy (when PFS is enabled)

!ipsec esp-policy 10 policy AES SHA 2 keylife 3600!ipsec peer local ip nated <--- enable "nated" when local peer is behind another NAT device local-net <--- use comma to separate multiple networks here remote-id <--- (optional) configure remote peers physical/private IP, only used if it's behind another NAT device remote-net, <--- use comma to separate multiple networks here policy ike 10 esp 10 psk Letmein99!ipsec start <--- start ipsec service after all is configured!


  1. if there's a change of IKE/IPSec policies config after "policy ike xx esp xx" is applied, we need to re-apply the same command to take in the changes

  2. if there's any change of settings, we need to restart ipsec service "ipsec stop" and "ipsec start" again.

  3. unlike SSLVPN which generates a local route for networks behind remote peers, IPSec VPN by default doesn't do that. Use ping and tcpdump to verify traffic passing through


show ipsec statusshow ipsec peershow ipsec log (see debug output)