Internet Protocol Security (IPsec) is a very complex protocol suite for secure Internet Protocol (IP) communications, by building secure tunnels/sessions to authenticate and encrypt each IP packet passing between two hosts or gateways. 

IPsec includes protocols for establishing mutual authentication between agents at the beginning of the session and negotiation of cryptographic keys to be used during the session. An IPSec tunnel consists of 3 Security Associations (SA). An SA defines a bundle of algorithms and parameters (such as keys) that is being used to encrypt and authenticate a particular flow in one direction. Therefore, in normal bi-directional traffic, the flows are secured by a pair of security associations.

There're a lot of details we can google about IPSec. In summary, to establish an IPSec tunnel, we configure three sets of policies:

• IKE policy (as in IPSec Phase I). This phase basically authenticates peers first then negotiate a bundle of encryption/hashing algorithms to generate session key, which is used by Phase II to encrypt packets. One peer (host or gateway) can be configured with multiple set/combination of policies, and when two peers are initiating a tunnel, they will agree on a matched policy. If no matching policy between two peers, tunnel will not come up. Each IKE policy consists of a combination of either one of these parameters. Supported parameters are listed below:

  • authentication: pre-shared key or RSA-sig

  • encryption: 3DES, AES, AES-192, AES-256

  • hashing: MD5, MD5-128, SHA, SHA-256, SHA-512

  • version: IKEv1 or IKEv2

• ESP policy (as in IPSec Phase II). IPSec can use either Authentication Header (AH) or Encapsulating Security Payload (ESP) to secure data, but AH is rarely used, so mbox will only support ESP. Under ESP, there're also two modes to "carry" the packets: transport mode or tunnel mode. Again, since transport mode is rarely used, mbox will only support tunnel mode. Diagram below illustrates a typical ESP packet. In our configuration, we usually call ESP-policy, which consists of a combination of either one of these options:

  • encryption: 3DES, AES, AES-192, AES-256

  • hashing: MD5, MD5-128, SHA, SHA-256, SHA-512

• IPSec peer. After we've defined IKE and ESP policies (both sides should be the same), we define peering information to each other, particularly the local and remote private networks (some vendors call "interesting" traffic), and also choose the pre-defined IKE and ESP policies (sometime if we have multiple remote peers with different IKE/ESP policies, we need to chose the one that matches each other).

SAMPLE CONFIG

we must configure firewall rules to bypass NAT for VPN traffic

this is a sample policy of using IKE version 1, 3DES encryption, MD5 hashing with DH2 key group, with pre-shared key for peer authentication

we can configure multiple policies on one gateway, and map different policies to different remote peers, if needed

we can also put DH group into IPSec policy (when PFS is enabled)

NOTES: 

1. if there's a change of IKE/IPSec policies config after "policy ike xx esp xx" is applied, we need to re-apply the same command to take in the changes

2. if there's any change of settings, we need to restart ipsec service "ipsec stop" and "ipsec start" again.

3. unlike SSLVPN which generates a local route for networks behind remote peers, IPSec VPN by default doesn't do that. Use ping and tcpdump to verify traffic passing through

TROUBLESHOOTING: