In this demo, we will combine senarios of VPN bonding (using OSPF) and Policy-Based Routing (PBR).
VPN bonding (with OSPF) routes packets on a per packet basis across multiple tunnels, packets are routed by routing table with equal paths generated by OSPF (see VPN bonding with OSPF), and PBR supersedes routing table to route packets based on specific source/destination/applications (see PBR detail).
In this demo setup
eth0 of HQ and remote CMG connect to Internet (VPN tunnel tap1 goes through Internet)
eth1 of HQ and remote CMG connect to MPLS (VPN tunnel tap2 goes through MPLS)
eth2 of HQ connect to hosting services (web & ftp)
eth2 of remote connect to user LAN
All web traffic (tcp/80/443) must route through MPLS (via tap2)
All ftp traffic (tcp/20/21) must route through Internet (via tap1)
All other traffic between remote and HQ must be load balanced
VPN bonding (OSPF) with PBR
use "firewall-set" to mark target traffic, on the inbound of eth2 (NOTE: for PBR, we need to mark traffic on the inbound direction the of interface closest to source hosts).
configure PBR on remote CMG for outgoing traffic to HQ (mark destination port)
Configure PBR on HQ CMG for returning traffic to remote (mark source port)
(see attached complete config for both HQ CMG and remote CMG).
When problem happens, use tcpdump to verify packets expected to arrive at each interface.