Fortigate SDWAN with IPSec Site-to-Site VPN and multiple subnets at both sites

Recently, I was tasked with setting up a Site-to-Site VPN between a Fortigate FG100-F in a High Availability (HA) configuration and a FG60-F. Both sites featured dual WAN links configured in an SD-WAN setup, along with multiple VLANs.

I needed to setup multiple VLANs at the branch office Fortigate. I used the VLAN switch feature to create VLANs as sub-interfaces, and then added them to a zone to help with inter-vlan traffic, and for easier firewall policy configuration

To completed the site-to-site VPN links despite consulting the Fortigate documentation, I couldn't find an example that exactly matched this scenario.

After some extensive searching, I found a YouTube video that demonstrated this particular setup. You can watch it here.

YouTube Video Summary

The video, although in Spanish, provides a detailed walkthrough of setting up a Site-to-Site VPN between Fortigate devices in a similar environment. It covers the configuration of dual WAN links, SD-WAN, . Thanks to YouTube's caption feature, I was able to follow along and apply the configurations needed for my setup.

 The video demonstrates how to combine IPsec tunnels with SD-WAN to balance traffic across multiple internet providers. It includes configuring two IPsec tunnels between SP1 and SP2 interfaces of two Fortigates and setting up SD-WAN to use these Fortigates for health checks.

Network Configuration:

1. Interface Configuration: Assign ISP1 and ISP2 to the "wan internet" zone with their respective gateways.
2. Routing and Policy Setup: A default route and an internet policy are created for outgoing traffic on LAN

IPsec Tunnel Configuration:

1. Two IPsec tunnels are configured between the Fortigates with identical settings.
2. SD-WAN zones are created, and the tunnels are added to these zones.

Loopback Interface and Static Routing:

1. Create a loopback interface on both Fortigates for monitoring.
2. Set up static routes using these interfaces.

Policy Configuration and SD-WAN Optimization:

1. Policies are created to route traffic based on the loopback interface.
2. SD-WAN rules are configured to route traffic through specific IPsec tunnels based on latency.
3. The video demonstrates how latency can trigger a switch between tunnels.

This video is an excellent resource for setting up IPsec with SD-WAN on Fortigate devices, especially in scenarios requiring dynamic traffic management based on link performance.

For more details, you can watch the video here.

Troubleshooting the Ping Issue

One problem I encountered was that after bringing the VPN links up ,ping was working from the Branch to the HQ, it wasn't working the other way around. To diagnose this, I used the following Fortigate CLI commands:

di de reset
diagnose debug flow filter addr <src-addr>
di de flow filter proto 1
diagnose debug flow show function enable
diagnose debug console timestamp enable
diagnose debug flow show iprope enable
diagnose debug flow trace start 30
diagnose debug enable

The issue was traced back to the SD-WAN policy rules at the HQ. The policy was configured to 
route all traffic through the WAN links, completely bypassing the VPN interfaces. I had to modify
 the policy to include more specific destinations, which allowed outgoing VPN traffic to be routed 
 correctly through the VPN interfaces.
This setup, while challenging, was a great learning experience, and the YouTube video was 
instrumental in helping me navigate the configuration process.