ECMP (Equal Cost Multi Path) is one of the other technique to distribute the workload across two or more network link in order to maximize throughput and minimize response time. ECMP use round robin per SRC/DST address combination to distribute traffice to muliple network link.
With ECMP you will have multiple gateway to the same destination address. Normally all reachable nexthops are copied to FIB(Forwarding Information Base) to forward packets. As a result of FIB table, connection with the same source address, destination address, source interface, routing mark and Tos are sent to the same gateway. That's why ECMP is a per-connection load balancing.
Ex: You have connection with 2 ISP (Internet Service Provider) as a diagram below. We got the same bandwidth to both ISP. In this configuration we will share 50% to ISP A and 50% to ISP B.
1. Setup IP address
/ ip address add address=192.168.0.1/24 network=192.168.0.0 interface=LAN add address=10.0.0.2/30 network=10.0.0.0 interface=wlan1 add address=220.127.116.11/30 network=18.104.22.168 interface=wlan2
2. Configure NAT
Because we have customer LAN, using private IP address. So we need to use NAT to translate private IP of customer LAN to public IP.
/ ip firewall nat add chain=srcnat out-interface=wlan1 action=masquerade add chain=srcnat out-interface=wlan2 action=masquerade
3. Configure Route
Because both ISP have the same bandwidth, so we configure share traffice 1:1. It's mean 50% to ISP A and 50% to ISP B
/ ip route add dst-address=0.0.0.0/0 gateway=10.0.0.1,22.214.171.124 check-gateway=ping
Some issue on ECMP
In the case your client have connection to ISP B while your client use ISP A DNS server, your client will not able to resolve domain name. Because ISP they will allow only their customer able to send DNS query to their DNS server. To resolve this issue you should use third party DNS server like 126.96.36.199 or 188.8.131.52
Routing Table Flashing
When router flush the routing table, it also flush the FIB cache. Then the connection will assign to gateway once again. In this case the traffice may or may not distribute to the same gateway as before. As in the above scenario you do masquerade, change of the gateway will result in change of the packet's source address and connection will be dropped.