OpenWRT conditional hostname routing using MWAN3 and ipset

a year ago · 9 min read

Linksys WRT 1900 AC V2

Network plan

My main requirement is to route the internet traffic to specific WAN connection conditionally depending on it's hostname. For example, If someone try to access facebook.com, twitter.com or youtube.com etc. I want those requests to be routed via a specific WAN (or internet router) and all the other request should routed through 2nd internet connection (WAN).

Second requirement was to monitor and collect stats about my home local network usage and persist them for later analysis. One of the motivational factor for collecting usage data was,there are rumors in our country that some Internet Service Providers(ISPs) charge for more data than what the user has actually consumed :)

Below is my network setup:

Network Setup

Diagram sources: lucid & excalidraw

Hardware

I'm using Linksys WRT 1900AC V2 (Cobra) router with OpenWRT latest firmware(v19.07.5). But you can use any OpenWRT compatible device to configure this. There are some OpenWRT like firmwares i:e dd-wrt & 🍅 tomato

OpenWRT Multiple WAN configuration

To begin with , You need to setup ssh on the OpenWRT router and access the router as root user ssh -v [email protected]. Most of the configurations can be done via the web UI but, It's easy to track the changes when configuring via command line, I used to take backups using scp and edit the config files in laptop and push them back to router via scp.

OpenWRT has a mwan3 package which allows you to configure various rules and policies on WAN connections. I have use mwan3 to configure the conditional load balancing/routing to achieve my above requirement.

To install the mwan3 package use

opkg update
opkg install mwan3

After installing mwan3 keep it aside and configure the IPSet, and later we can refer the IPSet names and create mwan3 config file at once.

To configure dynamic hostname base WAN routing you need to do 3 things

create a ipset
```
ipset -N twitter hash:ip
```
in the above twitter is the name of ipset hash:ip is a template kind of thing, When router get it's first ping request. Following command will list all the ip sets
```
ipset list
```
To get information about a specific ip set
```
ipset -L twitter
```
Where twitter is the IP set name, If system has resolved IPs for the name , Then those IPs will be shown under Members parameter
```
root@knnect:~# ipset -L twitter
Name: twitter
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 416
References: 1
Number of entries: 9
Members:
69.195.179.128
69.195.182.128
69.195.165.128
104.244.42.2
104.244.42.194
104.244.42.66
69.195.177.128
209.237.193.128
104.244.42.130
```
Important thing to note here is, When you run above command ip set will be added to the memory, and they don't get persisted. Hence restarting the router will clear all the added ipset. In other words ipset rules are saved in memory, upon a reboot they will be lost. Because of that I have created a startup script (source code)
```
/etc/init.d/autoipset
```
to add all the ip sets I want when starting up the router. Instructions are given in above gist source on how to add the startup script. For more info about how to write init script in OpenWRT follow this official doc

Update dnsmasq.conf with ipset name and dns name

What is dnsmasq

Open the

/etc/dnsmasq.conf

in router, and add ipset entries as below

ipset=/.twitter.com/twitter

here, AFAIK twitter after the last slash is the ipset name, Following is a sample dnsmasq.conf configuration content

# /etc/dnsmasq.conf
# Change the following lines if you want dnsmasq to serve SRV
# records.
# You may add multiple srv-host lines.
# The fields are <name>,<target>,<port>,<priority>,<weight>

# A SRV record sending LDAP for the example.com domain to
# ldapserver.example.com port 289
#srv-host=_ldap._tcp.example.com,ldapserver.example.com,389

ipset=/.facebook.com/facebook
ipset=/.instagram.com/instagram
ipset=/.fbcdn.net/fbcdn
ipset=/.twitter.com/twitter
ipset=/.twimg.com/twimg

after updating the config file run

service dnsmasq restart

to refresh the dnsmasq Note here: In my case default dnsmasq tool coming with the OpenWRT installation does not worked and I had to uninstall the pre-installed dnsmasq and install dnsmasq-full to get this work!

Now IP sets are ready, So final thing is to configure the mwan3

If you are new to mwan3 there is a really good documentation in OpenWRT docs,covering all the aspects.

At this point, better to restart the router, and ping to those host name that you have configured above i:e
```
    ping twitter.com
```
If you are curious about IPs and DNS, When you ping you will get one of following IPs at the time you execute the ping command
```
whois -h whois.radb.net -- '-i origin AS13414' | grep ^route
```
If you wonder how it resolve all the IPs , Google search about ASN Lookup, I also initially though that how to find all the IPs belongs to a particular domain and i might need to add them to routing table or something like that, But it's not required here. ok that is for your information about AS codes and IPs , now run the
```
    ipset -L twitter
```
command and check whether the member IP have resolved Configuring the mwan3 can be done via the UI too. But i will put the configuration elements here for convenient. I assume that you are somewhat familiar with OpenWRT and mwan3 configurations, Hence will directly jump into mwan3 config. mwan3 config is located in
```
/etc/config/mwan3
```
and following is a sample configuration
```
    config rule 'facebook'
        option proto 'all'
        option family 'ipv4'
        option sticky '1'
        option ipset 'facebook'
        option use_policy 'testp'
```
In above rule config, option sticky: Is important to set true in UI or '1' in the config option ipset: Is the IP set name created with ipset -N {name} and the ipset={domain}/{name} in the config file option use_policy: is the policy name which have the WAN as a member that you want to forward the request match with the above ipset (i:e facebook.com requests go to Dialog 4G WAN) Change the above config accordingly and put it in the mwan3 config at the top (will get the priority from configs in top to bottom) Finally restart the mwan3 service
```
mwan3 restart
```
and check the mwan3 status
```
mwan3 status
```
or check the status from the web UI it should show the above added new rul in Active ipv4 user rules: section

Verifying the changes

Router

IPset members

ipset -L {name}

run the above command and check whether any member IPs have resolved

mwan3 configs

mwan3 status

run above command and check whether the newly added rule is listed in the active ipv4 user rules section

Check dnsmasq configuration syntax using below command

dnsmasq --test

External device (Laptop)

traceroute facebook.com

execute the above command and check from which WAN connection or hope the request been routed, Is it routing via the expected WAN connection

Monitoring traffic

This is about monitoring the home network traffic using the OpenWRT, This is a different topic than the above discussed topic.

I have used luciappstatistics package in OpenWRT, It provide good coverage of parameters, from Network interfaces, to memory & CPU usages and there are lot more supported in collectd. This package gives more similar graph outputs this is available in LERAN traffic graphs. So i believe this luci_app_statistics package give more professional looking stats in OpenWRT routers.

The default rrd configuration store the monitoring data in the temporary directory (/tmp/) hence the graph data get cleared when router is restarted. In the above doc , they have pointed to some scripts and docs which explains how to backup the data to an external device plugged into router's USB interface. Only thing lacking there is how to use existing vFat/Fat32 formatted pen drive to backup the data. So I had to google and collect the information from various sources, and though of organizing them here. To get vFat/FAT32 partition mounted you need following package in the router

opkg install dosfstools

And then simply follow the instruction in usb-installing and usb-drives To take a backup every X (i:e 6 hours) interval, You can use this script This sheduling is done using the corntabs

Schedule WiFi and switch WAN usages timely

To edit the crontab type

crontab -e

and following scheduled commands will switch off the wifi radios at mid night and switch on at 5AM morning. And also it will switch the MWAN3 policies at mid night and restore back at 8AM morning(That's where peak time start for SLT connections)

0 */6 * * * /etc/init.d/rrdbackup backup
0 8 * * * uci set mwan3.all.use_policy='Dialog4G_Only'; uci commit; mwan3 restart
0 0 * * * uci set mwan3.all.use_policy='SLT_Only'; uci commit; mwan3 restart
0 0 * * * /sbin/wifi down
0 5 * * * /sbin/wifi up

References

This Openwrt forum thread healped me a lot to figure out this path.
And this StackOverflow answer helps to find out about ipset list command and important of restarting and setting up the order for startup script (set ips before 19)
This blog post in nicedoc.io was also helpful.
I created this github repo with all the configuration files and script for reference