Dec 15

Context

My Internet Service Provider changes my externally-visible IP Address from time to time without notice. I am having issues with the ISP’s service and I decided to keep a watch on my Internet connectivity unsing external monitoring services. As I work on refining my connectivity monitoring infrastructure, I detect anomalies which need watching. One of these anomalies is the ISP-induced IP address change frequency.

In this article I present a simple Unix script which keeps track of my changing external IP addresses so that I can look and see what they were at different points in time and when they were changed, to within 2 minutes. This script is ISP Router-specific. It applies to Sagemcom F@ST3864 provided by Optus.

Script

I have a pfSense firewall (https://www.pfsense.org/), which is based on the FreeBSD OS. The script being discussed runs on this firewall.

The purpose of the script is to discover the externally-visible host name of my externally-visible host, compare it to the last IP address that was scraped from the ISP-provided Router’s management UI and, if they are different, update the last scraped IP Address and log a change to the system log and to the private log created for the purpose.

Set prerequisites (manually create myip_last.log for the first time):

mkdir -pv ~/ipaddresses
# find externally-visible IP address of the gateway
gatewayIP=xxx.xxx.xxx.xxx
export CURLOPT_USERPWD="routeruiusername:routeruipassword"
extip=$(curl --user ${CURLOPT_USERPWD} -ss http://${gatewayIP}/info.html | grep -A1 'WAN IPv4 Address:' | tail -n 1 | sed 's|^[ ].*<td>||;s|</td>$||')
echo "$(date -jR) - ${extip}" > ~/ipaddresses/extip_last.log

Create the script:

> ~/extip_log_change.sh
chmod u+x ~/extip_log_change.sh

cat <<-'EODECK' > ~/extip_log_change.sh

#!/bin/sh
gatewayIP=xxx.xxx.xxx.xxx
export CURLOPT_USERPWD="routeruiusername:routeruipassword"
extip=$(curl --user ${CURLOPT_USERPWD} -ss http://${gatewayIP}/info.html | grep -A1 'WAN IPv4 Address:' | tail -n 1 | sed 's|^[ ].*<td>||;s|</td>$||')
oldip=$(cat ~/ipaddresses/extip_last.log|cut -d'-' -f2|tr -d ' ')

if [ $extip != $oldip ]; then
    echo "$(date -jR) - ${extip}" > ~/ipaddresses/extip_last.log
    cat ~/ipaddresses/extip_last.log >> ~/ipaddresses/extip.log
    logger -p user.crit "IP Address change (at GW): $(cat ~/ipaddresses/extip_last.log)"
fi

EODECK

Add this script to crontab and run it every minute

crontab -e
*/1   *    *    *    *     /root/extip_log_change.sh 2>/root/ipaddresses/extip_log_change.err

Give it a minute and see what the logs say. Likely they will say nothing new unless it so happened that the IP address was changed between the time you first logged it manually and the time you run the script.

When you detect connectivity issues based on notifications from the monitors check the logs.

cat ~/ipaddresses/extip_last.log

cat ~/ipaddresses/extip.log

Summary

In this article I present a simple Unix script which keeps track of my changing external IP addresses so that I can see what they were at different points in time and when they were changed. This script is ISP Router-specific. It applies to Sagemcom F@ST3864 provided by Optus.

Dec 15

Context

My Internet Service Provider changes my externally-visible IP Address from time to time without notice. I am having issues with the ISP’s service and I decided to keep a watch on my Internet connectivity unsing external monitoring services. Since I donlt have a static IP address visible to the Internet I needed to work out a way to make sure that the monitoring configuration gets to use the current IP address of my monitoring endpoint despite the ISP changing the IP address. I use Dynamic DNS service. More on this in a subsequent article.

In this article I present a simple Unix script which keeps track of my changing external IP addresses so that I can see what they were at different points in time and when they were changed. This script attempts to resolve the domain name of the host to its I address as means of obtaining the IP address which is visible form the Internet.

Script

I have a pfSense firewall (https://www.pfsense.org/), which is based on the FreeBSD OS. The script being discussed runs on this firewall.

The purpose of the script is to resolve the externally-visible host name of my externally-visible host, compare it to the last IP address that was resolved and, if they are different, update the last resolved IP Address and log a change to the system log and the private log created for the purpose.

Set prerequisites (manually create myip_last.log for the first time):

mkdir -pv ~/ipaddresses
# seed for the first time
myip=$(nslookup somehostname.dynu.net 208.67.222.222 | grep Address | tail -n 1 | cut -c10-)
echo "$(date -jR) - ${myip}" > ~/ipaddresses/myip_last.log

Create the script:

> ~/log_ip_change.sh # create empoty script file
chmod u+x ~/log_ip_change.sh # set permissions to allow script execution

cat <<-'EODECK' > ~/log_ip_change.sh # create the script via a HERE-document
#!/bin/sh
myip=$(nslookup somehostname.dynu.net 208.67.222.222 | grep Address | tail -n 1 | cut -c10-)
olip=$(cat ~/ipaddresses/myip_last.log|cut -d'-' -f2|tr -d ' ')

if [ $myip != $olip ]; then
    echo "$(date -jR) - ${myip}" > ~/ipaddresses/myip_last.log
    cat ~/ipaddresses/myip_last.log >> ~/ipaddresses/myip.log
    logger -p user.crit "IP Address change: $(cat ~/ipaddresses/myip_last.log)"
fi

EODECK

Add this script to crontab and run it every minute

crontab -eAdd the following line
*/1   *    *    *    *     /root/log_ip_change.sh 2>/root/ipaddresses/log_ip_change.err

Give it 1 minute and see what the logs say

cat ~/ipaddresses/myip_last.log
cat ~/ipaddresses/myip.log

Summary

In this article I present a script which watches and records my externally-visible IP addresses, as changed bymy ISP form time to time, by resolving my externally-visible host name to the IP address – call me curious 🙂

Dec 15

Context

I have been migrated to the Australia’s national broadband network (nbn). Soon after the migration I begun experiencing Internet connectivity issues which my old cable internet service did not have. I got fed up with waking up to find that my ISP-provided nbn router lost connectivity overnight and never regained it, or that it did this in the middle of the day when I needed to work, or when I was not at home and my family was inconvenienced. I decided to figure out a way to “monitor” my Internet connection and get alerted when it went bad. This is so that I had an opportunity to investigate and reboot the router, which was the only way to deal with the issue most of the time.

In this article I share my view on the four connectivity monitors which I named in the previous article and used over a period of time.

Free Internet-based monitoring services I used

(Free service) UptimeRobot (https://uptimerobot.com/)

An UptimeRobot monitor can be configured with the URL of the “responder” and the email address(es) to which to send notification emails. UptimeRobot tests connectivity every 5 minutes in the free service.

One issue I detected with the UptimeRobot is that is occasionally uses outdated cached IP address of the target host, which may have changed and has been updated in the Dynamic DNS service. On these occasions it does not recognise that the host is visible from the Internet because its IP address has changed.

It was the first I chose. I liked the dashboard and notification emails. I want to give them time to sort out the DDNS caching issue. When I complained about the DDNS cachin issue I got a response from the support folks within a day, which for a free service is pretty good in my book.

DDNS caching is still an issue but for the time being I am sticking with this service.

(Free service) StatusCake (https://www.statuscake.com/)

A StatusCake monitor can be configured with the URL of the “responder” and the email addresses to which to send notification emails, and a mobile phone number to which to send text messages. StatusCake tests connectivity every 5 minutes in the free service.

I signed up to this free service because of the IP address caching issue with the UptimeRobot.

Ultimately I found StatusCake somewhat annoying.

Notification emails contain embedded icons which may look nice but add no value and are “broken” if the email client prevents automatic embedded URL following.

Notification emails do not have date/time when “service down” was detected or “service up” was detected. The outage duration is clearly shown in the “service is up” notification, though, which is nice.

In the end there was not enough of an advantage over others to keep it. I deleted the service and the account.

(Free service) Port Monitor (https://www.port-monitor.com)

The free service from Port Monitor can be used to tests connectivity every 60+ seconds. This is more frequent than any of the other monitoring services I tried, where the most frequent test one can configure in the free service is 5 minutes, and is obviously more responsive in detecting “service down” and “service up” events.

The port monitor email notifications are pretty good except they do not provide a pre-calculated “service has been down for x minutes y second” data but rather shows the date/time the service detected that the connectivity was down and date/time the connectivity was up again. It could be that the paid service provides that information in notification emails. I know, it should be no great thing to work out the outage duration from the dates/times but it is extra work and all other services I tried provide this piece of information.

Still, if you want to know roughly to a minute that your service is down and up the port monitor will provide this service.

I am keeping the port monitor watch for the time being.

(“Free for now” service) Monitoshi (http://monitoshi.com/)

Monitoshi offers a service which may cease to be free at some point in time. It does not have a status UI and the monitors are enabled/disabled via API URLs.

I found email notifications to be pretty Spartan. No date/time for when the service down was detected or when the service up was detected, let alone how long the service was down. One could work these out from the email dates/times but that introduces inaccuracies and is more work than I am willing to undertake given that all other free services I tried do this already, and provide management UI and dashboards to boot.

I deleted the monitor I set up.

Summary

I have been running free connectivity monitoring services for a while. Of the 4 I used I am sticking with 2 and I ditched 2 others. The reasons are given in this article.

Dec 03

Context

I have been migrated to the Australia’s national broadband network (nbn). Soon after the migration I begun experiencing Internet connectivity issues which my old cable internet service did not have. I got fed up with waking up to find that my ISP-provided nbn router lost connectivity overnight and never regained it, or that it did this in the middle of the day when I needed to work, or when I was not at home and my family was inconvenienced. I decided to figure out a way to “monitor” my Internet connection and get alerted when it went bad. This is so that I had an opportunity to investigate and reboot the router, which was the only way to deal with the issue most of the time.

In this article I outline how I set up connectivity monitoring for testing Internet connectivity of my internal network from the “outside”, that is an Internet-based monitors testing connectivity to my infrastructure.

It seems perhaps strange that I would want to test my Internet connectivity from the outside as the first thing I wanted to do. The reason is simple. If I set up an internal monitoring mechanism then I couldn’t get email notifications about connectivity issues – there would have been no connectivity, right?

Need Internet-based monitoring service

First, one has to find an Internet-based monitoring service which to use; free if possible, cheap if no satisfactory free service is available.

A bit of research lead me to the UptimeRobot – https://uptimerobot.com/. This organisation offers a free service with some limitations which did not matter to me as what I needed to do did not require their premium service.

I signed up and set up a single monitor which would “tickle” my infrastructure every 5 minutes and email me when it failed to get a response and then again when it succeeded in getting a response after an interruption.

Since my mobile device can receive emails regardless of whether my internal network has Internet connectivity I started getting notifications about outages within 5 minutes of outages being detected and if I did not resolve the issue myself and the issue “went away” I started getting notification within 5 minutes of the connectivity being restored.

How I set up the “tickle” “responder” is discussed next.

Need Internet-visible “Responder”

For the external connectivity monitor to work I had to have a “responder” which was visible from the Internet, responded to queries and was of the kind that a monitor service could deal with.

UptimeRobot offers monitors which:

  • connect to a HTTP(s) URL
  • connect to a HTTP(s) URL and do look for a keyword
  • connect to a specific port on a specific host
  • Ping a specific host (presumably using an ICMP Packet but I was not motivated to look into this closely since the Ping request failed when I tested, presumably having been blocked by the ISP or the router)

While my ISP-provided router has web-based UI this UI is not accessible from the Internet, that I can detect, and it offers no other Internet-visible HTTP(s) endpoints to which for the monitor to connect.

I have a small PC, with the address in the ISP router’s DMZ, on which I created a HTTP responder and exposed it to the Internet via a port forwarding rule on the ISP’s router. The HTTP connection request from the monitor will get a response if the router is accessible from the Internet, meaning that my internal network can connect to the Internet. That’s it.

Later on I configured another UptimeRobot monitor, which simply needed a listener on a TCP port, whether that listener did anything or not. This eliminated the need to run a web server, however minimalist.

Configure a Monitor

A server or URL monitoring service can be used to monitor connectivity and notify of service interruptions and resumptions once the “responder” service is running and can be “tickled” from the Internet.

UptimeRobot (https://uptimerobot.com)

An UptimeRobot monitor can be configured with the URL of the “responder” and the email address(es) to which to send notification emails. UptimeRobot tests connectivity every 5 minutes in the free service.

One issue I detected with the UptimRobot is that is occasionally uses outdated cached IP address of the target host, which may have changed and has been updated in the Dynamic DNS service, thus it does not recognise for a while that the host is visible form the Internet because its IP address has changed.

StatusCake (https://www.statuscake.com/)

A StatusCake monitor can be configured with the URL of the “responder” and the email addresses to which to send notification emails, and a mobile phone number to which to send text messages. StatusCake tests connectivity every 5 minutes in the free service.

I recently signed up to the free service because of the IP address caching issue with the UptimeRobot and I am watching how the StatusCake behaves and what it produces.

Port Monitor (https://www.port-monitor.com)

A Port Monitor monitor can be configured with the URL of the “responder” and the email address(es) to which to send notification emails. Port Monitor tests connectivity every 60+ seconds in the free service.

Monitoshi (http://monitoshi.com/)

Monitoshi offers a service which may cease to be free at some point in time. It does not have a status UI and the monitors are enabled/disabled via API URLs.

Issues

A couple of issues make this solution less than perfect.

The first issue is that of dynamic IP addresses. If the externally visible IP address of the server, configured in the monitor definition, changes then the monitor will report downtime regardless of whether the physical server is up and accessible. This necessitates manual change of the I address in all monitors or the use of a Dynamic DNS service and some mechanism to detect the address change and update the Dynamic DNS.

Summary

To test internal-to-external connectivity, and receive notification of connectivity issues an external monitoring service is needed. This service needs to be able to connect to an internal responder. The internal responder must be “visible” from the Internet. Once both are configured, one gets notified when the connectivity disruption is detected and when it is restored.

preload preload preload