Skip to main content

Automatically rebooting piaware and prevent outage (avoid "not feeding ADS-B to FlightAware" emails)

A few months ago, I setup a Raspberry Pi 2 at my house in Pune to monitor the ADS-B signals from aircrafts passing overhead.
I referred the build process outlined here to build my own PiAware setup.



The problem is that roughly twice a week, I used to get this mail saying that my feeder was no longer feeding data to flighaware.

Barring a few rare instances when the trouble was due to internet outage, most of the time it required me to reboot the Raspberry Pi. Eventually I got tired, like a few other users here. Apparently it is due to WiFi dongle hanging or losing connection to the WiFi Router. To get around that, I wrote a script and set it up as scheduled task that runs every 5 minutes. The script checks if the RPi is able to ping Google's Public DNS servers, if yes, then it exits. If the ping fails, it waits for 10 seconds, tries again to ping again and if it fails this time, it issues a reboot command. In both cases, it loges results to a text file (saved on SD card) before exiting. I could then analyse how many times the RPi rebooted later on.

Here is the script (Saved as /home/pi/monitor_gateway.sh):



 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
# Send two Ping request packets and 4 seconds timeout
Logfile=/home/pi/pingtest.log
 
return_value=$(ping -c 2 -W 4 8.8.8.8)
return_value=$?
# Return code for ping are taken from
# ./OpenWrt-SDK-ar71xx-for-linux-i486-gcc-4.6-linaro_uClibc-0.9.33.2/staging_dir/toolchain-mips_r2_gcc-4.6-linaro_uClibc-0.9.33.2/include/sysexits.h
 
case $return_value in
	0)  #Ping reply received
		logger "Ping result: ONLINE"
		TZ="Asia/Kolkata" date | xargs echo -n >> $Logfile
		echo " Ping result: ONLINE" >> $Logfile
		;;
	1)  #Ping sent but reply was not received
		logger "Ping result: OFFLINE (Ping reply not received), Retrying after 10 seconds"
		TZ="Asia/Kolkata" date | xargs echo -n >> $Logfile
		echo " Ping result: OFFLINE (Ping reply not received), Retrying after 10 seconds" >> $Logfile
		sleep 10
		return_value2=$(ping -c 2 -W 4 8.8.8.8)
		return_value2=$?
		case $return_value2 in
			0)  #Ping reply received
				logger "Ping result: ONLINE"
				TZ="Asia/Kolkata" date | xargs echo -n >> $Logfile
				echo " Ping result: ONLINE" >> $Logfile
				;;
			1)  #Ping sent but reply was not received
				logger "Ping result: OFFLINE (Ping reply not received), Rebooting..."
				TZ="Asia/Kolkata" date | xargs echo -n >> $Logfile
				echo " Ping result: OFFLINE (Ping reply not received), Rebooting..." >> $Logfile
				reboot
				;;
			*)  #Error
				logger "Ping result: OFFLINE (ping return code: $return_value2), Rebooting..."
				TZ="Asia/Kolkata" date | xargs echo -n >> $Logfile
				echo " Ping result: OFFLINE (ping return code: $return_value2), Rebooting..." >> $Logfile
				reboot
				;;
		esac
		;;
	*)  #Error
		logger "Ping result: OFFLINE (Ping reply not received), Retrying after 10 seconds"
		TZ="Asia/Kolkata" date | xargs echo -n >> $Logfile
		echo " Ping result: OFFLINE (Ping reply not received), Retrying after 10 seconds" >> $Logfile
		sleep 10
		return_value2=$(ping -c 2 -W 4 8.8.8.8)
		return_value2=$?
		case $return_value2 in
			0)  #Ping reply received
				logger "Ping result: ONLINE"
				TZ="Asia/Kolkata" date | xargs echo -n >> $Logfile
				echo " Ping result: ONLINE" >> $Logfile
				;;
			1)  #Ping sent but reply was not received
				logger "Ping result: OFFLINE (Ping reply not received), Rebooting..."
				TZ="Asia/Kolkata" date | xargs echo -n >> $Logfile
				echo " Ping result: OFFLINE (Ping reply not received), Rebooting..." >> $Logfile
				reboot
				;;
			*)  #Error
				logger "Ping result: OFFLINE (ping return code: $return_value2), Rebooting..."
				TZ="Asia/Kolkata" date | xargs echo -n >> $Logfile
				echo " Ping result: OFFLINE (ping return code: $return_value2), Rebooting..." >> $Logfile
				reboot
				;;
		esac
		;;
esac


Here is how my crontab looks like:

You can edit your crontab by issuing the command (on the RPi): sudo crontab -e
I also set my RPi2 to reboot once everyday at 12:00 AM just in case the internet connection
stays alive but dump1090 itself crashes. We could write a script to detect that case as well, but this
preemptive strike seems easy. 


And just for reference, here is what my log file looks like after a day of using the script:
(The Raspberry Pi did reboot a couple of times.)



pi@rpi-lithiumhead-piaware ~ $ cat pingtest.log
Fri Nov 4 10:43:13 IST 2016 Ping result: ONLINE
Fri Nov 4 10:50:02 IST 2016 Ping result: ONLINE
Fri Nov 4 11:00:03 IST 2016 Ping result: ONLINE
Fri Nov 4 11:10:02 IST 2016 Ping result: ONLINE
Fri Nov 4 11:20:02 IST 2016 Ping result: ONLINE
Fri Nov 4 11:30:02 IST 2016 Ping result: ONLINE
Fri Nov 4 11:40:02 IST 2016 Ping result: ONLINE
Fri Nov 4 11:50:02 IST 2016 Ping result: ONLINE
Fri Nov 4 12:00:02 IST 2016 Ping result: ONLINE
Fri Nov 4 12:10:02 IST 2016 Ping result: ONLINE
Fri Nov 4 12:20:02 IST 2016 Ping result: ONLINE
Fri Nov 4 12:30:03 IST 2016 Ping result: ONLINE
Fri Nov 4 12:40:03 IST 2016 Ping result: ONLINE
Fri Nov 4 12:50:02 IST 2016 Ping result: ONLINE
Fri Nov 4 13:00:02 IST 2016 Ping result: ONLINE
Fri Nov 4 13:10:02 IST 2016 Ping result: ONLINE
Fri Nov 4 13:20:02 IST 2016 Ping result: ONLINE
Fri Nov 4 13:30:02 IST 2016 Ping result: ONLINE
Fri Nov 4 13:40:02 IST 2016 Ping result: ONLINE
Fri Nov 4 13:50:02 IST 2016 Ping result: ONLINE
Fri Nov 4 14:00:02 IST 2016 Ping result: ONLINE
Fri Nov 4 14:10:02 IST 2016 Ping result: ONLINE
Fri Nov 4 14:20:02 IST 2016 Ping result: ONLINE
Fri Nov 4 14:30:02 IST 2016 Ping result: ONLINE
Fri Nov 4 14:40:02 IST 2016 Ping result: ONLINE
Fri Nov 4 14:50:02 IST 2016 Ping result: ONLINE
Fri Nov 4 15:00:02 IST 2016 Ping result: ONLINE
Fri Nov 4 15:10:02 IST 2016 Ping result: ONLINE
Fri Nov 4 15:20:03 IST 2016 Ping result: ONLINE
Fri Nov 4 15:30:02 IST 2016 Ping result: ONLINE
Fri Nov 4 15:40:02 IST 2016 Ping result: ONLINE
Fri Nov 4 15:50:02 IST 2016 Ping result: ONLINE
Fri Nov 4 16:00:03 IST 2016 Ping result: ONLINE
Fri Nov 4 16:10:04 IST 2016 Ping result: ONLINE
Fri Nov 4 16:20:02 IST 2016 Ping result: ONLINE
Fri Nov 4 16:30:02 IST 2016 Ping result: ONLINE
Fri Nov 4 16:40:02 IST 2016 Ping result: ONLINE
Fri Nov 4 16:50:02 IST 2016 Ping result: ONLINE
Fri Nov 4 17:00:02 IST 2016 Ping result: ONLINE
Fri Nov 4 17:10:02 IST 2016 Ping result: ONLINE
Fri Nov 4 17:20:02 IST 2016 Ping result: ONLINE
Fri Nov 4 17:30:03 IST 2016 Ping result: ONLINE
Fri Nov 4 17:40:02 IST 2016 Ping result: ONLINE
Fri Nov 4 17:50:02 IST 2016 Ping result: ONLINE
Fri Nov 4 18:00:02 IST 2016 Ping result: ONLINE
Fri Nov 4 18:10:03 IST 2016 Ping result: ONLINE
Fri Nov 4 18:20:02 IST 2016 Ping result: ONLINE
Fri Nov 4 18:30:02 IST 2016 Ping result: ONLINE
Fri Nov 4 18:40:02 IST 2016 Ping result: ONLINE
Fri Nov 4 18:50:03 IST 2016 Ping result: ONLINE
Fri Nov 4 19:00:02 IST 2016 Ping result: ONLINE
Fri Nov 4 19:10:02 IST 2016 Ping result: ONLINE
Fri Nov 4 19:20:02 IST 2016 Ping result: ONLINE
Fri Nov 4 19:30:02 IST 2016 Ping result: ONLINE
Fri Nov 4 19:40:03 IST 2016 Ping result: ONLINE
Fri Nov 4 19:50:02 IST 2016 Ping result: ONLINE
Fri Nov 4 20:00:02 IST 2016 Ping result: ONLINE
Fri Nov 4 20:10:02 IST 2016 Ping result: ONLINE
Fri Nov 4 20:20:03 IST 2016 Ping result: ONLINE
Fri Nov 4 20:30:02 IST 2016 Ping result: ONLINE
Fri Nov 4 20:40:02 IST 2016 Ping result: ONLINE
Fri Nov 4 20:50:02 IST 2016 Ping result: ONLINE
Fri Nov 4 21:00:03 IST 2016 Ping result: ONLINE
Fri Nov 4 21:10:06 IST 2016 Ping result: OFFLINE (Ping reply not received), Retrying after 10 seconds
Fri Nov 4 21:10:17 IST 2016 Ping result: ONLINE
Fri Nov 4 21:20:02 IST 2016 Ping result: ONLINE
Fri Nov 4 21:30:02 IST 2016 Ping result: ONLINE
Fri Nov 4 21:40:02 IST 2016 Ping result: ONLINE
Fri Nov 4 21:50:02 IST 2016 Ping result: ONLINE
Fri Nov 4 22:00:02 IST 2016 Ping result: ONLINE
Fri Nov 4 22:10:02 IST 2016 Ping result: ONLINE
Fri Nov 4 22:20:03 IST 2016 Ping result: ONLINE
Fri Nov 4 22:30:02 IST 2016 Ping result: ONLINE
Fri Nov 4 22:40:02 IST 2016 Ping result: ONLINE
Fri Nov 4 22:50:02 IST 2016 Ping result: ONLINE
Fri Nov 4 23:00:02 IST 2016 Ping result: ONLINE
Fri Nov 4 23:10:02 IST 2016 Ping result: ONLINE
Fri Nov 4 23:20:02 IST 2016 Ping result: ONLINE
Fri Nov 4 23:30:02 IST 2016 Ping result: ONLINE
Fri Nov 4 23:40:03 IST 2016 Ping result: ONLINE
Fri Nov 4 23:50:02 IST 2016 Ping result: ONLINE
Sat Nov 5 00:00:02 IST 2016 Ping result: ONLINE
Sat Nov 5 00:10:03 IST 2016 Ping result: ONLINE
Sat Nov 5 00:20:02 IST 2016 Ping result: ONLINE
Sat Nov 5 00:30:02 IST 2016 Ping result: ONLINE
Sat Nov 5 00:40:02 IST 2016 Ping result: ONLINE
Sat Nov 5 00:50:02 IST 2016 Ping result: ONLINE
Sat Nov 5 01:00:02 IST 2016 Ping result: ONLINE
Sat Nov 5 01:10:02 IST 2016 Ping result: ONLINE
Sat Nov 5 01:20:02 IST 2016 Ping result: ONLINE
Sat Nov 5 01:30:02 IST 2016 Ping result: ONLINE
Sat Nov 5 01:40:02 IST 2016 Ping result: ONLINE
Sat Nov 5 01:50:02 IST 2016 Ping result: ONLINE
Sat Nov 5 02:00:03 IST 2016 Ping result: ONLINE
Sat Nov 5 02:10:02 IST 2016 Ping result: ONLINE
Sat Nov 5 02:20:02 IST 2016 Ping result: ONLINE
Sat Nov 5 02:30:02 IST 2016 Ping result: ONLINE
Sat Nov 5 02:40:03 IST 2016 Ping result: ONLINE
Sat Nov 5 02:50:02 IST 2016 Ping result: ONLINE
Sat Nov 5 03:00:06 IST 2016 Ping result: OFFLINE (Ping reply not received), Retrying after 10 seconds
Sat Nov 5 03:00:17 IST 2016 Ping result: ONLINE
Sat Nov 5 03:10:03 IST 2016 Ping result: ONLINE
Sat Nov 5 03:20:02 IST 2016 Ping result: ONLINE
Sat Nov 5 03:30:02 IST 2016 Ping result: ONLINE
Sat Nov 5 03:40:02 IST 2016 Ping result: ONLINE
Sat Nov 5 03:50:02 IST 2016 Ping result: ONLINE
Sat Nov 5 04:00:02 IST 2016 Ping result: ONLINE
Sat Nov 5 04:10:02 IST 2016 Ping result: ONLINE
Sat Nov 5 04:20:03 IST 2016 Ping result: ONLINE
Sat Nov 5 04:30:02 IST 2016 Ping result: ONLINE
Sat Nov 5 04:40:02 IST 2016 Ping result: ONLINE
Sat Nov 5 04:50:02 IST 2016 Ping result: ONLINE
Sat Nov 5 05:00:03 IST 2016 Ping result: ONLINE
Sat Nov 5 05:10:02 IST 2016 Ping result: ONLINE
Sat Nov 5 05:20:02 IST 2016 Ping result: ONLINE
Sat Nov 5 05:30:03 IST 2016 Ping result: ONLINE
Sat Nov 5 05:40:03 IST 2016 Ping result: ONLINE
Sat Nov 5 05:50:20 IST 2016 Ping result: ONLINE
Sat Nov 5 06:00:20 IST 2016 Ping result: ONLINE
Sat Nov 5 06:10:27 IST 2016 Ping result: ONLINE
Sat Nov 5 06:20:03 IST 2016 Ping result: ONLINE
Sat Nov 5 06:30:27 IST 2016 Ping result: ONLINE
Sat Nov 5 06:40:02 IST 2016 Ping result: OFFLINE (Ping reply not received), Retrying after 10 seconds
Sat Nov 5 06:40:13 IST 2016 Ping result: OFFLINE (Ping reply not received), Rebooting...
Sat Nov 5 06:50:02 IST 2016 Ping result: OFFLINE (Ping reply not received), Retrying after 10 seconds
Sat Nov 5 06:50:13 IST 2016 Ping result: OFFLINE (Ping reply not received), Rebooting...
Sat Nov 5 07:00:02 IST 2016 Ping result: OFFLINE (Ping reply not received), Retrying after 10 seconds
Sat Nov 5 07:00:13 IST 2016 Ping result: OFFLINE (Ping reply not received), Rebooting...
Sat Nov 5 07:10:02 IST 2016 Ping result: OFFLINE (Ping reply not received), Retrying after 10 seconds
Sat Nov 5 07:10:13 IST 2016 Ping result: OFFLINE (Ping reply not received), Rebooting...
Sat Nov 5 07:20:03 IST 2016 Ping result: OFFLINE (Ping reply not received), Retrying after 10 seconds
Sat Nov 5 07:20:14 IST 2016 Ping result: OFFLINE (Ping reply not received), Rebooting...
Sat Nov 5 07:30:03 IST 2016 Ping result: OFFLINE (Ping reply not received), Retrying after 10 seconds
Sat Nov 5 07:30:14 IST 2016 Ping result: OFFLINE (Ping reply not received), Rebooting...
Sat Nov 5 07:40:02 IST 2016 Ping result: OFFLINE (Ping reply not received), Retrying after 10 seconds
Sat Nov 5 07:40:13 IST 2016 Ping result: OFFLINE (Ping reply not received), Rebooting...
Sat Nov 5 07:50:03 IST 2016 Ping result: OFFLINE (Ping reply not received), Retrying after 10 seconds
Sat Nov 5 07:50:14 IST 2016 Ping result: OFFLINE (Ping reply not received), Rebooting...
Sat Nov 5 08:00:02 IST 2016 Ping result: ONLINE
Sat Nov 5 08:10:02 IST 2016 Ping result: ONLINE
Sat Nov 5 08:20:02 IST 2016 Ping result: ONLINE
Sat Nov 5 08:30:02 IST 2016 Ping result: ONLINE
Sat Nov 5 08:40:03 IST 2016 Ping result: ONLINE
Sat Nov 5 08:50:02 IST 2016 Ping result: ONLINE
Sat Nov 5 09:00:02 IST 2016 Ping result: ONLINE
Sat Nov 5 09:10:02 IST 2016 Ping result: ONLINE
Sat Nov 5 09:20:03 IST 2016 Ping result: ONLINE
Sat Nov 5 09:30:02 IST 2016 Ping result: ONLINE
Sat Nov 5 09:40:02 IST 2016 Ping result: ONLINE
Sat Nov 5 09:50:02 IST 2016 Ping result: ONLINE
Sat Nov 5 10:00:03 IST 2016 Ping result: ONLINE
Sat Nov 5 10:10:03 IST 2016 Ping result: ONLINE
Sat Nov 5 10:20:02 IST 2016 Ping result: ONLINE
Sat Nov 5 10:30:02 IST 2016 Ping result: ONLINE
Sat Nov 5 10:40:02 IST 2016 Ping result: ONLINE
Sat Nov 5 10:50:03 IST 2016 Ping result: ONLINE
Sat Nov 5 11:00:02 IST 2016 Ping result: ONLINE
Sat Nov 5 11:10:02 IST 2016 Ping result: ONLINE
Sat Nov 5 11:20:02 IST 2016 Ping result: ONLINE
Sat Nov 5 11:30:03 IST 2016 Ping result: ONLINE
Sat Nov 5 11:40:03 IST 2016 Ping result: ONLINE
Sat Nov 5 11:50:02 IST 2016 Ping result: ONLINE
Sat Nov 5 12:00:03 IST 2016 Ping result: ONLINE
pi@rpi-lithiumhead-piaware ~ $



Labels:

Comments

  1. UnknownJune 4, 2017 at 1:06 AM

    I was experiencing my FlightAware setup crashing almost every 4 days. I used your code and learned some things on how to exactly mimic your results and it works great! Thank you for posting this, Anurag!

    ReplyDelete

Post a Comment