A few days ago I was contacted by a client whose website had been offline for a month. The site has a history of going offline because of server overloads. The client is OK with the occasional down time as the site is just a pet project. Every time the server goes down we usually restart the MySQL server and everything goes back to normal, but this time it was different. Instead of restarting the MySQL server, the whole server was restarted by power cycling it, and that is when everything went wrong.
When the site came back up, the HTTP and MySQL server had to be manually started, and the program that communicates with the server control panel for status reporting was down as well. It had to also be started manually. However, after all of this, the site was still off line. Trying to access the website resulted in an error like the one you see when you are not connected to the internet.
I decided to curl the site, and in return I got an error saying “No route to host”. Pinging the server worked, but form some reason I could not connect to it, because of this I knew it was not a DNS problem, since the url resolved to the server IP correctly, but the connection was refued. Could it be a server issue?
I decided to take a look at the server config files, only to find out everything was configured correctly. At this point I was absolutely intrigued.
I decided to search the interwebs to find out what people said about the “no route to host” problem, and, as I thought, it was a connection problem. The machine was not refusing the connection, the network was. I confirmed that by fetching localhost from the server. I got the expected result, so the server was not refusing the connection.
It was at this point that I decided to concentrate on the network side of things. I started by determining the port the server was listening on. I did this by checking the server configuration files. When I confirmed the server was listening on port 80, I decided to check if port 80 was open. Using iptables I determined that it was not. From there, it was just a mater of opening the port. I did that by running
iptables -I INPUT 1 -p tcp --dport 80 -j ACCEP
Here are some documents I consulted while trying to solve this problem. Some proved useful, others did not. It is worth mentioning the server runs Fedora.