Skip to navigation

Database node fixed

January 04, 2015

I've been busy since Christmas.  I've had a lot of emails and comments to reply to. Unfortunately it's getting difficult to answer all the questions I get, but I try and answer as many as possible.

There are a couple of new articles on my Linux Web Servers site.  The first article is about setting up an Apache load balancer on Ubuntu, and the second one is about setting up the web UI for the balancer manager.

I've just finished fixing a problem with a node in the database cluster in my Banana Pi server farm. I spotted the problem with the Ganglia monitoring tool. It showed that the CPU in the fourth node was busy, even though there was virtually no load on the cluster at the time. I connected to the fourth node using ssh and ran the 'top' command to find out what was going on.  

A process named mmcqd was hogging the CPU. This process is used to access the SD card, so I guessed that there was a problem with the SD card.  The fourth node in the database cluster handles MySQL requests for the fourth node in the web server cluster, so before I shut down the faulty node in the database cluster, I used the balancer manager's web UI to disable the web server node.

Next, I shutdown the bad database node, took out the SD card and put it in a card reader.  I connected the card reader to another Banana Pi and ran these commands to fix the problem:

sudo badblock -sv /dev/sda2 bad-blocks
sudo fsck -t ext4 -L bad-blocks /dev/sda2

The first command gets a list of bad blocks, and the second command fixes them.  Once the card was fixed, I put the card back in the Banana Pi in the database cluster.  I started it up and the CPU load was back to normal. Problem solved.

Apart from this problem, the cluster has been running smoothly.  Page load times on all the sites hosted on the cluster are good, although there isn't much of a load on the cluster at the moment.  It will be interesting to see what happens when traffic increases.


Share this page:

Follow me