Running fsck on EC2, Part 1

I encountered a situation where I wished to run fsck on an EC2 instance’s root filesystem. As running on fsck on a mounted filesystem is not recommended, I had two choices:

  1. Shutdown the EC2 instance, detach the EBS volume, attach this EBS volume to another EC2 instance and run fsck.
  2. Find a way to run fsck on the root filesystem during EC2 instance startup.

Running fsck during during EC2 instance startup is widely reported to be troublesome. The typical complaint is that fsck prompts for input at the console – an example of this problem is here: http://hackinjack.wordpress.com/tag/ec2-fsck/ for an example.

fsck_root_filessytem_result

How to run fsck on an EC2 instance’s root directory.

Longer method (see “Running fsck on EC2 – part 2” for a Quick and Dirty fsck Method)

The solution for running fsck on an EC2 instance’s root directory is described below. Note that this is tested on Ubuntu 12.0.4 LTS. The steps below describe a method for safely checking a root filesytem.

Enable root filesystem fsck

Open  /etc/fstab – example below:

sudo vi /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults 0 0
/dev/xvdb /mnt auto defaults,nobootwait,comment=cloudconfig 0 2
/dev/xvda3 none swap sw,comment=cloudconfig 0 0

Change the value “fs_passno” for the root filesystem from “0” to “1”. The “fs_passno” number is the last column in the an fstab entry.

sudo vi /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults 0 1

The default value of “fs_passno” is “0” – the “0” value specifies a filesystem that should not be checked by fsck – in other words, a default EC2 instance should not have fsck checks performed, presumably because this could cause problems if fsck prompted the user for input on startup. Changing this value to 1 designates the the /etc/fstab entry as that of the root filesystem – and of a filesystem that should be checked by fsck if a check is requested.

Set fsck to Automatically Fix Errors at Startup

The other part of this change requires setting your EC2 instance to automatically fix errors that fsck finds (so as to not prompt you at startup). Instructions are below:

Open /etc/default/rcS and change the FSCKFIX value

sudo vi /etc/default/rcS
#FSCKFIX=no is the default value for FSCKFIX
FSCKFIX=no

Change FSCKFIX to FSCKFIX=yes

sudo vi /etc/default/rcS
FSCKFIX=yes

Confirm the time that fsck was last run:

Use  tune2fs to determine the when fsck was last run on the root filesystem. By comparing the “Last checked” time given at this moment and after the restart you’ll be able to confirm that fsck ran.

colin@devcolin:~$ sudo tune2fs -l /dev/xvda1 | grep "Last checked"
Last checked:             Sat Jul 28 08:09:37 2012

Reboot the EC2 Instance.

I use the Amazon console or EC2 tools for this.

Confirm Reboot

colin@devcolin:~$ uptime
 07:51:22 up 1 min, 1 user, load average: 2.56, 0.81, 0.28

Confirm fsck Ran Correctly

colin@devcolin:~$ sudo tune2fs -l /dev/xvda1 | grep "Last checked"
Last checked:             Tue Mar  4 07:50:06 2014

Return /etc Files to Defaults

  1. In /etc/fstab, change the “fs_passno” to 0
  2. In /etc/default/rcS change FSCKFIX to FSCKFIX=no

One thought on “Running fsck on EC2, Part 1

  1. Those steps worked perfectly, thanks! I was lamenting EC2 didn’t give me a virtual view like VM software does, but these steps did the job without having that.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s