Running fsck on EC2, Part 1
I encountered a situation where I wished to run fsck on an EC2 instance’s root filesystem. As running on fsck on a mounted filesystem is not recommended, I had two choices:
- Shutdown the EC2 instance, detach the EBS volume, attach this EBS volume to another EC2 instance and run fsck.
- Find a way to run fsck on the root filesystem during EC2 instance startup.
Running fsck during during EC2 instance startup is widely reported to be troublesome. The typical complaint is that fsck prompts for input at the console – an example of this problem is here: http://hackinjack.wordpress.com/tag/ec2-fsck/ for an example.
How to run fsck on an EC2 instance’s root directory.
Longer method (see “Running fsck on EC2 – part 2” for a Quick and Dirty fsck Method)
The solution for running fsck on an EC2 instance’s root directory is described below. Note that this is tested on Ubuntu 12.0.4 LTS. The steps below describe a method for safely checking a root filesytem.
Enable root filesystem fsck
Open /etc/fstab – example below:
sudo vi /etc/fstab LABEL=cloudimg-rootfs / ext4 defaults 0 0 /dev/xvdb /mnt auto defaults,nobootwait,comment=cloudconfig 0 2 /dev/xvda3 none swap sw,comment=cloudconfig 0 0
Change the value “fs_passno” for the root filesystem from “0” to “1”. The “fs_passno” number is the last column in the an fstab entry.
sudo vi /etc/fstab LABEL=cloudimg-rootfs / ext4 defaults 0 1
The default value of “fs_passno” is “0” – the “0” value specifies a filesystem that should not be checked by fsck – in other words, a default EC2 instance should not have fsck checks performed, presumably because this could cause problems if fsck prompted the user for input on startup. Changing this value to 1 designates the the /etc/fstab entry as that of the root filesystem – and of a filesystem that should be checked by fsck if a check is requested.
Set fsck to Automatically Fix Errors at Startup
The other part of this change requires setting your EC2 instance to automatically fix errors that fsck finds (so as to not prompt you at startup). Instructions are below:
Open /etc/default/rcS and change the FSCKFIX value
sudo vi /etc/default/rcS #FSCKFIX=no is the default value for FSCKFIX FSCKFIX=no
Change FSCKFIX to FSCKFIX=yes
sudo vi /etc/default/rcS FSCKFIX=yes
Confirm the time that fsck was last run:
Use tune2fs to determine the when fsck was last run on the root filesystem. By comparing the “Last checked” time given at this moment and after the restart you’ll be able to confirm that fsck ran.
colin@devcolin:~$ sudo tune2fs -l /dev/xvda1 | grep "Last checked" Last checked: Sat Jul 28 08:09:37 2012
Reboot the EC2 Instance.
I use the Amazon console or EC2 tools for this.
colin@devcolin:~$ uptime 07:51:22 up 1 min, 1 user, load average: 2.56, 0.81, 0.28
Confirm fsck Ran Correctly
colin@devcolin:~$ sudo tune2fs -l /dev/xvda1 | grep "Last checked" Last checked: Tue Mar 4 07:50:06 2014
Return /etc Files to Defaults
- In /etc/fstab, change the “fs_passno” to 0
- In /etc/default/rcS change FSCKFIX to FSCKFIX=no