Internal Server Error caused by a full disk

You might experience internal server errors from time to time.

Internal Server Error
You have a nil object when you didn't expect it!
You might have expected an instance of Array. 
The error occurred while evaluating nil.each

Such an error can (but not necessarily) be caused by a full disk storage. Especially home servers with very low disk space suffer from this sooner than later and even with bigger storage it’s not if but when it happens to you.

The first location you’d probably clean up is your home directory but the chance that you find there many big files you no longer need is surely low.

A very common location that consumes a lot of space over time is /var. Two directories you should keep an eye on are cache and log.

# sudo du -h --max-depth=1 /var
[sudo] password for slopjong: 
2.9G	./cache
16K	./named
4.0K	./opt
56K	./db
4.0K	./tmp
64K	./spool
4.0K	./local
39M	./lib
1.1G	./log
4.0K	./games
4.0K	./empty
16K	./state
4.1G

Depending on your linux distribution one of the subdirectories of cache belongs to the package manager which might cache a lot of old installed software packages. Especially distributions which use a rolling release model let that directory increase in size.

If your server is running smoothly so that you can waive the old packages, just clean the cache of your package manager. In other words remove packages that are no longer needed to free some space.

In Arch Linux use pacman.

sudo pacman -Sc

In other linux distributions please read the manpage of your package manager for the appropriate options.

The other directory is log which keeps all the relevant system log files of your server. A file which permanently grows is wtmp. In this file all the server logins are recorded and can consume hundreds of megabytes after a server was setup few weeks ago.

Because the wtmp file possibly helps you to find intrusions that maybe you are not aware of you should never delete or overwrite it without checking it before. To read the wtmp file use the command

# last

It will list all the logins chronologically and outputs something like

slopjong pts/1        :0               Wed Sep 12 10:38 - 10:38  (00:00)    
slopjong pts/0        :0               Wed Sep 12 10:28 - 12:37  (02:09)    
reboot   system boot  3.5.3-1-ARCH     Wed Sep 12 10:27 - 12:51  (02:23)    
slopjong pts/0        :0               Wed Sep 12 10:27 - down   (00:00)    
slopjong tty1                          Wed Sep 12 10:21 - down   (00:05)    
reboot   system boot  3.5.3-1-ARCH     Wed Sep 12 10:20 - 10:27  (00:06)    
slopjong pts/2        :0               Wed Sep 12 09:51 - 10:20  (00:28)    
slopjong pts/2        :0               Wed Sep 12 09:50 - 09:51  (00:00)  

Alternatively you can use

utmpdump /var/log/wtmp | less

If you’re sure no incidents happened empty the file by executing

# cp /dev/null /var/log/wtmp

Hopefully your available free space got increased but you can check this with

du -h

IBM has published a guideline of Resolving overflows in the /var file system. Just have a look at it.