For quite some time I’ve been having a problem with my httpd.pid file going missing every now and then. Whenever this happens, I can’t use the init script to start and stop the server anymore, which is pretty annoying. For some reason I had learned to live with this – I would just ‘killall httpd’, start the server again and the httpd.pid would be back for some time, until the problem would reappear.

Finally I decided to do something about it. I added this small script to my hourly cron jobs:

if [ ! -e "/usr/local/apache2/logs/httpd.pid" ]; then
        echo "httpd.pid missing"
fi

A few weeks later this trap was finally triggered – less than an hour after my daily logrotate, which produced this output:

Starting httpd: [  OK  ]
Stopping httpd: [  OK  ]
(repeated a number of times...)
Starting httpd: [  OK  ]
Stopping httpd: [FAILED]
Starting httpd: (98)Address already in use: AH00072: make_sock: could not bind to address [::]:80
(98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
AH00015: Unable to open logs
[FAILED]
error: error running postrotate script for /usr/local/apache2/logs/*_log 

This was not the first time I had seen this, but it was the first time I realized that the logrotate problem was causing the httpd.pid problem, and not the other way around.

So I checked my /etc/logrotate.d/apache config file:

/usr/local/apache2/logs/*_log {
    rotate 1024
    size 4M
    notifempty
    postrotate
        /etc/init.d/httpd restart
    endscript
}

After a few minutes of reading the logrotate man page, I realized what was wrong (had a hunch, though): The restart between the postrotate/endscript directives was performed for each logfile, i.e. multiple times, instead of just a single time after the last one was rotated. This in itself is pretty bad, but it would also happen asynchronously, thus create a mess. This problem was easily solved using the sharedscript directive. Also, a simple “reload” instead of “restart” is sufficient to make httpd reopen the logfiles, thus create new filehandles.

After fixing these two problems, the file ended up like this:

/usr/local/apache2/logs/*_log {
    rotate 1024
    size 4M
    notifempty
    sharedscripts
    postrotate
        /etc/init.d/httpd reload
    endscript
}

I’m fully expecting this to solve the problem once and for all.