I recently encountered a puzzling situation where Logstash couldn't read Nginx log files despite having what appeared to be correct permissions. Here's the complete breakdown of what happened and how I resolved it.

The configuration looked correct on the surface:

# File permissions
-rw-r----- 1 www-data adm 0 Jul 25 07:52 /var/log/nginx/foo-access.log

# Logstash user groups
$ groups logstash
logstash : logstash adm

Yet, Logstash kept throwing these errors:

{:timestamp=>"2013-07-31T17:05:17.287000+0000", 
:message=>"failed to open /var/log/nginx/foo-access.log: Permission denied", 
:level=>:warn}

The key insight came when I realized Linux caches group memberships for processes. When you add a user to a new group with usermod, existing processes (including services) won't see the new group membership until they're restarted.

Here's the complete fix I implemented:

# 1. Verify current group membership
$ id logstash
uid=110(logstash) gid=114(logstash) groups=114(logstash),4(adm)

# 2. Fully restart the logstash service (not just reload)
$ sudo systemctl stop logstash
$ sudo systemctl daemon-reload
$ sudo systemctl start logstash

# 3. Alternative: Restart the user session
$ sudo su - logstash
$ groups

To verify everything worked:

$ sudo -u logstash head /var/log/nginx/foo-access.log
# Should now show file contents without errors

If the issue persists, consider:

# Check for SELinux/AppArmor restrictions
$ sudo ausearch -m avc -ts recent
$ sudo aa-status

# Verify filesystem ACLs
$ getfacl /var/log/nginx/foo-access.log

# Set up proper log rotation that maintains permissions
/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        [ ! -f /var/run/nginx.pid ] || kill -USR1 cat /var/run/nginx.pid
    endscript
}

Remember that service accounts like Logstash often need special consideration for file access. Always test permission changes by running commands as the target user before relying on service behavior.

---

When dealing with Logstash file reading permissions, there's a common pitfall that even experienced Linux administrators sometimes stumble upon. The scenario typically looks like this:

$ ls -la /var/log/nginx/foo-access.log
-rw-r----- 1 www-data adm 0 Jul 25 07:52 /var/log/nginx/foo-access.log

$ groups logstash
logstash : logstash adm

Despite proper group membership (logstash user in adm group) and correct file permissions (group read permission for adm), Logstash still throws:

{:timestamp=>"2013-07-31T17:05:17.287000+0000", :message=>"failed to open /var/log/nginx/foo-access.log: Permission denied", :level=>:warn}

Here's what's happening under the hood: when you modify a user's group membership with usermod, the changes don't automatically apply to already running processes. This is a fundamental Linux security behavior.

To verify if this is your issue, try this diagnostic command:

$ ps aux | grep logstash
$ id -Gn <logstash_pid>

You'll likely see that the running Logstash process doesn't have the new group membership even though the user does when you check interactively.

Here's the complete step-by-step fix:

# 1. Add user to group (if not already done)
sudo usermod -a -G adm logstash

# 2. Stop the Logstash service
sudo service logstash stop

# 3. Verify no orphaned processes
sudo pkill -u logstash

# 4. Start Logstash fresh
sudo service logstash start

For systemd systems (modern Linux distributions), you might need:

sudo systemctl daemon-reload
sudo systemctl restart logstash

If restarting the service isn't an option, consider these alternatives:

Option 1: Adjust file permissions temporarily (not recommended for production)

sudo chmod o+r /var/log/nginx/foo-access.log

Option 2: Use extended ACLs (preferred for temporary fixes)

sudo setfacl -m u:logstash:r /var/log/nginx/foo-access.log

To avoid this issue in the future:

• Always restart services after modifying group membership
• Consider using extended ACLs for persistent access control
• Implement proper log rotation with correct permissions

Example logrotate configuration (/etc/logrotate.d/nginx):

/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 cat /var/run/nginx.pid
    endscript
}