As a system administrator, one frustrating issue I frequently encounter is that the “TCP in” and “TCP out” block logs generated by the CSF/LFD firewall disappear from /var/log/messages after a server reboot. Strangely, running csf -r magically brings them back. In this article, I will explain why this happens and provide a permanent, step-by-step solution so you never have to manually restart CSF after a reboot again.
Symptoms of the Problem
- After rebooting the server,
/var/log/messagescontains no blocked connection logs. - Running
csf -r(orsystemctl restart lfd) immediately causes the logs to start appearing again. - Checking the service status with
systemctl status lfdshows LFD asactive (running)with no obvious errors. - The LFD log file (
/var/log/lfd.log) does not show the usual “/var/log/messages has been reset” message after the reboot.
The Root Cause
This issue is caused by two primary factors:
1. Service Startup Order (Systemd Ordering)
By default, systemd starts services in parallel. On your server, the LFD service starts before the rsyslog service. At that moment, the /var/log/messages file hasn’t been created yet (rsyslog creates it upon startup). Since the file doesn’t exist, LFD fails to start watching it. However, because LFD continues to monitor other log files (like /var/log/secure), the service remains active (running). This is why no block logs appear until you manually restart LFD with csf -r.
2. Manual File Deletion (Optional but common)
If you manually delete the file using rm /var/log/messages, LFD’s open file handle becomes invalid. Even when rsyslog recreates the file, LFD does not automatically re-open it. Restarting LFD solves this temporarily, but the problem returns on the next reboot.
Permanent Solutions
To resolve this permanently, I recommend applying the following three fixes in order.
1. Configure Systemd to Start LFD After Rsyslog
This ensures that LFD waits until rsyslog is fully running and /var/log/messages exists before it tries to monitor it.
Create an override file for the LFD service:
bash
systemctl edit lfd
Add the following lines to the opened file:
text
[Unit]
After=rsyslog.service
Save and exit (Ctrl+O, Enter, Ctrl+X). Optionally, do the same for the CSF service:
bash
systemctl edit csf
text
[Unit]
After=rsyslog.service
Reload systemd and restart the services:
bash
systemctl daemon-reload
systemctl restart rsyslog
systemctl restart lfd
systemctl restart csf
Now, LFD will always start after rsyslog on every reboot.
2. Configure Logrotate to Use copytruncate
Logrotate typically moves (mv) the log file during rotation and creates a new one. This breaks LFD’s file handle. The copytruncate option copies the file’s contents and truncates the original in place, keeping the same file handle and inode.
Edit the syslog logrotate configuration:
bash
vi /etc/logrotate.d/syslog
Find the section for /var/log/messages and add copytruncate:
text
/var/log/messages {
daily
rotate 4
copytruncate
compress
missingok
notifempty
create 0640 root root
}
Save the file. From now on, logrotate will not delete or move the file, ensuring LFD never loses its watch.
3. Never Delete the File – Truncate It Instead
If you need to clear the log contents manually, do NOT use rm /var/log/messages. Deleting the file breaks LFD’s monitoring. Instead, just empty the contents in place:
bash
> /var/log/messages
This command truncates the file to zero bytes without deleting it. LFD instantly detects the change (you’ll see the “has been reset” log entry) and seamlessly reopens the file without any interruption to your logging.
Testing Your Fixes
After applying all the steps above, test the setup with a reboot:
bash
reboot
Once the server comes back up, monitor the messages log:
bash
tail -f /var/log/messages
From another terminal or server, try to connect to a blocked port on your server (or trigger any firewall rule). You should see the “TCP in/out block” logs appearing immediately without running csf -r.
Additional Security Recommendations
While fixing this issue, I also recommend the following tweaks:
RESTRICT_SYSLOG: In/etc/csf/csf.conf, changeRESTRICT_SYSLOGfrom"0"to"3"to restrict syslog access and prevent log spoofing. Don’t forget to runcsf -rafterward.- Invalid
csf.ignoreentries: If you see “Invalid entry in csf.ignore” in/var/log/lfd.log, remove those incorrect entries (e.g.,131.0.72.0/222400:cb00::/32) from/etc/csf/csf.ignoreto clean up your logs.
Conclusion
By implementing these three solutions—adjusting systemd service ordering, using logrotate with copytruncate, and replacing rm with truncation—you permanently eliminate the nuisance of missing firewall logs after a reboot. Your server will now reliably log all blocked connections without requiring manual intervention.
This guide was tested on AlmaLinux 8 / CentOS 8 with CSF v14.24. If you have any questions or run into issues, feel free to leave a comment below!
