Categories
Uncategorized

Automating Linux updates and weekly reboots

Managing multiple Linux servers means keeping them updated and secure. Rather than manually SSH-ing into each server to run updates, I’ve automated the entire process using systemd timers. Here’s how I set up daily automatic updates and weekly reboots across 10 Ubuntu/Debian servers.

The Goal

I wanted to automate this manual process I’d been running:

sudo apt-get update && \
sudo apt-get -y dist-upgrade && \
sudo apt-get clean && \
sudo apt-get autoclean && \
sudo apt autoremove --purge && \
sudo reboot

The requirements were simple:

  • Run updates daily
  • Reboot servers weekly to apply kernel updates and clear memory
  • Log everything for troubleshooting
  • Deploy easily across multiple servers

The Solution: Systemd Timers

I chose systemd timers over cron for several reasons:

  • Better logging integration with journalctl
  • Built-in randomization to prevent all servers updating simultaneously
  • More reliable execution with Persistent=true
  • Easier to manage and monitor

Implementation

Step 1: The Update Script

First, I created a bash script that handles the update process and logs everything:

#!/bin/bash
# /usr/local/bin/system-update.sh

LOG_FILE="/var/log/system-update.log"

echo "=== Update started: $(date) ===" >> "$LOG_FILE"

apt-get update >> "$LOG_FILE" 2>&1
apt-get -y dist-upgrade >> "$LOG_FILE" 2>&1
apt-get clean >> "$LOG_FILE" 2>&1
apt-get autoclean >> "$LOG_FILE" 2>&1
apt autoremove -y --purge >> "$LOG_FILE" 2>&1

echo "=== Update completed: $(date) ===" >> "$LOG_FILE"
echo "" >> "$LOG_FILE"

Step 2: Systemd Service for Updates

The service file defines what to run:

# /etc/systemd/system/system-update.service
[Unit]
Description=Daily system update
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/system-update.sh

Step 3: Timer for Daily Updates

The timer controls when the service runs:

# /etc/systemd/system/system-update.timer
[Unit]
Description=Run system updates daily

[Timer]
OnCalendar=daily
RandomizedDelaySec=1h
Persistent=true

[Install]
WantedBy=timers.target

The RandomizedDelaySec=1h is crucial – it staggers updates across servers by adding a random 0-60 minute delay, preventing all servers from hammering the repositories simultaneously.

Step 4: Weekly Reboot Service and Timer

For reboots, I created a simple service:

# /etc/systemd/system/weekly-reboot.service
[Unit]
Description=Weekly system reboot
DefaultDependencies=no
Before=shutdown.target

[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl reboot

And a timer to trigger it every Sunday at 3 AM:

# /etc/systemd/system/weekly-reboot.timer
[Unit]
Description=Weekly system reboot

[Timer]
OnCalendar=Sun 03:00
Persistent=true

[Install]
WantedBy=timers.target

Step 5: Enable Everything

sudo systemctl daemon-reload
sudo systemctl enable --now system-update.timer
sudo systemctl enable --now weekly-reboot.timer

Automated Deployment

Setting this up manually on 10 servers would be tedious and error-prone. I created two scripts to automate the entire process:

Setup Script

The first script (setup-auto-update.sh) handles installation on a single server. It creates all necessary files, sets permissions, and enables the timers. The script includes error checking at each step to catch issues immediately.

Deployment Script

The second script (deploy-auto-update.sh) pushes the setup to multiple servers:

#!/bin/bash
USERNAME="sysadmin"
SERVERS=(
    "server1.example.com"
    "server2.example.com"
    # ... more servers
)

for server in "${SERVERS[@]}"; do
    echo "Deploying to: $server"
    scp setup-auto-update.sh $USERNAME@$server:/tmp/
    ssh $USERNAME@$server "sudo bash /tmp/setup-auto-update.sh"
done

The deployment script includes:

  • SSH connectivity testing before attempting deployment
  • Color-coded output for success/failure
  • Summary of successful and failed deployments
  • Optional verification of timers across all servers

Monitoring and Verification

After deployment, verify everything is working:

# Check timer status
systemctl list-timers

# View update logs
tail -f /var/log/system-update.log

# Check last update time
journalctl -u system-update.service

The systemctl list-timers command shows when each timer last ran and when it will run next:

NEXT                         LEFT        LAST                         PASSED    UNIT
Sun 2024-12-22 03:00:00 EST  8h left     Sun 2024-12-15 03:00:00 EST  6 days ago weekly-reboot.timer
Mon 2024-12-23 02:47:18 EST  1 day left  Sun 2024-12-22 02:34:56 EST  15h ago    system-update.timer

Lessons Learned

  1. Always create the service file for timers – My initial version forgot the weekly-reboot.service file, causing the timer to fail with “unit to trigger not loaded”
  2. Randomization is essential – Without RandomizedDelaySec, all servers would update simultaneously, potentially overwhelming your network or repository mirrors
  3. Logging is crucial – Having a dedicated log file (/var/log/system-update.log) makes troubleshooting much easier than digging through system logs
  4. Test SSH connectivity first – The deployment script tests each connection before attempting to copy files, saving time when servers are unreachable
  5. Systemd timers are more reliable than cron – The Persistent=true setting ensures that if a server is down during a scheduled run, the job executes when it comes back up

Customization Options

You can easily adjust the schedule by editing the OnCalendar values:

  • OnCalendar=Mon 02:00 – Run at 2 AM every Monday
  • OnCalendar=*-*-* 02:00:00 – Run at 2 AM every day (alternative syntax)
  • OnCalendar=Sat 04:00 – Run at 4 AM every Saturday

For testing, you can use:

  • OnCalendar=hourly – Run every hour
  • OnCalendar=*:0/15 – Run every 15 minutes

Disabling Auto-Updates

If you need to temporarily disable automatic updates or reboots:

# Disable updates
sudo systemctl disable system-update.timer
sudo systemctl stop system-update.timer

# Disable reboots
sudo systemctl disable weekly-reboot.timer
sudo systemctl stop weekly-reboot.timer

# Re-enable later
sudo systemctl enable --now system-update.timer
sudo systemctl enable --now weekly-reboot.timer

Conclusion

This setup has been running reliably across my server infrastructure for weeks now. The combination of systemd timers and automated deployment scripts means I can focus on more important tasks while my servers stay updated and secure.

The complete scripts are available below. Feel free to adapt them to your needs!

Complete Scripts

You can find the complete setup-auto-update.sh and deploy-auto-update.sh scripts in the artifacts above. Both include comprehensive error checking and user-friendly output.

Key benefits:

  • ✅ Fully automated updates and reboots
  • ✅ Staggered execution prevents overload
  • ✅ Comprehensive logging
  • ✅ Easy deployment to multiple servers
  • ✅ Simple monitoring with systemd tools

If you manage Linux servers, I highly recommend implementing something similar. The time investment upfront pays dividends in reduced maintenance overhead and improved security posture.