Production Is Not a Playground

Below is a battle tested checklist of pro tips and hard won habits that Linux system administrators should follow when operating production servers. These are practical measures that help prevent outages, data loss, and late night incidents.

1. Golden rule: assume you will break something

Always operate with the mindset that:

• A typo will happen
• A command will run on the wrong host
• A disk will fill up
• A process will hang

Design your actions so failures are reversible.

2. Never touch production blindly

Always verify where you are:

hostname
whoami
pwd

Set distinct prompts for production versus non-production systems:

export PS1="\[\e[31m\][PROD]\u@\h:\w$ \[\e[0m\]"

Many administrators have avoided disasters simply by noticing a red production prompt.

3. Use tmux or screen, always

SSH sessions will drop.

tmux new -s prod

Benefits include surviving network drops, scrollback history, multiple panes for logs and commands, and safe reattachment.

4. Think in read-only first

Before changing anything:

cat file
less file
systemctl status service

Only then:

cp file file.bak.$(date +%F)
vim file

Never edit without a backup, even if Git exists.

5. Package management: slow and predictable

Before installing or upgrading:

apt update
apt list --upgradable

Dry-run whenever possible:

apt install nginx --simulate

Never run unattended upgrades on production unless they are fully tested.

6. Service management discipline

Always:

systemctl status service
systemctl cat service
journalctl -u service -n 100 --no-pager

Know the difference. Reload applies configuration changes. Restart fully stops and starts the service and may cause downtime.

7. Logs are your truth

journalctl -xe
journalctl --since "10 minutes ago"
tail -f /var/log/syslog
journalctl -u nginx -f

Never guess. Logs show what actually happened.

8. Disk space will kill you silently

Check regularly:

df -h
du -sh /var/*

Find space hogs:

du -xh / | sort -h | tail -20

The majority of outages are caused by full disks, from logs, containers, or temporary files.

9. Treat permissions like loaded weapons

Before chmod or chown:

ls -l
getfacl file

Avoid:

chmod 777

Prefer:

chmod 640
chown appuser:appgroup file

One wrong recursive chown -R can render a system unusable.

10. Backups are useless unless tested

You don't have backups unless the restore was tested. Checklist:

• Configuration files backed up
• Databases backed up
• Restore process documented
• Restore tested on another host

Untested backups are false confidence.

11. Use noclobber

Prevents accidental overwrite:

set -o noclobber

Now > file fails if the file already exists.

12. Always use absolute paths in scripts

Bad:

rm logs/*

Good:

rm /var/app/logs/*.log

Relative paths combined with cron jobs are a common source of failures.

13. SSH like a professional

Disable root login, use key-based authentication only, restrict users explicitly with AllowUsers, and maintain a break-glass account.

PermitRootLogin no
PasswordAuthentication no

Also:

ssh -o ServerAliveInterval=60

14. Know your kill signals

kill -15 PID
kill -9 PID

Jumping straight to SIGKILL can corrupt data and should be a last resort.

15. Always check what will be deleted

ls dir
echo dir/*
rm -rf dir && echo "deleted dir"

16. Cron jobs are silent killers

Always redirect output:

* * * * * /script.sh >> /var/log/script.log 2>&1

Check:

grep CRON /var/log/syslog

17. Monitor before you need it

At minimum: CPU, RAM, disk usage, load average, and process count.

htop
vmstat
iostat

If you notice a problem before users do, you're doing it right.

18. Write everything down

Your future self is a stranger. Maintain change logs, runbooks, recovery steps, and a history of past issues.

19. Assume automation will fail

Even Ansible, Terraform, and scripts need a check or dry-run mode, reviewed diffs, testing on staging, and a gradual rollout.

20. The ultimate pro tip

Production is not the place to experiment. Experiment on staging systems, virtual machines, containers, or replicas.

Final mindset

• Calm under pressure
• Obsessed with verification
• Slightly paranoid
• Extremely disciplined

Production Linux administration is less about knowing obscure commands and more about cultivating disciplined habits. The practices outlined here are simple, repeatable, and deliberately conservative. Applied consistently, they reduce uncertainty, limit blast radius, and make recovery predictable.

In production, reliability is earned not through heroics, but through careful preparation and restraint.