Why you need a plan in advance, not during the attack
At the moment of a breach, an administrator loses a third of their effectiveness to stress and makes decisions that destroy evidence: rebooting the server, deleting suspicious files, changing passwords before the logs are analyzed. An incident response plan is a sequence of actions written in advance for the first hour, when decisions are made fast but with no room for error.
The goal of the first 60 minutes is not to investigate the incident fully but to stop further damage and preserve everything the investigation and recovery will need. A full investigation takes days, but the decision to isolate the server must be made in minutes.
Minute 0-10: detection and isolation
The first step is to confirm this is really an incident, not a monitoring false positive. Once confirmed:
- Isolate the server at the network level: disable the public interface through the provider's panel or a firewall rule, but do not power off the server itself — RAM holds traces of the attacker's process.
- Do not delete or overwrite files — any change destroys evidence needed for later analysis.
- Record the exact detection time and first observations in text — an hour later, details fade from memory.
iptables -I INPUT 1 -j DROP
iptables -I OUTPUT 1 -j DROP
These two rules block all new traffic while keeping existing sessions alive for later memory analysis.
Minute 10-25: collecting initial evidence
While the server is isolated but still powered on, capture state snapshots for later analysis:
ps auxf > /mnt/evidence/ps.txt
netstat -tulpn > /mnt/evidence/netstat.txt
who -a > /mnt/evidence/who.txt
last -50 > /mnt/evidence/last.txt
cp /var/log/auth.log /mnt/evidence/
Copy evidence to an externally mounted volume, not the same disk — the attacker may have left a log-cleanup script that triggers on reboot. If the organization runs Wazuh SIEM or auditd, export events from the last 24-48 hours as a separate archive before powering the server down.
Minute 25-40: assessing the scope
Determine exactly what was affected:
| Question | Where to look for the answer |
|---|---|
| Which accounts are compromised | the auth.log journal, command history, new SSH keys in authorized_keys |
| What data may have been copied | outbound traffic in netstat, web server logs from the last hours |
| Is there attacker persistence | crontab, systemd timers, new systemd units, autostart entries |
| Are other servers affected | outbound SSH sessions, shared accounts, a shared database |
Do not rush to conclusions: a sign of compromise on one server often means the whole network needs checking, including backups made after the breach.
Minute 40-60: notification and first decisions
Report what happened using a contact list prepared in advance: the technical lead, the business owner, and, for a personal data leak, the data protection officer. A short notification template:
Incident: [date, time detected]
Affected system: [host/service]
Status: isolated / under investigation
Preliminary scope: [what is known so far]
Next step: [for example, restore from backup N-1]
At this stage, decide on the next path: restoring from a clean backup, bringing in an external incident response specialist, notifying a regulator about a personal data leak. The detailed cleanup process is described in the article hacked site: diagnosis and recovery.
First-hour checklist
- The server is isolated at the network level but not powered off — memory is preserved for analysis.
- Evidence is copied to an external volume before any changes are made on the server.
- The exact time and first observations are recorded.
- The scope of the incident is assessed at least preliminarily: affected accounts, data, other servers.
- Responsible people are notified using a contact list known in advance.
After the first hour, move on to a full investigation: use osquery for inventory and investigation to systematically compare the server's state against a known baseline.
Frequently asked questions
Do you need to power off the server entirely if network isolation cannot be set up?
Powering the server off completely is justified only when network-level isolation truly cannot be done through the provider's panel, a firewall rule, or a switch. Shutting it down immediately erases the RAM, which still holds traces of the attacker's process, making investigation harder. Any network-level way to cut traffic is tried first, and a full shutdown stays the last resort.
What should you do if the incident is discovered at night and no specialist is on site?
The plan does not require a specialist watching a monitor around the clock: monitoring itself should send an alert, and an on-call person must connect remotely for the first step, isolating the server. With no remote access, delay grows into hours instead of minutes, and damage can grow noticeably, so remote access is worth arranging in advance.
Should customers be told about the breach right away?
Telling customers is a separate decision made once the scope is clearer, not in the first minutes: while it is unclear whether personal data was exposed, a public announcement could be premature or inaccurate. The internal circle of responsible people is informed first, and a public statement follows once what happened is clear.
Can backups made after the attack was discovered be trusted?
Backups made after discovering the attack should not be treated as clean by default: the attacker may already have been inside the system for a while, so the copy could carry altered files or hidden entry points. Such a backup is worth checking before restoring, and for a rollback rely on a version created before the first signs of compromise.