What to do when your server is hacked: recovering from backups

The first hour: isolate it, don't switch it off

Contain first, investigate second. CISA's ransomware response checklist says to isolate affected systems at once and to power them down only if you cannot disconnect them, because shutting down destroys the evidence in memory.

• Block traffic at the provider, not on the server. Root can undo iptables rules; a cloud firewall sits outside the machine. On DigitalOcean, a firewall with no inbound and no outbound rules permits nothing, VPC traffic included. On Hetzner, a firewall with no rules blocks inbound but allows outbound until you add at least one outbound rule, and private networks are not filtered.
• Detach the other firewalls. On both clouds the rules of every attached firewall are combined, so an old one that allows port 443 still lets traffic in.
• Expect open connections to survive. Hetzner applies changes only to new connections, and AWS security groups keep tracked connections until they time out; an AWS network ACL breaks them at once.
• Use the provider's console, not SSH. DigitalOcean's Recovery Console works whatever the network settings. If you must SSH in, never forward your agent (ssh -A): OpenSSH warns that anyone who can bypass file permissions on the remote host can use your forwarded keys.

Then snapshot the disk before changing anything: DigitalOcean, Hetzner, AWS EBS, Google Cloud. Snapshot attached volumes too; a Droplet snapshot leaves them out. Name it so nobody restores from it by mistake, such as web-01-COMPROMISED-2026-10-04.

Rebuild, don't clean

Once someone has had root, you cannot prove you found everything they changed: a replaced ps, a library loaded into every program through /etc/ld.so.preload, a cron line, a second SSH key, one edited PHP file. CISA's checklist says to rebuild from pre-configured standard images, restore data from clean backups, and take care not to re-infect clean systems. Keep the old server stopped and firewalled, not deleted, until you have finished investigating.

Work out when it started

You need one date: the earliest sign of the attacker. It decides which backup is safe. Look on a copy if you can: on AWS, Google Cloud and Azure, make a disk from the evidence snapshot, attach it to a clean server and mount it read-only without replaying the journal (noload on ext4; ro,norecovery on XFS):

sudo mount -o ro,noload,noexec,nodev /dev/nvme1n1p1 /mnt/evidence

The commands below use live paths; on a copy, put /mnt/evidence in front and read its journal with journalctl -D /mnt/evidence/var/log/journal. Logins first: -i prints IP addresses, --time-format iso adds the time zone, and -f twice reads the rotated file too:

sudo last -i --time-format iso -f /var/log/wtmp -f /var/log/wtmp.1

Debian 13 dropped last, so there the SSH log is your login record. The unit is ssh on Debian and Ubuntu and sshd on the Red Hat family; -u can be repeated:

sudo journalctl -u ssh -u sshd --since 2026-08-01 --grep 'Accepted|Invalid user'

Key logins end with the key's fingerprint: Accepted publickey for root from 203.0.113.50 port 51234 ssh2: ED25519 SHA256:…. Print the fingerprints of the keys each account accepts and match them up. A key nobody can account for, or an address nobody recognizes, is a candidate start date:

sudo ssh-keygen -lf /root/.ssh/authorized_keys

An attacker can also point AuthorizedKeysFile elsewhere or add an AuthorizedKeysCommand; -T prints the configuration sshd really uses:

sudo sshd -T | grep -i authorizedkeys

• awk -F: '$3 == 0' /etc/passwd lists accounts with user ID 0; only root should appear. getent group sudo (or wheel) lists who can become root.
• Cron: /etc/crontab, /etc/cron.d, and user crontabs in /var/spool/cron/crontabs on Debian and Ubuntu.
• systemd: systemctl list-timers --all and systemctl list-unit-files --state=enabled.
• /etc/ld.so.preload: most servers have none, and the ld.so manual calls a system-wide preload usually undesirable.

Then files changed since a date. Search by change time (-newerct), not modification time: touch can backdate an mtime, but the kernel sets ctime to the current time on every write, including the call that backdates the mtime:

sudo find /etc /usr /root /var/www -xdev -type f -newerct 2026-09-01 -ls

Upgrades change ctime too, so compare with /var/log/apt/history.log. sudo dpkg --verify (rpm -Va on the Red Hat family) flags installed files whose contents differ from the package with a 5 as the third character; c marks config files you probably edited. dpkg's manual calls it an integrity check, not security verification: the checksums sit on the attacker's disk.

Choose a backup from before that date

Every backup taken after the break-in contains it. Here is a site with nightly backups kept 30 days:

With 7-day retention, the oldest backup on 2026-09-28 would be from 2026-09-21, and every copy would hold the attacker's key. Retention has to be longer than the time it takes you to notice. See how long to keep backups.

Later data is not necessarily lost. Restore the newest backup onto a throwaway server with no network access, export only the rows created since the clean backup (orders, sign-ups), read them, and import them into the new server as data. Data can carry the attack too: a new administrator in the users table, a script injected into stored content, or a .php file in an uploads folder.

Rebuild on a fresh server and restore only data

1. Create the server from the provider's own image, never from a snapshot of the old one, with a new SSH key pair made on your own machine. Patch it first.
2. Install software from the distribution's repositories and the application from version control at a commit you trust; reinstall dependencies from the lockfile instead of copying vendor or node_modules.
3. Restore databases from dumps and user files from the clean backup, checked as above.
4. Rewrite configuration from the backed-up copies line by line. Never copy back /etc wholesale, crontabs, authorized_keys, systemd units or /usr/local/bin unread.

The order and commands are in how to restore a server from backup. Keep the new server firewalled to your own address until the last section below is done.

Rotate every credential the server held

Assume the attacker copied every secret the server could read. Revoke anything usable from elsewhere right away: isolating the server does not stop anyone using a copied token. Set new database passwords and app secrets on the new server, so they never touch the old one; CISA likewise advises resetting passwords once the environment is rebuilt.

Backup keys decide whether your backups survived. If the server held a key that could delete or overwrite in the bucket, assume the attacker did. A server that only held a write-only key, or never held the storage key at all, could not reach earlier backups: see protecting backups from ransomware.

Check the backups were not tampered with

If the storage was versioned, list every version under the server's prefix. A DeleteMarkers entry, a key with more than one version, or a LastModified time outside your backup schedule means something deleted or rewrote a backup:

aws s3api list-object-versions --bucket acme-server-backups --prefix web-01/

• Object Lock: a version locked at upload cannot have been deleted or overwritten before its lock ends. Restore that version.
• Checksums kept elsewhere: check each archive's SHA-256 again, as in the tar guide; restic can read back all data with check --read-data (restic guide).
• A test restore: restore onto a server with no network access and look for the changes you found before you trust it (restore drill).

Tell the people you must

If the server held personal data about people in the EU, the GDPR counts unauthorized access to it as a personal data breach (Article 4(12)), even if nothing was visibly taken. What it requires:

• Article 33(1): the controller notifies the supervisory authority without undue delay and, where feasible, within 72 hours of becoming aware, unless the breach is unlikely to result in a risk to people's rights and freedoms. A late notice must give reasons; details can follow in phases.
• Article 33(2): a processor, such as an agency hosting a client's site, notifies the controller without undue delay.
• Article 33(5): every breach is documented, including ones not reported.
• Article 34: if the risk to people is likely to be high, they are told directly, without undue delay, unless, for example, the data was encrypted so the attacker cannot read it.

In the UK, the ICO asks for notice within 72 hours where feasible when a risk is likely. Other laws vary by country, state and industry, as do your contracts, so check what applies to you; this is not legal advice. In the US, CISA takes reports at cisa.gov/report. If your server attacked others, tell your provider's abuse team.

Close the way in before it goes live

• Fix the way they got in: an outdated plugin, a reused password, an exposed admin panel or database port, a leaked key. Skip this and the new server can fall the same way.
• SSH keys only, password logins off (SSH key authentication); automatic security updates; current app and plugin versions.
• Only the ports the app needs open; databases on a private network or localhost; an app user that cannot write its own code.
• Logs shipped off the server. journald can also seal its files (journalctl --setup-keys), so journalctl --verify with the key you keep elsewhere detects later edits.
• MFA on every cloud, storage, DNS and email account.
• Backups the server cannot delete, kept longer than it could take you to notice, plus uptime monitoring and backup failure alerts.

No list makes a server impossible to break into. This one makes the next incident smaller, quicker to spot and quicker to recover from.

Frequently asked questions

Should I shut down a hacked server?

Not as the first step. Block its traffic with your provider's firewall and snapshot its disk. CISA advises powering systems down only when you cannot disconnect them from the network, because shutting down loses what was in memory.

Can I clean a hacked server instead of rebuilding it?

You can remove what you find, but you cannot prove nothing is left. Rebuild from the provider's image, restore data from a backup taken before the attack, and keep the old disk as evidence.

How do I know which backup is clean?

Find the earliest sign of the attacker in logins, keys and changed files, and take the newest backup from before it, with a margin. Restore it to a server with no network access and check it before you use it.

Do I have to report that my server was hacked?

If it held personal data of people in the EU or UK and the breach is likely to risk their rights, the GDPR requires notifying the supervisory authority without undue delay and, where feasible, within 72 hours. Other laws and contracts vary.