Reach your server even when the OS is down

Out-of-band access is often the difference between a quick fix and a full reinstall. It runs through the server's baseboard management controller (BMC), a separate controller on the mainboard, so it doesn't depend on your OS or its network configuration. Typical situations:

• A firewall rule or a Netplan change cuts off SSH: log in at the KVM console and revert it.
• A kernel update fails to boot: pick the previous kernel from the boot menu.
• A disk problem drops the server into emergency mode: read the messages on screen and repair from the console or from rescue mode.
• You need an OS that isn't in the library: attach its ISO and install it as on a local machine.

Power commands work from the command line too. With the credentials from the panel and the password in the IPMI_PASSWORD variable, ipmitool -I lanplus -H BMC_IP -U USER -E chassis power status reports the power state, and chassis power cycle restarts a hung server.

IPMI credentials are as powerful as physical access: whoever holds them can power-cycle the server, watch the console and boot other media. Treat them like root credentials:

• Store them in a password manager or secret store, never in tickets, chat or source control.
• Share them only with the people who need hardware-level access.
• Log out of KVM sessions when you are done.
• Pass the password through an environment variable, as with ipmitool -E, rather than on the command line where other users can see it.

Also make sure the local console is usable when you need it. SSH keys don't help at a console login prompt, so keep a user with a strong local password and sudo rights, and know how to reach your boot loader's menu. Test this once while the server is healthy, not during an outage. The IPMI and KVM documentation explains how to connect and mount ISOs.