IPMI

Last change on 2023-01-30 • Created on 2020-03-20 • ID: RO-1F15A

Introduction

Note: This article now only applies to DELL servers and some older DELL server models still available on the Server Auction. See [below][/robot/dedicated-server/maintainance/ipmi/#hetzner-ipmi-information].

This article provides an overview, and some guides and examples for the IPMI interface.

IPMI (Intelligent Platform Management Interface) is an interface for the management and administration of servers. It is implemented with a BMC (Baseboard Management Controller) of a motherboard. You can access this interface via a command line program (such as IPMItool) or a web interface with which you can administer the server. You can perform a reset, start a KVM, and read the output of the motherboard sensors.

Hetzner IPMI information

In 2019, we at Hetzner decided to no longer provide customers with additional IPMI or KVM modules to install on their dedicated root servers. There are two exceptions: DELL servers and DELL servers from the Server Auction. Just in case there is a fault, we have kept a few of these IPMI modules.

We decided to make this decision because of many security issues in the past. These vulnerabilities were the result of issues in the manufacturers' firmware, and unfortunately, it took the manufacturers a long time to fix the issues.

Model overview

If you aren't sure which server model is the basis for your SB server (your server from the Server Auction), contact the support team by writing a support request on Robot. Our team can tell you.

  • PX60(-SSD)/PX70(-SSD) - BMC include IPMI (KVM-over-IP as an optional paid module)
  • PX90(-SSD)/PX120(-SSD) - BMC include IPMI and KVM
  • PX91(-SSD)/PX121(-SSD) - paid optional BMC module with IPMI/KVM

Activation of the network interface

With the integrated BMC, the network configuration is disabled by default. To use the IPMI and Serial over LAN and/or KVM function, you will need to order an additional IP address (which has a small fee) via Robot.

Important: You need to specify the MAC address of the BMC when you order the additional IP address.

You can read the MAC address using IPMItool (see Network). After we allocate the IP address to you, you can statically configure or assign it to the BMC via DHCP.

Safety instructions

If the BMC is made accessible by assigning a public IP to it. It can be attacked, and under certain circumstances, abused, leading to the server potentially becoming compromised. Therefore, you should take measures to counteract the most well-known attack scenarios. To learn more about these attacks, read about the Metasploit penetration tool collection (https://community.rapid7.com/community/metasploit/blog/2013/07/02/a-penetration-testers-guide-to-ipmi).

The motherboard used in the models PX60 and PX70 is not vulnerable to the options described in the link, as some protection mechanisms are already installed by default. Generally you should change the default passwords and disable or rename already existing users. Anonymous access is deactivated by default on all models. See below for information about the other attack vectors and their prevention.

Current Threats

Vulnerabilities

Since the BMC that provides the IPMI functionality is simply software, there can be vulnerabilities in it.

PX90/PX120

For the motherboard of the models PX90/120 (Supermicro X9SRi-F), a vulnerability was discovered in the 2.14 firmware version, allowing usernames and passwords to be read in plain text. If your server has this firmware, you need to do a firmware update before you activate the network interface.

You can read the firmware version using impitool:

ipmitool mc info
...
Firmware Revision         : 3.50

If you have already activated the network of the IPMI, you can perform the update via the web interface. You can find the latest version on download.hetzner.com. (The login details are in the same email we sent you when we deployed your server.)

Or you can do an update with Linux if the network of the IPMI has not been activated:

wget http://mirror.hetzner.com/tools/Supermicro/tools/SMT_X9_350.tgz
tar -xzf SMT_X9_350.tgz
cd SMT_X9_350
./lUpdate -f SMT_X9_350.bin

Please note that this update only applies to the PX90 and PX120.

PX91/PX121

The additional KVM module for the PX91 and PX121 models (Asus Z10PA-U8) uses insecure SSL protocols and an obsolete SSL certificate in versions before 1.11.

Therefore, depending on the browser you use, there may not be any secure (HTTPS) connections available until after you have performed an update.

You can download the update via the web interface, and you can find the current firmware at download.hetzner.com. (The login details are in the same email we sent you when we deployed your server.)

Cipher 0 attacks

Cipher 0 means that no encryption is used and thus any authentication is bypassed. By default Cipher 0 is only activated on the motherboard for callback, meaning logging in is not possible; you can only get a response of whether the BMC is there or not. Even with that, to ensure that no abuse can take place, the Cipher 0 is disabled by default upon delivery of the server and can therefore no longer be used.

Transmission of the password's hash

In the IPMI specification, the authentication of the user is only possible on the user's side. Therefore, a hash of the password is transmitted to any requesting users. As it specifies exactly what this hash contains, it is possible to find the password via a brute-force attack. Since this is part of the IPMI specification, this problem is found on all BMCs and can only be remedied by changing the specification. Therefore, the only current recommendation about this issue is to use a really long and strong password for the BMC to make it as difficult as possible for any attackers. If you use a short or easy-to-guess password for the BMC, it can be compromised within hours or even minutes.

Here are some tips for a secure password:

If you need to make the password easy to remember, it makes sense to string together several words that have no connection to each other (http://correcthorsebatterystaple.net/). This is secure due to the length and is yet still easy to remember.

If you plan to store the password in a database and you do not need to remember it, then you can use sufficiently random numbers and letters in a reasonable length (>30 characters) to create a secure password.

SNMP reflection

A few IPMI modules (for example, ASMB8-iKVM in the PX91 and PX121 models) permit queries via SNMP. Therefore, a small query can cause the loss of a large amount of data if the query from the source address is misused in an attack. If you use SNMP, you need to be sure to use a strong password (which means using an SNMP Community String). If you do not use SNMP, you can use a firewall to block this port on the ASMB8-iKVM modules of the PX91 and PX121 models. You can use the web interface to perform both options.

Explanation of individual functions

Web interface

You can use the web interface to read data from the BMC easily and securely. It displays all sensors, you can add and change users, you can set the network configuration, and if you have a KVM, you can start it.

System Information: On this page, you can find some information about your server (BIOS version, current status, CPU and RAM information) and you can see the users who are currently signed in.

Server Health: Here, you can see the output of individual sensors on the motherboard and in the CPU. If there are any thermal problems, you can detect them here. Furthermore, there is an event log. In the log, you can find system events, such as critical temperatures, reboots, and CPU throttling. This may help you diagnose a potential problem. The page Power Statistics does not work with this model because the power supply does not have the necessary PMBUS interface.

Configuration: Here you can configure many options of the BMC. You do not usually need to change the network settings because the configuration for IPv4 is set automatically via DHCP. You can manually configure IPv6, but using the Hetzner default gateway fe80::1 will only be possible with future firmware versions. You can also add new users here and change and delete existing ones. Additionally, the option Alerts allows you to have notifications sent via SNMP or email when certain events occur on the server. This can be useful for monitoring the server.

Remote Control: On this page, you can use the KVM functionality of the BMC. However, the option Console Redirection is only available if you activate an additional module. You can always use Server Power Control. This allows you to send a hardware and software reset to the server; you can also shut it down or start it.

Configuration

In this section, you can see some basic configuration options. You can usually use the web interface of the BMC. It is also recommended to install ipmitool, which you can install using the package manager of all major distributions. This gives you access to additional functions which you cannot configure using the web interface.

Example for Debian:

Installation via the package manager:

apt install ipmitool

In order for ipmitool to function, you should load the following modules via modprobe:

modprobe ipmi_devintf
modprobe ipmi_si

To check if everything important was correctly loaded and installed, use the following example command, which will show you the data from all available sensors:

ipmitool sensor list

Users

You can create several users with different rights on the BMC. After creating a new user with administrative rights via ipmitool, you can manage more users via the web interface. There are 4 different rights/permission levels:

  • Callback (1): Can only initiate a callback
  • User (2): Can send read-only requests but cannot change any configuration files
  • Operator (3): Can change all configurations apart from deactivating the channel and changing rights
  • Administrator (4): Can change all configurations

Usually one or more users already exist. You can get an overview of the existing user IDs and logins via:

ipmitool user list 1

In the PX90/PX120 models, an active user with administrative rights already exists:

ID  Name             Callin  Link Auth  IPMI Msg   Channel Priv Limit
2   ADMIN            false   false      true       ADMINISTRATOR

In the BMC/KVM modules of the PX91/PX121 models, there are two active users with administrator rights:

ID  Name             Callin  Link Auth  IPMI Msg   Channel Priv Limit
1                    false   false      true       ADMINISTRATOR
2   admin            false   false      true       ADMINISTRATOR

In the PX60/PX70 models, there are 5 standard inactive users. You can change all of them except for the first one.

ID  Name             Callin  Link Auth  IPMI Msg   Channel Priv Limit
1                    true    false      true       ADMINISTRATOR
2   root             false   true       true       ADMINISTRATOR
3   test1            true    false      true       ADMINISTRATOR
4   test2            true    false      true       ADMINISTRATOR
5   test3            true    false      true       ADMINISTRATOR

You should deactivate the root (or ADMIN) user ID and, if possible, rename it after creating a customer user and before activating the network configuration.

Change the login name via ipmitool:

ipmitool user set name 2 john-doe

To create a new user, simply assign a previously unused ID a name. The procedure here is identical to changing the login of an ID. You can delete IDs only by altering the BMC settings.

Create a new user:

ipmitool user set name 6 max+meier

After that, you should set a password:

ipmitool user set password 6 Correct-Battery-Horse-Staple

Now activate the access for this user:

ipmitool channel setaccess 1 6 link=on ipmi=on callin=on privilege=4

Activate the user itself:

ipmitool user enable 6

To change the password of the user, simply enter the following command:

ipmitool user set password 6 Battery+Staple-Horse$Correct

Finally, you can disable the default admin user:

ipmitool user disable 2

Network

In order to make the BMC accessible via the internet, you need to order an additional IP for it via Robot. This IP costs a small fee. You can do the IPv4 configuration of the BMC either manually or via DHCP using ipmitool. You can make changes to this configuration using the web interface by going to Configuration / IPv4 Network. You cannot currently use IPv6. The configuration with IPv6 will become available later on the web interface.

You can set the initial configuration using ipmitool. The corresponding IPMI channel is dependent on the motherboard and which interface you would like to configure.

Shared LAN port of the main IP:

  • PX60/70/90/120 and SX131/291: Channel 1
  • PX91/121: Channel 8

To display the current configuration and the MAC address of the BMC, use the following command:

  • PX60/70/90/120 and SX131/291:
  • ipmitool lan print 1
  • PX91/121:
  • ipmitool lan print 8

As shown above, use set 8, rather than set 1, for this and all other commands for the PX91 and PX121 models.

To receive an IP via DHCP, use the following command:

ipmitool lan set 1 ipsrc dhcp

If you want to use the default static configuration, enter:

ipmitool lan set 1 ipsrc static

To set an IP address, enter:

ipmitool lan set 1 ipaddr <IP address>

To set a netmask, enter:

ipmitool lan set 1 netmask <netmask>

To set a gateway IP, enter:

ipmitool lan set 1 defgw ipaddr <gateway IP address>

Serial over LAN

In order to activate SOL (Serial over LAN), enter the following command:

ipmitool -C 3 -I lanplus -H <ipaddr> -U <user> -P <pass> sol activate

Using cipher suite 3 is essential (if that is not the default) because communication via LANplus is not possible otherwise.

If the following error message appears, you need to activate SOL for the user:

ipmitool -C 3 -I lanplus -H <ipaddr> -U <user> -P <pass> sol activate
Info: SOL payload disabled
ipmitool -C 3 -I lanplus -H <ipaddr> -U <user> -P <pass> sol payload enable <channel> <user-id>

After that, you can see the BIOS output. Accessing the boot loader and/or the booted system requires additional settings.

GRUB2

For GRUB2, simply change some lines to match the following in /etc/default/grub and re-generate the settings.

With the PX90/120 (Supermicro X9SRi-F), the serial console is on ttyS2/unit=2. With the PX91/121 (Asus Z10PA-U8), it is on ttyS1/unit=1. And with the PX60/70 (Intel S1200V3RPL), it is on ttyS0/unit=0. NOTE: You need to set the baud rate at 57600 with the PX91/121, and 115200 with all other models.

PX60/70

GRUB_CMDLINE_LINUX_DEFAULT="nomodeset console=tty0 console=ttyS0,115200n8"
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"

PX90/120, SX131/291

GRUB_CMDLINE_LINUX_DEFAULT="nomodeset console=tty0 console=ttyS2,115200n8"
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=2 --word=8 --parity=no --stop=1"

PX91/121

GRUB_CMDLINE_LINUX_DEFAULT="nomodeset console=tty0 console=ttyS1,57600n8"
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=57600 --unit=1 --word=8 --parity=no --stop=1"

GRUB (grub-legacy)

For GRUB1 (grub-legacy), add the following lines to /boot/grub/menu.lst or /boot/grub/grub.conf (CentOS):

PX60/70

serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
terminal --timeout=5 serial console

PX90/120, SX131/291

serial --unit=2 --speed=57600 --word=8 --parity=no --stop=1
terminal --timeout=5 serial console

PX91/121

serial --unit=1 --speed=115200 --word=8 --parity=no --stop=1
terminal --timeout=5 serial console

At the same time, you need to add the same serial port to the boot options of the kernel. That is ttyS0 with the PX60/70, ttyS1 with the PX91/121, and ttyS2 with the PX90/120.

console=tty0 console=ttyS0,115200n8

This tells the kernel to output information on the first serial port. Changing GRUB_TERMINAL to serial means any input/output is redirected to the serial port. A local screen will not display a boot menu anymore and thus, you will not be able to select a boot entry via KVM Console or KVM anymore. After a reboot, the output will be sent in parallel to both the local screen and the serial port.

After that, you need to set up a terminal for the serial port in your system.

Ubuntu

Create the file /etc/init/ttyS0.conf with the following content (or alternatively, ttyS2.conf with ttyS2 and 115200 baud with the PX90/PX120 models, or ttyS1.conf with ttyS1 and 57600 baud with the PX91/PX121 models):

# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]

respawn
exec /sbin/getty -L ttyS0 115200 vt100

After that, you can activate the terminal by entering start ttyS0.

CentOS

In CentOS, the configuration is similar to Ubuntu. However, /etc/init/serial.conf automatically starts a getty on the serial port, which adds the port /etc/securetty. So you just need to configure the serial console in grub.conf and attach the appropriate kernel option.

Debian / OpenSuSE / Fedora

For Debian, OpenSuSE, and other distributions such as Fedora which use systemd and GRUB2, just change /etc/default/grub accordingly and renew the configuration using grub2-mkconfig. At the next boot, systemd will automatically start using the serial port of GRUB2.

Serial Console

Now, you will see a login quickly if you connect via ipmitool:

ipmitool -C 3 -I lanplus -H <ipaddr> -U <user> -P <pass> sol activate
[SOL Session operational.  Use ~? for help]

Debian GNU/Linux 7 Debian-70-wheezy-64-minimal ttyS0

Debian-70-wheezy-64-minimal login:
Table of Contents