Instructions for an error report with Hetzner network problems

Last change on 2021-11-19 • Created on 2020-03-18

Introduction

If you are experiencing network problems with your server, then please send us a support request via your Robot account. Make sure to select the affected server in your support request. Depending on the problem you are experiencing, you can choose one of two request types.

Server is not reachable

If your server is not reachable, you can try to restart it yourself. To do this, go to Robot and then to the correct server, and click on the "Reset" tab. If that doesn't work, it is possible that we blocked your server. For more information about such situation, please also see our Guideline for server locking.

If you are still have the problem, please send us a ticket via Robot. Go to the correct server, click on "Support" and then choose "Server Problems" -> "Server is not reachable".

Packet loss

If you are experiencing packet loss, then we will need some evidence of this. Simple statements such as "my ping is bad" or "there is packet loss to my server" are unfortunately not enough for an error analysis.

Please do a trace (in both directions) with at least 1000 packets with a tool like MTR or WinMTR. You can install MTR via the package manager of the Linux distribution or MacOS. For Windows, you can download it from a specific website. The following table lists the ways you can install/download this tool for different operating systems.

OS type Specific OS Installation
Linux Debian/Ubuntu apt install mtr-tiny
Linux CentOS/RHEL yum install mtr
Linux SuSE yast -i mtr
Linux Arch Linux pacman -S mtr
Linux Gentoo emerge -av mtr
Windows Windows 98 and up https://sourceforge.net/projects/winmtr/
MacOS * brew install mtr (HomeBrew required)

Please follow these guidelines to create traces that are useful for our technicians:

  • Send at least 1000 packets.
  • Do each trace in both directions. That means you need to do it from your local endpoint (PC, Laptop etc.) to the server and from the server back to your endpoint.
  • You can find the IP address of your local endpoint using an online tool, like https://ip.hetzner.com/.

You can use the following command to create an MTR with Linux or MacOS:

mtr -s 1000 -r -c 1000 <TARGET-IP_OR_DOMAIN>

To do this, run the above command on both systems and replace the placeholder with the correct remote IP address or domain:

[user@SERVER]$ mtr -s 1000 -r -c 1000 <CLIENT-IP_OR_DOMAIN>
[user@CLIENT]$ mtr -s 1000 -r -c 1000 <SERVER-IP_OR_DOMAIN>

The test usually takes about 18 minutes.

Important:

  • If your MTR shows the last hop to be unreachable, presumably the installed system or your local router is set to ignore ICMP requests due to security reasons. Nevertheless, the MTR can be used to investigate the connection.
  • The hops of an MTR show the process for the specific connection. So your MTR might look quite different than the following examples.

These are problems that might become clear in test results: incorrect routing, very high latency on one of the network hops, or packet loss, which then leads to a re-transmission of those packets. But please be careful; there can be three types of packet loss:

1. Packet loss that disappears until the target hop

  • As you can see in this example, there is packet loss shown at hop 4 and 5:

     1.|-- your_client.example.com    0.0%  1000    0.2   0.1   0.1  11.0   0.9
     2.|-- dmbkt.your-cloud.host      0.0%  1000    0.2   0.2   0.1  11.0   0.8
     3.|-- leaf1.cloud2.fsn1.hetzner  0.0%  1000   13.4  18.0   1.6 328.3  19.7
     4.|-- spine1.cloud2.fsn1.hetzne  4.2%  1000    0.8   1.3   0.7  50.0   3.1
     5.|-- core21.fsn1.hetzner.com   31.7%  1000    0.5   2.9   0.3  51.2   6.6
     6.|-- core21.fsn1.hetzner.com    0.0%  1000    0.6   1.4   0.4  56.6   4.2
     7.|-- ex9k2.dc1.fsn1.hetzner.co  0.0%  1000    0.6   1.8   0.4 214.1  12.0
     8.|-- your_host.example.com      0.0%  1000    0.5   0.4   0.3  11.0   0.9
  • The packet loss returns to 0% before the connection reaches the last hop, so the MTR does not show any issue that might affect the connection to your server. This behavior is caused by routers that ignore ICMP packets. (They do this, for example, to save bandwidth or improve performance).

2. Packet loss only on the last hop

  • In this example, there is no packet loss except for the last hop:

     1.|-- your_client.example.com    0.0%  1000    0.2   0.1   0.1  11.0   0.9
     2.|-- dmbkt.your-cloud.host      0.0%  1000    0.2   0.2   0.1  11.0   0.8
     3.|-- leaf1.cloud2.fsn1.hetzner  0.0%  1000   13.4  18.0   1.6 328.3  19.7
     4.|-- spine1.cloud2.fsn1.hetzne  0.0%  1000    0.8   1.3   0.7  50.0   3.1
     5.|-- core21.fsn1.hetzner.com    0.0%  1000    0.5   2.9   0.3  51.2   6.6
     6.|-- core21.fsn1.hetzner.com    0.0%  1000    0.6   1.4   0.4  56.6   4.2
     7.|-- ex9k2.dc1.fsn1.hetzner.co  0.0%  1000    0.6   1.8   0.4 214.1  12.0
     8.|-- your_host.example.com     42.0%  1000    0.5   0.4   0.3  11.0   0.9
  • This problem is usually caused on the server itself. This might be the result of a fully utilized system performance, a wrongly configured software firewall, or in rare cases, the network card or uplink cable. If you see this kind of loss, please check your installed system first. If you can't find any of the mentioned problems on your system, please send us the MTRs, and we will investigate them from our end.

3. Packet loss on the connection

  • Here, the packet loss starts at hop 5 and continues until the last hop:

     1.|-- your_client.example.com    0.0%  1000    0.2   0.1   0.1  11.0   0.9
     2.|-- dmbkt.your-cloud.host      0.0%  1000    0.2   0.2   0.1  11.0   0.8
     3.|-- leaf1.cloud2.fsn1.hetzner  0.0%  1000   13.4  18.0   1.6 328.3  19.7
     4.|-- spine1.cloud2.fsn1.hetzne  0.0%  1000    0.8   1.3   0.7  50.0   3.1
     5.|-- core21.fsn1.hetzner.com   55.1%   551    0.5   2.9   0.3  51.2   6.6
     6.|-- core21.fsn1.hetzner.com   54.9%   549    0.6   1.4   0.4  56.6   4.2
     7.|-- ex9k2.dc1.fsn1.hetzner.co 59.2%   592    0.6   1.8   0.4 214.1  12.0
     8.|-- your_host.example.com     59.2%   592    0.5   0.4   0.3  11.0   0.9
  • In this case, please directly send us the MTRs, so we can investigate the issue. If the packet loss already starts outside of our network (meaning at a network hop, to which we do not have any access or influence), please contact the correct network provider and contact us.

If one of your MTRs shows a problem, please send a ticket via Robot by selecting the correct server, clicking on "Support" and then choosing "Packet loss". Please attach the output of both MTRs to the request.

MTR formatting

To make them easy to read, please always attach MTRs as a files (e.g. TXT or HTML) to the your support request or to your response to our messages.

If you want to post the output in forums (for example, the Hetzner Forum), please consider working on the text to make it as neat and tidy as possible. This make it easy for other users to easily understand it, and they may be more willing to help you. For example, you can use [CODE] blocks, so that the information is evenly spread out in the columns.

An unfavorable formatting would be:

  1.|-- your_client.example.com 0.0% 1000 0.2 0.1 0.1 11.0 0.9
  2.|-- dmbkt.your-cloud.host 0.0% 1000 0.2 0.2 0.1 11.0 0.8
  3.|-- leaf1.cloud2.fsn1.hetzner 0.0% 1000 13.4 18.0 1.6 328.3 19.7
  4.|-- spine1.cloud2.fsn1.hetzne 0.0% 1000 0.8 1.3 0.7 50.0 3.1
  5.|-- core21.fsn1.hetzner.com 55.1% 1000 0.5 2.9 0.3 51.2 6.6
  6.|-- core21.fsn1.hetzner.com 54.9% 1000 0.6 1.4 0.4 56.6 4.2
  7.|-- ex9k2.dc1.fsn1.hetzner.co 59.2% 1000 0.6 1.8 0.4 214.1 12.0
  8.|-- your_host.example.com 59.2% 1000 0.5 0.4 0.3 11.0 0.9

However, formatted correctly:

  1.|-- your_client.example.com    0.0%  1000    0.2   0.1   0.1  11.0   0.9
  2.|-- dmbkt.your-cloud.host      0.0%  1000    0.2   0.2   0.1  11.0   0.8
  3.|-- leaf1.cloud2.fsn1.hetzner  0.0%  1000   13.4  18.0   1.6 328.3  19.7
  4.|-- spine1.cloud2.fsn1.hetzne  0.0%  1000    0.8   1.3   0.7  50.0   3.1
  5.|-- core21.fsn1.hetzner.com   55.1%  1000    0.5   2.9   0.3  51.2   6.6
  6.|-- core21.fsn1.hetzner.com   54.9%  1000    0.6   1.4   0.4  56.6   4.2
  7.|-- ex9k2.dc1.fsn1.hetzner.co 59.2%  1000    0.6   1.8   0.4 214.1  12.0
  8.|-- your_host.example.com     59.2%  1000    0.5   0.4   0.3  11.0   0.9

Asymmetrical routing

A fault diagnosis is relatively easy if the round trip between the client and the server follows the same path. However, packets will often take a different return route. (This is caused by different requirements of client's provider and the server location).

Error example

In the following example, the Hetzner network seems to be the cause of the delays. Starting with the first Hetzner router, ping times are significantly increased and packets are discarded:

From client to server:

1     67 ms     65 ms     66 ms     62.26.xx.xx
2     63 ms     63 ms     65 ms     62.26.251.97
3     80 ms     74 ms     76 ms     so-6-0-0.core3.f.tiscali.de          (62.27.95.2)
4     74 ms     75 ms     73 ms     so-3-0-0.fra30.ip.tiscali.net        (213.200.64.25)
5     73 ms     74 ms     75 ms     ffm-s2-rou-1071.DE.eurorings.net     (80.81.192.22)
6     75 ms     74 ms     75 ms     ffm-s1-rou-1001.DE.eurorings.net     (134.222.227.65)
7     78 ms     79 ms     78 ms     nbg-s1-rou-1071.DE.eurorings.net     (134.222.227.30)
8     84 ms     78 ms     79 ms     gi-0-1-286-nbg5.noris.net            (134.222.107.26)
9     392 ms    393 ms    *         ne.gi-2-1.RS8000.RZ2.hetzner.de      (213.133.96.65)
10    393 ms    *         392 ms    et-2-16.RS3000.RZ2.hetzner.de        (213.133.96.38)
11    393 ms    392 ms    *         (...)

But the actual source of the problem isn't clear until you look at a trace route in the other direction:

From server to client:

1     213.133.xx.xx                         (213.133.xx.xx)     0.233 ms     0.205 ms     0.281 ms
2     et-1-11.RS8000.RZ2.hetzner.de         (213.133.96.37)     0.653 ms     0.660 ms     0.650 ms
3     nefkom-gw.hetzner.de                  (213.133.96.66)     1.119 ms     0.423 ms     0.415 ms
4     GW-SF-BBR-06-S2-3.nefkom.de           (212.114.147.23)    0.635 ms     0.807 ms     0.457 ms
5     hsa2.mun1.pos6-0.eu.level3.net        (212.162.44.25)     6.811 ms     6.347 ms     6.143 ms
6     ae0-19.mp1.Munich1.Level3.net         (195.122.176.193)   315.587 ms   314.949 ms   315.164 ms
7     so-0-0-0.mp1.Frankfurt1.Level3.net    (212.187.128.90)    301.324 ms   300.789 ms   300.742 ms
8     gige1-2.core1.Frankfurt1.Level3.net   (195.122.136.101)   301.673 ms   300.853 ms   301.087 ms
9     de-cix.fra30.ip.tiscali.net           (80.81.192.30)      -            317.844 ms   317.634 ms
10    so-4-0-0.core3f.tiscali.de            (213.200.64.26)     318.453 ms   -            318.021 ms
11    so-1-0-0.core1.hh.tiscali.de          (62.27.95.38)       307.780 ms   307.230 ms   307.252 ms
12    ge-2-0-0.7.core0.hh.tiscali.de        (62.27.93.83)       307.431 ms   307.298 ms   307.084 ms
13    62.26.251.101                         (62.26.251.101)     -            307.753 ms   308.933 ms
14    (...)                                 (...)               390.856 ms   399.355 ms   -

In this case, there was an error between two Level3 routers in Munich. The problem seemed to be with Hetzner since the packages up to hope 8 (noris) were successfully routed back. The test packets to the Hetzner network and back, however, took a different route back, and that resulted in the delay in Munich.

Other

Please do not use flood pings to troubleshoot your issue. Most networks (including Hetzner's) will interpret this as an attack. And the server initiating the pings may be taken offline very quickly!

Table of Contents