Introduction
If you are experiencing network problems with your server, then please send us a support request via your Robot account. Make sure to select the affected server in your support request. Depending on the problem you are experiencing, you can choose one of two request types.
Server is not reachable
If your server is not reachable, you can try to restart it yourself. To do this, go to Robot and then to the correct server, and click on the "Reset" tab. If that doesn't work, it is possible that we blocked your server. For more information about such situation, please also see our Guideline for server locking.
If you are still have the problem, please send us a ticket via Robot. Go to "Support" in the top right. Choose "Server" as the type of product, then pick the correct server, and then select "Network" as the type of support issue. Then pick "Server is not reachable".
Packet loss
If you are experiencing packet loss, then we will need some evidence of this. Simple statements such as "my ping is bad" or "there is packet loss to my server" are unfortunately not enough for an error analysis.
Please do a trace (in both directions) with at least 200 packets with a tool like MTR or WinMTR. You can install MTR via the package manager of the Linux distribution or MacOS. For Windows, you can download it from a specific website. The following table lists the ways you can install/download this tool for different operating systems.
| OS type | Specific OS | Installation |
|---|---|---|
| Linux | Debian/Ubuntu | apt install mtr-tiny |
| Linux | CentOS/RHEL | yum install mtr |
| Linux | SuSE | yast -i mtr |
| Linux | Arch Linux | pacman -S mtr |
| Linux | Gentoo | emerge -av mtr |
| Windows | Windows 98 and up | https://sourceforge.net/projects/winmtr/ |
| MacOS | * | brew install mtr (HomeBrew required) |
Please follow these guidelines to create traces that are useful for our technicians:
- Send at least 200 packets.
- Do each trace in both directions. That means you need to do it from your local endpoint (PC, Laptop etc.) to the server and from the server back to your endpoint.
- You can find the IP address of your local endpoint using an online tool, like https://ip.hetzner.com/.
You can use the following command to create an MTR with Linux or MacOS:
mtr -s 1000 -r -c 200 <TARGET-IP_OR_DOMAIN>To do this, run the above command on both systems and replace the placeholder with the correct remote IP address or domain:
[user@SERVER]$ mtr -s 1000 -r -c 200 <CLIENT-IP_OR_DOMAIN>
[user@CLIENT]$ mtr -s 1000 -r -c 200 <SERVER-IP_OR_DOMAIN>The test usually takes about 3-4 minutes.
Important:
- If your MTR shows the last hop to be unreachable, presumably the installed system or your local router is set to ignore ICMP requests due to security reasons. Nevertheless, the MTR can be used to investigate the connection.
- The hops of an MTR show the process for the specific connection. So your MTR might look quite different than the following examples.
These are problems that might become clear in test results: incorrect routing, very high latency on one of the network hops, or packet loss, which then leads to a re-transmission of those packets. But please be careful; there can be three types of packet loss:
1. Packet loss that disappears until the target hop
-
As you can see in this example, there is packet loss shown at hop 4 and 5:
1.|-- your_client.example.com 0.0% 1000 0.2 0.1 0.1 11.0 0.9 2.|-- dmbkt.your-cloud.host 0.0% 1000 0.2 0.2 0.1 11.0 0.8 3.|-- leaf1.cloud2.fsn1.hetzner 0.0% 1000 13.4 18.0 1.6 328.3 19.7 4.|-- spine1.cloud2.fsn1.hetzne 4.2% 1000 0.8 1.3 0.7 50.0 3.1 5.|-- core21.fsn1.hetzner.com 31.7% 1000 0.5 2.9 0.3 51.2 6.6 6.|-- core21.fsn1.hetzner.com 0.0% 1000 0.6 1.4 0.4 56.6 4.2 7.|-- ex9k2.dc1.fsn1.hetzner.co 0.0% 1000 0.6 1.8 0.4 214.1 12.0 8.|-- your_host.example.com 0.0% 1000 0.5 0.4 0.3 11.0 0.9 -
The packet loss returns to 0% before the connection reaches the last hop, so the MTR does not show any issue that might affect the connection to your server. This behavior is caused by routers that ignore ICMP packets. (They do this, for example, to save bandwidth or improve performance).
2. Packet loss only on the last hop
-
In this example, there is no packet loss except for the last hop:
1.|-- your_client.example.com 0.0% 1000 0.2 0.1 0.1 11.0 0.9 2.|-- dmbkt.your-cloud.host 0.0% 1000 0.2 0.2 0.1 11.0 0.8 3.|-- leaf1.cloud2.fsn1.hetzner 0.0% 1000 13.4 18.0 1.6 328.3 19.7 4.|-- spine1.cloud2.fsn1.hetzne 0.0% 1000 0.8 1.3 0.7 50.0 3.1 5.|-- core21.fsn1.hetzner.com 0.0% 1000 0.5 2.9 0.3 51.2 6.6 6.|-- core21.fsn1.hetzner.com 0.0% 1000 0.6 1.4 0.4 56.6 4.2 7.|-- ex9k2.dc1.fsn1.hetzner.co 0.0% 1000 0.6 1.8 0.4 214.1 12.0 8.|-- your_host.example.com 42.0% 1000 0.5 0.4 0.3 11.0 0.9 -
This problem is usually caused on the server itself. This might be the result of a fully utilized system performance, a wrongly configured software firewall, or in rare cases, the network card or uplink cable. If you see this kind of loss, please check your installed system first. If you can't find any of the mentioned problems on your system, please send us the MTRs, and we will investigate them from our end.
3. Packet loss on the connection
-
Here, the packet loss starts at hop 5 and continues until the last hop:
1.|-- your_client.example.com 0.0% 1000 0.2 0.1 0.1 11.0 0.9 2.|-- dmbkt.your-cloud.host 0.0% 1000 0.2 0.2 0.1 11.0 0.8 3.|-- leaf1.cloud2.fsn1.hetzner 0.0% 1000 13.4 18.0 1.6 328.3 19.7 4.|-- spine1.cloud2.fsn1.hetzne 0.0% 1000 0.8 1.3 0.7 50.0 3.1 5.|-- core21.fsn1.hetzner.com 55.1% 551 0.5 2.9 0.3 51.2 6.6 6.|-- core21.fsn1.hetzner.com 54.9% 549 0.6 1.4 0.4 56.6 4.2 7.|-- ex9k2.dc1.fsn1.hetzner.co 59.2% 592 0.6 1.8 0.4 214.1 12.0 8.|-- your_host.example.com 59.2% 592 0.5 0.4 0.3 11.0 0.9 -
In this case, please directly send us the MTRs, so we can investigate the issue. If the packet loss already starts outside of our network (meaning at a network hop, to which we do not have any access or influence), please contact the correct network provider and contact us.
If one of your MTRs shows a problem, please send a ticket via Robot by selecting the correct server, clicking on "Support" and then choosing "Packet loss". Please attach the output of both MTRs to the request.
MTR formatting
To make them easy to read, please always attach MTRs as a files (e.g. TXT or HTML) to the your support request or to your response to our messages.
If you want to post the output in forums (for example, the Hetzner Forum), please consider working on the text to make it as neat and tidy as possible. This make it easy for other users to easily understand it, and they may be more willing to help you. For example, you can use [CODE] blocks, so that the information is evenly spread out in the columns.
An unfavorable formatting would be:
1.|-- your_client.example.com 0.0% 1000 0.2 0.1 0.1 11.0 0.9
2.|-- dmbkt.your-cloud.host 0.0% 1000 0.2 0.2 0.1 11.0 0.8
3.|-- leaf1.cloud2.fsn1.hetzner 0.0% 1000 13.4 18.0 1.6 328.3 19.7
4.|-- spine1.cloud2.fsn1.hetzne 0.0% 1000 0.8 1.3 0.7 50.0 3.1
5.|-- core21.fsn1.hetzner.com 55.1% 1000 0.5 2.9 0.3 51.2 6.6
6.|-- core21.fsn1.hetzner.com 54.9% 1000 0.6 1.4 0.4 56.6 4.2
7.|-- ex9k2.dc1.fsn1.hetzner.co 59.2% 1000 0.6 1.8 0.4 214.1 12.0
8.|-- your_host.example.com 59.2% 1000 0.5 0.4 0.3 11.0 0.9However, formatted correctly:
1.|-- your_client.example.com 0.0% 1000 0.2 0.1 0.1 11.0 0.9
2.|-- dmbkt.your-cloud.host 0.0% 1000 0.2 0.2 0.1 11.0 0.8
3.|-- leaf1.cloud2.fsn1.hetzner 0.0% 1000 13.4 18.0 1.6 328.3 19.7
4.|-- spine1.cloud2.fsn1.hetzne 0.0% 1000 0.8 1.3 0.7 50.0 3.1
5.|-- core21.fsn1.hetzner.com 55.1% 1000 0.5 2.9 0.3 51.2 6.6
6.|-- core21.fsn1.hetzner.com 54.9% 1000 0.6 1.4 0.4 56.6 4.2
7.|-- ex9k2.dc1.fsn1.hetzner.co 59.2% 1000 0.6 1.8 0.4 214.1 12.0
8.|-- your_host.example.com 59.2% 1000 0.5 0.4 0.3 11.0 0.9Asymmetrical routing
A fault diagnosis is relatively easy if the round trip between the client and the server follows the same path. However, packets will often take a different return route. (This is caused by different requirements of client's provider and the server location).
Error example
In the following example, the Hetzner network seems to be the cause of the delays. Starting with the first Hetzner router, ping times are significantly increased and packets are discarded:
From client to server:
1 67 ms 65 ms 66 ms 62.26.xx.xx
2 63 ms 63 ms 65 ms 62.26.251.97
3 80 ms 74 ms 76 ms so-6-0-0.core3.f.tiscali.de (62.27.95.2)
4 74 ms 75 ms 73 ms so-3-0-0.fra30.ip.tiscali.net (213.200.64.25)
5 73 ms 74 ms 75 ms ffm-s2-rou-1071.DE.eurorings.net (80.81.192.22)
6 75 ms 74 ms 75 ms ffm-s1-rou-1001.DE.eurorings.net (134.222.227.65)
7 78 ms 79 ms 78 ms nbg-s1-rou-1071.DE.eurorings.net (134.222.227.30)
8 84 ms 78 ms 79 ms gi-0-1-286-nbg5.noris.net (134.222.107.26)
9 392 ms 393 ms * ne.gi-2-1.RS8000.RZ2.hetzner.de (213.133.96.65)
10 393 ms * 392 ms et-2-16.RS3000.RZ2.hetzner.de (213.133.96.38)
11 393 ms 392 ms * (...)But the actual source of the problem isn't clear until you look at a trace route in the other direction:
From server to client:
1 213.133.xx.xx (213.133.xx.xx) 0.233 ms 0.205 ms 0.281 ms
2 et-1-11.RS8000.RZ2.hetzner.de (213.133.96.37) 0.653 ms 0.660 ms 0.650 ms
3 nefkom-gw.hetzner.de (213.133.96.66) 1.119 ms 0.423 ms 0.415 ms
4 GW-SF-BBR-06-S2-3.nefkom.de (212.114.147.23) 0.635 ms 0.807 ms 0.457 ms
5 hsa2.mun1.pos6-0.eu.level3.net (212.162.44.25) 6.811 ms 6.347 ms 6.143 ms
6 ae0-19.mp1.Munich1.Level3.net (195.122.176.193) 315.587 ms 314.949 ms 315.164 ms
7 so-0-0-0.mp1.Frankfurt1.Level3.net (212.187.128.90) 301.324 ms 300.789 ms 300.742 ms
8 gige1-2.core1.Frankfurt1.Level3.net (195.122.136.101) 301.673 ms 300.853 ms 301.087 ms
9 de-cix.fra30.ip.tiscali.net (80.81.192.30) - 317.844 ms 317.634 ms
10 so-4-0-0.core3f.tiscali.de (213.200.64.26) 318.453 ms - 318.021 ms
11 so-1-0-0.core1.hh.tiscali.de (62.27.95.38) 307.780 ms 307.230 ms 307.252 ms
12 ge-2-0-0.7.core0.hh.tiscali.de (62.27.93.83) 307.431 ms 307.298 ms 307.084 ms
13 62.26.251.101 (62.26.251.101) - 307.753 ms 308.933 ms
14 (...) (...) 390.856 ms 399.355 ms -In this case, there was an error between two Level3 routers in Munich. The problem seemed to be with Hetzner since the packages up to hope 8 (noris) were successfully routed back. The test packets to the Hetzner network and back, however, took a different route back, and that resulted in the delay in Munich.
Other
Please do not use flood pings to troubleshoot your issue. Most networks (including Hetzner's) will interpret this as an attack. And the server initiating the pings may be taken offline very quickly!