---
title: "Servers"
description: "Add servers with one command, see their load, and remove them again."
canonical_url: "https://harblet.com/docs/administration/servers"
---
# Servers

> Add servers with one command, see their load, and remove them again.

Harblet starts on one server. Add more servers to run more services, and Harblet places services on all of them. When a server fails, its services start on another one.

<note>

The Servers page is for admins. Production use on more than one server needs a commercial license, see [License](https://harblet.com/docs/reference/license).

</note>

## The first server

The server you installed Harblet on is the **first server**. It has a special role:

- It runs Harblet, Harblet's database, the image registry and the build service.
- It receives all traffic for your domains and routes it to the services on every server. Point all DNS records at it.
- It can't be removed.

<warning>

If the first server fails, Harblet and all domains are unreachable until it's back, even though services on other servers keep running. Set up [backups of Harblet](https://harblet.com/docs/backups/harblet-backups) so you can restore it onto a new server.

</warning>

## See your servers

**Servers** in the sidebar lists every server with its status (**Ready** or **Not ready**), IP address, operating system, k3s version, and CPU and memory use. The page refreshes every 5 seconds. Admins also see a card per server on the **Overview** page.

## Add a server

<steps level="3">

### Prepare the server

Set up a fresh server with Ubuntu 24.04 or 26.04, like the first one. It must be able to:

- reach Harblet's address, to download the join script,
- reach the other servers directly on **TCP 6443** and **10250** and **UDP 51820**.

Servers in one private network, or servers with public IP addresses, work. Servers behind NAT at different providers don't, yet.

### Get the join command

On **Servers**, click **Add server**. Harblet shows a command:

```bash
curl -fsSL https://harblet.example.com/api/join-scripts/<token> | sudo sh
```

The command works for 24 hours. Every click creates a new one.

### Run it on the new server

Run the command on the new server as root. It installs the k3s agent at the same version as the first server. The server shows up on the page within seconds, and Harblet starts placing services on it.

</steps>

<tip>

If Harblet has its own domain, the new server downloads the join script over HTTPS, so it has to trust the domain's certificate. Certificates from public CAs work. For certificates from your own CA, add the CA to the new server first.

</tip>

## When a server fails

- **Services without data** start on another server. In tests, a failed server was marked **Not ready** after about 50 seconds, and its services ran on another server about 30 seconds later.
- **Databases** stay on the server that holds their data. They wait for it to come back.
- **When the server comes back,** it rejoins by itself.

Services keep running when Harblet itself is down. Only deploys, logs and backups need Harblet.

## Remove a server

<steps level="3">

### Delete its databases

Harblet refuses to remove a server that stores the data of a database. The message names them. Harblet can't move a database to another server, so download a backup first if you need the data, then delete them.

### Remove it

On **Servers**, click the trash button of the server and confirm. Harblet moves the server's services to the other servers and removes it. Join commands that are still valid stop working.

### Uninstall it

Run the command Harblet shows on the removed server:

```bash
sudo /usr/local/bin/k3s-agent-uninstall.sh
```

Until you run it, the removed server keeps trying to reconnect and its boot doesn't finish, though SSH still works.

</steps>

Removing a server that's powered off works too. You can add the same machine again later.

## k3s upgrades

Each Harblet release comes with the k3s version it was tested with. After you [update Harblet](https://harblet.com/docs/administration/updates), it upgrades k3s on every server by itself, the first server first and then one server at a time. Services keep running.

While a server upgrades, its k3s version shows **Upgrading to** the new version. If an upgrade gets stuck, an alert **The k3s upgrade is stuck** above the list shows Kubernetes' message. See [Troubleshooting](https://harblet.com/docs/reference/troubleshooting#a-k3s-upgrade-is-stuck).


## Sitemap

See the full [sitemap](https://harblet.com/sitemap.md) for all pages.