Connecting to a Linux Server with a Private Key

Step-by-step guide for Windows, macOS, and Linux

Using PuTTY on Windows  •  OpenSSH on macOS & Linux

1. Overview

This guide walks you through every step required to securely connect to a Linux server using SSH key-based authentication. It covers the three most common client platforms: Windows (using PuTTY), macOS (using the built-in Terminal), and Linux (using the built-in terminal).

Key-based authentication is the standard, recommended way to access Linux servers. Unlike passwords, an SSH key pair cannot be brute-forced and is required by most cloud providers (AWS, Azure, GCP) by default.

What you will learn

• How SSH private-key authentication works at a high level

• How to install PuTTY and convert an OpenSSH .pem key to .ppk format on Windows

• How to configure a PuTTY session and connect using the converted key

• How to connect from macOS Terminal using ssh and a .pem key

• How to connect from a Linux terminal using ssh and a .pem key

• How to set up an SSH config file so you never have to type the full command again

• How to troubleshoot the most common errors

Prerequisites

• A Linux server you have access to (running an OpenSSH server on port 22 by default)

• The server's public IP address or hostname

• The default username for the server (e.g., ubuntu, ec2-user, centos, root)

• A private key file in OpenSSH format (.pem or no extension) — typically supplied by your cloud provider or system administrator

• Administrator access on your local machine to install software (Windows only)

How SSH key authentication works

A key pair consists of a private key, which lives only on your computer, and a public key, which is placed in the ~/.ssh/authorized_keys file on the server. When you connect, the server sends a challenge that only the holder of the private key can sign correctly. If the signature verifies against the stored public key, the server grants access.

Figure 1: SSH private-key authentication flow

2. Windows — Connecting with PuTTY

On Windows, the most common SSH client is PuTTY. PuTTY does not understand the OpenSSH .pem format directly — it uses its own .ppk format. Therefore the Windows workflow has four parts: install PuTTY, convert the key with PuTTYgen, configure the PuTTY session, and connect.

2.1  Download and install PuTTY

1. Open a web browser and go to the official PuTTY download page: https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html

2. Under "MSI ('Windows Installer')" download the 64-bit x86 installer (puttyXX-64bit-installer.msi) for most modern PCs. Choose 64-bit Arm if you are on an ARM-based Windows device.

Figure 2: The official PuTTY download page

1. Run the downloaded MSI installer. Accept the defaults — this installs PuTTY, PuTTYgen, Pageant, and PSCP/PSFTP.

2. After installation, you will find PuTTY and PuTTY Key Generator in the Windows Start menu.

2.2  Convert your .pem key to .ppk using PuTTYgen

PuTTY requires a .ppk file. If you were given a .pem file (or an OpenSSH key with no extension), convert it once using PuTTYgen.

1. From the Start menu, open PuTTY Key Generator (puttygen.exe).

2. In the menu bar, click Conversions → Import key.

3. In the file dialog, navigate to your .pem file. You may need to change the file type filter to "All files (*.*)" to see it. Select it and click Open.

4. PuTTYgen will load and display the public key and fingerprint. The Save private key button becomes available.

Figure 3: PuTTY Key Generator after importing an OpenSSH key — click Save private key to export as .ppk

1. Click Save private key. PuTTYgen may warn you that the key is not protected with a passphrase. For production use, consider adding a passphrase. For initial testing you can click Yes to save without one.

2. Choose a safe location (for example, C:\Users\<You>\keys\) and a clear filename like matilda-server.ppk. Click Save.

2.3  Configure a PuTTY session

1. Open PuTTY from the Start menu. The PuTTY Configuration window appears with the Session category selected.

2. In the Host Name (or IP address) field, enter the IP address or hostname of your Linux server (e.g., 203.0.113.42 or myserver.example.com).

3. Confirm the Port is 22 (the SSH default) and the Connection type is set to SSH.

Figure 4: PuTTY Session screen with all required fields filled in

1. In the left tree, expand SSH → Auth and click Credentials.

2. Next to Private key file for authentication, click Browse… and select the .ppk file you created in section 2.2.

Figure 5: PuTTY SSH ▸ Auth ▸ Credentials screen with the .ppk private key selected

1. (Optional but recommended) In the left tree click Connection → Data and enter the default username (e.g., ubuntu) in the Auto-login username field. This avoids the "login as:" prompt on every connection.

2. Scroll back to the top of the tree, click Session, type a memorable name (e.g., matilda-prod-server) in Saved Sessions, and click Save. The session is now reusable.

2.4  Connect to the server

1. On the Session screen, click Open.

2. The first time you connect to a server, PuTTY shows a PuTTY Security Alert asking whether to trust the server's host key. If you trust the server, click Accept to add it to PuTTY's known hosts. (Click Connect Once for a one-time session without saving.)

3. A terminal window opens. If you did not configure an auto-login username, type it now and press Enter.

4. PuTTY uses your private key automatically. You should see the server's welcome banner and a shell prompt — you are connected.

3. macOS — Connecting with Terminal

macOS includes the OpenSSH client out of the box, so you do not need to install anything. You will use the Terminal app and the standard ssh command. The .pem file from your cloud provider works directly — no conversion needed.

3.1  Open Terminal

• Press ⌘ + Space to open Spotlight, type Terminal, and press Enter.

• Or open Finder → Applications → Utilities → Terminal.

3.2  Move the key to ~/.ssh and set permissions

OpenSSH refuses to use a private key whose permissions are too permissive. The key must be readable only by you (mode 600).

1. In Terminal, run the commands below. Replace the filename with your actual key file:

Figure 6: Moving the key to ~/.ssh and tightening permissions on macOS

1. Confirm the listing shows -rw------- on the left. If it shows anything more permissive (such as -rw-r--r--), re-run the chmod command.

3.3  Connect with ssh -i

1. In Terminal, run the ssh command, substituting your username, key path, and server IP:

1. On first connection, ssh asks you to verify the server's fingerprint. Type yes and press Enter to add it to ~/.ssh/known_hosts.

2. You should see the server's welcome banner and a shell prompt.

Figure 7: A successful first SSH connection from macOS

3.4  Optional — set up an SSH config file

Typing the full ssh -i ... command every time is tedious. Add an entry to ~/.ssh/config so you can simply type ssh matilda.

1. Open ~/.ssh/config in your editor (create it if needed):

1. Add the following block, then save:

1. Tighten permissions on the config file:

1. From now on, simply run:

4. Linux — Connecting with the terminal

On Linux, the OpenSSH client is preinstalled on virtually every distribution (Ubuntu, Debian, RHEL, CentOS, Fedora, Arch, etc.). The workflow is essentially identical to macOS: move the key to ~/.ssh, restrict its permissions, and run ssh.

4.1  Verify ssh is installed

Run the following in your terminal:

You should see output similar to OpenSSH_9.x. If the command is not found, install the client:

• Debian/Ubuntu: sudo apt update && sudo apt install -y openssh-client

• RHEL/CentOS/Fedora: sudo dnf install -y openssh-clients

• Arch: sudo pacman -S openssh

4.2  Move the key to ~/.ssh and set permissions

1. Move your .pem file to ~/.ssh and set strict permissions:

Figure 8: Moving the key and setting permissions on Linux (Ubuntu shown)

1. The output should start with -rw-------.

4.3  Connect with ssh -i

1. Run the ssh command, replacing the username, key path, and server IP with your own:

1. Type yes at the host fingerprint prompt on first connection.

Figure 9: Connecting from a Linux terminal

4.4  Optional — set up an SSH config file

Just like on macOS, you can streamline future connections by editing ~/.ssh/config. The exact same syntax applies on Linux:

Figure 10: Using an SSH config alias to connect with a single short command

5. Troubleshooting common errors

5.1  "Permission denied (publickey)"

This is the most common error. It usually means the server received your connection but rejected the key. Check, in order:

• You are using the correct username for the server (Amazon Linux uses ec2-user, Ubuntu uses ubuntu, CentOS often uses centos, Debian uses admin or debian — using root is rarely correct).

• You are pointing ssh / PuTTY at the right key file. It is easy to mix up keys when you have several.

• On macOS / Linux, key permissions are wrong. Re-run chmod 600 ~/.ssh/your-key.pem.

• The public key has actually been added to the server's ~/.ssh/authorized_keys (your administrator must do this if you did not provision the server yourself).

5.2  "UNPROTECTED PRIVATE KEY FILE!" (macOS / Linux)

OpenSSH refuses to use a key that is readable by other users. Fix it with:

5.3  "Server refused our key" (PuTTY)

Same root causes as section 5.1, plus one Windows-specific gotcha: you tried to give PuTTY the .pem file directly. PuTTY only accepts .ppk — go back to section 2.2 and convert with PuTTYgen.

5.4  "Connection timed out" / "Network is unreachable"

The traffic is not reaching the server at all. Check, in order:

• The server's IP address is correct and the server is running.

• The cloud security group / firewall (AWS Security Group, Azure NSG, GCP firewall, ufw, firewalld) allows inbound TCP traffic on port 22 from your IP.

• Your local network or corporate VPN does not block outbound port 22.

5.5  "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!"

You connected before, the server's host key changed (often because the server was rebuilt), and ssh is protecting you against a possible man-in-the-middle attack. If the change is expected, remove the old fingerprint:

Then reconnect and accept the new fingerprint.

6. Best practices

• Always use a passphrase on production private keys. Combine with ssh-agent (or Pageant on Windows) so you only type it once per session.

• Prefer modern key types when generating new keys: ssh-keygen -t ed25519 creates a strong, compact key that all modern servers support.

• Rotate keys periodically — at least once a year, and immediately if anyone with access leaves the team.

• Never commit private keys to git. Add ~/.ssh and *.pem / *.ppk to your global .gitignore.

• Use one key per role or environment (dev, staging, prod) rather than one universal key — easier to revoke a single compromised key.

• On servers, disable password authentication entirely once key-based auth is working: set PasswordAuthentication no in /etc/ssh/sshd_config and reload sshd.

• Use an SSH config file so connection details are consistent and discoverable, instead of remembering long command-line invocations.

Summary of platform commands