Skip to content
XC0DE
Cardano Stake Pool
Menu
Home
Pool Identity
How to Stake
Infra
Stats
Links
Docs
Support
KNOWLEDGE
Knowledge Base
Just guides and notes from my own work.
Choose a document.
# Managing `systemd` on RHEL 9 ## Overview This guide covers the commands and checks used to manage `systemd` services on RHEL 9. It includes service discovery, status checks, log review, boot analysis, timer management, unit-file overrides, failure recovery, SELinux troubleshooting, and journal maintenance. Commands that only display information can normally be run as a standard user. Commands that change service state, unit files, timers, targets, or journal retention generally require `sudo`. > **Production warning** > > Before stopping, disabling, masking, or restarting a service, confirm what the service does and what depends on it. An incorrect change can interrupt SSH, networking, storage, authentication, logging, or other critical system functions. --- ## Quick Reference | Task | Command | |---|---| | List running services | `systemctl list-units --type=service --state=running` | | List enabled services | `systemctl list-unit-files --type=service --state=enabled` | | Check service status | `systemctl status <service>.service` | | Start a service | `sudo systemctl start <service>.service` | | Stop a service | `sudo systemctl stop <service>.service` | | Restart a service | `sudo systemctl restart <service>.service` | | Reload a service | `sudo systemctl reload <service>.service` | | Enable at boot | `sudo systemctl enable <service>.service` | | Disable at boot | `sudo systemctl disable <service>.service` | | Follow service logs | `journalctl -u <service>.service -f` | | List failed units | `systemctl --failed` | | Show unit configuration | `systemctl cat <service>.service` | | Reload unit files | `sudo systemctl daemon-reload` | | Analyze boot time | `systemd-analyze` | | List timers | `systemctl list-timers --all` | --- # Service Administration ## 1. List Services Use these commands to identify services that are running, active, enabled, disabled, or installed on the system. ### 1.1 List running services ```bash systemctl list-units --type=service --state=running ``` This displays service units that are currently in the `running` state. ### 1.2 List all active services ```bash systemctl list-units --type=service --state=active ``` This also includes active services that completed successfully and remain in an `exited` state. ### 1.3 List services enabled at boot ```bash systemctl list-unit-files --type=service --state=enabled ``` > **Note** > > A service can be enabled without currently running. A service can also be running without being enabled. ### 1.4 List all installed service unit files ```bash systemctl list-unit-files --type=service ``` Common unit-file states include: | State | Meaning | |---|---| | `enabled` | Configured to start through one or more target dependencies | | `disabled` | Installed but not configured to start automatically | | `static` | Cannot be enabled directly; normally started as a dependency | | `masked` | Explicitly blocked from starting | | `indirect` | Enabled through another unit or alias | ### 1.5 List all loaded service units ```bash systemctl list-units --type=service --all ``` This includes active, inactive, failed, and loaded service units. --- ## 2. Check Service Status ### 2.1 Display the full service status ```bash systemctl status <service>.service ``` Example: ```bash systemctl status sshd.service ``` The output normally includes: - Unit-file location - Enablement state - Active state - Main process ID - Start command - Recent journal entries - Exit status when the service failed ### 2.2 Check whether a service is active ```bash systemctl is-active <service>.service ``` Example: ```bash systemctl is-active sshd.service ``` Expected output for a running service: ```text active ``` ### 2.3 Check whether a service is enabled ```bash systemctl is-enabled <service>.service ``` ### 2.4 Check whether a service is failed ```bash systemctl is-failed <service>.service ``` ### 2.5 Use service checks in a script ```bash if systemctl is-active --quiet sshd.service; then echo "sshd is running" else echo "sshd is not running" fi ``` The `is-active`, `is-enabled`, and `is-failed` commands return exit codes that can be used in scripts and monitoring checks. --- ## 3. Start, Stop, Reload, and Restart Services ### 3.1 Start a service ```bash sudo systemctl start <service>.service ``` ### 3.2 Stop a service ```bash sudo systemctl stop <service>.service ``` ### 3.3 Restart a service ```bash sudo systemctl restart <service>.service ``` A restart stops the service and starts it again. Active connections or in-progress requests may be interrupted. ### 3.4 Reload a service configuration ```bash sudo systemctl reload <service>.service ``` A reload asks the application to reload its configuration without a full restart. Check whether the service supports reload: ```bash systemctl show <service>.service --property=CanReload ``` ### 3.5 Reload when supported, otherwise restart ```bash sudo systemctl reload-or-restart <service>.service ``` ### 3.6 Restart only when already running ```bash sudo systemctl try-restart <service>.service ``` This avoids starting a service that is currently stopped. > **Production practice** > > Use the application's native configuration test before reloading or restarting it. This is especially important for SSH, web servers, DNS, firewalls, and authentication services. --- ## 4. Enable and Disable Services ### 4.1 Enable a service at boot ```bash sudo systemctl enable <service>.service ``` ### 4.2 Enable and start a service immediately ```bash sudo systemctl enable --now <service>.service ``` ### 4.3 Disable a service at boot ```bash sudo systemctl disable <service>.service ``` Disabling a running service does not stop it. ### 4.4 Disable and stop a service immediately ```bash sudo systemctl disable --now <service>.service ``` ### 4.5 Confirm the enablement state ```bash systemctl is-enabled <service>.service ``` > **Important** > > Do not disable a service only because it is unfamiliar. Some services are activated through dependencies, sockets, timers, D-Bus, udev, or other systemd units. --- ## 5. Mask and Unmask Services Masking is stronger than disabling. A masked unit cannot be started manually, automatically, or as a dependency. ### 5.1 Mask a service ```bash sudo systemctl mask <service>.service ``` ### 5.2 Stop and mask a service ```bash sudo systemctl mask --now <service>.service ``` ### 5.3 Unmask a service ```bash sudo systemctl unmask <service>.service ``` ### 5.4 Confirm whether a service is masked ```bash systemctl is-enabled <service>.service ``` Expected output: ```text masked ``` > **Critical warning** > > Do not mask SSH, networking, storage, authentication, logging, or system targets unless console or out-of-band access is available and the rollback procedure has been tested. --- # Logging and Troubleshooting ## 6. Review Service Logs ### 6.1 View all available logs for a service ```bash journalctl -u <service>.service ``` Example: ```bash journalctl -u sshd.service ``` ### 6.2 View logs from the current boot ```bash journalctl -b -u <service>.service ``` ### 6.3 View logs from the previous boot ```bash journalctl -b -1 -u <service>.service ``` ### 6.4 List recorded boots ```bash journalctl --list-boots ``` ### 6.5 Follow logs in real time ```bash journalctl -u <service>.service -f ``` ### 6.6 Show the most recent entries ```bash journalctl -u <service>.service -n 100 ``` ### 6.7 Disable the pager ```bash journalctl -u <service>.service --no-pager ``` ### 6.8 Show logs from a relative time ```bash journalctl --since "30 minutes ago" -u <service>.service ``` ### 6.9 Show logs from a specific time range ```bash journalctl \ --since "2026-07-16 12:00:00" \ --until "2026-07-16 14:00:00" \ -u <service>.service ``` ### 6.10 Filter by severity Errors and higher-priority messages: ```bash journalctl -p err -u <service>.service ``` Warnings and higher-priority messages: ```bash journalctl -p warning -u <service>.service ``` ### 6.11 Display detailed journal fields ```bash journalctl -u <service>.service -o verbose ``` ### 6.12 Display precise timestamps ```bash journalctl -u <service>.service -o short-iso-precise ``` --- ## 7. Find Failed Services ### 7.1 List all failed units ```bash systemctl --failed ``` ### 7.2 List failed services only ```bash systemctl --failed --type=service ``` ### 7.3 Inspect a failed service ```bash systemctl status <service>.service ``` ### 7.4 Review current-boot logs ```bash journalctl -b -u <service>.service ``` ### 7.5 Review the recorded failure result ```bash systemctl show <service>.service \ --property=Result,ExecMainCode,ExecMainStatus ``` Example: ```text Result=exit-code ExecMainCode=exited ExecMainStatus=1 ``` > **Operational rule** > > Do not restart every failed service automatically. Determine why each unit failed before changing its state. --- ## 8. Check Service Dependencies ### 8.1 Show service dependencies ```bash systemctl list-dependencies <service>.service ``` ### 8.2 Show reverse dependencies ```bash systemctl list-dependencies --reverse <service>.service ``` Reverse dependencies identify units that depend on the selected service. ### 8.3 Show all dependency levels ```bash systemctl list-dependencies --all <service>.service ``` ### 8.4 Show the service startup chain ```bash systemd-analyze critical-chain <service>.service ``` This is useful when a service is waiting on another unit during startup. --- ## 9. Debug Service Failures ### 9.1 Start a service with systemd debug output ```bash sudo SYSTEMD_LOG_LEVEL=debug systemctl start <service>.service ``` This increases diagnostic output from `systemctl` and systemd. It does not automatically enable application-level debugging. ### 9.2 Show boot errors ```bash journalctl -b -p err ``` ### 9.3 Show boot warnings and errors ```bash journalctl -b -p warning ``` ### 9.4 Show the configured start command ```bash systemctl show <service>.service --property=ExecStart ``` ### 9.5 Show the service account ```bash systemctl show <service>.service --property=User,Group ``` ### 9.6 Show environment configuration ```bash systemctl show <service>.service \ --property=Environment,EnvironmentFiles ``` ### 9.7 Show the working directory ```bash systemctl show <service>.service --property=WorkingDirectory ``` ### 9.8 Show selected resource limits ```bash systemctl show <service>.service \ --property=LimitNOFILE,LimitNPROC,MemoryMax,CPUQuotaPerSecUSec ``` ### 9.9 Show pending systemd jobs ```bash systemctl list-jobs ``` ### 9.10 Enter rescue mode ```bash sudo systemctl rescue ``` ### 9.11 Enter emergency mode ```bash sudo systemctl emergency ``` > **Critical warning** > > Rescue and emergency mode can stop services and terminate remote sessions. Use them only from a console or out-of-band management connection. --- ## 10. Check SELinux, Ports, and Permissions A service can fail even when the unit file is valid. Check SELinux, ports, ownership, permissions, and executable paths. ### 10.1 Check recent SELinux denials ```bash sudo ausearch -m AVC,USER_AVC -ts recent ``` ### 10.2 Generate a readable SELinux report ```bash sudo sealert -a /var/log/audit/audit.log ``` Install the required package when needed: ```bash sudo dnf install setroubleshoot-server ``` > **SELinux rule** > > Do not disable SELinux just to make a service start. Identify the denial and correct the file context, permissions, or policy issue. ### 10.3 Check listening ports ```bash sudo ss -lntup ``` ### 10.4 Check a specific port ```bash sudo ss -lntup | grep ':<port>' ``` ### 10.5 Identify the process using a port ```bash sudo lsof -i :<port> ``` ### 10.6 Check file ownership and permissions ```bash ls -l <path> ``` ### 10.7 Check every component in a path ```bash namei -l <path> ``` ### 10.8 Check the SELinux file context ```bash ls -Z <path> ``` ### 10.9 Restore the default SELinux context ```bash sudo restorecon -Rv <path> ``` ### 10.10 Verify the configured executable ```bash systemctl show <service>.service --property=ExecStart ``` Then verify that the executable exists: ```bash ls -l <executable-path> ``` --- # Unit Files and Configuration ## 11. Review Unit Files and Overrides ### 11.1 Show the complete effective unit configuration ```bash systemctl cat <service>.service ``` This displays the vendor unit file followed by any drop-in override files. ### 11.2 Show the primary unit-file path ```bash systemctl show <service>.service --property=FragmentPath ``` ### 11.3 Show all service properties ```bash systemctl show <service>.service ``` ### 11.4 Show selected service properties ```bash systemctl show <service>.service \ --property=ActiveState,SubState,UnitFileState,MainPID,ExecMainStatus ``` ### 11.5 Check for local changes and overrides ```bash systemd-delta ``` ### 11.6 Show extended overrides only ```bash systemd-delta --type=extended ``` ### Unit-file locations | Path | Purpose | |---|---| | `/usr/lib/systemd/system/` | Vendor unit files installed by RPM packages | | `/etc/systemd/system/` | Administrator-created units and overrides | | `/run/systemd/system/` | Runtime-generated unit files | > **Important** > > Do not edit vendor files under `/usr/lib/systemd/system/` directly. Package updates can overwrite those changes. --- ## 12. Edit a Service Safely ### 12.1 Create or edit a drop-in override ```bash sudo systemctl edit <service>.service ``` The override is normally stored at: ```text /etc/systemd/system/<service>.service.d/override.conf ``` Example: ```ini [Service] Restart=on-failure RestartSec=5s ``` ### 12.2 Edit a full replacement unit ```bash sudo systemctl edit --full <service>.service ``` Use a full replacement only when a drop-in override cannot provide the required change. ### 12.3 Revert local changes ```bash sudo systemctl revert <service>.service ``` ### 12.4 Review the resulting configuration ```bash systemctl cat <service>.service ``` > **Production practice** > > Record the original unit configuration before making changes. Review the effective configuration again before restarting the service. --- ## 13. Reload systemd After Changes ### 13.1 Reload unit-file configuration ```bash sudo systemctl daemon-reload ``` Run `daemon-reload` after creating, modifying, or deleting: - Service files - Timer files - Socket files - Mount files - Drop-in overrides ### 13.2 Re-execute the systemd manager ```bash sudo systemctl daemon-reexec ``` `daemon-reexec` serializes the manager state and re-executes the systemd manager process. > **Important** > > Use `daemon-reload` for normal unit-file changes. Do not use `daemon-reexec` as a routine replacement for `daemon-reload`. --- ## 14. Validate Unit Files ### 14.1 Validate a service unit ```bash systemd-analyze verify /etc/systemd/system/<service>.service ``` ### 14.2 Validate a service and its override ```bash systemd-analyze verify \ /etc/systemd/system/<service>.service \ /etc/systemd/system/<service>.service.d/override.conf ``` ### 14.3 Reload systemd after validation ```bash sudo systemctl daemon-reload ``` ### 14.4 Review the loaded configuration ```bash systemctl cat <service>.service ``` ### 14.5 Validate the application configuration Use the application's native validation command before restarting or reloading it. ```bash sudo sshd -t sudo nginx -t sudo apachectl configtest sudo named-checkconf ``` ### 14.6 Confirm the service after the change ```bash systemctl status <service>.service ``` --- # Performance and Scheduling ## 15. Analyze Boot Performance ### 15.1 Check the total boot time ```bash systemd-analyze ``` Example: ```text Startup finished in 1.234s (kernel) + 4.567s (userspace) = 5.801s ``` ### 15.2 Show units ordered by startup time ```bash systemd-analyze blame ``` ### 15.3 Show the 25 slowest units ```bash systemd-analyze blame | head -n 25 ``` > **Interpretation note** > > A long duration in `systemd-analyze blame` does not always mean that the unit delayed the entire boot. Many units start in parallel. ### 15.4 Show the critical boot chain ```bash systemd-analyze critical-chain ``` The critical chain is usually more useful than `blame` when identifying what directly delayed boot completion. ### 15.5 Check a specific service in the boot chain ```bash systemd-analyze critical-chain <service>.service ``` ### 15.6 Generate a boot timeline ```bash systemd-analyze plot > boot-analysis.svg ``` Open the file in a browser: ```bash firefox boot-analysis.svg ``` ### 15.7 Check the overall system state ```bash systemctl is-system-running ``` Common results include: | State | Meaning | |---|---| | `running` | System startup completed successfully | | `degraded` | Startup completed, but one or more units failed | | `starting` | System startup is still in progress | | `maintenance` | System is in rescue or emergency mode | --- ## 16. Manage systemd Timers ### 16.1 List all timers ```bash systemctl list-timers --all ``` ### 16.2 Check a timer ```bash systemctl status <timer>.timer ``` ### 16.3 Show timer properties ```bash systemctl show <timer>.timer ``` ### 16.4 Show selected timer fields ```bash systemctl show <timer>.timer \ --property=ActiveState,SubState,LastTriggerUSec,NextElapseUSecRealtime ``` ### 16.5 Start a timer ```bash sudo systemctl start <timer>.timer ``` Starting the timer activates its schedule. It does not always execute the associated task immediately. ### 16.6 Run the associated service immediately ```bash sudo systemctl start <timer>.service ``` ### 16.7 Enable a timer at boot ```bash sudo systemctl enable <timer>.timer ``` ### 16.8 Enable and start a timer immediately ```bash sudo systemctl enable --now <timer>.timer ``` ### 16.9 Disable a timer ```bash sudo systemctl disable <timer>.timer ``` ### 16.10 Disable and stop a timer ```bash sudo systemctl disable --now <timer>.timer ``` ### 16.11 Review timer logs ```bash journalctl -u <timer>.timer ``` ### 16.12 Review the associated service logs ```bash journalctl -u <timer>.service ``` The associated service log normally contains the command output and failure details. --- # Recovery and Maintenance ## 17. Reset Failed States ### 17.1 Reset one failed service ```bash sudo systemctl reset-failed <service>.service ``` ### 17.2 Reset all failed units ```bash sudo systemctl reset-failed ``` Resetting the failed state only clears the status stored by systemd. It does not correct the original problem. ### Recommended order 1. Check the service status. 2. Review the logs. 3. Correct the problem. 4. Restart the service. 5. Confirm that it is working. 6. Reset the failed state when needed. --- ## 18. Manage Journal Storage ### 18.1 Check journal disk usage ```bash journalctl --disk-usage ``` ### 18.2 Remove archived journals older than two weeks ```bash sudo journalctl --vacuum-time=2weeks ``` ### 18.3 Limit archived journal storage to 1 GB ```bash sudo journalctl --vacuum-size=1G ``` ### 18.4 Rotate the journal before vacuuming ```bash sudo journalctl --rotate sudo journalctl --vacuum-time=2weeks ``` ### 18.5 Review the active journal configuration ```bash grep -vE '^\s*(#|$)' /etc/systemd/journald.conf ``` Persistent journal configuration is normally stored in: ```text /etc/systemd/journald.conf ``` Administrator drop-ins can be stored in: ```text /etc/systemd/journald.conf.d/ ``` ### 18.6 Create a journal retention drop-in ```bash sudo mkdir -p /etc/systemd/journald.conf.d sudo vi /etc/systemd/journald.conf.d/retention.conf ``` Example: ```ini [Journal] SystemMaxUse=1G MaxRetentionSec=2week Compress=yes ``` Restart journald after changing its configuration: ```bash sudo systemctl restart systemd-journald.service ``` > **Important** > > Collect the logs required for troubleshooting before rotating, vacuuming, or deleting journal data. --- ## 19. Standard Troubleshooting Workflow Use this sequence when a service does not start, stops unexpectedly, or enters a failed state. ### Step 1: Check the current status ```bash systemctl status <service>.service --no-pager ``` ### Step 2: Review recent logs ```bash journalctl -b -u <service>.service -n 100 --no-pager ``` ### Step 3: Check the recorded exit result ```bash systemctl show <service>.service \ --property=Result,ExecMainCode,ExecMainStatus ``` ### Step 4: Review the complete unit configuration ```bash systemctl cat <service>.service ``` ### Step 5: Check for local overrides ```bash systemd-delta ``` ### Step 6: Validate the application configuration Examples: ```bash sudo sshd -t sudo nginx -t sudo apachectl configtest sudo named-checkconf ``` ### Step 7: Check dependencies ```bash systemctl list-dependencies <service>.service ``` ### Step 8: Check listening ports ```bash sudo ss -lntup ``` ### Step 9: Check SELinux denials ```bash sudo ausearch -m AVC,USER_AVC -ts recent ``` ### Step 10: Check ownership, permissions, and contexts ```bash namei -l <path> ls -lZ <path> ``` ### Step 11: Reload systemd when unit files changed ```bash sudo systemctl daemon-reload ``` ### Step 12: Restart the service ```bash sudo systemctl restart <service>.service ``` ### Step 13: Confirm the service is active ```bash systemctl is-active <service>.service ``` ### Step 14: Review the newest logs ```bash journalctl -b -u <service>.service -n 50 --no-pager ``` ### Completion checks Confirm all of the following before closing the change: - The service is active. - The expected port is listening. - No new errors appear in the journal. - Dependent applications are working. - Monitoring is reporting a healthy state. - The rollback information has been recorded. --- ## 20. Commands to Avoid Using Blindly ### 20.1 Restarting every failed service Avoid running: ```bash systemctl restart $(systemctl --failed --no-legend | awk '{print $1}') ``` Each service may have failed for a different reason. Restarting everything can hide the original problem or create additional failures. ### 20.2 Masking services without checking dependencies Do not run: ```bash sudo systemctl mask --now <service>.service ``` until the service purpose and reverse dependencies have been reviewed. ### 20.3 Editing vendor files directly Do not edit: ```text /usr/lib/systemd/system/<service>.service ``` Use: ```bash sudo systemctl edit <service>.service ``` ### 20.4 Using `daemon-reexec` for routine changes Do not use: ```bash sudo systemctl daemon-reexec ``` when the required command is: ```bash sudo systemctl daemon-reload ``` ### 20.5 Clearing logs before troubleshooting Do not rotate, vacuum, or delete journal data until the required logs have been collected. ### 20.6 Disabling SELinux to make a service work Do not permanently set SELinux to permissive or disabled because a service fails to start. Check the denial: ```bash sudo ausearch -m AVC,USER_AVC -ts recent ``` Then correct the file context, permissions, or policy issue. --- # Final Production Checklist Before restarting a production service: - [ ] Review the service status and recent logs. - [ ] Validate the application configuration. - [ ] Review dependencies and reverse dependencies. - [ ] Confirm the correct unit file and overrides are loaded. - [ ] Check ports, permissions, ownership, and SELinux contexts. - [ ] Document the current configuration. - [ ] Prepare a rollback procedure. - [ ] Keep a second SSH session open for networking, firewall, or SSH changes. - [ ] Confirm console or out-of-band access for high-risk changes. - [ ] Verify service health after the change. The following commands provide the fastest starting point for most service problems: ```bash systemctl status <service>.service journalctl -b -u <service>.service systemctl cat <service>.service systemctl show <service>.service ``` A consistent process prevents most avoidable service outages: inspect first, validate the change, apply it carefully, and verify the result.
# SSH, UFW, Fail2Ban, and WireGuard Setup Debian-style server setup for SSH on a custom port, a minimal UFW policy, Fail2Ban protection, and a private WireGuard tunnel. Keep your current SSH session open until a second SSH connection works on the new port. ## Example Values | Item | Value | |---|---| | SSH port | `55222/tcp` | | WireGuard port | `51820/udp` | | WireGuard network | `10.10.10.0/24` | | Server tunnel IP | `10.10.10.1/24` | | Client tunnel IP | `10.10.10.2/24` | | Interface name | `wg0` | Replace `USERNAME`, `SERVER_IP`, and all key placeholders before running commands that use them. ## Install Packages Run on the server: ```bash apt-get update apt-get install -y openssh-server ufw fail2ban python3-systemd wireguard ``` Check the packages: ```bash dpkg-query -W openssh-server ufw fail2ban python3-systemd wireguard test -f /etc/fail2ban/action.d/ufw.conf ``` `python3-systemd` lets Fail2Ban read SSH failures from the systemd journal. ## Pick the SSH Port Port `22` works, but this guide uses `55222` to reduce routine SSH noise. A custom port does not replace strong keys, safe passwords, or Fail2Ban. Check that the port is free: ```bash ss -lntp | grep -E ':55222([[:space:]]|$)' || true ``` No output means nothing is listening on that port. ## Allow the New SSH Port Add the SSH rule before changing OpenSSH: ```bash ufw allow 55222/tcp comment 'SSH' ufw show added ``` This protects your access if UFW is already enabled. ## Configure OpenSSH Create a drop-in config: ```bash cat > /etc/ssh/sshd_config.d/99-custom-port.conf <<'EOF' Port 55222 EOF ``` Set ownership and permissions: ```bash chown root:root /etc/ssh/sshd_config.d/99-custom-port.conf chmod 644 /etc/ssh/sshd_config.d/99-custom-port.conf ``` Check active port settings: ```bash grep -RniE '^[[:space:]]*Port[[:space:]]+' /etc/ssh/sshd_config /etc/ssh/sshd_config.d 2>/dev/null sshd -t sshd -T | grep '^port ' ``` If another active `Port` line exists, remove or comment it before restarting SSH. ## Restart SSH Debian usually uses `ssh.service`. Some Ubuntu releases may use `ssh.socket`. ```bash systemctl daemon-reload if systemctl is-active --quiet ssh.socket || systemctl is-enabled --quiet ssh.socket 2>/dev/null; then systemctl restart ssh.socket else systemctl restart ssh.service fi ``` Verify: ```bash sshd -T | grep '^port ' ss -lntp | grep -E ':55222([[:space:]]|$)' ss -lntp | grep -E ':22([[:space:]]|$)' || true ``` Keep the original SSH session open. From a second terminal: ```bash ssh -p 55222 USERNAME@SERVER_IP ``` Do not continue until the second connection works. ## Set the UFW Policy This resets existing UFW rules. Add back every service the server actually needs. ```bash ufw --force reset ufw default deny incoming ufw default allow outgoing ufw default deny routed ufw allow 55222/tcp comment 'SSH' ufw allow 51820/udp comment 'WireGuard' ufw logging low ufw show added ufw --force enable ``` Verify: ```bash ufw status verbose ufw status numbered ssh -p 55222 USERNAME@SERVER_IP ``` ## Common UFW Rules Open only what the server uses. | Service | Port | Rule | |---|---:|---| | SSH | `55222/tcp` | `ufw allow 55222/tcp comment 'SSH'` | | WireGuard | `51820/udp` | `ufw allow 51820/udp comment 'WireGuard'` | | HTTP | `80/tcp` | `ufw allow 80/tcp comment 'HTTP'` | | HTTPS | `443/tcp` | `ufw allow 443/tcp comment 'HTTPS'` | | DNS | `53/tcp` and `53/udp` | `ufw allow 53 comment 'DNS'` | | NTP | `123/udp` | `ufw allow 123/udp comment 'NTP'` | ## Configure Fail2Ban Create the SSH jail: ```bash cat > /etc/fail2ban/jail.d/sshd.local <<'EOF' [sshd] enabled = true port = 55222 filter = sshd backend = systemd banaction = ufw maxretry = 5 findtime = 10m bantime = 1h ignoreip = 127.0.0.1/8 ::1 EOF ``` Set ownership and permissions: ```bash chown root:root /etc/fail2ban/jail.d/sshd.local chmod 644 /etc/fail2ban/jail.d/sshd.local ``` Check the config and start Fail2Ban: ```bash fail2ban-client -t systemctl enable fail2ban systemctl restart fail2ban ``` Verify: ```bash systemctl is-enabled fail2ban systemctl is-active fail2ban fail2ban-client ping fail2ban-client status fail2ban-client status sshd ``` Expected results include `enabled`, `active`, `Server replied: pong`, and an active `sshd` jail. ## Server WireGuard Keys Run on the server: ```bash install -d -o root -g root -m 700 /etc/wireguard umask 077 wg genkey \ | tee /etc/wireguard/server.key \ | wg pubkey \ > /etc/wireguard/server.pub ``` Check files: ```bash stat -c '%U %G %a %n' /etc/wireguard /etc/wireguard/server.key /etc/wireguard/server.pub cat /etc/wireguard/server.pub ``` Share only the public key. Keep `server.key` private. ## Client WireGuard Keys Run on the Linux client: ```bash apt-get update apt-get install -y wireguard install -d -o root -g root -m 700 /etc/wireguard umask 077 wg genkey \ | tee /etc/wireguard/client.key \ | wg pubkey \ > /etc/wireguard/client.pub ``` Check files: ```bash wg --version stat -c '%U %G %a %n' /etc/wireguard /etc/wireguard/client.key /etc/wireguard/client.pub cat /etc/wireguard/client.pub ``` Share only the public key. Keep `client.key` private. ## Server WireGuard Config Run on the server: ```bash CLIENT_PUBLIC_KEY="PASTE_CLIENT_PUBLIC_KEY_HERE" SERVER_PRIVATE_KEY="$(cat /etc/wireguard/server.key)" cat > /etc/wireguard/wg0.conf <<EOF [Interface] Address = 10.10.10.1/24 ListenPort = 51820 PrivateKey = ${SERVER_PRIVATE_KEY} [Peer] PublicKey = ${CLIENT_PUBLIC_KEY} AllowedIPs = 10.10.10.2/32 EOF unset SERVER_PRIVATE_KEY unset CLIENT_PUBLIC_KEY chown root:root /etc/wireguard/wg0.conf chmod 600 /etc/wireguard/wg0.conf ``` Check the file: ```bash grep -q 'PASTE_' /etc/wireguard/wg0.conf && echo 'Replace placeholders' wg-quick strip wg0 >/dev/null sed -e 's/^PrivateKey = .*/PrivateKey = [REDACTED]/' -e 's/^PublicKey = .*/PublicKey = [REDACTED]/' /etc/wireguard/wg0.conf ``` ## Start WireGuard on the Server ```bash systemctl enable --now wg-quick@wg0 ``` Verify: ```bash systemctl is-enabled wg-quick@wg0 systemctl is-active wg-quick@wg0 ip -brief address show wg0 ss -lunp | grep -E ':51820([[:space:]]|$)' wg show ``` The server should show `10.10.10.1/24`. A handshake appears after the client starts. ## Client WireGuard Config Run on the client: ```bash SERVER_PUBLIC_KEY="PASTE_SERVER_PUBLIC_KEY_HERE" SERVER_ENDPOINT="PASTE_SERVER_IP_OR_HOSTNAME_HERE" CLIENT_PRIVATE_KEY="$(cat /etc/wireguard/client.key)" cat > /etc/wireguard/wg0.conf <<EOF [Interface] Address = 10.10.10.2/24 PrivateKey = ${CLIENT_PRIVATE_KEY} [Peer] PublicKey = ${SERVER_PUBLIC_KEY} Endpoint = ${SERVER_ENDPOINT}:51820 AllowedIPs = 10.10.10.0/24 PersistentKeepalive = 25 EOF unset CLIENT_PRIVATE_KEY unset SERVER_PUBLIC_KEY unset SERVER_ENDPOINT chown root:root /etc/wireguard/wg0.conf chmod 600 /etc/wireguard/wg0.conf ``` `AllowedIPs = 10.10.10.0/24` routes only the tunnel network through WireGuard. Check the file: ```bash grep -q 'PASTE_' /etc/wireguard/wg0.conf && echo 'Replace placeholders' wg-quick strip wg0 >/dev/null sed -e 's/^PrivateKey = .*/PrivateKey = [REDACTED]/' -e 's/^PublicKey = .*/PublicKey = [REDACTED]/' /etc/wireguard/wg0.conf ``` ## Start WireGuard on the Client ```bash systemctl enable --now wg-quick@wg0 ``` Verify: ```bash systemctl is-enabled wg-quick@wg0 systemctl is-active wg-quick@wg0 ip -brief address show wg0 wg show ping -c 4 10.10.10.1 ``` The client should show `10.10.10.2/24` and reach `10.10.10.1`. ## Final Checks Server: ```bash sshd -t sshd -T | grep '^port ' ss -lntp | grep -E ':55222([[:space:]]|$)' ss -lntp | grep -E ':22([[:space:]]|$)' || true ufw status verbose fail2ban-client status sshd systemctl is-active wg-quick@wg0 ip -brief address show wg0 ss -lunp | grep -E ':51820([[:space:]]|$)' wg show ``` Client: ```bash systemctl is-active wg-quick@wg0 ip -brief address show wg0 ping -c 4 10.10.10.1 wg show ``` ## Useful Commands | Task | Command | |---|---| | Show UFW status | `ufw status verbose` | | Show numbered UFW rules | `ufw status numbered` | | Delete a UFW rule | `ufw delete <NUMBER>` | | Reload UFW | `ufw reload` | | Show Fail2Ban status | `fail2ban-client status` | | Show SSH jail | `fail2ban-client status sshd` | | Unban an IP | `fail2ban-client set sshd unbanip <IP>` | | Show WireGuard status | `wg show` | | Restart WireGuard | `systemctl restart wg-quick@wg0` | | Stop WireGuard | `systemctl stop wg-quick@wg0` | | Start WireGuard | `systemctl start wg-quick@wg0` | | Show listening ports | `ss -lntup` | ## Notes - This creates a private point-to-point WireGuard tunnel. - It does not route client Internet traffic through the server. - It does not configure NAT or gateway routing. - If the hosting provider has an external firewall, allow `55222/tcp` and `51820/udp` there too.
# Forgejo Binary Install on RHEL with NGINX This is the Forgejo setup pattern I use for a small self-hosted Git service: Forgejo runs as its own `git` user, stores data under `/var/lib/forgejo`, uses SQLite, runs under systemd, and sits behind NGINX. Use the example DNS names and ports as a template, then replace them with the values for your own server. ## Scope This setup uses: - Forgejo installed as `/usr/local/bin/forgejo` - Dedicated system user: `git` - Work path: `/var/lib/forgejo` - Config path: `/etc/forgejo/app.ini` - SQLite database stored under Forgejo data - systemd service: `forgejo.service` - NGINX reverse proxy in front of Forgejo - Forgejo bound locally on `127.0.0.1` Record only the settings needed to reproduce the install. Leave sensitive values out of any shared copy. ## DNS Examples Pick the hostname first. Forgejo links, clone URLs, redirects, and webhook URLs depend on the base URL being correct. Standard HTTPS example: ```text git.example.com -> server public address ROOT_URL = https://git.example.com/ DOMAIN = git.example.com ``` Custom HTTPS port example: ```text git.example.com -> server public address ROOT_URL = https://git.example.com:8443/ DOMAIN = git.example.com ``` The `DOMAIN` value is the hostname. The `ROOT_URL` value is the full browser URL, including `https://` and any non-standard port. ## 1. Install Required Packages Forgejo needs Git for repository operations. Git LFS is useful when repositories need large file support. NGINX receives browser traffic and forwards it to Forgejo. ```bash sudo dnf install git git-lfs wget nginx ``` Verify the tools: ```bash git --version git lfs version nginx -v ``` ## 2. Install the Forgejo Binary Download the Forgejo release that matches the server CPU architecture from the official Forgejo release page. Verify the release signature before installing the binary. Replace `x.y.z` with the version being installed. ```bash sudo cp forgejo-x.y.z-linux-amd64 /usr/local/bin/forgejo sudo chmod 755 /usr/local/bin/forgejo /usr/local/bin/forgejo --version ``` The version command should print the installed Forgejo version. ## 3. Create the Forgejo Service User Forgejo should not run as root. A dedicated `git` user keeps the application process and repository ownership separate from normal administrator accounts. ```bash sudo groupadd --system git sudo useradd --system --shell /bin/bash \ --comment "Git Version Control" \ --gid git --home-dir /home/git --create-home git ``` Verify the account: ```bash getent passwd git ``` ## 4. Create Forgejo Directories Forgejo uses one path for application data and one path for configuration. During the first web install, Forgejo needs write access to create `app.ini`. After setup, permissions are tightened. ```bash sudo mkdir /var/lib/forgejo sudo chown git:git /var/lib/forgejo sudo chmod 750 /var/lib/forgejo sudo mkdir /etc/forgejo sudo chown root:git /etc/forgejo sudo chmod 770 /etc/forgejo ``` Path usage: | Path | Purpose | |---|---| | `/var/lib/forgejo` | repositories, database, attachments, LFS data, logs, custom assets | | `/etc/forgejo/app.ini` | main Forgejo configuration | | `/usr/local/bin/forgejo` | Forgejo executable | ## 5. Install the systemd Service The systemd unit starts Forgejo, starts it after reboot, and gives one place to check status and logs. ```bash sudo wget -O /etc/systemd/system/forgejo.service \ https://codeberg.org/forgejo/forgejo/raw/branch/forgejo/contrib/systemd/forgejo.service sudo systemctl daemon-reload sudo systemctl enable forgejo.service sudo systemctl start forgejo.service ``` Check service status: ```bash sudo systemctl status forgejo.service sudo journalctl -n 100 --unit forgejo.service ``` If the service starts but the web installer is not reachable, confirm the listener and logs before changing configuration. ## 6. Complete the Web Installer Open the Forgejo installer through the temporary local listener or through the reverse proxy once NGINX is configured. Common installer choices for this setup: | Setting | Example | |---|---| | Database type | `SQLite3` | | Site title | `Forgejo Git` | | Run user | `git` | | Server domain | `git.example.com` | | Forgejo base URL | `https://git.example.com/` or `https://git.example.com:8443/` | | Registration | Disable if accounts should be created manually | After installation, Forgejo writes `/etc/forgejo/app.ini`. ## 7. Review app.ini Use a local listener behind NGINX. This keeps Forgejo itself off the browser-facing port while NGINX handles HTTPS and proxy headers. Standard HTTPS example: ```ini APP_NAME = Forgejo Git RUN_USER = git RUN_MODE = prod [database] DB_TYPE = sqlite3 PATH = /var/lib/forgejo/data/forgejo.db [server] PROTOCOL = http HTTP_ADDR = 127.0.0.1 HTTP_PORT = 3000 DOMAIN = git.example.com ROOT_URL = https://git.example.com/ [service] DISABLE_REGISTRATION = true REQUIRE_SIGNIN_VIEW = false ``` Custom external port example: ```ini [server] PROTOCOL = http HTTP_ADDR = 127.0.0.1 HTTP_PORT = 3001 DOMAIN = git.example.com ROOT_URL = https://git.example.com:8443/ ``` Important rules: - `HTTP_ADDR` is the local address Forgejo listens on. - `HTTP_PORT` is the local Forgejo port, not necessarily the public browser port. - `DOMAIN` is the hostname only. - `ROOT_URL` is the exact browser URL users open. - Do not publish `SECRET_KEY`, `INTERNAL_TOKEN`, OAuth secrets, SMTP credentials, or private keys. Restart after config changes: ```bash sudo systemctl restart forgejo.service sudo systemctl status forgejo.service ``` ## 8. Lock the Config File After the installer finishes, Forgejo should read `app.ini` but not rewrite it. ```bash sudo systemctl stop forgejo.service sudo chmod 750 /etc/forgejo sudo chmod 640 /etc/forgejo/app.ini sudo systemctl start forgejo.service ``` Verify permissions: ```bash ls -ld /etc/forgejo ls -l /etc/forgejo/app.ini ``` ## 9. Configure NGINX Reverse Proxy Forgejo stays on the local listener. NGINX handles browser traffic, forwards the required headers, supports upgrades, and allows larger web uploads. Standard HTTPS site example: ```nginx server { listen 443 ssl; listen [::]:443 ssl; http2 on; server_name git.example.com; merge_slashes off; location / { proxy_pass http://127.0.0.1:3000; proxy_set_header Connection $http_connection; proxy_set_header Upgrade $http_upgrade; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; client_max_body_size 512M; } } ``` Custom HTTPS port example: ```nginx server { listen 8443 ssl; listen [::]:8443 ssl; http2 on; server_name git.example.com; merge_slashes off; location / { proxy_pass http://127.0.0.1:3001; proxy_set_header Connection $http_connection; proxy_set_header Upgrade $http_upgrade; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; client_max_body_size 512M; } } ``` Validate and reload NGINX: ```bash sudo nginx -t sudo systemctl reload nginx ``` If SELinux is enforcing and NGINX cannot proxy to Forgejo, allow NGINX to make outbound network connections: ```bash sudo setsebool -P httpd_can_network_connect 1 ``` ## 10. Verify the Deployment Check the service, proxy, listener, and Forgejo CLI. ```bash sudo systemctl status forgejo.service sudo journalctl -n 100 --unit forgejo.service sudo nginx -t sudo ss -tulpn | grep forgejo sudo -u git forgejo \ -w /var/lib/forgejo \ -c /etc/forgejo/app.ini \ admin user list ``` Browser checks: - The configured `ROOT_URL` loads the Forgejo login page. - Redirects keep the correct hostname and port. - Repository clone URLs use the expected hostname. - Registration is disabled if accounts are meant to be created manually. ## 11. Backup Before Changes Backups should include the config, database, repositories, attachments, LFS data, and custom files. Minimum backup set: - `/etc/forgejo/app.ini` - `/var/lib/forgejo/data/forgejo.db` - `/var/lib/forgejo/data/forgejo-repositories` - `/var/lib/forgejo/data/attachments` - `/var/lib/forgejo/data/lfs` - `/var/lib/forgejo/custom` if used Stop Forgejo or use an application-aware backup method before copying SQLite data. ## 12. Upgrade Notes Before upgrading: 1. Read the release notes for the target Forgejo version. 2. Back up `/etc/forgejo` and `/var/lib/forgejo`. 3. Stop `forgejo.service`. 4. Replace `/usr/local/bin/forgejo` with the verified new binary. 5. Start the service and check logs. ```bash sudo systemctl stop forgejo.service sudo cp forgejo-new-version-linux-amd64 /usr/local/bin/forgejo sudo chmod 755 /usr/local/bin/forgejo sudo systemctl start forgejo.service sudo journalctl -n 100 --unit forgejo.service /usr/local/bin/forgejo --version ``` ## References - [Forgejo binary installation](https://forgejo.org/docs/v15.0/admin/installation/binary/) - [Forgejo reverse proxy](https://forgejo.org/docs/v15.0/admin/setup/reverse-proxy/) - [Forgejo config cheat sheet](https://forgejo.org/docs/v15.0/admin/config-cheat-sheet/)