# Prometheus with Tailscale Integration

![Prometheus with Tailscale](https://prometheus.io/assets/prometheus_logo_grey.svg "Prometheus")

This project sets up a Prometheus instance with Tailscale VPN integration using Docker Compose. It creates a secure, private network connection for your Prometheus instance using Tailscale.

## Prerequisites

- Docker and Docker Compose installed on your system
- A Tailscale account and auth key (get one from https://login.tailscale.com/admin/authkeys)
- Basic understanding of Docker and networking concepts

## Project Structure
```
ts-prometheus/
├── docker-compose.yml
├── tailscale/
│   ├── tailscale-data/             # Persistent Tailscale state
│   └── config/                     # Tailscale configuration files
└── prometheus/
    │   └── etc/
    │       └── prometheus/         # Prometheus configuration files
    └── data/                       # Prometheus data
```

## Setup Instructions

1. **Clone the Repository**
   ```bash
   git clone https://gitea.damconsulting.llc/DAM/ts-prometheus
   cd ts-prometheus
   ```
2. Create Required Directories
    ```bash
    mkdir -p tailscale/tailscale-data prometheus/data
    ```
3. Configure Tailscale
    - Replace `{{YOUR_TAILSCALE_AUTHKEY}}` in the docker-compose.yml with your actual Tailscale auth key
    - Optionally, update the file in `tailscale/config/serve.json` if you need specific Tailscale serve configurations
        - CAUTION: Changing `"${TS_CERT_DOMAIN}:443": false` to `true` will expose the service to the internet

4. Configure Prometheus `/prometheus/etc/prometheus/prometheus.yml`
    - See https://prometheus.io/docs/prometheus/latest/configuration/configuration/ for configuration options

5. Start the Services
    ```bash
    docker compose up -d
    ```

6. Wait for Certificate to propagate [~2m] 

7. Login

## Services

### prometheus-ts (Tailscale)

- Runs Tailscale VPN client
- Image: tailscale/tailscale:latest
- Container name: prometheus-ts
- Hostname: prometheus
- Requires NET_ADMIN and SYS_MODULE capabilities
- Persists state in ./tailscale/tailscale-data
- Uses configuration from ./tailscale/config

### prometheus

- Depends on prometheus-ts service

## Usage

- After starting the services your service should be available via tailnet at `https://prometheus.{{YOUR_TAILNET_DOMAIN}}.ts.net` ie `https://prometheus.tail12345.ts.net/`
- To manually get the Tailscale IP/hostname of your container:
    ```bash
    docker logs prometheus-ts
    ```
    Look for the Tailscale IP address in the logs.

## Optional Features

- Uncomment and adjust the ports mapping if you need direct access (without Tailscale):
    ```yaml
    ports:
    - 9000:9000
    ```
- Stopping the Services
    ```bash
    docker compose down
    ```

## Troubleshooting
- Check container logs:
    ```bash
    docker logs prometheus-ts
    docker logs prometheus
    ```
- Ensure your Tailscale auth key is valid and not expired
- Verify the configuration files have proper permissions
- Make sure required directories exist before starting

## Notes
- The Prometheus service uses the Tailscale service's network stack via `network_mode: service:prometheus-ts`
- Direct port mapping is disabled by default as Tailscale handles the networking
- Services restart automatically unless explicitly stopped
- For more information:
    - Tailscale documentation: https://tailscale.com/kb/
    - Prometheus documentation: https://prometheus.io/docs/introduction/overview/
    - Prometheus repository: https://github.com/prometheus/prometheus