wazuh-proxmox-iac/docs/RUNBOOK_WAZUH_RBMK.md

# Wazuh Lab Runbook (RBMK2, VMID/IP reuse)

## Scope
- Rebuild VM `104` on `rbmk2`
- Keep IP `10.1.50.125/24`
- Use Ubuntu 22.04 template `9000`
- Install Wazuh all-in-one

## 1) Create Ubuntu template on RBMK2
Run on: `root@rbmk2`

```bash
qm status 9000 || true
wget -O /var/lib/vz/template/iso/jammy-server-cloudimg-amd64.img https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
qm create 9000 --name ubuntu2204-template-rbmk2 --memory 4096 --cores 2 --net0 virtio,bridge=vmbr0,firewall=1
qm importdisk 9000 /var/lib/vz/template/iso/jammy-server-cloudimg-amd64.img local-lvm
qm set 9000 --scsihw virtio-scsi-single --scsi0 local-lvm:vm-9000-disk-0
qm set 9000 --ide2 local-lvm:cloudinit
qm set 9000 --boot order=scsi0
qm set 9000 --agent enabled=1
qm template 9000
qm resize 9000 scsi0 60G
qm config 9000 | egrep '^(name|template|ostype|agent|boot|scsihw|scsi0|ide2|net0):'
```

Expected:
- `template: 1`
- `ide2: ...cloudinit`
- `scsi0 ... size=60G`

## 2) Prepare Terraform
Run on: local laptop  
Path: `/home/nikola/codex-cli/projects/wazuh-proxmox-iac/terraform`

```bash
cd /home/nikola/codex-cli/projects/wazuh-proxmox-iac/terraform
mkdir -p .secrets
cp terraform.tfvars.example terraform.tfvars
nano terraform.tfvars
nano .secrets/api_token
chmod 600 .secrets/api_token
terraform init
terraform fmt -recursive
terraform validate
terraform plan -out tfplan
```

Set in `terraform.tfvars`:
- `proxmox_template_vm_id = 9000`
- `vm_id = 104`
- `vm_ipv4_cidr = "10.1.50.125/24"`
- `vm_ssh_public_key = "<your pub key>"`

Expected:
- `Success! The configuration is valid.`
- plan shows `+ create` for `vm_id = 104`

## 3) Replace old VM 104
Run on: `root@rbmk2`

```bash
qm stop 104 || true
qm destroy 104 --purge 1 --destroy-unreferenced-disks 1
```

Expected:
- old test VM removed

## 4) Apply Terraform
Run on: local laptop  
Path: `/home/nikola/codex-cli/projects/wazuh-proxmox-iac/terraform`

```bash
terraform apply tfplan
terraform output
```

Expected:
- `Apply complete!`
- outputs include:
  - `ssh devops@10.1.50.125 -p42315`
  - `https://10.1.50.125`

## 5) Fix guest agent if apply waits
Symptom:
- long `Still creating...`
- on rbmk2: `qm agent 104 ping` => `QEMU guest agent is not running`

Run on: VM `10.1.50.125` (ssh `-p22` or `-p42315`)

```bash
sudo apt-get update
sudo apt-get install -y qemu-guest-agent
sudo systemctl start qemu-guest-agent
sudo systemctl status qemu-guest-agent --no-pager
```

Verify on `rbmk2`:

```bash
qm agent 104 ping
```

Expected:
- agent running; apply completes

## 6) Apply company SSH baseline
Run on: local laptop  
Path: `/home/nikola/codex-cli/projects/wazuh-proxmox-iac/terraform`

```bash
chmod +x scripts/company-bootstrap-ubuntu.sh scripts/install-wazuh-aio.sh scripts/verify-wazuh.sh
scp -P22 scripts/company-bootstrap-ubuntu.sh devops@10.1.50.125:/tmp/
ssh devops@10.1.50.125 -p22 "sudo bash /tmp/company-bootstrap-ubuntu.sh devops 42315 \"$(cat ~/.ssh/id_ed25519.pub)\""
ssh devops@10.1.50.125 -p42315 "sudo sshd -T | egrep '^(port|permitrootlogin|passwordauthentication|pubkeyauthentication)'"
```

Expected:
- `port 42315`
- `permitrootlogin no`
- `passwordauthentication no`
- `pubkeyauthentication yes`

## 7) Install Wazuh
Run on: local laptop  
Path: `/home/nikola/codex-cli/projects/wazuh-proxmox-iac/terraform`

```bash
scp -P42315 scripts/install-wazuh-aio.sh devops@10.1.50.125:/tmp/
ssh devops@10.1.50.125 -p42315 "sudo bash /tmp/install-wazuh-aio.sh"
./scripts/verify-wazuh.sh 10.1.50.125
ssh devops@10.1.50.125 -p42315 "sudo systemctl status wazuh-indexer wazuh-manager wazuh-dashboard filebeat --no-pager"
```

Expected:
- Installer summary with dashboard URL and admin password
- `verify-wazuh.sh` checks pass
- all 4 services `active (running)`

## Common issues and fixes
1. `can't clone to non-shared storage 'local-lvm'`
- Cause: cross-node clone to non-shared storage.
- Fix: create template directly on `rbmk2`.

2. `unable to find configuration file for VM 129 on node 'rbmk2'`
- Cause: source VM exists on another node.
- Fix: run actions on the correct source node or avoid cross-node clone.

3. SSH host key changed warning
- Fix:
```bash
ssh-keygen -f ~/.ssh/known_hosts -R '[10.1.50.125]:42315'
```

4. `QEMU guest agent is not running`
- Fix: install/start `qemu-guest-agent` inside VM, then retry/check.

5. Thin pool warnings during import/resize
- Cause: local-lvm oversubscription warning.
- Fix: monitor storage free space before new clones and log growth.

## Post-install hardening checklist (Wazuh lab)
Run on: VM `10.1.50.125` (`ssh devops@10.1.50.125 -p42315`)

1. Rotate default admin password
- In dashboard: `https://10.1.50.125` -> change `admin` password immediately.

2. Restrict API exposure (if not needed externally)
```bash
sudo ss -tulpen | egrep '(:443|:1514|:1515|:55000)'
```
- confirm only required ports are listening.

3. Enable host firewall baseline
```bash
sudo apt-get install -y ufw
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 42315/tcp
sudo ufw allow 443/tcp
sudo ufw allow 1514/tcp
sudo ufw allow 1515/tcp
sudo ufw allow 55000/tcp
sudo ufw --force enable
sudo ufw status verbose
```
- keep only Wazuh + SSH management ports open.

4. Verify services are enabled and healthy
```bash
sudo systemctl is-active wazuh-indexer wazuh-manager wazuh-dashboard filebeat
sudo systemctl is-enabled wazuh-indexer wazuh-manager wazuh-dashboard filebeat
```
- all should be active/enabled.

5. Backup install artifacts and credentials file
```bash
sudo ls -l /root/wazuh-install-files.tar /var/log/wazuh-install.log
```
- copy to a secure internal vault/location.