From 2eb8ede3aaf97f319c046bea21260711acdaf054 Mon Sep 17 00:00:00 2001 From: Andrea Dell'Amico Date: Wed, 5 Aug 2026 18:32:18 +0200 Subject: [PATCH] Updates to s2i2s/project-setup/README.md --- s2i2s/project-setup/README.md | 122 +++++++++++++++++++++++++--------- 1 file changed, 92 insertions(+), 30 deletions(-) diff --git a/s2i2s/project-setup/README.md b/s2i2s/project-setup/README.md index 4499c4d..d4a6116 100644 --- a/s2i2s/project-setup/README.md +++ b/s2i2s/project-setup/README.md @@ -1,6 +1,6 @@ # S2I2S Project Setup -This Terraform configuration sets up the core infrastructure components for the S2I2S OpenStack project. +This OpenTofu configuration sets up the core infrastructure components for the S2I2S OpenStack project. ## Overview @@ -8,42 +8,84 @@ The project-setup module creates the following resources: ### Virtual Machines -| VM | Purpose | Flavor | Floating IP | DNS Record | -| --- | --- | --- | --- | --- | -| SSH Jump Proxy | Secure SSH gateway for accessing internal VMs | m1.small | Yes | `ssh-jump.s2i2s.cloud.isti.cnr.it` | -| Internal CA | Certificate Authority for internal services | m1.small | No | - | -| HAProxy L7 (x2) | Layer 7 load balancers behind Octavia | m1.medium | No | - | -| Prometheus | Monitoring and metrics collection | m1.medium | Yes | `prometheus.s2i2s.cloud.isti.cnr.it` | +| VM | Purpose | Flavor | Private IP | Boot disk | Floating IP | DNS Record | +| --- | --- | --- | --- | --- | --- | --- | +| SSH Jump Proxy | Secure SSH gateway for accessing internal VMs | m2.small | 10.10.0.5 | 30 GB | Yes | `ssh-jump-proxy.s2i2s.cloud.isti.cnr.it` | +| Internal CA | Certificate Authority for internal services | m1.small | 10.10.0.4 | 10 GB | No | - | +| HAProxy L7 (x2) | Layer 7 load balancers behind Octavia | m1.medium | 10.10.0.11, 10.10.0.12 | 10 GB | No | - | +| Prometheus | Monitoring and metrics collection | m1.medium | 10.10.0.10 | 10 GB | Yes | `prometheus.s2i2s.cloud.isti.cnr.it`, `alertmanager.s2i2s.cloud.isti.cnr.it` (CNAME) | -All VMs run Ubuntu 24.04 and are provisioned with the standard cloud-init user data script. +All VMs run **Ubuntu 24.04** (`Ubuntu-Noble-24.04.img`) and are provisioned with the standard `ubuntu2404.sh` cloud-init user data script. Boot disks are volume-backed with `delete_on_termination = false`. ### Load Balancer An OVN-based Octavia load balancer (`s2i2s-cloud-l4-load-balancer`) provides L4 load balancing: +- **Provider**: `ovn` (not amphora) +- **VIP**: 10.10.0.20, on the main private subnet - **Floating IP**: Yes - **DNS Record**: `octavia-main-lb.s2i2s.cloud.isti.cnr.it` - **Backend**: HAProxy L7 instances (anti-affinity for HA) -| Listener | Port | Protocol | Health Check | -| --- | --- | --- | --- | -| HTTP | 80 | TCP | HTTP GET `/_haproxy_health_check` | -| HTTPS | 443 | TCP | HTTPS GET `/_haproxy_health_check` | -| Stats | 8880 | TCP | TCP connect | +| Listener | Port | Protocol | Pool method | Health Check | +| --- | --- | --- | --- | --- | +| HTTP | 80 | TCP | SOURCE_IP_PORT | TCP connect | +| HTTPS | 443 | TCP | SOURCE_IP_PORT | TCP connect | +| Stats | 8880 | TCP | SOURCE_IP_PORT | TCP connect | + +#### Consequences of the OVN provider + +The OVN driver is lighter than amphora (no amphora VMs, uses the main subnet directly), but it +constrains the configuration in ways that are visible throughout this setup: + +- **No `allowed_cidrs` on listeners.** Access restrictions must be enforced by security groups + or inside HAProxy itself. This is why the stats port is filtered in the security group below. +- **Only `ROUND_ROBIN` and `SOURCE_IP_PORT`** are available as pool methods; `LEAST_CONNECTIONS` + is not supported. +- **Only `TCP` and `UDP-CONNECT` health monitors**; HTTP/HTTPS monitors are not supported, so all + three pools are probed with a plain TCP connect. +- **The client source IP is preserved.** OVN does not SNAT the traffic to an amphora VIP, so the + HAProxy L7 backends see the real client address rather than an address of the load balancer. + Only health-monitor probes and hairpinned traffic originate from the private subnet. ### Security Groups | Security Group | Purpose | Use On | | --- | --- | --- | -| `s2i2s-default-sg` | Default rules: SSH via jump proxy, ICMP, Prometheus node exporter | All VMs | +| `default_for_all` | Default rules: SSH via jump proxy, ICMP, Prometheus node exporter | All VMs | | `ssh_access_to_the_jump_node` | SSH access from VPN endpoints | SSH Jump Proxy only | | `debugging_from_jump_node` | Web debugging via SSH tunnels (ports 80, 443, 8100) | VMs needing debug access | -| `traffic_from_the_main_load_balancers` | HTTP/HTTPS from HAProxy L7 (ports 80, 443, 8080, 8888) | Backend web services | -| `traffic_from_main_lb_to_haproxy_l7` | Traffic from Octavia LB to HAProxy | HAProxy L7 VMs | +| `traffic_from_the_main_load_balancers` | HTTP/HTTPS from the HAProxy L7 IPs (ports 80, 443, 8080, 8888) | Backend web services | +| `traffic_from_main_lb_to_haproxy_l7` | Public HTTP/HTTPS plus filtered access to the stats port | HAProxy L7 VMs | | `public_web_service` | HTTP/HTTPS from anywhere | Public-facing services with floating IP | | `restricted_web_service` | HTTP from anywhere, HTTPS from VPNs only | Restricted services with floating IP | | `prometheus_access_from_grafana` | HTTPS access from public Grafana server | Prometheus VM | +> **Do not confuse the two load balancer groups.** `traffic_from_main_lb_to_haproxy_l7` sits on the +> HAProxy L7 VMs themselves and faces the internet. `traffic_from_the_main_load_balancers` sits on +> the backend services *behind* HAProxy: since HAProxy is an L7 proxy that opens new connections +> from its own address, the client IP is **not** preserved on that hop, so its sources stay +> restricted to 10.10.0.11 and 10.10.0.12 and it must never be opened to `0.0.0.0/0`. + +#### `traffic_from_main_lb_to_haproxy_l7` + +Because the OVN load balancer preserves the client source IP (see above), the public listeners on +the HAProxy L7 VMs cannot be restricted to the private subnet: + +| Port | Source | Rationale | +| --- | --- | --- | +| 80 | `0.0.0.0/0` | HTTP arrives with the original client address | +| 443 | `0.0.0.0/0` | HTTPS arrives with the original client address | +| 8880 | main private subnet | Health monitor probes for the stats pool | +| 8880 | D4S VPN 1/2, S2I2S VPN 1/2, InfraScience network | Human access to the HAProxy stats page | +| 8880 | 10.10.0.10/32 and the Prometheus floating IP | Prometheus scrapes the stats, over the private network and through the LB public IP | +| 10000 | 10.10.0.11/32, 10.10.0.12/32 | Peer traffic between the two HAProxy instances | + +The stats port is deliberately **not** open to `0.0.0.0/0`: since the OVN provider does not support +`allowed_cidrs` on the listener, this security group is the only place where port 8880 can be +restricted, and client-IP preservation is precisely what makes filtering by real source address +possible. The per-source rules are generated with a `for_each` map in `haproxy.tf`. + ### Storage - **Prometheus Data Volume**: 100 GB SSD (CephSSD) with online resize enabled @@ -78,20 +120,30 @@ An OVN-based Octavia load balancer (`s2i2s-cloud-l4-load-balancer`) provides L4 1. The `main_net_dns_router` configuration must be applied first (creates network, subnet, DNS zone) 2. SSH key must be configured in the OpenStack project 3. OpenStack credentials must be configured (via `clouds.yaml` or environment variables) +4. [OpenTofu](https://opentofu.org/) — this repository is driven with `tofu`, not `terraform` +5. `terraform-provider-openstack/openstack` >= 2.0.0; the configuration is verified against 3.4.0 ## Usage ```bash -# Initialize Terraform -terraform init +# Initialize OpenTofu +tofu init # Review the plan -terraform plan +tofu plan # Apply the configuration -terraform apply +tofu apply ``` +> When re-initialising a directory that was last used with `terraform`, run `tofu init -upgrade`. +> The provider address in `.terraform.lock.hcl` moves from `registry.terraform.io` to +> `registry.opentofu.org`; that is expected. +> +> Provider 3.x tracks a `stateful` attribute on `openstack_networking_secgroup_v2` that 1.53.0 did +> not. The first refresh after upgrading therefore reports `+ stateful = false` as out-of-band +> drift on every security group. It produces no planned change and can be ignored. + ## SSH Jump Proxy Configuration To access VMs in the S2I2S cloud, you must use the SSH jump proxy. Add the following configuration to your `~/.ssh/config` file: @@ -100,7 +152,7 @@ To access VMs in the S2I2S cloud, you must use the SSH jump proxy. Add the follo # S2I2S SSH Jump Proxy # Replace with your actual username Host s2i2s-jump - HostName ssh-jump.s2i2s.cloud.isti.cnr.it + HostName ssh-jump-proxy.s2i2s.cloud.isti.cnr.it User IdentityFile ~/.ssh/your_private_key ForwardAgent yes @@ -114,32 +166,32 @@ Host s2i2s-jump Host 10.10.0.* User ForwardAgent yes - ProxyJump @ssh-jump.s2i2s.cloud.isti.cnr.it + ProxyJump @ssh-jump-proxy.s2i2s.cloud.isti.cnr.it # Alternative: named aliases for specific internal hosts Host s2i2s-prometheus HostName 10.10.0.10 User ForwardAgent yes - ProxyJump @ssh-jump.s2i2s.cloud.isti.cnr.it + ProxyJump @ssh-jump-proxy.s2i2s.cloud.isti.cnr.it Host s2i2s-ca HostName 10.10.0.4 User ForwardAgent yes - ProxyJump @ssh-jump.s2i2s.cloud.isti.cnr.it + ProxyJump @ssh-jump-proxy.s2i2s.cloud.isti.cnr.it Host s2i2s-haproxy-01 HostName 10.10.0.11 User ForwardAgent yes - ProxyJump @ssh-jump.s2i2s.cloud.isti.cnr.it + ProxyJump @ssh-jump-proxy.s2i2s.cloud.isti.cnr.it Host s2i2s-haproxy-02 HostName 10.10.0.12 User ForwardAgent yes - ProxyJump @ssh-jump.s2i2s.cloud.isti.cnr.it + ProxyJump @ssh-jump-proxy.s2i2s.cloud.isti.cnr.it ``` ### SSH Usage Examples @@ -155,10 +207,10 @@ ssh 10.10.0.10 ssh s2i2s-prometheus # Connect without ssh config (replace ) -ssh -J @ssh-jump.s2i2s.cloud.isti.cnr.it @10.10.0.10 +ssh -J @ssh-jump-proxy.s2i2s.cloud.isti.cnr.it @10.10.0.10 # Copy a file to an internal VM -scp -J @ssh-jump.s2i2s.cloud.isti.cnr.it localfile.txt @10.10.0.10:/tmp/ +scp -J @ssh-jump-proxy.s2i2s.cloud.isti.cnr.it localfile.txt @10.10.0.10:/tmp/ # Forward a local port to an internal service ssh -L 8080:10.10.0.30:80 s2i2s-jump @@ -184,12 +236,12 @@ ssh -L 8080:10.10.0.50:80 s2i2s-jump ssh -L 8443:10.10.0.50:443 s2i2s-jump # Without ssh config (replace ) -ssh -L 8080:10.10.0.50:80 @ssh-jump.s2i2s.cloud.isti.cnr.it +ssh -L 8080:10.10.0.50:80 @ssh-jump-proxy.s2i2s.cloud.isti.cnr.it ``` ## Outputs -The module exports the following outputs for use by other Terraform configurations: +The module exports the following outputs for use by other OpenTofu configurations: ### VM IDs and IPs @@ -219,6 +271,11 @@ The module exports the following outputs for use by other Terraform configuratio - `main_private_network`, `main_private_subnet`, `main_subnet_network_id` - `basic_services_ip`, `main_haproxy_l7_ip` +### Image Outputs (re-exported from labs_common_variables) + +- `ubuntu_2404`, `ubuntu2404_data_file` - the image and cloud-init script used by every VM here +- `ubuntu_2204`, `ubuntu2204_data_file` - kept for consumers still on 22.04 + ## File Structure ```text @@ -251,3 +308,8 @@ This module depends on: - The Prometheus data volume uses CephSSD storage for better I/O performance - Volumes have `enable_online_resize = true` for live resizing capability - Security groups are designed to minimize attack surface while allowing necessary traffic flows +- Every instance carries `ignore_changes = [key_pair, user_data, network]` so that rotating the SSH + key or the cloud-init script does not trigger a rebuild +- The main load balancer uses the OVN Octavia provider; see + [Consequences of the OVN provider](#consequences-of-the-ovn-provider) before changing listeners, + pool methods, health monitors or the HAProxy security group