ISTI-provisioning
/
openstack-infrastructure-terraform
4
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
openstack-infrastructure-te.../s2i2s/project-setup/README.md

369 lines
15 KiB
Markdown
Raw Blame History

# S2I2S Project Setup
This OpenTofu configuration sets up the core infrastructure components for the S2I2S OpenStack project.
## Overview
The project-setup module creates the following resources:
### Virtual Machines
| VM | Purpose | Flavor | Private IP | Boot disk | Floating IP | DNS Record |
| --- | --- | --- | --- | --- | --- | --- |
| SSH Jump Proxy | Secure SSH gateway for accessing internal VMs | m2.small | 10.10.0.5 | 30 GB | Yes | `ssh-jump-proxy.s2i2s.cloud.isti.cnr.it` |
| Internal CA | Certificate Authority for internal services | m1.small | 10.10.0.4 | 10 GB | No | - |
| HAProxy L7 (x2) | Layer 7 load balancers behind Octavia | m1.medium | 10.10.0.11, 10.10.0.12 | 10 GB | No | - |
| Prometheus | Monitoring and metrics collection | m1.medium | 10.10.0.10 | 10 GB | Yes | `prometheus.s2i2s.cloud.isti.cnr.it`, `alertmanager.s2i2s.cloud.isti.cnr.it` (CNAME) |
All VMs run **Ubuntu 24.04** (`Ubuntu-Noble-24.04.img`) and are provisioned with the standard `ubuntu2404.sh` cloud-init user data script. Boot disks are volume-backed with `delete_on_termination = false`.
### Load Balancer
An OVN-based Octavia load balancer (`s2i2s-cloud-l4-load-balancer`) provides L4 load balancing:
- **Provider**: `ovn` (not amphora)
- **VIP**: 10.10.0.20, on the main private subnet
- **Floating IP**: Yes
- **DNS Record**: `main-lb.s2i2s.cloud.isti.cnr.it` — see [DNS](#dns) below. Renamed from
`octavia-main-lb`, which no longer resolves
- **Backend**: HAProxy L7 instances (anti-affinity for HA)
| Listener | Port | Protocol | Pool method | Health Check |
| --- | --- | --- | --- | --- |
| HTTP | 80 | TCP | SOURCE_IP_PORT | TCP connect |
| HTTPS | 443 | TCP | SOURCE_IP_PORT | TCP connect |
| Stats | 8880 | TCP | SOURCE_IP_PORT | TCP connect |
#### DNS
The load balancer floating IP is published as a single **A record**:
| Record | Type | Points to |
| --- | --- | --- |
| `main-lb.s2i2s.cloud.isti.cnr.it.` | A | Floating IP of the Octavia load balancer |
It is created by `openstack_dns_recordset_v2.main_lb_dns_recordset` in `octavia.tf`; the hostname
comes from `local.octavia_lb_hostname` in `main.tf`, and the record is exported as the
`main_loadbalancer_hostname` output.
**Every service published through the load balancer must be a CNAME pointing at this A record** —
never a second A record repeating the IP address. The floating IP is then written down in exactly
one place, so a change of address (rebuild of the load balancer, migration, DR) is a single-record
update instead of a sweep across every service.
```hcl
resource "openstack_dns_recordset_v2" "my_service" {
zone_id = local.dns_zone_id
name = "my-service.${local.dns_zone.name}"
description = "My service, published through the main load balancer"
ttl = 8600
type = "CNAME"
records = ["main-lb.s2i2s.cloud.isti.cnr.it."]
}
```
Notes:
- Designate expects fully qualified names **with the trailing dot**, both in `name` and in
`records`. `local.dns_zone.name` already ends with one.
- These CNAMEs normally live in the configuration of the service that owns them, not here. When
the service is defined in another workspace, read the target from this state rather than
hardcoding it: `data.terraform_remote_state.project_setup.outputs.main_loadbalancer_hostname`.
- The CNAME only brings the client to the load balancer. Routing the hostname to the right backend
is a separate step in the HAProxy L7 configuration (SNI / `Host` header), and the backend VM must
carry the `traffic_from_the_main_load_balancers` security group to accept the traffic.
- A CNAME cannot coexist with other records for the same name, so a hostname published this way
cannot also carry its own A record.
#### ACME DNS-01 challenge
| Record | Type | Points to |
| --- | --- | --- |
| `_acme-challenge.s2i2s.cloud.isti.cnr.it.` | CNAME | `_acme-challenge.isti.cnr.it.` |
Created by `openstack_dns_recordset_v2.acme_challenge_recordset` in `acme-challenge.tf` and exported
as the `acme_challenge_hostname` output. Let's Encrypt follows the CNAME when validating DNS-01, so
the `_acme-challenge` TXT records are published once in the `isti.cnr.it` zone instead of requiring
write access to this zone from every client.
#### Consequences of the OVN provider
The OVN driver is lighter than amphora (no amphora VMs, uses the main subnet directly), but it
constrains the configuration in ways that are visible throughout this setup:
- **No `allowed_cidrs` on listeners.** Access restrictions must be enforced by security groups
or inside HAProxy itself. This is why the stats port is filtered in the security group below.
- **Only `ROUND_ROBIN` and `SOURCE_IP_PORT`** are available as pool methods; `LEAST_CONNECTIONS`
is not supported.
- **Only `TCP` and `UDP-CONNECT` health monitors**; HTTP/HTTPS monitors are not supported, so all
three pools are probed with a plain TCP connect.
- **The client source IP is preserved.** OVN does not SNAT the traffic to an amphora VIP, so the
HAProxy L7 backends see the real client address rather than an address of the load balancer.
Only health-monitor probes and hairpinned traffic originate from the private subnet.
### Security Groups
| Security Group | Purpose | Use On |
| --- | --- | --- |
| `default_for_all` | Default rules: SSH via jump proxy, ICMP, Prometheus node exporter | All VMs |
| `ssh_access_to_the_jump_node` | SSH access from VPN endpoints | SSH Jump Proxy only |
| `debugging_from_jump_node` | Web debugging via SSH tunnels (ports 80, 443, 8100) | VMs needing debug access |
| `traffic_from_the_main_load_balancers` | HTTP/HTTPS from the HAProxy L7 IPs (ports 80, 443, 8080, 8888) | Backend web services |
| `traffic_from_main_lb_to_haproxy_l7` | Public HTTP/HTTPS plus filtered access to the stats port | HAProxy L7 VMs |
| `public_web_service` | HTTP/HTTPS from anywhere | Public-facing services with floating IP |
| `restricted_web_service` | HTTP from anywhere, HTTPS from VPNs only | Restricted services with floating IP |
| `prometheus_access_from_grafana` | HTTPS access from public Grafana server | Prometheus VM |
> **Do not confuse the two load balancer groups.** `traffic_from_main_lb_to_haproxy_l7` sits on the
> HAProxy L7 VMs themselves and faces the internet. `traffic_from_the_main_load_balancers` sits on
> the backend services *behind* HAProxy: since HAProxy is an L7 proxy that opens new connections
> from its own address, the client IP is **not** preserved on that hop, so its sources stay
> restricted to 10.10.0.11 and 10.10.0.12 and it must never be opened to `0.0.0.0/0`.
#### `traffic_from_main_lb_to_haproxy_l7`
Because the OVN load balancer preserves the client source IP (see above), the public listeners on
the HAProxy L7 VMs cannot be restricted to the private subnet:
| Port | Source | Rationale |
| --- | --- | --- |
| 80 | `0.0.0.0/0` | HTTP arrives with the original client address |
| 443 | `0.0.0.0/0` | HTTPS arrives with the original client address |
| 8880 | main private subnet | Health monitor probes for the stats pool |
| 8880 | D4S VPN 1/2, S2I2S VPN 1/2, InfraScience network | Human access to the HAProxy stats page |
| 8880 | 10.10.0.10/32 and the Prometheus floating IP | Prometheus scrapes the stats, over the private network and through the LB public IP |
| 10000 | 10.10.0.11/32, 10.10.0.12/32 | Peer traffic between the two HAProxy instances |
The stats port is deliberately **not** open to `0.0.0.0/0`: since the OVN provider does not support
`allowed_cidrs` on the listener, this security group is the only place where port 8880 can be
restricted, and client-IP preservation is precisely what makes filtering by real source address
possible. The per-source rules are generated with a `for_each` map in `haproxy.tf`.
### Storage
- **Prometheus Data Volume**: 100 GB SSD (CephSSD) with online resize enabled
## Architecture
```text
Internet
|
+-------------------+-------------------+
| | |
[SSH Jump Proxy] [Octavia LB] [Prometheus]
| (Floating IP) (Floating IP)
| |
| +-------+-------+
| | |
| [HAProxy L7-01] [HAProxy L7-02]
| | |
| +-------+-------+
| |
+-------------------+
|
[Internal Network]
|
+-------+-------+
| |
[Internal CA] [Backend VMs]
```
## Prerequisites
1. The `main_net_dns_router` configuration must be applied first (creates network, subnet, DNS zone)
2. SSH key must be configured in the OpenStack project
3. OpenStack credentials must be configured (via `clouds.yaml` or environment variables)
4. [OpenTofu](https://opentofu.org/) — this repository is driven with `tofu`, not `terraform`
5. `terraform-provider-openstack/openstack` >= 2.0.0; the configuration is verified against 3.4.0
## Usage
```bash
# Initialize OpenTofu
tofu init
# Review the plan
tofu plan
# Apply the configuration
tofu apply
```
> When re-initialising a directory that was last used with `terraform`, run `tofu init -upgrade`.
> The provider address in `.terraform.lock.hcl` moves from `registry.terraform.io` to
> `registry.opentofu.org`; that is expected.
>
> Provider 3.x tracks a `stateful` attribute on `openstack_networking_secgroup_v2` that 1.53.0 did
> not. The first refresh after upgrading therefore reports `+ stateful = false` as out-of-band
> drift on every security group. It produces no planned change and can be ignored.
## SSH Jump Proxy Configuration
To access VMs in the S2I2S cloud, you must use the SSH jump proxy. Add the following configuration to your `~/.ssh/config` file:
```ssh-config
# S2I2S SSH Jump Proxy
# Replace <your_username> with your actual username
Host s2i2s-jump
HostName ssh-jump-proxy.s2i2s.cloud.isti.cnr.it
User <your_username>
IdentityFile ~/.ssh/your_private_key
ForwardAgent yes
# Keep connection alive
ServerAliveInterval 60
ServerAliveCountMax 3
# Pattern match for all S2I2S internal hosts by IP
# Matches any IP in the 10.10.0.x range
# Usage: ssh 10.10.0.10
Host 10.10.0.*
User <your_username>
ForwardAgent yes
ProxyJump <your_username>@ssh-jump-proxy.s2i2s.cloud.isti.cnr.it
# Alternative: named aliases for specific internal hosts
Host s2i2s-prometheus
HostName 10.10.0.10
User <your_username>
ForwardAgent yes
ProxyJump <your_username>@ssh-jump-proxy.s2i2s.cloud.isti.cnr.it
Host s2i2s-ca
HostName 10.10.0.4
User <your_username>
ForwardAgent yes
ProxyJump <your_username>@ssh-jump-proxy.s2i2s.cloud.isti.cnr.it
Host s2i2s-haproxy-01
HostName 10.10.0.11
User <your_username>
ForwardAgent yes
ProxyJump <your_username>@ssh-jump-proxy.s2i2s.cloud.isti.cnr.it
Host s2i2s-haproxy-02
HostName 10.10.0.12
User <your_username>
ForwardAgent yes
ProxyJump <your_username>@ssh-jump-proxy.s2i2s.cloud.isti.cnr.it
```
### SSH Usage Examples
```bash
# Connect to the jump proxy directly
ssh s2i2s-jump
# Connect to an internal VM by IP (using pattern match from ssh config)
ssh 10.10.0.10
# Connect to a named internal host (if configured in ssh config)
ssh s2i2s-prometheus
# Connect without ssh config (replace <your_username>)
ssh -J <your_username>@ssh-jump-proxy.s2i2s.cloud.isti.cnr.it <your_username>@10.10.0.10
# Copy a file to an internal VM
scp -J <your_username>@ssh-jump-proxy.s2i2s.cloud.isti.cnr.it localfile.txt <your_username>@10.10.0.10:/tmp/
# Forward a local port to an internal service
ssh -L 8080:10.10.0.30:80 s2i2s-jump
# Create a SOCKS proxy through the jump host
ssh -D 1080 s2i2s-jump
# Then configure your browser to use SOCKS5 proxy at localhost:1080
```
### SSH Debugging via Tunnel
For debugging web applications on internal VMs, you can create SSH tunnels:
```bash
# Forward local port 8100 to a Tomcat debug port on internal VM
# (requires s2i2s-jump defined in ssh config)
ssh -L 8100:10.10.0.50:8100 s2i2s-jump
# Forward local port 8080 to HTTP on internal VM
ssh -L 8080:10.10.0.50:80 s2i2s-jump
# Forward local port 8443 to HTTPS on internal VM
ssh -L 8443:10.10.0.50:443 s2i2s-jump
# Without ssh config (replace <your_username>)
ssh -L 8080:10.10.0.50:80 <your_username>@ssh-jump-proxy.s2i2s.cloud.isti.cnr.it
```
## Outputs
The module exports the following outputs for use by other OpenTofu configurations:
### VM IDs and IPs
- `ssh_jump_proxy_id`, `ssh_jump_proxy_public_ip`, `ssh_jump_proxy_hostname`
- `internal_ca_id`
- `main_haproxy_l7_ids`
- `prometheus_server_id`, `prometheus_public_ip`, `prometheus_hostname`
### Load Balancer Outputs
- `main_loadbalancer_id`, `main_loadbalancer_ip`, `main_loadbalancer_public_ip`, `main_loadbalancer_hostname`
### Security Group Outputs
- `default_security_group`, `default_security_group_id`, `default_security_group_name`
- `access_to_the_jump_proxy`
- `debugging`
- `traffic_from_main_haproxy`
- `public_web`
- `restricted_web`
- `main_lb_to_haproxy_l7_security_group`
- `prometheus_access_from_grafana`
### Network Outputs (re-exported from main_net_dns_router)
- `dns_zone`, `dns_zone_id`
- `main_private_network`, `main_private_subnet`, `main_subnet_network_id`
- `basic_services_ip`, `main_haproxy_l7_ip`
### Image Outputs (re-exported from labs_common_variables)
- `ubuntu_2404`, `ubuntu2404_data_file` - the image and cloud-init script used by every VM here
- `ubuntu_2204`, `ubuntu2204_data_file` - kept for consumers still on 22.04
## File Structure
```text
project-setup/
├── provider.tf # OpenStack provider configuration
├── main.tf # Module references and local variables
├── security-groups.tf # All security group definitions
├── ssh-jump-proxy.tf # SSH jump proxy VM and floating IP
├── internal-ca.tf # Internal CA VM
├── haproxy.tf # HAProxy L7 VMs (pair with anti-affinity)
├── prometheus.tf # Prometheus VM with data volume
├── octavia.tf # OVN-based Octavia load balancer
├── outputs.tf # Output definitions
└── README.md # This file
```
## Dependencies
This module depends on:
- `../main_net_dns_router` - Network, subnet, router, and DNS zone
- `../variables` - Project-specific variables
- `../../modules/labs_common_variables` - Common variables (images, flavors, etc.)
- `../../modules/ssh-key-ref` - SSH key reference
## Notes
- The HAProxy L7 VMs are deployed with anti-affinity to ensure they run on different hypervisors
- All VMs use volume-backed boot disks with `delete_on_termination = false` for data persistence
- The Prometheus data volume uses CephSSD storage for better I/O performance
- Volumes have `enable_online_resize = true` for live resizing capability
- Security groups are designed to minimize attack surface while allowing necessary traffic flows
- Every instance carries `ignore_changes = [key_pair, user_data, network]` so that rotating the SSH
key or the cloud-init script does not trigger a rebuild
- The main load balancer uses the OVN Octavia provider; see
[Consequences of the OVN provider](#consequences-of-the-ovn-provider) before changing listeners,
pool methods, health monitors or the HAProxy security group
Reference in New Issue View Git Blame Copy Permalink