ISTI-provisioning
/
openstack-infrastructure-terraform
4
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
openstack-infrastructure-te.../s2i2s/project-setup/README.md

15 KiB
Raw Blame History

S2I2S Project Setup

This OpenTofu configuration sets up the core infrastructure components for the S2I2S OpenStack project.

Overview

The project-setup module creates the following resources:

Virtual Machines

VM Purpose Flavor Private IP Boot disk Floating IP DNS Record
SSH Jump Proxy Secure SSH gateway for accessing internal VMs m2.small 10.10.0.5 30 GB Yes ssh-jump-proxy.s2i2s.cloud.isti.cnr.it
Internal CA Certificate Authority for internal services m1.small 10.10.0.4 10 GB No -
HAProxy L7 (x2) Layer 7 load balancers behind Octavia m1.medium 10.10.0.11, 10.10.0.12 10 GB No -
Prometheus Monitoring and metrics collection m1.medium 10.10.0.10 10 GB Yes prometheus.s2i2s.cloud.isti.cnr.it, alertmanager.s2i2s.cloud.isti.cnr.it (CNAME)

All VMs run Ubuntu 24.04 (Ubuntu-Noble-24.04.img) and are provisioned with the standard ubuntu2404.sh cloud-init user data script. Boot disks are volume-backed with delete_on_termination = false.

Load Balancer

An OVN-based Octavia load balancer (s2i2s-cloud-l4-load-balancer) provides L4 load balancing:

  • Provider: ovn (not amphora)
  • VIP: 10.10.0.20, on the main private subnet
  • Floating IP: Yes
  • DNS Record: main-lb.s2i2s.cloud.isti.cnr.it — see DNS below. Renamed from octavia-main-lb, which no longer resolves
  • Backend: HAProxy L7 instances (anti-affinity for HA)
Listener Port Protocol Pool method Health Check
HTTP 80 TCP SOURCE_IP_PORT TCP connect
HTTPS 443 TCP SOURCE_IP_PORT TCP connect
Stats 8880 TCP SOURCE_IP_PORT TCP connect

DNS

The load balancer floating IP is published as a single A record:

Record Type Points to
main-lb.s2i2s.cloud.isti.cnr.it. A Floating IP of the Octavia load balancer

It is created by openstack_dns_recordset_v2.main_lb_dns_recordset in octavia.tf; the hostname comes from local.octavia_lb_hostname in main.tf, and the record is exported as the main_loadbalancer_hostname output.

Every service published through the load balancer must be a CNAME pointing at this A record — never a second A record repeating the IP address. The floating IP is then written down in exactly one place, so a change of address (rebuild of the load balancer, migration, DR) is a single-record update instead of a sweep across every service.

resource "openstack_dns_recordset_v2" "my_service" {
  zone_id     = local.dns_zone_id
  name        = "my-service.${local.dns_zone.name}"
  description = "My service, published through the main load balancer"
  ttl         = 8600
  type        = "CNAME"
  records     = ["main-lb.s2i2s.cloud.isti.cnr.it."]
}

Notes:

  • Designate expects fully qualified names with the trailing dot, both in name and in records. local.dns_zone.name already ends with one.
  • These CNAMEs normally live in the configuration of the service that owns them, not here. When the service is defined in another workspace, read the target from this state rather than hardcoding it: data.terraform_remote_state.project_setup.outputs.main_loadbalancer_hostname.
  • The CNAME only brings the client to the load balancer. Routing the hostname to the right backend is a separate step in the HAProxy L7 configuration (SNI / Host header), and the backend VM must carry the traffic_from_the_main_load_balancers security group to accept the traffic.
  • A CNAME cannot coexist with other records for the same name, so a hostname published this way cannot also carry its own A record.

ACME DNS-01 challenge

Record Type Points to
_acme-challenge.s2i2s.cloud.isti.cnr.it. CNAME _acme-challenge.isti.cnr.it.

Created by openstack_dns_recordset_v2.acme_challenge_recordset in acme-challenge.tf and exported as the acme_challenge_hostname output. Let’s Encrypt follows the CNAME when validating DNS-01, so the _acme-challenge TXT records are published once in the isti.cnr.it zone instead of requiring write access to this zone from every client.

Consequences of the OVN provider

The OVN driver is lighter than amphora (no amphora VMs, uses the main subnet directly), but it constrains the configuration in ways that are visible throughout this setup:

  • No allowed_cidrs on listeners. Access restrictions must be enforced by security groups or inside HAProxy itself. This is why the stats port is filtered in the security group below.
  • Only ROUND_ROBIN and SOURCE_IP_PORT are available as pool methods; LEAST_CONNECTIONS is not supported.
  • Only TCP and UDP-CONNECT health monitors; HTTP/HTTPS monitors are not supported, so all three pools are probed with a plain TCP connect.
  • The client source IP is preserved. OVN does not SNAT the traffic to an amphora VIP, so the HAProxy L7 backends see the real client address rather than an address of the load balancer. Only health-monitor probes and hairpinned traffic originate from the private subnet.

Security Groups

Security Group Purpose Use On
default_for_all Default rules: SSH via jump proxy, ICMP, Prometheus node exporter All VMs
ssh_access_to_the_jump_node SSH access from VPN endpoints SSH Jump Proxy only
debugging_from_jump_node Web debugging via SSH tunnels (ports 80, 443, 8100) VMs needing debug access
traffic_from_the_main_load_balancers HTTP/HTTPS from the HAProxy L7 IPs (ports 80, 443, 8080, 8888) Backend web services
traffic_from_main_lb_to_haproxy_l7 Public HTTP/HTTPS plus filtered access to the stats port HAProxy L7 VMs
public_web_service HTTP/HTTPS from anywhere Public-facing services with floating IP
restricted_web_service HTTP from anywhere, HTTPS from VPNs only Restricted services with floating IP
prometheus_access_from_grafana HTTPS access from public Grafana server Prometheus VM

Do not confuse the two load balancer groups. traffic_from_main_lb_to_haproxy_l7 sits on the HAProxy L7 VMs themselves and faces the internet. traffic_from_the_main_load_balancers sits on the backend services behind HAProxy: since HAProxy is an L7 proxy that opens new connections from its own address, the client IP is not preserved on that hop, so its sources stay restricted to 10.10.0.11 and 10.10.0.12 and it must never be opened to 0.0.0.0/0.

traffic_from_main_lb_to_haproxy_l7

Because the OVN load balancer preserves the client source IP (see above), the public listeners on the HAProxy L7 VMs cannot be restricted to the private subnet:

Port Source Rationale
80 0.0.0.0/0 HTTP arrives with the original client address
443 0.0.0.0/0 HTTPS arrives with the original client address
8880 main private subnet Health monitor probes for the stats pool
8880 D4S VPN 1/2, S2I2S VPN 1/2, InfraScience network Human access to the HAProxy stats page
8880 10.10.0.10/32 and the Prometheus floating IP Prometheus scrapes the stats, over the private network and through the LB public IP
10000 10.10.0.11/32, 10.10.0.12/32 Peer traffic between the two HAProxy instances

The stats port is deliberately not open to 0.0.0.0/0: since the OVN provider does not support allowed_cidrs on the listener, this security group is the only place where port 8880 can be restricted, and client-IP preservation is precisely what makes filtering by real source address possible. The per-source rules are generated with a for_each map in haproxy.tf.

Storage

  • Prometheus Data Volume: 100 GB SSD (CephSSD) with online resize enabled

Architecture

                                    Internet
                                        |
                    +-------------------+-------------------+
                    |                   |                   |
            [SSH Jump Proxy]    [Octavia LB]         [Prometheus]
                    |           (Floating IP)        (Floating IP)
                    |                   |
                    |           +-------+-------+
                    |           |               |
                    |     [HAProxy L7-01] [HAProxy L7-02]
                    |           |               |
                    |           +-------+-------+
                    |                   |
                    +-------------------+
                            |
                    [Internal Network]
                            |
                    +-------+-------+
                    |               |
              [Internal CA]   [Backend VMs]

Prerequisites

  1. The main_net_dns_router configuration must be applied first (creates network, subnet, DNS zone)
  2. SSH key must be configured in the OpenStack project
  3. OpenStack credentials must be configured (via clouds.yaml or environment variables)
  4. OpenTofu — this repository is driven with tofu, not terraform
  5. terraform-provider-openstack/openstack >= 2.0.0; the configuration is verified against 3.4.0

Usage

# Initialize OpenTofu
tofu init

# Review the plan
tofu plan

# Apply the configuration
tofu apply

When re-initialising a directory that was last used with terraform, run tofu init -upgrade. The provider address in .terraform.lock.hcl moves from registry.terraform.io to registry.opentofu.org; that is expected.

Provider 3.x tracks a stateful attribute on openstack_networking_secgroup_v2 that 1.53.0 did not. The first refresh after upgrading therefore reports + stateful = false as out-of-band drift on every security group. It produces no planned change and can be ignored.

SSH Jump Proxy Configuration

To access VMs in the S2I2S cloud, you must use the SSH jump proxy. Add the following configuration to your ~/.ssh/config file:

# S2I2S SSH Jump Proxy
# Replace <your_username> with your actual username
Host s2i2s-jump
    HostName ssh-jump-proxy.s2i2s.cloud.isti.cnr.it
    User <your_username>
    IdentityFile ~/.ssh/your_private_key
    ForwardAgent yes
    # Keep connection alive
    ServerAliveInterval 60
    ServerAliveCountMax 3

# Pattern match for all S2I2S internal hosts by IP
# Matches any IP in the 10.10.0.x range
# Usage: ssh 10.10.0.10
Host 10.10.0.*
    User <your_username>
    ForwardAgent yes
    ProxyJump <your_username>@ssh-jump-proxy.s2i2s.cloud.isti.cnr.it

# Alternative: named aliases for specific internal hosts
Host s2i2s-prometheus
    HostName 10.10.0.10
    User <your_username>
    ForwardAgent yes
    ProxyJump <your_username>@ssh-jump-proxy.s2i2s.cloud.isti.cnr.it

Host s2i2s-ca
    HostName 10.10.0.4
    User <your_username>
    ForwardAgent yes
    ProxyJump <your_username>@ssh-jump-proxy.s2i2s.cloud.isti.cnr.it

Host s2i2s-haproxy-01
    HostName 10.10.0.11
    User <your_username>
    ForwardAgent yes
    ProxyJump <your_username>@ssh-jump-proxy.s2i2s.cloud.isti.cnr.it

Host s2i2s-haproxy-02
    HostName 10.10.0.12
    User <your_username>
    ForwardAgent yes
    ProxyJump <your_username>@ssh-jump-proxy.s2i2s.cloud.isti.cnr.it

SSH Usage Examples

# Connect to the jump proxy directly
ssh s2i2s-jump

# Connect to an internal VM by IP (using pattern match from ssh config)
ssh 10.10.0.10

# Connect to a named internal host (if configured in ssh config)
ssh s2i2s-prometheus

# Connect without ssh config (replace <your_username>)
ssh -J <your_username>@ssh-jump-proxy.s2i2s.cloud.isti.cnr.it <your_username>@10.10.0.10

# Copy a file to an internal VM
scp -J <your_username>@ssh-jump-proxy.s2i2s.cloud.isti.cnr.it localfile.txt <your_username>@10.10.0.10:/tmp/

# Forward a local port to an internal service
ssh -L 8080:10.10.0.30:80 s2i2s-jump

# Create a SOCKS proxy through the jump host
ssh -D 1080 s2i2s-jump
# Then configure your browser to use SOCKS5 proxy at localhost:1080

SSH Debugging via Tunnel

For debugging web applications on internal VMs, you can create SSH tunnels:

# Forward local port 8100 to a Tomcat debug port on internal VM
# (requires s2i2s-jump defined in ssh config)
ssh -L 8100:10.10.0.50:8100 s2i2s-jump

# Forward local port 8080 to HTTP on internal VM
ssh -L 8080:10.10.0.50:80 s2i2s-jump

# Forward local port 8443 to HTTPS on internal VM
ssh -L 8443:10.10.0.50:443 s2i2s-jump

# Without ssh config (replace <your_username>)
ssh -L 8080:10.10.0.50:80 <your_username>@ssh-jump-proxy.s2i2s.cloud.isti.cnr.it

Outputs

The module exports the following outputs for use by other OpenTofu configurations:

VM IDs and IPs

  • ssh_jump_proxy_id, ssh_jump_proxy_public_ip, ssh_jump_proxy_hostname
  • internal_ca_id
  • main_haproxy_l7_ids
  • prometheus_server_id, prometheus_public_ip, prometheus_hostname

Load Balancer Outputs

  • main_loadbalancer_id, main_loadbalancer_ip, main_loadbalancer_public_ip, main_loadbalancer_hostname

Security Group Outputs

  • default_security_group, default_security_group_id, default_security_group_name
  • access_to_the_jump_proxy
  • debugging
  • traffic_from_main_haproxy
  • public_web
  • restricted_web
  • main_lb_to_haproxy_l7_security_group
  • prometheus_access_from_grafana

Network Outputs (re-exported from main_net_dns_router)

  • dns_zone, dns_zone_id
  • main_private_network, main_private_subnet, main_subnet_network_id
  • basic_services_ip, main_haproxy_l7_ip

Image Outputs (re-exported from labs_common_variables)

  • ubuntu_2404, ubuntu2404_data_file - the image and cloud-init script used by every VM here
  • ubuntu_2204, ubuntu2204_data_file - kept for consumers still on 22.04

File Structure

project-setup/
├── provider.tf          # OpenStack provider configuration
├── main.tf              # Module references and local variables
├── security-groups.tf   # All security group definitions
├── ssh-jump-proxy.tf    # SSH jump proxy VM and floating IP
├── internal-ca.tf       # Internal CA VM
├── haproxy.tf           # HAProxy L7 VMs (pair with anti-affinity)
├── prometheus.tf        # Prometheus VM with data volume
├── octavia.tf           # OVN-based Octavia load balancer
├── outputs.tf           # Output definitions
└── README.md            # This file

Dependencies

This module depends on:

  • ../main_net_dns_router - Network, subnet, router, and DNS zone
  • ../variables - Project-specific variables
  • ../../modules/labs_common_variables - Common variables (images, flavors, etc.)
  • ../../modules/ssh-key-ref - SSH key reference

Notes

  • The HAProxy L7 VMs are deployed with anti-affinity to ensure they run on different hypervisors
  • All VMs use volume-backed boot disks with delete_on_termination = false for data persistence
  • The Prometheus data volume uses CephSSD storage for better I/O performance
  • Volumes have enable_online_resize = true for live resizing capability
  • Security groups are designed to minimize attack surface while allowing necessary traffic flows
  • Every instance carries ignore_changes = [key_pair, user_data, network] so that rotating the SSH key or the cloud-init script does not trigger a rebuild
  • The main load balancer uses the OVN Octavia provider; see Consequences of the OVN provider before changing listeners, pool methods, health monitors or the HAProxy security group