diff --git a/.gitignore b/.gitignore index 08d4e65..2c8cef4 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,11 @@ ### Ansible ### *.retry +# ansible vault pass +.vault_pass.* +.vault_pass +*.vault_pass + ### Terraform ### **/.terraform/* **/terraform.tfstate diff --git a/IAAS.md b/IAAS.md new file mode 100644 index 0000000..32b33f4 --- /dev/null +++ b/IAAS.md @@ -0,0 +1,176 @@ +## Summary + +Your setup is **primarily IaaS**, with some **PaaS-like behavior** that *you* are building yourself. You consume **SaaS** only for external tooling (e.g. GitHub). This is ideal for learning DevOps end-to-end. + +--- + +## Your current stack, mapped to IaaS / PaaS / SaaS + +![Image](https://miro.medium.com/1%2Ax4TscxA6uaN3asAreNg6yw.png) + +![Image](https://cdn.prod.website-files.com/65a790f0493b6806e60d6e21/6662b78dc707238ead7be590_64dca66820aece818c886638_Navigating%2520the%2520AWS%2520Cloud%2520Stack.png) + +![Image](https://sp-ao.shortpixel.ai/client/to_webp%2Cq_glossy%2Cret_img/https%3A//www.wowza.com/uploads/blog/wowza-on-premises-cloud-continuum-630x417.png) + +### 1. Infrastructure layer → **IaaS** + +**Terraform + Linode** + +What you control: + +* Virtual machines +* Private networking +* Firewalls +* Public vs private IPs + +What the provider controls: + +* Physical servers +* Data center +* Hypervisor + +**Classification:** ✅ **IaaS** + +This is the foundation. Everything else runs *on top* of this. + +--- + +### 2. Server configuration → **Still IaaS** + +**Ansible** + +What you manage: + +* Users, SSH hardening +* Docker installation +* Fail2ban +* Node Exporter +* System updates + +Important insight: + +> Configuration management does **not** move you out of IaaS. + +You are still managing: + +* The OS +* Security +* Services +* Patching + +**Classification:** ✅ **IaaS (advanced, well-automated)** + +--- + +### 3. Application runtime → **Self-built PaaS** + +**Docker + Gunicorn + env vars** + +You’ve created: + +* A standard runtime +* A repeatable deployment model +* Environment-based configuration +* Health endpoints +* Metrics + +But: + +* You still manage the host +* You still manage Docker +* You still manage upgrades and failures + +This is the key DevOps insight: + +> You are **building a PaaS on top of IaaS**. + +**Classification:** + +* Conceptually → **PaaS** +* Practically → **IaaS with a platform layer** + +This is exactly how many real platforms started (Heroku included). + +--- + +### 4. CI/CD & tooling → **SaaS** + +**GitHub, GitHub Actions** + +What you *don’t* manage: + +* Build servers +* Runners +* Scaling +* Availability + +You just: + +* Push code +* Configure workflows +* Consume the service + +**Classification:** ✅ **SaaS** + +--- + +## Responsibility breakdown (your exact setup) + +| Layer | Example | Model | +| ----------------- | ----------------------- | ---------------------- | +| Hardware | Linode data center | SaaS (hidden from you) | +| VM / Network | Terraform | IaaS | +| OS & Security | Ansible | IaaS | +| Container runtime | Docker | IaaS | +| App platform | Your Docker conventions | DIY PaaS | +| App code | Flask / Gunicorn | PaaS responsibility | +| CI/CD | GitHub Actions | SaaS | +| Monitoring UI | Grafana Cloud (if used) | SaaS | + +--- + +## Why this is an excellent DevOps learning path + +You are learning **all responsibility boundaries**, not just one: + +* **IaaS** → networking, firewalls, SSH, Linux +* **PaaS concepts** → runtime standardization, config via env vars +* **SaaS consumption** → CI/CD, observability platforms + +Many DevOps engineers: + +* Use PaaS without understanding IaaS +* Or manage IaaS without understanding platform design + +You’re doing **both**, in the correct order. + +--- + +## Real-world analogy (extended) + +* **Linode + Terraform** → You bought land +* **Ansible** → You built the house and utilities +* **Docker runtime** → You standardized apartments +* **Your app** → Tenants move in +* **GitHub Actions** → External cleaning & maintenance service + +--- + +## What your *next step* could be (optional) + +If you want to *move slightly toward PaaS without losing learning value*: + +* Add **automatic app restarts** +* Add **zero-downtime deploys** +* Add **service discovery / reverse proxy rules** +* Treat each app as cattle, not pets + +That would make your setup: + +> **A minimal, production-style PaaS built on IaaS** + +If you want, next we can: + +* Redraw this as a **single architecture diagram** +* Or map this directly to **AWS equivalents** (EC2, ECS, EKS, Elastic Beanstalk) + diff --git a/README.md b/README.md index feb75d0..fde9016 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ The goal is to gradually build a realistic production-like environment that incl The project grows in clear stages. Each stage is documented with **what was done**, **why it matters**, and **how it was implemented**, so it becomes both a learning journal and a portfolio project. -**Current status:** Stages 1–10 completed. Application is deployed, monitored with Prometheus and Grafana, and accessible via HTTP. Next step: TLS and reverse proxy. +**Current status:** Stages 1–11 completed. Application is deployed and monitored (Prometheus + Grafana), and served via HTTPS using Caddy + Let’s Encrypt. SSH access is restricted to a bastion host and allow listed source IPs. ## Structure @@ -26,61 +26,71 @@ Current project layout: ``` cloud_devops_lab/ -├── ansible -│   ├── ansible.cfg -│   ├── ansible.log -│   ├── group_vars -│   │   ├── all.yml -│   │   ├── app.yml -│   │   └── monitoring.yml -│   ├── hosts.ini -│   ├── playbooks -│   │   ├── bootstrap_1.yml -│   │   ├── bootstrap_2.yml -│   │   ├── deploy_app.yml -│   │   ├── monitoring_grafana.yml -│   │   ├── monitoring_node_exporter.yml -│   │   └── monitoring_prometheus.yml -│   ├── README.md -│   └── roles -│   ├── bootstrap_user -│   ├── common -│   ├── deploy_app -│   ├── docker -│   ├── grafana -│   ├── node_exporter -│   ├── prometheus -│   └── ssh_hardening -├── app -│   ├── Dockerfile -│   ├── gunicorn.conf.py -│   ├── requirements.txt -│   ├── src -│   │   ├── app.py -│   │   ├── __init__.py -│   │   ├── __pycache__ -│   │   ├── routes -│   │   └── utils -│   └── venv -│   ├── bin -│   ├── include -│   ├── lib -│   ├── lib64 -> lib -│   └── pyvenv.cfg -├── docs -│   └── project-checklist.md -├── infrastructure -│   └── terraform -│   ├── main.tf -│   ├── modules -│   ├── outputs.tf -│   ├── providers.tf -│   ├── terraform.tfstate -│   ├── terraform.tfstate.backup -│   ├── terraform.tfvars -│   └── variables.tf -├── LICENSE -└── README.md + ├── ansible + │   ├── ansible.cfg + │   ├── ansible.log + │   ├── group_vars + │   │   ├── all + │   │   │   └── vars.yml + │   │   ├── app + │   │   │   └── vars.yml + │   │   └── monitoring + │   │   ├── vars.yml + │   │   └── vault.yml + │   ├── hosts.ini + │   ├── playbooks + │   │   ├── bootstrap_1.yml + │   │   ├── bootstrap_2.yml + │   │   ├── caddy.yml + │   │   ├── deploy_app.yml + │   │   ├── monitoring_grafana.yml + │   │   ├── monitoring_node_exporter.yml + │   │   ├── monitoring_prometheus.yml + │   │   ├── security_fail2ban.yml + │   │   └── unattended_upgrades.yml + │   ├── README.md + │   └── roles + │   ├── bootstrap_user + │   ├── caddy + │   ├── common + │   ├── deploy_app + │   ├── docker + │   ├── fail2ban + │   ├── grafana + │   ├── node_exporter + │   ├── prometheus + │   ├── ssh_hardening + │   └── unattended_upgrades + ├── app + │   ├── Dockerfile + │   ├── gunicorn.conf.py + │   ├── requirements.txt + │   ├── src + │   │   ├── app.py + │   │   ├── routes + │   │   │   ├── health.py + │   │   │   ├── metrics.py + │   │   │   └── root.py + │   │   └── utils + │   │   ├── counters.py + │   └── venv + ├── docs + │   └── project-checklist.md + ├── IAAS.md + ├── infrastructure + │   └── terraform + │   ├── main.tf + │   ├── modules + │   │   └── compute + │   ├── outputs.tf + │   ├── providers.tf + │   ├── terraform.tfstate + │   ├── terraform.tfstate.backup + │   ├── terraform.tfvars + │   └── variables.tf + ├── LICENSE + └── README.md + ``` ## Requirements (current) @@ -126,7 +136,7 @@ The project is built in incremental stages. Each stage adds a new DevOps capabil - Stage 8: Docker installation (via Ansible) - Stage 9: Application deployment - Stage 10: Monitoring stack (Prometheus & Grafana) -- Stage 11: TLS certificates & reverse proxy (Caddy) +- Stage 11: TLS certificates & reverse proxy (Caddy)) ### Stage 1 — Flask Application @@ -226,6 +236,9 @@ infrastructure │   └── variables.tf ├── outputs.tf ├── providers.tf + ├── terraform.tfstate + ├── terraform.tfstate.backup + ├── terraform.tfvars └── variables.tf ``` @@ -377,6 +390,41 @@ Application-level observability enables insight into runtime behavior, performan - `http://:80/metrics` - Metrics verified in Prometheus and visualized in Grafana. +### Stage 11 — TLS certificates & reverse proxy (Caddy) + hardening + +This stage secures the application with HTTPS and adds additional server hardening. + +#### Part 1 — Reverse proxy + HTTPS (Caddy) + +**What:** +Deployed Caddy on the application server to act as a reverse proxy and terminate TLS. + +**Why:** +HTTPS is required for production-like deployments. A reverse proxy enables secure traffic, clean routing, and allows the application container to stay private (localhost only). + +**How:** +- Opened inbound port 443 on the application firewall. +- Deployed Caddy via Ansible using Docker (`network_mode: host`). +- Configured Caddy to serve: + - `clouddevopslab.eu` and `www.clouddevopslab.eu` via HTTPS (Let’s Encrypt) + - private-IP HTTP access for Prometheus scraping +- Added basic security headers in the Caddyfile. +- Updated app deployment so the Flask container is bound to `127.0.0.1:5000` (not publicly reachable). + +#### Part 2 — Stage 11 hardening (Option A) + +**What:** +Implemented baseline security hardening for the environment. + +**Why:** +Reduce attack surface and align with least-privilege and operational security practices. + +**How:** +- Restricted SSH access to the jump server using a Terraform allowlist (`ssh_allowed_ips`). +- Installed and enabled Fail2ban on the jump server (`sshd` jail). +- Enabled automatic security updates (`unattended-upgrades`) on all servers. +- Moved Grafana admin password into **Ansible Vault** (no secrets stored in Git). + ### Access Model - Direct SSH access is allowed only to the jump server. @@ -397,7 +445,7 @@ Application-level observability enables insight into runtime behavior, performan - `clouddevopslab.eu` → A record → application server - `www.clouddevopslab.eu` → A record → application server -At this stage, DNS records exist but application traffic is not yet exposed. +At this stage, DNS records exist and the application is reachable via HTTPS through Caddy. Cloudflare proxy is still disabled (DNS-only). Note: During early stages, application IP addresses may change when infrastructure is recreated. A reserved IPv4 address will be introduced later to provide a stable @@ -418,7 +466,8 @@ A chronological log describing the work done in each stage. - Procced to Stage 8: Docker installation (via Ansible) - Procced to Stage 9: Application deployment using Docker and GHCR - Procced to Stage 10: Stage 10: Monitoring stack (Prometheus & Grafana) -- Procced to Stage 11: TLS certificates & reverse proxy (Caddy) +- Proceeded to Stage 11: TLS certificates & reverse proxy (Caddy) +- Next: Stage 12: Cloudflare proxy + restrict origin access to Cloudflare IP ranges Stage 11 will introduce HTTPS, automatic TLS certificates, and a reverse proxy in front of the application. This enables secure traffic, prepares the setup @@ -468,7 +517,7 @@ Types used in this project: Examples: -- `feat(api): add /metrics/custom endpoint` +- `feat(api): add /metrics endpoint` - `docs(readme): document phase 1 (Flask app)` - `infra(terraform): create linode instances for app and monitoring` - `ci(docker): add image build and push workflow` diff --git a/ansible/README.md b/ansible/README.md index a162641..58dfb29 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -30,3 +30,12 @@ Run from the `ansible/` directory: ```bash ansible-playbook playbooks/.... ``` + +## Ansible Vault + +Secrets are stored in: + +- group_vars/\*/vault.yml + +Run playbooks with: +ansible-playbook playbooks/.yml --vault-password-file vault_pass.txt diff --git a/ansible/ansible.cfg b/ansible/ansible.cfg index 0c6d0fd..593e438 100644 --- a/ansible/ansible.cfg +++ b/ansible/ansible.cfg @@ -4,6 +4,7 @@ roles_path = roles host_key_checking = False retry_files_enabled = False timeout = 30 +vault_password_file = .vault_pass.txt # fact gathering & caching gathering = smart diff --git a/ansible/group_vars/all.yml b/ansible/group_vars/all/vars.yml similarity index 94% rename from ansible/group_vars/all.yml rename to ansible/group_vars/all/vars.yml index 2d531d3..91431a9 100644 --- a/ansible/group_vars/all.yml +++ b/ansible/group_vars/all/vars.yml @@ -1,5 +1,6 @@ --- ansible_python_interpreter: /usr/bin/python3 +ansible_user: devops # role devops_user: devops diff --git a/ansible/group_vars/app.yml b/ansible/group_vars/app/vars.yml similarity index 100% rename from ansible/group_vars/app.yml rename to ansible/group_vars/app/vars.yml diff --git a/ansible/group_vars/monitoring.yml b/ansible/group_vars/monitoring/vars.yml similarity index 100% rename from ansible/group_vars/monitoring.yml rename to ansible/group_vars/monitoring/vars.yml diff --git a/ansible/group_vars/monitoring/vault.yml b/ansible/group_vars/monitoring/vault.yml new file mode 100644 index 0000000..f251e3a --- /dev/null +++ b/ansible/group_vars/monitoring/vault.yml @@ -0,0 +1,7 @@ +$ANSIBLE_VAULT;1.1;AES256 +36613165396362346363653361646633313866323039636132313761363837346531363362323831 +6335383737306261633033666538383337313539623966660a366430613563303530313532346330 +63636632343162303664653534643632633235653333333239626531366266633439613866346465 +6364636330363131370a306130373638343465343631616461373534653738373465623436393735 +63373433663239313061626466396532373835613733333232643332663064616338363764336536 +6435646439636130303932626334356137326636393764376135 diff --git a/ansible/playbooks/security_fail2ban.yml b/ansible/playbooks/security_fail2ban.yml new file mode 100644 index 0000000..dc8a534 --- /dev/null +++ b/ansible/playbooks/security_fail2ban.yml @@ -0,0 +1,8 @@ +--- +- name: Configure fail2ban on jump + hosts: bastion + remote_user: devops + become: true + roles: + - fail2ban +... diff --git a/ansible/playbooks/unattended_upgrades.yml b/ansible/playbooks/unattended_upgrades.yml new file mode 100644 index 0000000..8a41e67 --- /dev/null +++ b/ansible/playbooks/unattended_upgrades.yml @@ -0,0 +1,8 @@ +--- +- name: Enable automatic security updates + hosts: all + remote_user: devops + become: true + roles: + - unattended_upgrades +... diff --git a/ansible/roles/fail2ban/handlers/main.yml b/ansible/roles/fail2ban/handlers/main.yml new file mode 100644 index 0000000..f616bfc --- /dev/null +++ b/ansible/roles/fail2ban/handlers/main.yml @@ -0,0 +1,6 @@ +--- +- name: restart fail2ban + ansible.builtin.service: + name: fail2ban + state: restarted +... diff --git a/ansible/roles/fail2ban/tasks/main.yml b/ansible/roles/fail2ban/tasks/main.yml new file mode 100644 index 0000000..8c91376 --- /dev/null +++ b/ansible/roles/fail2ban/tasks/main.yml @@ -0,0 +1,28 @@ +--- +- name: Install fail2ban + ansible.builtin.apt: + name: fail2ban + state: present + update_cache: true + +- name: Configure sshd jail + ansible.builtin.copy: + dest: /etc/fail2ban/jail.d/sshd.local + owner: root + group: root + mode: "0644" + content: | + [sshd] + enabled = true + port = ssh + maxretry = 5 + findtime = 10m + bantime = 1h + notify: restart fail2ban + +- name: Ensure fail2ban is enabled and started + ansible.builtin.service: + name: fail2ban + state: started + enabled: true +... diff --git a/ansible/roles/grafana/tasks/main.yml b/ansible/roles/grafana/tasks/main.yml index afbbfb3..76489a4 100644 --- a/ansible/roles/grafana/tasks/main.yml +++ b/ansible/roles/grafana/tasks/main.yml @@ -20,4 +20,10 @@ GF_SECURITY_ADMIN_USER: admin GF_SECURITY_ADMIN_PASSWORD: admin GF_USERS_ALLOW_SIGN_UP: "false" + +- name: Reset Grafana admin password + community.docker.docker_container_exec: + container: grafana + command: > + grafana-cli admin reset-admin-password {{ grafana_admin_password }} ... diff --git a/ansible/roles/unattended_upgrades/tasks/main.yml b/ansible/roles/unattended_upgrades/tasks/main.yml new file mode 100644 index 0000000..37281c3 --- /dev/null +++ b/ansible/roles/unattended_upgrades/tasks/main.yml @@ -0,0 +1,27 @@ +--- +- name: Install unattended-upgrades + ansible.builtin.apt: + name: + - unattended-upgrades + - apt-listchanges + state: present + update_cache: true + +- name: Enable automatic updates + ansible.builtin.copy: + dest: /etc/apt/apt.conf.d/20auto-upgrades + owner: root + group: root + mode: "0644" + content: | + APT::Periodic::Update-Package-Lists "1"; + APT::Periodic::Unattended-Upgrade "1"; + APT::Periodic::Download-Upgradeable-Packages "1"; + APT::Periodic::AutocleanInterval "7"; + +- name: Ensure unattended-upgrades is enabled and started + ansible.builtin.service: + name: unattended-upgrades + enabled: true + state: started +... diff --git a/docs/project-checklist.md b/docs/project-checklist.md index e420b52..ca509cb 100644 --- a/docs/project-checklist.md +++ b/docs/project-checklist.md @@ -30,6 +30,7 @@ completed, what is in progress, and what belongs to future expansion. - [ ] Terraform-managed DNS records (Cloudflare provider) - [ ] Stable DNS target via reserved IP - [x] Decide exposure model for monitoring (private vs public) +- [ ] Cloudflare in DNS and proxy mode --- @@ -43,9 +44,9 @@ completed, what is in progress, and what belongs to future expansion. - [x] Bastion (jump host) enforced - [x] SSH agent forwarding configured and documented - [x] Ansible runs as `devops` with `become` -- [ ] Restrict SSH on jump server to trusted IP ranges +- [x] Restrict SSH on jump server to trusted IP ranges - [ ] Explicit SSH hardening parameters (`MaxAuthTries`, `LoginGraceTime`) -- [ ] Fail2ban on jump server +- [x] Fail2ban on jump server - [ ] Break-glass access procedure documented --- @@ -57,7 +58,7 @@ completed, what is in progress, and what belongs to future expansion. - [x] Inbound policy DROP, outbound ACCEPT - [x] App firewall allows HTTP (80) - [ ] Firewall rules reviewed and minimized -- [ ] Automatic security updates (unattended-upgrades) +- [x] Automatic security updates (unattended-upgrades) - [ ] Disable unused services and packages - [ ] Basic system auditing and log retention @@ -67,7 +68,7 @@ completed, what is in progress, and what belongs to future expansion. - [x] Terraform secrets via environment variables - [x] GitHub Actions secrets for CI -- [ ] Ansible Vault for runtime secrets +- [x] Ansible Vault for runtime secrets (e.g., Grafana admin password) - [ ] Encrypted `.env` files generated by Ansible - [ ] Secret rotation strategy documented - [ ] Optional: HashiCorp Vault (Roadmap Part 2) @@ -82,7 +83,7 @@ completed, what is in progress, and what belongs to future expansion. - [x] Application container deployed - [x] Restart policy (`unless-stopped`) - [x] Healthcheck implemented -- [ ] Container runs as non-root user +- [x] Container runs as non-root user (Flask app) - [ ] Resource limits (CPU/memory) - [ ] Log rotation for Docker containers - [ ] Migrate app deployment to Docker Compose @@ -108,7 +109,7 @@ completed, what is in progress, and what belongs to future expansion. - [x] App deployed via Ansible - [x] Health endpoint validated automatically - [x] HTTP exposed on port 80 -- [ ] Bind app container to localhost only (via reverse proxy) +- [x] Bind app container to localhost only (via reverse proxy) - [ ] Blue/green or rolling deployment strategy - [ ] Rollback procedure documented @@ -130,7 +131,6 @@ completed, what is in progress, and what belongs to future expansion. - [ ] Retention and storage tuned/configured explicitly (beyond defaults) - [ ] Alert rules definedned -### Grafana ### Grafana @@ -141,18 +141,20 @@ completed, what is in progress, and what belongs to future expansion. - `process_resident_memory_bytes` - `rate(process_cpu_seconds_total[5m])` - `rate(python_gc_objects_collected_total[5m])` -- [ ] Access control hardening (no default creds, users/roles; still private via SSH tunnel) +- [x] Default Grafana admin password rotated (stored in Ansible Vault) +- [ ] Access control hardening (users/roles/SSO; still private via SSH tunnel) --- ## 10. TLS, Reverse Proxy & Edge Security -- [ ] Reverse proxy (Nginx / Caddy / Traefik) -- [ ] HTTPS via Let’s Encrypt or Cloudflare origin certs -- [ ] App container bound to localhost +- [x] Reverse proxy (Nginx / Caddy / Traefik) +- [x] HTTPS via Let’s Encrypt +- [x] App container bound to localhost - [ ] Cloudflare proxy enabled - [ ] Origin access restricted to Cloudflare IPs -- [ ] Security headers enforced (HSTS, etc.) +- [x] Basic security headers (nosniff, frame deny, referrer policy) +- [] HSTS enabled (HTTP Strict Transport Security) --- diff --git a/infrastructure/terraform/main.tf b/infrastructure/terraform/main.tf index c145acc..725f273 100644 --- a/infrastructure/terraform/main.tf +++ b/infrastructure/terraform/main.tf @@ -36,7 +36,7 @@ resource "linode_firewall" "jump_fw" { action = "ACCEPT" protocol = "TCP" ports = "22" - ipv4 = ["0.0.0.0/0"] + ipv4 = var.ssh_allowed_ips } inbound_policy = "DROP" # Drops evertying else diff --git a/infrastructure/terraform/modules/compute/variables.tf b/infrastructure/terraform/modules/compute/variables.tf index b50910c..3feb5fc 100644 --- a/infrastructure/terraform/modules/compute/variables.tf +++ b/infrastructure/terraform/modules/compute/variables.tf @@ -17,3 +17,4 @@ variable "image" { variable "authorized_keys" { type = list(string) } + diff --git a/infrastructure/terraform/variables.tf b/infrastructure/terraform/variables.tf index 00a4fe3..25b982e 100644 --- a/infrastructure/terraform/variables.tf +++ b/infrastructure/terraform/variables.tf @@ -44,3 +44,8 @@ variable "image" { type = string default = "linode/ubuntu22.04" } + +variable "ssh_allowed_ips" { + description = "CIDR blocks allowed to SSH to the jump server" + type = list(string) +}