From 9d90aec167714b07879c67ad9bcc4518327c7cb7 Mon Sep 17 00:00:00 2001 From: Pierre Ozoux Date: Tue, 14 Feb 2017 09:38:56 +0000 Subject: [PATCH 01/29] Create a basic roadmap --- ROADMAP.md | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100644 ROADMAP.md diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..3406694 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,84 @@ +# TL;DR + + - k8s + - [ ] ceph + - [ ] flannel + - [ ] baremetal install + +# Object + +The aim of this document is to write the big lines of the future of libre.sh. + +# Version 1 + +The current version, let's call it 1, is a nice opiniated framework on how to run a single host with docker-compose. +It provides a list of packages and mofule compatible with this framework. +The best features of this framework are: + - https only + - some integration between the tools (auto provisioning of emails for new applications) + - domain name buying (Namecheap api) + - dns configuration (Namecheap api) + +# Version 2 - k8s + +This roadmap will discuss about the migration to kubernetes (k8s). + +## Distributions + +There are various k8s distributions (Tectonic, deis, openshift..) and the aim of libre.sh is not to become yet another distribution. + +It would be nice if we could list them, evaluate them, and decide to use one of them or not. + +## Installation/Operation + +libre.sh should be opiniated on the way to install and operate the cluster. + +It should provide easy steps to install on baremetal first. We aim for libre software, and as such, we can't rely +on cloud providers like gcloud, aws, or digital ocean. +As a second priority, we should give easy instructions to deploy on any cloud providers, as people are free to choose their chains :) + +## Storage + +One big challenge in k8s cluster context is to provide an implementation of major cloud providers about [PersistantVolume](https://kubernetes.io/docs/user-guide/persistent-volumes/). +In a libre cluster, this function would be achieved by a distributed file system technology. +After some investigation, the choice would be to use ceph. +There are already some work done on it like the [ceph-docker](https://github.com/ceph/ceph-docker/tree/master/examples) repo. + +## Network + +Another big challenge is network. k8s is strongly opiniated on what should be the network configuration. +Ideally, we would use some IPsec to secure the links between machine in a context we can't trust the network (like at hetzner). +There are 2 options: + - tinc vpn + - flannel that might implement IPsec in a near future + +The cheapest in term of work would be to bet on flannel. + +## Packages + +There is now a way to create and distribute packages in a standard way. +We can then remove the idea of modules and applications. +They will all be packages. + +The k8s standard for that is [helm](http://helm.sh/). There is already a big list of packages. +As for libre.sh, the idea would be to contribute the missing packages there. + +### opportunistic packages + +libre.sh would then be, just a repo of documentation on how to install, operate and manage a k8s cluster on baremetal. +There is still a place where we can have a difference. + +This idea is called opportunistic package. +This would be a package based on an official one. + +Let's take the example of WordPress. +The libre.sh version of WordPress would be based on the official one. +But it will have some mechanisms to discovers services available inside the cluster it is running on. + +These services could be: + - ldap + - piwik + - email + +So, when you install a new WordPress, it will try to discover opportunistically if there is a ldap service in the cluster, +and if yes, configure WordPress to use this ldap service. From b87184045a2f6a62a9cf77f22a16e92ca3d237a0 Mon Sep 17 00:00:00 2001 From: Pierre Ozoux Date: Tue, 28 Mar 2017 09:45:06 +0100 Subject: [PATCH 02/29] Add service catalog to ROADMAP --- ROADMAP.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/ROADMAP.md b/ROADMAP.md index 3406694..737e838 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -82,3 +82,6 @@ These services could be: So, when you install a new WordPress, it will try to discover opportunistically if there is a ldap service in the cluster, and if yes, configure WordPress to use this ldap service. + +This pattern will help make it happen: +https://github.com/kubernetes-incubator/service-catalog From e59501d09b06b45e60c2c0d932459f4432ad84a1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=EA=99=A8d=D1=86=D3=8Do=CE=B7=D2=AC?= Date: Tue, 28 Mar 2017 12:50:47 +0200 Subject: [PATCH 03/29] typo cat > /etc/systemd/system/data.mount << EOF --- INSTALL_HETZNER.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/INSTALL_HETZNER.md b/INSTALL_HETZNER.md index 844dbf9..a335505 100644 --- a/INSTALL_HETZNER.md +++ b/INSTALL_HETZNER.md @@ -63,7 +63,7 @@ MAILADDR xxx@xxx.org # Start service systemctl start mdmonitor.service -cat > /etc/systemd/system/data.mont << EOF +cat > /etc/systemd/system/data.mount << EOF [Mount] What=/dev/md0 Where=/data From dd7006dbe48a7072cffc391d878e57de5c3dcfd2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=EA=99=A8d=D1=86=D3=8Do=CE=B7=D2=AC?= Date: Wed, 29 Mar 2017 01:40:38 +0200 Subject: [PATCH 04/29] add libre stats which show docker stats with name. --- utils/libre | 3 +++ 1 file changed, 3 insertions(+) diff --git a/utils/libre b/utils/libre index 597440b..071a89f 100755 --- a/utils/libre +++ b/utils/libre @@ -59,9 +59,12 @@ case "$1" in fi;; provision) provision ${@:2};; + stats) + docker stats $(docker ps|grep -v "NAMES"|awk '{ print $NF }'|tr "\n" " ");; *) echo "Usage:" echo " - start|status|enable|disable|restart|stop: command sent to systemctl." echo " - ps|exec|logs: command sent to docker compose." echo " - update: to update the current folder." + echo " - stats: show docker stats with names." esac From e76505cda863c278475001d524d4aab09cc2bf84 Mon Sep 17 00:00:00 2001 From: Pierre Ozoux Date: Thu, 6 Apr 2017 20:31:37 +0100 Subject: [PATCH 05/29] Add Fosdem installation video --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 418be92..93446c1 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,11 @@ A PaaS that is aimed at hosting free software \o/ -To install it, follow instructions in `INSTALL.md`. +## Installation + +[![ScreenShot](https://cloud.pierre-o.fr/index.php/apps/files_sharing/ajax/publicpreview.php?x=1364&y=283&a=true&file=preview.png&t=KlxYYFT59GirMJa&scalingup=0)](https://fosdem.org/2017/schedule/event/libre_sh/) + +To install it, follow the video above and instructions in `INSTALL.md`. ### What is libre.sh From b0440ddb79e88013f46873d10f2014d96869f285 Mon Sep 17 00:00:00 2001 From: Pierre Ozoux Date: Tue, 11 Apr 2017 15:09:30 +0100 Subject: [PATCH 06/29] Stricter sshd config --- user_data | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/user_data b/user_data index efe9dea..f1aa577 100644 --- a/user_data +++ b/user_data @@ -1,6 +1,17 @@ #cloud-config write_files: + - path: /etc/ssh/sshd_config + permissions: 0600 + owner: root:root + content: | + # Use most defaults for sshd configuration. + UsePrivilegeSeparation sandbox + Subsystem sftp internal-sftp + PermitRootLogin no + AllowUsers core + PasswordAuthentication no + ChallengeResponseAuthentication no - path: /etc/sysctl.d/libresh.conf permissions: 0644 owner: root From b34211c1bcee08e935dfd9793923bb2ef6910b85 Mon Sep 17 00:00:00 2001 From: pierreozoux Date: Fri, 30 Jun 2017 11:51:54 +0100 Subject: [PATCH 07/29] Add the www --- unit-files/u@.service | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/unit-files/u@.service b/unit-files/u@.service index 5100754..bead8bf 100644 --- a/unit-files/u@.service +++ b/unit-files/u@.service @@ -16,7 +16,7 @@ EnvironmentFile=-/data/domains/%i/env Environment=HOSTNAME=%H WorkingDirectory=/data/domains/%i/ ExecStartPre=-/opt/bin/docker-compose rm -f -ExecStart=/bin/bash -euxc "VIRTUAL_HOST=%i /opt/bin/docker-compose up" +ExecStart=/bin/bash -euxc "VIRTUAL_HOST=%i,www.%i /opt/bin/docker-compose up" ExecStop=/opt/bin/docker-compose stop [Install] From 13b022bcb56d32356a919175c3f48ad703aec1f1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?J=EA=99=A8d=D1=86=D3=8Do=CE=B7=D2=AC?= Date: Mon, 3 Jul 2017 17:10:47 +0700 Subject: [PATCH 08/29] Add Linux instructions --- INSTALL_LINUX.md | 89 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100644 INSTALL_LINUX.md diff --git a/INSTALL_LINUX.md b/INSTALL_LINUX.md new file mode 100644 index 0000000..eaf820c --- /dev/null +++ b/INSTALL_LINUX.md @@ -0,0 +1,89 @@ +# Instructions to install libre.sh on linux with Systemd + +## Recommendation +- Systemd (debian 8 or debian 9, CentOS 7 ...) + +# Installation +Where basicly reproduce what the user_data do for us. + +as root + +# configure sshd +Don't forget to create the user core and adding your ssh key before +You could also remove AllowUsers core or/and change the username. + +> cat > /etc/ssh/sshd_config < chmod 600 /etc/ssh/sshd_config +> systemctl restart sshd + +# add kernel parameter + +> cat > /etc/sysctl.d/libresh.conf < chmod 644 /etc/sysctl.d/libresh.conf +> sysctl -p + +# define Localhost + +> cat > /etc/hosts < cat > /etc/environment < mkdir -p /opt/bin &&\ +> dockerComposeVersion=$(curl -s https://api.github.com/repos/docker/compose/releases/latest|grep tag_name|cut -d'"' -f4) &&\ +> curl -L https://github.com/docker/compose/releases/download/$dockerComposeVersion/docker-compose-`uname -s`-`uname -m` > /opt/bin/docker-compose &&\ +> chmod +x /opt/bin/docker-compose + +# install Libre.sh + +> git clone https://github.com/indiehosters/libre.sh.git /libre.sh &&\ +> mkdir /{data,system} &&\ +> mkdir /data/trash &&\ +> cp /libre.sh/unit-files/* /etc/systemd/system && systemctl daemon-reload &&\ +> systemctl enable web-net.service &&\ +> systemctl start web-net.service &&\ +> cp /libre.sh/utils/* /opt/bin/ + +# add /opt/bin path + +> cat > /etc/profile.d/libre.sh < Date: Mon, 3 Jul 2017 11:18:57 +0100 Subject: [PATCH 09/29] Remove trailing space --- INSTALL_LINUX.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/INSTALL_LINUX.md b/INSTALL_LINUX.md index eaf820c..fb6c370 100644 --- a/INSTALL_LINUX.md +++ b/INSTALL_LINUX.md @@ -1,16 +1,16 @@ -# Instructions to install libre.sh on linux with Systemd +# Instructions to install libre.sh on linux with Systemd -## Recommendation -- Systemd (debian 8 or debian 9, CentOS 7 ...) +## Recommendation +- Systemd (debian 8 or debian 9, CentOS 7 ...) -# Installation -Where basicly reproduce what the user_data do for us. +# Installation +Where basicly reproduce what the user_data do for us. -as root +as root -# configure sshd +# configure sshd Don't forget to create the user core and adding your ssh key before -You could also remove AllowUsers core or/and change the username. +You could also remove AllowUsers core or/and change the username. > cat > /etc/ssh/sshd_config < chmod 600 /etc/ssh/sshd_config > systemctl restart sshd -# add kernel parameter > cat > /etc/sysctl.d/libresh.conf < chmod 644 /etc/sysctl.d/libresh.conf > sysctl -p +# add kernel parameter # define Localhost @@ -40,7 +40,6 @@ EOF ::1 localhost EOF -# define envrionment > cat > /etc/environment < mkdir -p /opt/bin &&\ > dockerComposeVersion=$(curl -s https://api.github.com/repos/docker/compose/releases/latest|grep tag_name|cut -d'"' -f4) &&\ > curl -L https://github.com/docker/compose/releases/download/$dockerComposeVersion/docker-compose-`uname -s`-`uname -m` > /opt/bin/docker-compose &&\ > chmod +x /opt/bin/docker-compose -# install Libre.sh > git clone https://github.com/indiehosters/libre.sh.git /libre.sh &&\ > mkdir /{data,system} &&\ @@ -81,9 +80,10 @@ EOF > systemctl start web-net.service &&\ > cp /libre.sh/utils/* /opt/bin/ -# add /opt/bin path > cat > /etc/profile.d/libre.sh < Date: Mon, 3 Jul 2017 11:19:15 +0100 Subject: [PATCH 10/29] Fix codestyle --- INSTALL_LINUX.md | 150 ++++++++++++++++++++++++++--------------------- 1 file changed, 82 insertions(+), 68 deletions(-) diff --git a/INSTALL_LINUX.md b/INSTALL_LINUX.md index fb6c370..d42bfaa 100644 --- a/INSTALL_LINUX.md +++ b/INSTALL_LINUX.md @@ -11,79 +11,93 @@ as root # configure sshd Don't forget to create the user core and adding your ssh key before You could also remove AllowUsers core or/and change the username. - -> cat > /etc/ssh/sshd_config < chmod 600 /etc/ssh/sshd_config -> systemctl restart sshd - - -> cat > /etc/sysctl.d/libresh.conf < chmod 644 /etc/sysctl.d/libresh.conf -> sysctl -p + +``` +cat > /etc/ssh/sshd_config < /etc/sysctl.d/libresh.conf < cat > /etc/hosts < cat > /etc/environment < /etc/hosts < /etc/environment < mkdir -p /opt/bin &&\ -> dockerComposeVersion=$(curl -s https://api.github.com/repos/docker/compose/releases/latest|grep tag_name|cut -d'"' -f4) &&\ -> curl -L https://github.com/docker/compose/releases/download/$dockerComposeVersion/docker-compose-`uname -s`-`uname -m` > /opt/bin/docker-compose &&\ -> chmod +x /opt/bin/docker-compose - - -> git clone https://github.com/indiehosters/libre.sh.git /libre.sh &&\ -> mkdir /{data,system} &&\ -> mkdir /data/trash &&\ -> cp /libre.sh/unit-files/* /etc/systemd/system && systemctl daemon-reload &&\ -> systemctl enable web-net.service &&\ -> systemctl start web-net.service &&\ -> cp /libre.sh/utils/* /opt/bin/ - - -> cat > /etc/profile.d/libre.sh < /opt/bin/docker-compose &&\ +chmod +x /opt/bin/docker-compose +``` # install Libre.sh + +``` +git clone https://github.com/indiehosters/libre.sh.git /libre.sh &&\ +mkdir /{data,system} &&\ +mkdir /data/trash &&\ +cp /libre.sh/unit-files/* /etc/systemd/system && systemctl daemon-reload &&\ +systemctl enable web-net.service &&\ +systemctl start web-net.service &&\ +cp /libre.sh/utils/* /opt/bin/ +``` + # add /opt/bin path + +``` +cat > /etc/profile.d/libre.sh < Date: Mon, 14 Aug 2017 16:30:21 +0700 Subject: [PATCH 11/29] Fix typo + sshkey (#151) forget the s a permission + add sskey --- user_data | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/user_data b/user_data index f1aa577..701c8c9 100644 --- a/user_data +++ b/user_data @@ -1,5 +1,8 @@ #cloud-config +ssh_authorized_keys: + - "PUT YOUR SSH KEY PUBLIC HERE" + write_files: - path: /etc/ssh/sshd_config permissions: 0600 @@ -26,7 +29,7 @@ write_files: 255.255.255.255 broadcasthost ::1 localhost - path: /etc/environment - permission: 0644 + permissions: 0644 owner: root content: | NAMECHEAP_URL="namecheap.com" From b0df1b0667137d67e5b46d81e2f9dbac0dba521e Mon Sep 17 00:00:00 2001 From: JOduMonT Date: Mon, 21 Aug 2017 22:36:13 +0700 Subject: [PATCH 12/29] vm.overcommit_memory & transparent_hugepage (#154) as recommanded by REDIS \#WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. \# WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. --- INSTALL_LINUX.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/INSTALL_LINUX.md b/INSTALL_LINUX.md index d42bfaa..90f1016 100644 --- a/INSTALL_LINUX.md +++ b/INSTALL_LINUX.md @@ -31,9 +31,12 @@ systemctl restart sshd cat > /etc/sysctl.d/libresh.conf < /sys/kernel/mm/transparent_hugepage/enabled ``` # define Localhost From 63e81f7d723c06b46866b7aaa208dbdc176262db Mon Sep 17 00:00:00 2001 From: JOduMonT Date: Mon, 21 Aug 2017 22:36:19 +0700 Subject: [PATCH 13/29] disable transparent hugepage (#153) as recommanded by REDIS \# WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. --- user_data | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/user_data b/user_data index 701c8c9..683ae84 100644 --- a/user_data +++ b/user_data @@ -50,6 +50,11 @@ write_files: MAIL_PASS= MAIL_HOST=mail.indie.host MAIL_PORT=587 + - path: /sys/kernel/mm/transparent_hugepage/enabled + permission: 0644 + owner: root + content: | + never coreos: update: reboot-strategy: off From b6853eee696cff176ebdb61c2d290786b089e2cf Mon Sep 17 00:00:00 2001 From: JOduMonT Date: Mon, 21 Aug 2017 22:36:40 +0700 Subject: [PATCH 14/29] add vm.overcommit_memory=1 (#152) as is recommanded by REDIS # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. --- user_data | 1 + 1 file changed, 1 insertion(+) diff --git a/user_data b/user_data index 683ae84..d364d5d 100644 --- a/user_data +++ b/user_data @@ -21,6 +21,7 @@ write_files: content: | fs.aio-max-nr=1048576 vm.max_map_count=262144 + vm.overcommit_memory=1 - path: /etc/hosts permissions: 0644 owner: root From 1128a898e14d3f469e72934a3d779c6f0acc5b69 Mon Sep 17 00:00:00 2001 From: JOduMonT Date: Wed, 30 Aug 2017 14:49:12 +0700 Subject: [PATCH 15/29] Fix typo in user_data (#155) --- user_data | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/user_data b/user_data index d364d5d..0f3537d 100644 --- a/user_data +++ b/user_data @@ -52,7 +52,7 @@ write_files: MAIL_HOST=mail.indie.host MAIL_PORT=587 - path: /sys/kernel/mm/transparent_hugepage/enabled - permission: 0644 + permissions: 0644 owner: root content: | never From 3766449a9a5e8c073fe857ced0ba3add2fcc699f Mon Sep 17 00:00:00 2001 From: JOduMonT Date: Mon, 18 Sep 2017 20:48:10 +0200 Subject: [PATCH 16/29] unable to write transparent_hugepage (#157) so the script stop at this line and libre.sh isn't install --- user_data | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/user_data b/user_data index 0f3537d..3791ec8 100644 --- a/user_data +++ b/user_data @@ -22,6 +22,7 @@ write_files: fs.aio-max-nr=1048576 vm.max_map_count=262144 vm.overcommit_memory=1 + vm.nr_hugepages=0 - path: /etc/hosts permissions: 0644 owner: root @@ -51,11 +52,6 @@ write_files: MAIL_PASS= MAIL_HOST=mail.indie.host MAIL_PORT=587 - - path: /sys/kernel/mm/transparent_hugepage/enabled - permissions: 0644 - owner: root - content: | - never coreos: update: reboot-strategy: off From 121076762c01a5153b43b8178d441c442acdcd44 Mon Sep 17 00:00:00 2001 From: Samuel Laulhau Date: Sat, 24 Feb 2018 16:05:22 +0100 Subject: [PATCH 17/29] specify sysctl file (#163) Avoid error when /etc/sysctl.conf dosen't exist --- INSTALL_LINUX.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/INSTALL_LINUX.md b/INSTALL_LINUX.md index 90f1016..11e6f53 100644 --- a/INSTALL_LINUX.md +++ b/INSTALL_LINUX.md @@ -34,7 +34,7 @@ vm.max_map_count=262144 vm.overcommit_memory=1 EOF chmod 644 /etc/sysctl.d/libresh.conf -sysctl -p +sysctl -p /etc/sysctl.d/libresh.conf echo never > /sys/kernel/mm/transparent_hugepage/enabled ``` From edc5e966739c3f04323e1b6a57769f541293b459 Mon Sep 17 00:00:00 2001 From: jess Date: Wed, 28 Feb 2018 19:19:45 +0900 Subject: [PATCH 18/29] Added backers and sponsors on the README (#168) --- README.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/README.md b/README.md index 93446c1..d9b217f 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,6 @@ # libre.sh +[![Backers on Open Collective](https://opencollective.com/libresh/backers/badge.svg)](#backers) + [![Sponsors on Open Collective](https://opencollective.com/libresh/sponsors/badge.svg)](#sponsors) ## Introduction @@ -104,3 +106,33 @@ You can help us by: - writing documentation/blog on how you got up and running in 5mins - writing more documentation - sending us cake :) We loove cake! + +## Contributors + +This project exists thanks to all the people who contribute. [[Contribute](CONTRIBUTING.md)]. + + + +## Backers + +Thank you to all our backers! 🙏 [[Become a backer](https://opencollective.com/libresh#backer)] + + + + +## Sponsors + +Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [[Become a sponsor](https://opencollective.com/libresh#sponsor)] + + + + + + + + + + + + + From b41c8f33dac9ab657771fca8e42812114892c9d6 Mon Sep 17 00:00:00 2001 From: Samuel Laulhau Date: Sat, 3 Mar 2018 10:59:32 +0100 Subject: [PATCH 19/29] fix contributors link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d9b217f..4b6c7fd 100644 --- a/README.md +++ b/README.md @@ -110,7 +110,7 @@ You can help us by: ## Contributors This project exists thanks to all the people who contribute. [[Contribute](CONTRIBUTING.md)]. - + ## Backers From dce1ce87f51adfa115049aa71b4ecc510993479f Mon Sep 17 00:00:00 2001 From: JOduMonT Date: Wed, 6 Jun 2018 13:39:49 +0200 Subject: [PATCH 20/29] I add zerotier as network solution (#172) to building a zerotier network it's very simple but not necessary decentralize also; for now, zerotier don't manage NS but for communication over IP; it kick some ass ;) --- ROADMAP.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/ROADMAP.md b/ROADMAP.md index 737e838..734bcd6 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -48,7 +48,8 @@ There are already some work done on it like the [ceph-docker](https://github.com Another big challenge is network. k8s is strongly opiniated on what should be the network configuration. Ideally, we would use some IPsec to secure the links between machine in a context we can't trust the network (like at hetzner). -There are 2 options: +There are 3 options: + - zerotier - tinc vpn - flannel that might implement IPsec in a near future From 7556c39c58f039bee9cc665e20dc8839ff2489fa Mon Sep 17 00:00:00 2001 From: michel memeteau Date: Sat, 29 Sep 2018 16:01:29 +0200 Subject: [PATCH 21/29] Update ROADMAP.md --- ROADMAP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ROADMAP.md b/ROADMAP.md index 734bcd6..bffd6a5 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -12,7 +12,7 @@ The aim of this document is to write the big lines of the future of libre.sh. # Version 1 The current version, let's call it 1, is a nice opiniated framework on how to run a single host with docker-compose. -It provides a list of packages and mofule compatible with this framework. +It provides a list of packages and module compatible with this framework. The best features of this framework are: - https only - some integration between the tools (auto provisioning of emails for new applications) From 5d24dce3d819a2d4a1ca0f61c778bb32004f4551 Mon Sep 17 00:00:00 2001 From: Timothee Gosselin Date: Mon, 7 Jan 2019 15:17:47 +0100 Subject: [PATCH 22/29] add headers for CRSF check --- utils/add_cloud_user | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/utils/add_cloud_user b/utils/add_cloud_user index 01ca7c5..74bc628 100755 --- a/utils/add_cloud_user +++ b/utils/add_cloud_user @@ -6,6 +6,6 @@ user=$1 email=$2 quota=$3 -curl -X POST --user ${cloud_admin}:${cloud_pass} https://${cloud_hostname}/ocs/v1.php/cloud/users -d userid="$1" -d password="`tr -dc A-Za-z0-9_ < /dev/urandom | head -c 10 | xargs`" -curl -X PUT --user ${cloud_admin}:${cloud_pass} https://${cloud_hostname}/ocs/v1.php/cloud/users/${user} -d key="email" -d value="${email}" -curl -X PUT --user ${cloud_admin}:${cloud_pass} https://${cloud_hostname}/ocs/v1.php/cloud/users/${user} -d key="quota" -d value="${quota}" +curl -X POST -H "OCS-APIRequest:true" --user ${cloud_admin}:${cloud_pass} https://${cloud_hostname}/ocs/v1.php/cloud/users -d userid="$1" -d password="`tr -dc A-Za-z0-9_ < /dev/urandom | head -c 10 | xargs`" +curl -X PUT -H "OCS-APIRequest:true" --user ${cloud_admin}:${cloud_pass} https://${cloud_hostname}/ocs/v1.php/cloud/users/${user} -d key="email" -d value="${email}" +curl -X PUT -H "OCS-APIRequest:true" --user ${cloud_admin}:${cloud_pass} https://${cloud_hostname}/ocs/v1.php/cloud/users/${user} -d key="quota" -d value="${quota}" From 39f4a5c4b14d6e058d0cd13329250cd5eb76bf38 Mon Sep 17 00:00:00 2001 From: Timothee Gosselin Date: Mon, 7 Jan 2019 15:27:25 +0100 Subject: [PATCH 23/29] add headers for CRSF check --- utils/add_to_group | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/utils/add_to_group b/utils/add_to_group index b87f4ef..67ce9a2 100755 --- a/utils/add_to_group +++ b/utils/add_to_group @@ -5,4 +5,4 @@ source /etc/environment user=$1 group=$2 -curl -X POST --user ${cloud_admin}:${cloud_pass} https://${cloud_hostname}/ocs/v1.php/cloud/users/${1}/groups -d groupid="${2}" +curl -X POST -H "OCS-APIRequest:true" --user ${cloud_admin}:${cloud_pass} https://${cloud_hostname}/ocs/v1.php/cloud/users/${1}/groups -d groupid="${2}" From 1ac7f2cbe9c450a31d4073c039ad4e40a0daf54d Mon Sep 17 00:00:00 2001 From: Pierre Ozoux Date: Mon, 28 Jan 2019 06:58:33 +0100 Subject: [PATCH 24/29] Adds more kubernetes stuff. But we need to migrate it clearly. --- README.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 4b6c7fd..a760ed7 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,17 @@ ## Introduction -A PaaS that is aimed at hosting free software \o/ +A ecosystem that is aimed at hosting free software \o/ + +We are working on bootstrapping an ecosystem of tools to facilitate the hosting of free software. +Think of it as + - [ISPconfig](https://www.ispconfig.org/) + - FLOSS [cpanel](https://www.cpanel.net/products/) + - [cloudron](https://cloudron.io/) with email + +for [kubernetes](https://kubernetes.io/). + +This ecosystem can be deployed on [Raspberries](https://kubecloud.io/setting-up-a-kubernetes-1-11-raspberry-pi-cluster-using-kubeadm-952bbda329c8) or on popular cloud providers and scale globally or anything in between. We can affirm that it scales globally because it is based on kubernetes, a tool developped from the experience of Google hosting containers at scale. ## Installation From 931d8d7355d5fa372dad86d9939e0d71303d8eeb Mon Sep 17 00:00:00 2001 From: michel memeteau Date: Mon, 28 Jan 2019 23:00:50 +0100 Subject: [PATCH 25/29] New libre.sh chat --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a760ed7..03fab0f 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ It contains 2 [unit-files](https://github.com/indiehosters/LibrePaaS/tree/master You can use the following channels to request community support: - [mailinglist/forum](https://forum.indie.host/t/about-the-libre-sh-category/71) - - [chat](https://chat.indie.host/channel/libre.sh) + - [chat](https://riot.allmende.io/#/room/#libre.sh:matrix.allmende.io) For paid support, just send an inquiry to support@libre.sh. From 983718bf9efc2b6af5456e675805f798c0b1c1e4 Mon Sep 17 00:00:00 2001 From: Pierre Ozoux Date: Fri, 15 Feb 2019 20:18:48 +0100 Subject: [PATCH 26/29] Fix indentation --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 03fab0f..61ba12c 100644 --- a/README.md +++ b/README.md @@ -62,8 +62,8 @@ All of this is hosted by libre.sh :) Here is a list of modules supported: - https proxy: - - [HAProxy](https://github.com/indiehosters/haproxy) - - [Nginx](https://github.com/indiehosters/nginx) + - [HAProxy](https://github.com/indiehosters/haproxy) + - [Nginx](https://github.com/indiehosters/nginx) - [logs](https://github.com/indiehosters/logs) - [monitoring](https://github.com/indiehosters/monitoring) - [git-puller](https://github.com/indiehosters/git-puller) From fab6a854a91ea77cdcb575482829d1f31bca4cf3 Mon Sep 17 00:00:00 2001 From: Pierre Ozoux Date: Wed, 20 Feb 2019 17:44:21 +0100 Subject: [PATCH 27/29] Removes repo that are deprecated. --- README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/README.md b/README.md index 61ba12c..05419c0 100644 --- a/README.md +++ b/README.md @@ -64,11 +64,9 @@ Here is a list of modules supported: - https proxy: - [HAProxy](https://github.com/indiehosters/haproxy) - [Nginx](https://github.com/indiehosters/nginx) - - [logs](https://github.com/indiehosters/logs) - [monitoring](https://github.com/indiehosters/monitoring) - [git-puller](https://github.com/indiehosters/git-puller) - [backups](https://github.com/indiehosters/backups) - - [sshd](https://github.com/indiehosters/sshd) Go to their respective page for more details. From cfd8fddd1de6af7640967d2d459f26b8a572a718 Mon Sep 17 00:00:00 2001 From: Timothee Gosselin Date: Wed, 20 Feb 2019 18:33:54 +0100 Subject: [PATCH 28/29] archive project --- CHANGELOG.md | 61 ---------- INSTALL.md | 36 ------ INSTALL_HETZNER.md | 75 ------------ INSTALL_LINUX.md | 106 ----------------- README.md | 147 +---------------------- ROADMAP.md | 88 -------------- unit-files/s@.service | 23 ---- unit-files/u@.service | 23 ---- unit-files/web-net.service | 11 -- user_data | 106 ----------------- utils/add_cloud_user | 11 -- utils/add_mailbox | 18 --- utils/add_to_group | 8 -- utils/add_user | 27 ----- utils/configure_dkim_dns | 43 ------- utils/helpers | 67 ----------- utils/libre | 70 ----------- utils/migrate | 17 --- utils/provision | 227 ------------------------------------ utils/stop_before_migration | 11 -- 20 files changed, 1 insertion(+), 1174 deletions(-) delete mode 100644 CHANGELOG.md delete mode 100644 INSTALL.md delete mode 100644 INSTALL_HETZNER.md delete mode 100644 INSTALL_LINUX.md delete mode 100644 ROADMAP.md delete mode 100644 unit-files/s@.service delete mode 100644 unit-files/u@.service delete mode 100644 unit-files/web-net.service delete mode 100644 user_data delete mode 100755 utils/add_cloud_user delete mode 100755 utils/add_mailbox delete mode 100755 utils/add_to_group delete mode 100755 utils/add_user delete mode 100755 utils/configure_dkim_dns delete mode 100755 utils/helpers delete mode 100755 utils/libre delete mode 100755 utils/migrate delete mode 100755 utils/provision delete mode 100755 utils/stop_before_migration diff --git a/CHANGELOG.md b/CHANGELOG.md deleted file mode 100644 index 548f582..0000000 --- a/CHANGELOG.md +++ /dev/null @@ -1,61 +0,0 @@ -# 0.3.0 - -* adds automation script for user provisionning -* moves backup to duplicity -* big simplification -* some fixes - -# 0.2.4 - -* improve tests -* wordpess version 4.1 -* Internal modifications - * rename project - * rename images - * integrate dockerfiles to the project - * add hotfixes - -# 0.2.3 - -* fixes backup -* better tests -* import dump.sql when relevant - -# 0.2.2 - -* add Known as an application - -# 0.2.1 - -* draft instructions for how to add an application (whether server-wide or per-user) -* several bugfixes - -# 0.2.0 - -* a separation between /data/domains and /data/runtime, making site immigration much easier -* the wordpress image and the mysql image it depends on -* the backup service which commits all user content, including a mysql dump, to a private git repo, and pushes that out to a remote destination every hour -* the nginx image from 0.1.0 split into static and static-git - - -# 0.1.0 - -* Static webhosting - * based on haproxy with nginx backends - * all running as Docker containers - * SNI-capable (multiple https domains on one single IPv4 address) - * pulls in content from any git repo, then updates every 10 minutes - * can be run redundantly in round-robin DNS setup - -* email forwarder - * based on postfix - * stateless apart from simple configuration files - * can be run redundantly on multiple MX handlers - -* automated administration - * Docker containers are orchestrated with etcd and systemd - * script to deploy it on a remote coreos server - * script for adding a site from a git repo - * script for adding an empty placeholder site - * docs describing how to use these scripts - * Vagrantfile for using it inside vagrant diff --git a/INSTALL.md b/INSTALL.md deleted file mode 100644 index 8c4cd05..0000000 --- a/INSTALL.md +++ /dev/null @@ -1,36 +0,0 @@ -# Instructions to install libre.sh - -## Recommendation -- you'd need API key on Namecheap (if you want to automatically buy and configure domain name) - -## Installation - -These instructions depend a bit on your cloud provider. - -### [Digital Ocean](https://m.do.co/c/1b468ce0671f) - - 1. Install [doctl](https://github.com/digitalocean/doctl/) - 2. Issue the following command: - -``` -doctl compute droplet create libre.sh --user-data-file ./user_data --wait --ssh-keys $KEY_ID --size 1gb --region lon1 --image coreos-stable -``` - -### Provider with user_data support - -If you use a cloud provider that support `user_data`, like [Scaleway](http://scaleway.com/), just use [this user_data](https://raw.githubusercontent.com/indiehosters/libre.sh/master/user_data). - -### Hetzner - -You can also buy a baremetal at [Hetzner](https://serverboerse.de/index.php?country=EN) as they are the cheapest options around. Follow these [instructions](INSTALL_HETZNER.md) in this case. - -### Provider without user_data support - -Use boot a live cd, and issue that command: - -``` -wget https://raw.github.com/coreos/init/master/bin/coreos-install -bash coreos-install -d /dev/sda -c user_data -``` - -And voila, your first libre.sh node is ready! diff --git a/INSTALL_HETZNER.md b/INSTALL_HETZNER.md deleted file mode 100644 index a335505..0000000 --- a/INSTALL_HETZNER.md +++ /dev/null @@ -1,75 +0,0 @@ -# Instructions to install libre.sh - -## Recommendation -- ssd on /dev/sda -- hdd on /dev/sdb -- hdd on /dev/sdc -- API key on Namecheap (if you want to automatically buy domain name) - -# Installation - -First, you need a server. -We recommend [Hetzner](https://serverboerse.de/index.php?country=EN) as they are the cheapest options around. -You can filter servers with ssd. - -These instructions can also work on any VM/VPS/Hardware. - -## Install the system - -``` -IP= - -ssh -o "StrictHostKeyChecking no" root@$IP - -hostname= -ssh_public_key="" - -fdisk -l #find your ssd - -# Setup raid -cat > /etc/mdadm.conf << EOF -MAILADDR dev@null.org -EOF -mdadm --create --verbose /dev/md0 --level=mirror --raid-devices=2 /dev/sdb /dev/sdc -mkfs.ext4 /dev/md0 - -cat > cloud-config.tmp << EOF -#cloud-config - -hostname: "$hostname" -ssh_authorized_keys: - - $ssh_public_key -EOF - -apt-get install gawk -wget https://raw.github.com/coreos/init/master/bin/coreos-install -bash coreos-install -d /dev/sda -c cloud-config.tmp - -reboot -``` - -``` -ssh core@$IP - -#configure mdmonitor. - -sudo su - - -mdadm --examine --scan > /etc/mdadm.conf -vim /etc/mdadm.conf -#ADD your mail -MAILADDR xxx@xxx.org - -# Start service -systemctl start mdmonitor.service - -cat > /etc/systemd/system/data.mount << EOF -[Mount] -What=/dev/md0 -Where=/data -Type=ext4 -EOF - -wget https://raw.githubusercontent.com/indiehosters/libre.sh/master/user_data -O /var/lib/coreos-install/user_data - -coreos-cloudinit /var/lib/coreos-install/user_data diff --git a/INSTALL_LINUX.md b/INSTALL_LINUX.md deleted file mode 100644 index 11e6f53..0000000 --- a/INSTALL_LINUX.md +++ /dev/null @@ -1,106 +0,0 @@ -# Instructions to install libre.sh on linux with Systemd - -## Recommendation -- Systemd (debian 8 or debian 9, CentOS 7 ...) - -# Installation -Where basicly reproduce what the user_data do for us. - -as root - -# configure sshd -Don't forget to create the user core and adding your ssh key before -You could also remove AllowUsers core or/and change the username. - -``` -cat > /etc/ssh/sshd_config < /etc/sysctl.d/libresh.conf < /sys/kernel/mm/transparent_hugepage/enabled -``` - -# define Localhost - -``` -cat > /etc/hosts < /etc/environment < /opt/bin/docker-compose &&\ -chmod +x /opt/bin/docker-compose -``` -# install Libre.sh - -``` -git clone https://github.com/indiehosters/libre.sh.git /libre.sh &&\ -mkdir /{data,system} &&\ -mkdir /data/trash &&\ -cp /libre.sh/unit-files/* /etc/systemd/system && systemctl daemon-reload &&\ -systemctl enable web-net.service &&\ -systemctl start web-net.service &&\ -cp /libre.sh/utils/* /opt/bin/ -``` - -# add /opt/bin path - -``` -cat > /etc/profile.d/libre.sh < - - -## Backers - -Thank you to all our backers! 🙏 [[Become a backer](https://opencollective.com/libresh#backer)] - - - - -## Sponsors - -Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [[Become a sponsor](https://opencollective.com/libresh#sponsor)] - - - - - - - - - - - - - +Moved to https://lab.libreho.st/libre.sh/compose.libre.sh diff --git a/ROADMAP.md b/ROADMAP.md deleted file mode 100644 index bffd6a5..0000000 --- a/ROADMAP.md +++ /dev/null @@ -1,88 +0,0 @@ -# TL;DR - - - k8s - - [ ] ceph - - [ ] flannel - - [ ] baremetal install - -# Object - -The aim of this document is to write the big lines of the future of libre.sh. - -# Version 1 - -The current version, let's call it 1, is a nice opiniated framework on how to run a single host with docker-compose. -It provides a list of packages and module compatible with this framework. -The best features of this framework are: - - https only - - some integration between the tools (auto provisioning of emails for new applications) - - domain name buying (Namecheap api) - - dns configuration (Namecheap api) - -# Version 2 - k8s - -This roadmap will discuss about the migration to kubernetes (k8s). - -## Distributions - -There are various k8s distributions (Tectonic, deis, openshift..) and the aim of libre.sh is not to become yet another distribution. - -It would be nice if we could list them, evaluate them, and decide to use one of them or not. - -## Installation/Operation - -libre.sh should be opiniated on the way to install and operate the cluster. - -It should provide easy steps to install on baremetal first. We aim for libre software, and as such, we can't rely -on cloud providers like gcloud, aws, or digital ocean. -As a second priority, we should give easy instructions to deploy on any cloud providers, as people are free to choose their chains :) - -## Storage - -One big challenge in k8s cluster context is to provide an implementation of major cloud providers about [PersistantVolume](https://kubernetes.io/docs/user-guide/persistent-volumes/). -In a libre cluster, this function would be achieved by a distributed file system technology. -After some investigation, the choice would be to use ceph. -There are already some work done on it like the [ceph-docker](https://github.com/ceph/ceph-docker/tree/master/examples) repo. - -## Network - -Another big challenge is network. k8s is strongly opiniated on what should be the network configuration. -Ideally, we would use some IPsec to secure the links between machine in a context we can't trust the network (like at hetzner). -There are 3 options: - - zerotier - - tinc vpn - - flannel that might implement IPsec in a near future - -The cheapest in term of work would be to bet on flannel. - -## Packages - -There is now a way to create and distribute packages in a standard way. -We can then remove the idea of modules and applications. -They will all be packages. - -The k8s standard for that is [helm](http://helm.sh/). There is already a big list of packages. -As for libre.sh, the idea would be to contribute the missing packages there. - -### opportunistic packages - -libre.sh would then be, just a repo of documentation on how to install, operate and manage a k8s cluster on baremetal. -There is still a place where we can have a difference. - -This idea is called opportunistic package. -This would be a package based on an official one. - -Let's take the example of WordPress. -The libre.sh version of WordPress would be based on the official one. -But it will have some mechanisms to discovers services available inside the cluster it is running on. - -These services could be: - - ldap - - piwik - - email - -So, when you install a new WordPress, it will try to discover opportunistically if there is a ldap service in the cluster, -and if yes, configure WordPress to use this ldap service. - -This pattern will help make it happen: -https://github.com/kubernetes-incubator/service-catalog diff --git a/unit-files/s@.service b/unit-files/s@.service deleted file mode 100644 index fd4cfdd..0000000 --- a/unit-files/s@.service +++ /dev/null @@ -1,23 +0,0 @@ -[Unit] -Description=%p-%i - -# Requirements -Requires=docker.service - -# Dependency ordering -After=docker.service - -[Service] -Restart=always -RestartSec=10 -TimeoutStartSec=60 -TimeoutStopSec=15 -EnvironmentFile=-/system/%i/env -Environment=HOSTNAME=%H -WorkingDirectory=/system/%i/ -ExecStartPre=-/opt/bin/docker-compose rm -f -ExecStart=/bin/bash -euxc "/opt/bin/docker-compose up" -ExecStop=/opt/bin/docker-compose stop - -[Install] -WantedBy=multi-user.target diff --git a/unit-files/u@.service b/unit-files/u@.service deleted file mode 100644 index bead8bf..0000000 --- a/unit-files/u@.service +++ /dev/null @@ -1,23 +0,0 @@ -[Unit] -Description=%p-%i - -# Requirements -Requires=docker.service - -# Dependency ordering -After=docker.service - -[Service] -Restart=always -RestartSec=10 -TimeoutStartSec=60 -TimeoutStopSec=15 -EnvironmentFile=-/data/domains/%i/env -Environment=HOSTNAME=%H -WorkingDirectory=/data/domains/%i/ -ExecStartPre=-/opt/bin/docker-compose rm -f -ExecStart=/bin/bash -euxc "VIRTUAL_HOST=%i,www.%i /opt/bin/docker-compose up" -ExecStop=/opt/bin/docker-compose stop - -[Install] -WantedBy=multi-user.target diff --git a/unit-files/web-net.service b/unit-files/web-net.service deleted file mode 100644 index f45effd..0000000 --- a/unit-files/web-net.service +++ /dev/null @@ -1,11 +0,0 @@ -[Unit] -Description=Create lb_web network -Requires=docker.service -After=docker.service -[Service] -Type=oneshot -RemainAfterExit=true -ExecStart=/usr/bin/docker network create lb_web -ExecStop=/usr/bin/docker network rm lb_web -[Install] -WantedBy=local.target diff --git a/user_data b/user_data deleted file mode 100644 index 3791ec8..0000000 --- a/user_data +++ /dev/null @@ -1,106 +0,0 @@ -#cloud-config - -ssh_authorized_keys: - - "PUT YOUR SSH KEY PUBLIC HERE" - -write_files: - - path: /etc/ssh/sshd_config - permissions: 0600 - owner: root:root - content: | - # Use most defaults for sshd configuration. - UsePrivilegeSeparation sandbox - Subsystem sftp internal-sftp - PermitRootLogin no - AllowUsers core - PasswordAuthentication no - ChallengeResponseAuthentication no - - path: /etc/sysctl.d/libresh.conf - permissions: 0644 - owner: root - content: | - fs.aio-max-nr=1048576 - vm.max_map_count=262144 - vm.overcommit_memory=1 - vm.nr_hugepages=0 - - path: /etc/hosts - permissions: 0644 - owner: root - content: | - 127.0.0.1 localhost - 255.255.255.255 broadcasthost - ::1 localhost - - path: /etc/environment - permissions: 0644 - owner: root - content: | - NAMECHEAP_URL="namecheap.com" - NAMECHEAP_API_USER="pierreo" - NAMECHEAP_API_KEY= - IP=`curl -s http://icanhazip.com/` - FirstName="Pierre" - LastName="Ozoux" - Address="" - PostalCode="" - Country="Portugal" - Phone="+351.967184553" - EmailAddress="pierre@ozoux.net" - City="Lisbon" - CountryCode="PT" - BACKUP_DESTINATION=root@xxxxx:port - MAIL_USER= - MAIL_PASS= - MAIL_HOST=mail.indie.host - MAIL_PORT=587 -coreos: - update: - reboot-strategy: off - units: - - name: systemd-sysctl.service - command: restart - - name: swap.service - enable: true - command: start - content: | - [Unit] - Description=Turn on swap - [Service] - Type=oneshot - RemainAfterExit=true - ExecStartPre=-/bin/bash -euxc ' \ - fallocate -l 8192m /swap &&\ - chmod 600 /swap &&\ - mkswap /swap' - ExecStart=/sbin/swapon /swap - ExecStop=/sbin/swapoff /swap - [Install] - WantedBy=local.target - - name: install-compose.service - command: start - content: | - [Unit] - Description=Install Docker Compose - [Service] - Type=oneshot - RemainAfterExit=true - ExecStart=-/bin/bash -euxc ' \ - mkdir -p /opt/bin &&\ - url=$(curl -s https://api.github.com/repos/docker/compose/releases/latest | jq -r \'.assets[].browser_download_url | select(contains("Linux") and contains("x86_64"))\') &&\ - curl -L $url > /opt/bin/docker-compose &&\ - chmod +x /opt/bin/docker-compose' - - name: install-libresh.service - command: start - content: | - [Unit] - Description=Install libre.sh - [Service] - Type=oneshot - RemainAfterExit=true - ExecStart=-/bin/bash -euxc ' \ - git clone https://github.com/indiehosters/libre.sh.git /libre.sh &&\ - mkdir /{data,system} &&\ - mkdir /data/trash &&\ - cp /libre.sh/unit-files/* /etc/systemd/system && systemctl daemon-reload &&\ - systemctl enable web-net.service &&\ - systemctl start web-net.service &&\ - cp /libre.sh/utils/* /opt/bin/' diff --git a/utils/add_cloud_user b/utils/add_cloud_user deleted file mode 100755 index 74bc628..0000000 --- a/utils/add_cloud_user +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/bash -eux - -source /etc/environment - -user=$1 -email=$2 -quota=$3 - -curl -X POST -H "OCS-APIRequest:true" --user ${cloud_admin}:${cloud_pass} https://${cloud_hostname}/ocs/v1.php/cloud/users -d userid="$1" -d password="`tr -dc A-Za-z0-9_ < /dev/urandom | head -c 10 | xargs`" -curl -X PUT -H "OCS-APIRequest:true" --user ${cloud_admin}:${cloud_pass} https://${cloud_hostname}/ocs/v1.php/cloud/users/${user} -d key="email" -d value="${email}" -curl -X PUT -H "OCS-APIRequest:true" --user ${cloud_admin}:${cloud_pass} https://${cloud_hostname}/ocs/v1.php/cloud/users/${user} -d key="quota" -d value="${quota}" diff --git a/utils/add_mailbox b/utils/add_mailbox deleted file mode 100755 index 649a425..0000000 --- a/utils/add_mailbox +++ /dev/null @@ -1,18 +0,0 @@ -#!/bin/bash -set -e -set -u -set -x - -source /etc/environment - -email=$1 -email_password=$2 - -local_part=`echo $email | cut -d@ -f1` -email_domain=`echo $email | cut -d@ -f2` - -curl --data "username=${mail_username}&password=${mail_password}&login=Log+In&rememberme=0" -c /tmp/cookie.txt https://${mail_hostname}/auth/login -domain_id=`curl -b /tmp/cookie.txt https://${mail_hostname}/domain | grep $email_domain | grep purge-domain | grep -o 'purge-domain-[0-9]*' | grep -o '[0-9]*'` -curl --data "local_part=${local_part}&domain=${domain_id}&password=${email_password}" -b /tmp/cookie.txt https://${mail_hostname}/mailbox/add - -rm /tmp/cookie.txt diff --git a/utils/add_to_group b/utils/add_to_group deleted file mode 100755 index 67ce9a2..0000000 --- a/utils/add_to_group +++ /dev/null @@ -1,8 +0,0 @@ -#!/bin/bash -eux - -source /etc/environment - -user=$1 -group=$2 - -curl -X POST -H "OCS-APIRequest:true" --user ${cloud_admin}:${cloud_pass} https://${cloud_hostname}/ocs/v1.php/cloud/users/${1}/groups -d groupid="${2}" diff --git a/utils/add_user b/utils/add_user deleted file mode 100755 index 161588c..0000000 --- a/utils/add_user +++ /dev/null @@ -1,27 +0,0 @@ -#!/bin/bash -eux - -source /etc/environment - -email=$1 -cc_welcome_email=$2 -quota=$3 - -password=`tr -dc A-Za-z0-9_ < /dev/urandom | head -c 10 | xargs` -local_part=`echo $email | cut -d@ -f1` -email_domain=`echo $email | cut -d@ -f2` - -curl --data "username=${mail_username}&password=${mail_password}&login=Log+In&rememberme=0" -c /tmp/cookie.txt https://${mail_hostname}/auth/login - -if ! curl -b /tmp/cookie.txt https://${mail_hostname}/domain | grep $email_domain | grep purge-domain | grep -o 'purge-domain-[0-9]*' | grep -o '[0-9]*' ; then - curl --data "backupmx=0&active=1&max_aliases=0&max_mailboxes=0&max_quota=0"a=0&transport=virtual&domain=${email_domain}" -b /tmp/cookie.txt https://${mail_hostname}/domain/add -fi - -domain_id=`curl -b /tmp/cookie.txt https://${mail_hostname}/domain | grep $email_domain | grep purge-domain | grep -o 'purge-domain-[0-9]*' | grep -o '[0-9]*'` -curl --data "local_part=${local_part}&domain=${domain_id}&password=${password}&welcome_email=1&cc_welcome_email=${cc_welcome_email}" -b /tmp/cookie.txt https://${mail_hostname}/mailbox/add - -rm /tmp/cookie.txt - -curl -X PUT --user ${cloud_admin}:${cloud_pass} https://${cloud_hostname}/ocs/v1.php/cloud/users/${email} -d key="quota" -d value="${quota}" - -#docker exec -it --user www-data `echo ${cloud_hostname}_app_1 |sed 's/-//g' | sed 's/\.//g'` bash -c "\ - # php occ mail:account:create ${email} ${email} ${email} ${mail_hostname} 993 ssl ${email} ${password} ${mail_hostname} 587 tls ${email} ${password}" diff --git a/utils/configure_dkim_dns b/utils/configure_dkim_dns deleted file mode 100755 index d6fc48b..0000000 --- a/utils/configure_dkim_dns +++ /dev/null @@ -1,43 +0,0 @@ -#!/bin/bash -eux - -source /etc/environment - -function provision_dkim () { - docker exec mailindiehost_postfix_1 /add_domain.sh ${arg_u} -} - - -function configure_dns () { - domain_key=`cat /data/domains/mail.indie.host/opendkim/keys/${arg_u}/mail.txt | cut -d\" -f2 | sed 'N;s/\n//g' | sed 's/ //g' | sed 's/+/%2B/g' | sed 's/\//%2F/g'` - info "Configuring DNS." - arguments="&Command=namecheap.domains.dns.setHosts\ -&DomainName=${arg_u}\ -&SLD=$(SLD)\ -&TLD=$(TLD)\ -&HostName1=@\ -&RecordType1=A\ -&Address1=${IP}\ -&HostName2=www\ -&RecordType2=CNAME\ -&Address2=${arg_u}\ -&HostName3=@\ -&RecordType3=MX\ -&Address3=${mail_hostname}\ -&MXPref3=10\ -&HostName4=@\ -&RecordType4=TXT\ -&Address4=v=spf1%20include:${mail_hostname}\ -&Hostname5=_dmarc\ -&RecordType5=TXT\ -&Address5=v=DMARC1;%20p=none;%20rua=mailto:support@indie.host\ -&HostName6=mail._domainkey\ -&RecordType6=TXT\ -&Address6=${domain_key}\ -&HostName7=autoconfig\ -&RecordType7=CNAME\ -&Address7=autoconfig.`echo $mail_hostname | cut -d. -f2,3`\ -&EmailType=mx" - call_API ${arguments} - -} - diff --git a/utils/helpers b/utils/helpers deleted file mode 100755 index f9e9805..0000000 --- a/utils/helpers +++ /dev/null @@ -1,67 +0,0 @@ -function contains () { - local n=$# - local value=${!n} - for ((i=1;i < $#;i++)) { - if [ "${!i}" == "${value}" ]; then - echo "y" - return 0 - fi - } - echo "n" - return 1 -} - -function TLD () { - echo ${arg_u} | cut -d. -f2,3 -} - -function SLD () { - echo ${arg_u} | cut -d. -f1 -} - -function call_API () { - url="https://api.$NAMECHEAP_URL/xml.response\?ApiUser=${NAMECHEAP_API_USER}&ApiKey=${NAMECHEAP_API_KEY}&UserName=${NAMECHEAP_API_USER}&ClientIp=${IP}$1" - output=$(curl -s ${url}) - - if [ $(echo ${output} | grep -c 'Status="OK"') -eq 0 ]; then - error "API call failed. Please read the output" - echo ${output} - exit 1 - else - info "API call is a success." - fi -} - -function _fmt () { - local color_ok="\x1b[32m" - local color_bad="\x1b[31m" - - local color="${color_bad}" - if [ "${1}" = "debug" ] || [ "${1}" = "info" ] || [ "${1}" = "notice" ]; then - color="${color_ok}" - fi - - local color_reset="\x1b[0m" - if [[ "${TERM}" != "xterm"* ]] || [ -t 1 ]; then - # Don't use colors on pipes or non-recognized terminals - color=""; color_reset="" - fi - echo -e "$(date -u +"%Y-%m-%d %H:%M:%S UTC") ${color}$(printf "[%9s]" ${1})${color_reset}"; -} -function emergency () { echo "$(_fmt emergency) ${@}" 1>&2 || true; exit 1; } -function alert () { [ "${LOG_LEVEL}" -ge 1 ] && echo "$(_fmt alert) ${@}" 1>&2 || true; } -function critical () { [ "${LOG_LEVEL}" -ge 2 ] && echo "$(_fmt critical) ${@}" 1>&2 || true; } -function error () { [ "${LOG_LEVEL}" -ge 3 ] && echo "$(_fmt error) ${@}" 1>&2 || true; } -function warning () { [ "${LOG_LEVEL}" -ge 4 ] && echo "$(_fmt warning) ${@}" 1>&2 || true; } -function notice () { [ "${LOG_LEVEL}" -ge 5 ] && echo "$(_fmt notice) ${@}" 1>&2 || true; } -function info () { [ "${LOG_LEVEL}" -ge 6 ] && echo "$(_fmt info) ${@}" 1>&2 || true; } -function debug () { [ "${LOG_LEVEL}" -ge 7 ] && echo "$(_fmt debug) ${@}" 1>&2 || true; } - -function help () { - echo "" 1>&2 - echo " ${@}" 1>&2 - echo "" 1>&2 - echo " ${usage}" 1>&2 - echo "" 1>&2 - exit 1 -} diff --git a/utils/libre b/utils/libre deleted file mode 100755 index 071a89f..0000000 --- a/utils/libre +++ /dev/null @@ -1,70 +0,0 @@ -#!/bin/bash -eu - - -function error_path { - >&2 echo "Error: you must be in either /data/domains/*/ or /system/*/ to execute these commands" - exit 1 -} - -function systemctl_param { - first_level_path=`pwd | cut -d'/' -f2` - second_level_path=`pwd | cut -d'/' -f3` - - if [ "$first_level_path" == "system" ]; then - module=`pwd | cut -d'/' -f3` - if [ -n "$module" ]; then - echo s@$module - else - error_path - fi - elif [ "$first_level_path" == "data" ] && [ "$second_level_path" == "domains" ]; then - domain=`pwd | cut -d'/' -f4` - if [ -n "$domain" ]; then - echo u@$domain - else - error_path - fi - else - error_path - fi -} - - -case "$1" in - start|status|enable|disable|restart|stop) - if [ -n "$(systemctl_param)" ]; then - echo "systemctl $1 $(systemctl_param)" - systemctl $1 $(systemctl_param) - fi;; - journal) - if [ -n "$(systemctl_param)" ]; then - journalctl -fu $(systemctl_param) - fi;; - ps|exec|logs) - if [ -f ./env ]; then - env $(cat ./env | xargs) docker-compose $1 ${@:2} - else - docker-compose $1 ${@:2} - fi;; - update) - if [ "$(pwd)" == "/libre.sh" ]; then - git pull - cp /libre.sh/unit-files/* /etc/systemd/system && systemctl daemon-reload - cp /libre.sh/utils/* /opt/bin/ - elif [ -n "$(systemctl_param)" ]; then - git pull - docker-compose pull - docker-compose build - /opt/bin/libre restart - fi;; - provision) - provision ${@:2};; - stats) - docker stats $(docker ps|grep -v "NAMES"|awk '{ print $NF }'|tr "\n" " ");; - *) - echo "Usage:" - echo " - start|status|enable|disable|restart|stop: command sent to systemctl." - echo " - ps|exec|logs: command sent to docker compose." - echo " - update: to update the current folder." - echo " - stats: show docker stats with names." -esac diff --git a/utils/migrate b/utils/migrate deleted file mode 100755 index a2db01d..0000000 --- a/utils/migrate +++ /dev/null @@ -1,17 +0,0 @@ -#!/bin/bash -eux -#!/bin/bash -eux - -hash=$1 -domain=$2 - -cd /data/domains/ -wget https://wader.indie.host/migrate/$hash -unzip $hash -rm $hash - -cd $domain - -cp TLS/$domain.pem /system/haproxy/haproxy/certs - -systemctl start u@$domain -systemctl enable u@$domain diff --git a/utils/provision b/utils/provision deleted file mode 100755 index 2fbeeb6..0000000 --- a/utils/provision +++ /dev/null @@ -1,227 +0,0 @@ -#!/usr/bin/env bash -# Provision an application for a user for LibrePaaS -# -# This file: -# - Registers the domain name to NameCheap -# - Configures the DNS -# -# Version 0.0.3 -# -# Authors: -# - Pierre Ozoux (pierre-o.fr) -# -# Usage: -# LOG_LEVEL=7 ./provision -a github.com/indiehosters/known -u example.org -g -b -c -# -# Licensed under AGPLv3 - - -### Configuration -##################################################################### - -# Environment variables and their defaults -LOG_LEVEL="${LOG_LEVEL:-6}" # 7 = debug -> 0 = emergency - -# Commandline options. This defines the usage page, and is used to parse cli -# opts & defaults from. The parsing is unforgiving so be precise in your syntax -read -r -d '' usage <<-'EOF' - -u [arg] URL to process. Required. - -a [arg] Application to install. (in the form github.com/indiehosters/wordpress) - -s Start the application right away. - -b Buys the associated domain name. - -i Configure OpenDKIM. - -c Configures DNS on Namecheap. - -d Enables debug mode - -h This page -EOF - -### Functions -##################################################################### - -source /etc/environment -source /opt/bin/helpers -source /opt/bin/configure_dkim_dns - -function buy_domain_name () { - - not_supported_extensions=( "us" "eu" "nu" "asia" "ca" "co.uk" "me.uk" "org.uk" "com.au" "net.au" "org.au" "es" "nom.es" "com.es" "org.es" "de" "fr" ) - if [ $(contains "${not_supported_extensions[@]}" "$(TLD)") == "y" ]; then - error "Extension .$(TLD) is not yet supported.." - exit 1 - fi - - info "Buying Domain name." - arguments="&Command=namecheap.domains.create\ -&DomainName=${arg_u}\ -&Years=1\ -&AuxBillingFirstName=${FirstName}\ -&AuxBillingLastName=${LastName}\ -&AuxBillingAddress1=${Address}\ -&AuxBillingCity=${City}\ -&AuxBillingPostalCode=${PostalCode}\ -&AuxBillingCountry=${Country}\ -&AuxBillingPhone=${Phone}\ -&AuxBillingEmailAddress=${EmailAddress}\ -&AuxBillingStateProvince=${City}\ -&TechFirstName=${FirstName}\ -&TechLastName=${LastName}\ -&TechAddress1=${Address}\ -&TechCity=${City}\ -&TechPostalCode=${PostalCode}\ -&TechCountry=${Country}\ -&TechPhone=${Phone}\ -&TechEmailAddress=${EmailAddress}\ -&TechStateProvince=${City}\ -&AdminFirstName=${FirstName}\ -&AdminLastName=${LastName}\ -&AdminAddress1=${Address}\ -&AdminCity=${City}\ -&AdminPostalCode=${PostalCode}\ -&AdminCountry=${Country}\ -&AdminPhone=${Phone}\ -&AdminEmailAddress=${EmailAddress}\ -&AdminStateProvince=${City}\ -&RegistrantFirstName=${FirstName}\ -&RegistrantLastName=${LastName}\ -&RegistrantAddress1=${Address}\ -&RegistrantCity=${City}\ -&RegistrantPostalCode=${PostalCode}\ -&RegistrantCountry=${Country}\ -&RegistrantPhone=${Phone}\ -&RegistrantEmailAddress=${EmailAddress}\ -&RegistrantStateProvince=${City}" - - call_API ${arguments} - - info "Changing email forwarding." - arguments="&Command=namecheap.domains.dns.setEmailForwarding\ -&DomainName=${arg_u}\ -&mailbox1=hostmaster\ -&ForwardTo1=${EmailAddress}" - - call_API ${arguments} -} - -function application () { - git clone https://${arg_a}.git /data/domains/${arg_u} - cd /data/domains/${arg_u} - if [ -f ./scripts/install ]; then - export URL=${arg_u} - if [ -z ${MAIL_DOMAIN:-} ]; then - warning "you have no email server setup, we'll print a random configuration in your application. Make sure to check the parameters for your app to send proper emails." - warning "To stop having this warning, please configure your libre.sh to be abble to create email accounts." - warning "You can also contact support@indie.host to setup an email account for you" - export MAIL_PASS="randompass" - export MAIL_USER="example@indie.host" - export MAIL_DOMAIN="indie.host" - export MAIL_HOST="mail.indie.host" - export MAIL_PORT="587" - else - export MAIL_PASS=`tr -dc A-Za-z0-9_ < /dev/urandom | head -c 20 | xargs` - export MAIL_USER="noreply.${arg_u}@${MAIL_DOMAIN}" - /opt/bin/add_mailbox ${MAIL_USER} ${MAIL_PASS} - fi - ./scripts/install - fi -} - -function start () { - systemctl start u@${arg_u} - systemctl enable u@${arg_u} -} - -### Parse commandline options -##################################################################### - -# Translate usage string -> getopts arguments, and set $arg_ defaults -while read line; do - opt="$(echo "${line}" |awk '{print $1}' |sed -e 's#^-##')" - if ! echo "${line}" |egrep '\[.*\]' >/dev/null 2>&1; then - init="0" # it's a flag. init with 0 - else - opt="${opt}:" # add : if opt has arg - init="" # it has an arg. init with "" - fi - opts="${opts}${opt}" - - varname="arg_${opt:0:1}" - if ! echo "${line}" |egrep '\. Default=' >/dev/null 2>&1; then - eval "${varname}=\"${init}\"" - else - match="$(echo "${line}" |sed 's#^.*Default=\(\)#\1#g')" - eval "${varname}=\"${match}\"" - fi -done <<< "${usage}" - -# Reset in case getopts has been used previously in the shell. -OPTIND=1 - -# Overwrite $arg_ defaults with the actual CLI options -while getopts "${opts}" opt; do - line="$(echo "${usage}" |grep "\-${opt}")" - - - [ "${opt}" = "?" ] && help "Invalid use of script: ${@} " - varname="arg_${opt:0:1}" - default="${!varname}" - - value="${OPTARG}" - if [ -z "${OPTARG}" ] && [ "${default}" = "0" ]; then - value="1" - fi - - eval "${varname}=\"${value}\"" - debug "cli arg ${varname} = ($default) -> ${!varname}" -done - -shift $((OPTIND-1)) - -[ "$1" = "--" ] && shift - - -### Switches (like -d for debugmdoe, -h for showing helppage) -##################################################################### - -# debug mode -if [ "${arg_d}" = "1" ]; then - set -o xtrace - LOG_LEVEL="7" -fi - -# help mode -if [ "${arg_h}" = "1" ]; then - # Help exists with code 1 - help "Help using ${0}" -fi - - -### Validation (decide what's required for running your script and error out) -##################################################################### - -[ -z "${arg_u}" ] && help "URL is required." -[ -z "${LOG_LEVEL}" ] && emergency "Cannot continue without LOG_LEVEL." - - -### Runtime -##################################################################### - -# Exit on error. Append ||true if you expect an error. -# set -e is safer than #!/bin/bash -e because that is neutralised if -# someone runs your script like `bash yourscript` -set -o errexit -set -o nounset - -# Bash will remember & return the highest exitcode in a chain of pipes. -# This way you can catch the error in case mysqldump fails in `mysqldump |gzip` -set -o pipefail - -FOLDER=/data/domains/${arg_u} -TLS_FOLDER=${FOLDER}/TLS - -[ ${arg_b} -eq 1 ] && buy_domain_name -[ ! -z "${arg_a}" ] && application -[ ${arg_i} -eq 1 ] && provision_dkim -[ ${arg_c} -eq 1 ] && configure_dns -[ ${arg_s} -eq 1 ] && start - -exit 0 diff --git a/utils/stop_before_migration b/utils/stop_before_migration deleted file mode 100755 index 3729207..0000000 --- a/utils/stop_before_migration +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/bash -eux - -domain=$1 -ip=$2 -cd /data/domains/$domain -./scripts/backup - -systemctl stop u@$domain -systemctl disable u@$domain - -tar cvzf /home/core/${domain}.tgz . From e327bbfa38f7d43de8fdb9ec8c30e3b97a466fe7 Mon Sep 17 00:00:00 2001 From: Timothee Gosselin Date: Wed, 20 Feb 2019 18:34:38 +0100 Subject: [PATCH 29/29] Delete .gitignore --- .gitignore | 1 - 1 file changed, 1 deletion(-) delete mode 100644 .gitignore diff --git a/.gitignore b/.gitignore deleted file mode 100644 index 1377554..0000000 --- a/.gitignore +++ /dev/null @@ -1 +0,0 @@ -*.swp