Merge pull request #24 from SUNET/ft-dynamic_cosmos_modules
Allow dynamically generated /etc/puppet/cosmos-modules.conf.
This commit is contained in:
Micke Nordin
GitHub
commit
714506da1e
|
|
@ -5,7 +5,7 @@
|
|||
Introduction
|
||||
============
|
||||
|
||||
This document describes how to setup and run systems and service operations for a small to midsized
|
||||
This document describes how to setup and run systems and service operations for a small to mid-sized
|
||||
systems collection while maintaining scalability, security and auditability for changes.
|
||||
The process described below is based on open source components and assumes a Linux-based hosting
|
||||
infrastructure. These limitations could easily be removed though. This document describes the
|
||||
|
|
@ -18,7 +18,7 @@ Design Requirements
|
|||
The cosmos system has been used to operate security-critical infrastructure for a few years before
|
||||
it was combined with puppet into the multiverse template.
|
||||
|
||||
Several of the design requirements below are fulfilled by comos alone, while some (eg consistency)
|
||||
Several of the design requirements below are fulfilled by cosmos alone, while some (eg consistency)
|
||||
are easier to achieve using puppet than with cosmos alone.
|
||||
|
||||
Consistency
|
||||
|
|
@ -27,7 +27,7 @@ Consistency
|
|||
Changes should be applied atomically (locally on each host) across multiple system components on multiple
|
||||
physical and logical hosts (aka system state). The change mechanism should permit verification of state
|
||||
consistency and all modifications should be idempotents, i.e the same operation
|
||||
performend twice on the same system state should not in itself cause a problem.
|
||||
performed twice on the same system state should not in itself cause a problem.
|
||||
|
||||
Auditability
|
||||
------------
|
||||
|
|
@ -62,7 +62,7 @@ Administrative Scope
|
|||
|
||||
The repository constitutes the administrative domain of a multiverse setup: each host is
|
||||
connected to (i.e runs cosmos off of) a single GIT repository and derives trust from signed
|
||||
tags on that repository. A host cannot belong to more than 1 administratve domain but each
|
||||
tags on that repository. A host cannot belong to more than 1 administrative domain but each
|
||||
administrative domains can host multiple DNS domains - all hosts in a single repository
|
||||
doesn't need to be in the same zone.
|
||||
|
||||
|
|
@ -122,9 +122,9 @@ Consistency
|
|||
-----------
|
||||
|
||||
As a master-less architecture, multiverse relies on _eventual consistency_: changes will eventually
|
||||
be applied to all hosts. In such a model it becomes very imporant that changes are idempotent, so
|
||||
be applied to all hosts. In such a model it becomes very important that changes are idempotent, so
|
||||
that applying a change multiple times (in an effort to get dependent changes through) won't cause
|
||||
an issue. Using native cosmos, such changes are achived using timestamp-files that control entry
|
||||
an issue. Using native cosmos, such changes are archived using timestamp-files that control entry
|
||||
into code-blocks:
|
||||
|
||||
```
|
||||
|
|
@ -136,7 +136,7 @@ fi
|
|||
```
|
||||
|
||||
This pattern is mostly replaced in multiverse by using puppet manifests and modules that
|
||||
are inherently indempotent but it can nevertheless be a useful addition to the toolchain.
|
||||
are inherently idempotent but it can nevertheless be a useful addition to the toolchain.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
|
@ -184,7 +184,7 @@ multiverse. Fabric provides the 'fab' command which will be introduced later on.
|
|||
# apt-get install fabric
|
||||
```
|
||||
|
||||
These two tools (git & fabric) are only needed on mashines where system operators work.
|
||||
These two tools (git & fabric) are only needed on machines where system operators work.
|
||||
|
||||
Next clone git@github.com:SUNET/multiverse.git - this will form the basis of your cosmos+puppet
|
||||
repository:
|
||||
|
|
@ -226,7 +226,7 @@ Finally create a branch for the 'multiverse' upstream so you can merge changes t
|
|||
```
|
||||
|
||||
Note that you can maintain your repo on just about any git hosting platform, including
|
||||
github, gitorius or your own local setup as long as it supports read-only access to your
|
||||
github, gitorious or your own local setup as long as it supports read-only access to your
|
||||
repository. It is important that the remotes called 'origin' and 'ro' refer to your
|
||||
repository and not to anything else (like a private version of multiverse).
|
||||
|
||||
|
|
@ -266,7 +266,7 @@ ssh as root. This requires that root key trust be established in advance. The ad
|
|||
command creates and commits the necessary changes to the repository to add a host named
|
||||
$fqdn. Only fully qualified hostnames should ever be used in cosmos+puppet.
|
||||
|
||||
The boostrap process will create a cron-job on $fqdn that runs
|
||||
The bootstrap process will create a cron-job on $fqdn that runs
|
||||
|
||||
```
|
||||
# cosmos update && cosmos apply
|
||||
|
|
@ -295,7 +295,7 @@ with names on the form ns\<number\>.example.com to the 'nameserver' class.
|
|||
nameserver:
|
||||
```
|
||||
|
||||
Note that the value is a hash with an empty value ('namserver:') and not just a string
|
||||
Note that the value is a hash with an empty value ('nameserver:') and not just a string
|
||||
value.
|
||||
|
||||
Since regular expressions can also match on whole strings so the following is also
|
||||
|
|
@ -323,11 +323,11 @@ Given the above example the following command would reload all nameservers:
|
|||
Creating a change-request
|
||||
-------------------------
|
||||
|
||||
After performing whatever changes you want to the reqpository, commit the changes as usual
|
||||
After performing whatever changes you want to the repository, commit the changes as usual
|
||||
and then sign an appropriately formatted tag. This last operation is wrapped in the 'bump-tag' command:
|
||||
|
||||
```
|
||||
# git commit -m "some changes" global/overlay/somethig or/other/files
|
||||
# git commit -m "some changes" global/overlay/something or/other/files
|
||||
# ./bump-tag
|
||||
```
|
||||
|
||||
|
|
@ -337,30 +337,30 @@ gpg commands to create, sign and push the correct tag.
|
|||
Puppet modules
|
||||
--------------
|
||||
|
||||
Puppet modules can be maintained using a designated cosmos pre-task that reads a file
|
||||
global/overlay/etc/puppet/cosmos-modules.conf. This file is a simple text-format file
|
||||
with 3 columns:
|
||||
Puppet modules can be maintained using a designated cosmos pre-task that reads the file
|
||||
/etc/puppet/cosmos-modules.conf. This file is a simple text-format file
|
||||
with either three (for puppetlabs modules) or four columns:
|
||||
|
||||
```
|
||||
#
|
||||
# name source (puppetlabs fq name or git url) upgrade (yes/no)
|
||||
# name source (puppetlabs fq name or git url) upgrade (yes/no) tag_pattern
|
||||
#
|
||||
concat puppetlabs/concat no
|
||||
stdlib puppetlabs/stdlib no
|
||||
cosmos https://github.com/SUNET/puppet-cosmos.git yes
|
||||
ufw https://github.com/SUNET/puppet-module-ufw.git yes
|
||||
apt puppetlabs/apt no
|
||||
concat puppetlabs/concat no
|
||||
cosmos https://github.com/SUNET/puppet-cosmos.git yes sunet-2*
|
||||
#golang elithrar/golang yes
|
||||
python https://github.com/SUNET/puppet-python.git yes sunet-2*
|
||||
stdlib puppetlabs/stdlib no
|
||||
ufw https://github.com/SUNET/puppet-module-ufw.git yes sunet-2*
|
||||
vcsrepo puppetlabs/vcsrepo no
|
||||
xinetd puppetlabs/xinetd no
|
||||
#golang elithrar/golang yes
|
||||
python https://github.com/SUNET/puppet-python.git yes
|
||||
hiera-gpg https://github.com/SUNET/hiera-gpg.git no
|
||||
```
|
||||
|
||||
This is an example file - the first field is the name of the module, the second is
|
||||
the source: either a puppetlabs path or a git URL. The final field is 'yes' if the
|
||||
module should be automatically updated or 'no' if it should only be installed. As usual
|
||||
lines beginning with '#' are silently ignored.
|
||||
the source: either a puppetlabs path or a git URL. The third field is 'yes' if the
|
||||
module should be automatically updated or 'no' if it should only be installed. The
|
||||
fourth field is a tag pattern to use (same style as the cosmos tag pattern).
|
||||
As usual lines beginning with '#' are silently ignored.
|
||||
|
||||
This file is processed in a cosmos pre-hook so the modules should be available for
|
||||
use in the puppet post-hook. By default the file contains several lines that are
|
||||
|
|
@ -370,6 +370,16 @@ In order to add a new module, the best way is to commit a change to this file an
|
|||
tag this change, allowing time for the module to get installed everywhere before
|
||||
adding a change that relies on this module.
|
||||
|
||||
As there might be a need to use different sets of modules (or different tag patterns)
|
||||
on different hosts in an ops-repo, the contents of this file can be controlled in
|
||||
different ways:
|
||||
|
||||
1. If the file is present in the model, it is used as such.
|
||||
2. If there is a script called /etc/puppet/setup_cosmos_modules, that script is executed.
|
||||
If the file /etc/puppet/cosmos-modules.conf does not exist after this script runs,
|
||||
proceed to step 3, otherwise use this dynamically generated list of modules.
|
||||
3. Use a (very small) default set of modules from the pre-hook global/post-tasks.d/010cosmos-modules.
|
||||
|
||||
HOWTO and Common Tasks
|
||||
======================
|
||||
|
||||
|
|
@ -388,7 +398,7 @@ Add the ascii-armoured key in a file in `global/overlay/etc/cosmos/keys` with a
|
|||
Removing an operator
|
||||
--------------------
|
||||
|
||||
Identitfy the public key file in `global/overlay/etc/cosmos/keys`
|
||||
Identify the public key file in `global/overlay/etc/cosmos/keys`
|
||||
|
||||
```
|
||||
# git rm global/overlay/etc/cosmos/keys/X.pub
|
||||
|
|
@ -465,4 +475,3 @@ On all hosts:
|
|||
```
|
||||
# fab -- reboot # danger Will Robinsson!
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -1,36 +0,0 @@
|
|||
#
|
||||
# name source (puppetlabs fq name or git url) upgrade (yes/no) tag-pattern
|
||||
#
|
||||
# NOTE that Git packages MUST be tagged with signatures by someone
|
||||
# in the Cosmos trust list. That is why all the URLs point to forked
|
||||
# versions in the SUNET github organization.
|
||||
#
|
||||
concat https://github.com/SUNET/puppetlabs-concat.git yes sunet-*
|
||||
stdlib https://github.com/SUNET/puppetlabs-stdlib.git yes sunet-*
|
||||
cosmos https://github.com/SUNET/puppet-cosmos.git yes sunet-*
|
||||
ufw https://github.com/SUNET/puppet-module-ufw.git yes sunet_dev-*
|
||||
apt https://github.com/SUNET/puppetlabs-apt.git yes sunet_dev-*
|
||||
vcsrepo https://github.com/SUNET/puppetlabs-vcsrepo.git yes sunet-*
|
||||
xinetd https://github.com/SUNET/puppetlabs-xinetd.git yes sunet-*
|
||||
hiera-gpg https://github.com/SUNET/hiera-gpg.git yes sunet-*
|
||||
#
|
||||
# Alternate sources you might or might not want to use:
|
||||
#concat puppetlabs/concat no
|
||||
#stdlib puppetlabs/stdlib no
|
||||
#ufw attachmentgenie/ufw no
|
||||
#apt puppetlabs/apt no
|
||||
#vcsrepo puppetlabs/vcsrepo no
|
||||
#xinetd puppetlabs/xinetd no
|
||||
#cosmos https://github.com/SUNET/puppet-cosmos.git yes
|
||||
#python https://github.com/SUNET/puppet-python.git yes sunet-*
|
||||
#erlang https://github.com/SUNET/garethr-erlang.git yes sunet-*
|
||||
#rabbitmq https://github.com/SUNET/puppetlabs-rabbitmq.git yes sunet_dev-*
|
||||
#pound https://github.com/SUNET/puppet-pound.git yes sunet_dev-*
|
||||
#augeas https://github.com/SUNET/puppet-augeas.git yes sunet-*
|
||||
#bastion https://github.com/SUNET/puppet-bastion.git yes sunet-*
|
||||
#postgresql https://github.com/SUNET/puppetlabs-postgresql.git yes sunet_dev-*
|
||||
#munin https://github.com/SUNET/ssm-munin.git yes sunet-*
|
||||
#nagios https://github.com/SUNET/puppet-nagios.git yes sunet-*
|
||||
#staging https://github.com/SUNET/puppet-staging.git yes sunet-*
|
||||
#apparmor https://github.com/SUNET/puppet-apparmor.git yes sunet-*
|
||||
#docker https://github.com/SUNET/garethr-docker.git yes sunet_dev-*
|
||||
|
|
@ -11,9 +11,9 @@ Exec {
|
|||
#include cosmos::ntp
|
||||
#include cosmos::rngtools
|
||||
#include cosmos::preseed
|
||||
include ufw
|
||||
include apt
|
||||
include cosmos
|
||||
#include ufw
|
||||
#include apt
|
||||
#include cosmos
|
||||
|
||||
# you need a default node
|
||||
|
||||
|
|
@ -49,4 +49,3 @@ node default {
|
|||
# proto => "tcp"
|
||||
# }
|
||||
#}
|
||||
|
||||
|
|
|
|||
39
global/post-tasks.d/010cosmos-modules
Executable file
39
global/post-tasks.d/010cosmos-modules
Executable file
|
|
@ -0,0 +1,39 @@
|
|||
#!/bin/sh
|
||||
#
|
||||
# Dynamically configure /etc/puppet/cosmos-modules.conf
|
||||
#
|
||||
# The content of that file is chosen according to:
|
||||
#
|
||||
# 1. If the file is actually present in the model, use that.
|
||||
# 2. If there is a script called /etc/puppet/setup_cosmos_modules, run that.
|
||||
# 3. If the file still doesn't exist, create it with the defaults in this script.
|
||||
#
|
||||
|
||||
set -e
|
||||
|
||||
if [ -f "${COSMOS_MODEL}/overlay/etc/puppet/cosmos-modules.conf" ]; then
|
||||
test "$COSMOS_VERBOSE" = "y" && \
|
||||
echo "$0: /etc/puppet/cosmos-modules.conf is present in the model, exiting"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ -x /etc/puppet/setup_cosmos_modules ]; then
|
||||
test "$COSMOS_VERBOSE" = "y" && \
|
||||
echo "$0: Updating /etc/puppet/cosmos-modules.conf with /etc/puppet/setup_cosmos_modules"
|
||||
/etc/puppet/setup_cosmos_modules
|
||||
|
||||
test -f /etc/puppet/cosmos-modules.conf && exit 0
|
||||
fi
|
||||
|
||||
test "$COSMOS_VERBOSE" = "y" && \
|
||||
echo "$0: Creating/updating /etc/puppet/cosmos-modules.conf with defaults from this script"
|
||||
|
||||
cat > /etc/puppet/cosmos-modules.conf << EOF
|
||||
# File created/updated by $0
|
||||
#
|
||||
concat puppetlabs/concat yes
|
||||
stdlib puppetlabs/stdlib yes
|
||||
#ufw attachmentgenie/ufw yes
|
||||
#apt puppetlabs/apt yes
|
||||
#cosmos https://github.com/SUNET/puppet-cosmos.git yes
|
||||
EOF
|
||||
Loading…
Reference in a new issue