multiverse docs
This commit is contained in:
parent
3d1b26e762
commit
70c7828b0b
10
docs/Makefile
Normal file
10
docs/Makefile
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
|
||||
.PHONY: all
|
||||
DOCS := $(wildcard *.mkd)
|
||||
all: $(DOCS:.mkd=.pdf)
|
||||
|
||||
%.pdf: %.mkd
|
||||
pandoc -o $@ $<
|
||||
|
||||
clean:
|
||||
rm -f *.html *.pdf *~
|
||||
421
docs/cosmos-puppet-ops.mkd
Normal file
421
docs/cosmos-puppet-ops.mkd
Normal file
|
|
@ -0,0 +1,421 @@
|
|||
% System Operations using Cosmos & Puppet
|
||||
% Leif Johansson / SUNET / 2013 / v0.0.3
|
||||
|
||||
|
||||
Introduction
|
||||
============
|
||||
|
||||
This document describes how to setup and run systems and service operations for a small to midsized
|
||||
systems collection while maintaining scalability, security and auditability for changes.
|
||||
The process described below is based on opensource components and assumes a Linux-based hosting
|
||||
infrastructure. These limitations could easily be removed though. This document describes the
|
||||
multiverse template for combining cosmos and puppet.
|
||||
|
||||
|
||||
Design Requirements
|
||||
===================
|
||||
|
||||
The cosmos system has been used to operate security-critical infrastructure for a few years before
|
||||
it was combined with puppet into the multiverse template.
|
||||
|
||||
Several of the design requirements below are fulfilled by comos alone, while some (eg consistency)
|
||||
are easier to achieve using puppet than with cosmos alone.
|
||||
|
||||
Consistency
|
||||
-----------
|
||||
|
||||
Changes should be applied atomically across multiple system components on multiple
|
||||
physical and logical hosts (aka system state). The change mechanism should permit verification of state
|
||||
consistency and all modifications should be idempotents, i.e the same operation
|
||||
performend twice on the same system state should not in itself cause a problem.
|
||||
|
||||
Auditability
|
||||
------------
|
||||
|
||||
It must be possible to review changes in advance of applying them to system state. It
|
||||
must also be possible to trace changes that have already been applied to privileged
|
||||
system operators.
|
||||
|
||||
Authenticity
|
||||
------------
|
||||
|
||||
All changes must be authenticated by private keys in the personal possession of privileged
|
||||
system operators before applied to system state aswell as at any point in the future.
|
||||
|
||||
Simplicity
|
||||
----------
|
||||
|
||||
The system must be simple and must not rely on external services to be online to maintain
|
||||
state except when new state is being requested and applied. When new state is being requested
|
||||
external dependencies must be kept to a minimum.
|
||||
|
||||
Architecture
|
||||
============
|
||||
|
||||
The basic architecture of puppet is to use a VCS (git) to manage and distribute changes to a
|
||||
staging area on each managed host. At the staging area the changes are authenticated (using
|
||||
tag signatures) and if valid, distributed to the host using local rsync. Before and after
|
||||
hooks (using run-parts) are used to provide programmatic hooks.
|
||||
|
||||
Administrative Scope
|
||||
--------------------
|
||||
|
||||
The repository constitutes the administrative domain of a multiverse setup: each host is
|
||||
connected to (i.e runs cosmos off of) a single GIT repository and derives trust from signed
|
||||
tags on that repository. A host cannot belong to more than 1 administratve domain but each
|
||||
administrative domains can host multiple DNS domains - all hosts in a single repository
|
||||
doesn't need to be in the same zone.
|
||||
|
||||
The role of Puppet
|
||||
------------------
|
||||
|
||||
In the multiverse template, the cosmos system is used to authenticate and distribute changes
|
||||
and prepare the system state for running puppet. Puppet is used to apply idempotent changes
|
||||
to the system state using "puppet apply".
|
||||
|
||||
~~~~~ {.ditaa .no-separation}
|
||||
+------------+ +------+
|
||||
| cosmo repo |---->| host |-----+
|
||||
+------------+ +------+ |
|
||||
^ |
|
||||
| |
|
||||
(change) (manifests)
|
||||
| |
|
||||
+--------+ |
|
||||
| puppet |<---+
|
||||
+--------+
|
||||
~~~~~
|
||||
|
||||
Note that there is no puppet master in this setup so collective resources cannot be used
|
||||
in multiverse. Instead 'fabric' is used to provide a simple way to loop over subsets of
|
||||
the hosts in a managed domain.
|
||||
|
||||
Private data (eg system credentials, application passwords, or private keys) are encrypted
|
||||
to a master host PGP key before stored in the cosmos repo.
|
||||
|
||||
System state can be tied to classes used to classify systems into roles (eg "database server"
|
||||
or "webserver"). System classes can be assigned by regular expressions on the fqdn (eg all
|
||||
hosts named db-\* is assigned to the "database server" class) using a custom puppet ENC.
|
||||
|
||||
The system classes are also made available to 'fabric' in a custom fabfile. Fabric (or fab)
|
||||
is a simple frontend to ssh that allows an operator to run commands on multiple remote
|
||||
hosts at once.
|
||||
|
||||
Trust
|
||||
-----
|
||||
|
||||
All data in the system is maintained in a cosmos GIT repository. A change is
|
||||
requested by signing a tag in the repository with a system-wide well-known name-prefix.
|
||||
The tag name typically includes the date and a counter to make it unique.
|
||||
|
||||
The signature on the tag is authenticated against a set of trusted keys maintained in the
|
||||
repository itself - so that one trusted system operator must be present to authenticate addition or
|
||||
removal of another trusted system operator. This authentication of tags is done in addition
|
||||
to authenticating access to the GIT repository when the changes are pushed. Trust is typically
|
||||
bootstrapped when a repository is first established. This model also serves to provide auditability
|
||||
of all changes for as long as repository history is retained.
|
||||
|
||||
Access to hosts is done through ssh with ssh-key access. The ssh keys are typically maintained
|
||||
using either puppet or cosmos natively.
|
||||
|
||||
Consistency
|
||||
-----------
|
||||
|
||||
As a master-less architecture, multiverse relies on _eventual consistency_: changes will eventually
|
||||
be applied to all hosts. In such a model it becomes very imporant that changes are idempotent, so
|
||||
that applying a change multiple times (in an effort to get dependent changes through) won't cause
|
||||
an issue. Using native cosmos, such changes are achived using timestamp-files that control entry
|
||||
into code-blocks:
|
||||
|
||||
```
|
||||
stamp="${COSMOS\_BASE}/stamps/foo-v04.stamp"
|
||||
if ! test -f $stamp; then
|
||||
# do something here
|
||||
touch $stamp
|
||||
fi
|
||||
```
|
||||
|
||||
This pattern is mostly replaced in multiverse by using puppet manifests and modules that
|
||||
are inherently indempotent but it can nevertheless be a useful addition to the toolchain.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Implementation is based on two major components: cosmos and puppet. The cosmos system was
|
||||
created by Simon Josefsson and Fredrik Tulin as a simple and secure way to distribute files
|
||||
and run pre- and post-processors (using run-parts). This allows for a simple, yet complete
|
||||
mechanism for updating system state.
|
||||
|
||||
The second component is puppet which is run in masterless (aka puppet apply) mode on files
|
||||
distributed and authenticated using cosmos. Puppet is a widely deployed way to describe
|
||||
system state using a set of idempotent operations. In theory, anything that can de done
|
||||
using puppet can be done using cosmos post-processors but puppet allows for greater
|
||||
abstraction which greatly increases readability.
|
||||
|
||||
The combination of puppet and cosmos is maintained on github in the 'leifj/multiverse'
|
||||
project.
|
||||
|
||||
|
||||
Operations
|
||||
==========
|
||||
|
||||
Setting up a new administrative domain
|
||||
--------------------------------------
|
||||
|
||||
The simplest way is to clone the multiverse repository. First install 'git'. On ubuntu/debian
|
||||
this is in the 'git-core' package:
|
||||
|
||||
```
|
||||
# apt-get install git-core
|
||||
```
|
||||
|
||||
Also install 'fabric' - a very useful too for multiple-host-ssh that is integrated into
|
||||
multiverse. Fabric provides the 'fab' command which will be introduced later on.
|
||||
|
||||
```
|
||||
# apt-get install fabric
|
||||
```
|
||||
|
||||
These two tools (git & fabric) are only needed on mashines where system operators work.
|
||||
|
||||
Next clone git://github.com/leifj/multiverse.git - this will form the basis of your cosmos+puppet
|
||||
repository:
|
||||
|
||||
```
|
||||
# git clone git://github.com/leifj/multiverse.git myproj-cosmos
|
||||
# cd myproj-cosmos
|
||||
```
|
||||
|
||||
Next rename the upstream from github - you will want to keep this around to get new
|
||||
features as the multiverse codebase evolves.
|
||||
|
||||
```
|
||||
# git remote rename origin multiverse
|
||||
```
|
||||
|
||||
Now add a new remote pointing to the git repo where you are going to be pushing
|
||||
changes for your administrative domain. Also add a read-only version of this remote
|
||||
as 'ro'. The read-only remote is used by multiverse scripts during host bootstrap.
|
||||
|
||||
```
|
||||
# git remote add origin git@yourhost:myproj-cosmos.git
|
||||
# git remote add ro git://yourhost/myproj-cosmos.git
|
||||
```
|
||||
|
||||
Note that you can maintain your repo on just about any git hosting platform, including
|
||||
github, gitorius or your own local setup as long as it supports read-only "git://" access
|
||||
to your repository. It is important that the remotes called 'origin' and 'ro' refer to
|
||||
your repository and not to anything else (like a private version of multiverse).
|
||||
|
||||
Now add at least one key to 'global/overlay/etc/cosmos/keys/' in a file with a .pub extension
|
||||
(eg 'operator.pub') - the name of the file doesn't matter other than the extension.
|
||||
|
||||
```
|
||||
# cp mykey.pub global/overlay/etc/cosmos/keys/
|
||||
# git add global/overlay/etc/cosmos/keys/mykey.pub
|
||||
# git commit -m "initial trust" global/overlay/etc/cosmos/keys/mykey.pub
|
||||
```
|
||||
|
||||
At this point you should create and sign your first tag:
|
||||
|
||||
```
|
||||
# ./bump-tag
|
||||
```
|
||||
|
||||
Make sure that you are using the key whose public key you just added to the repository! You
|
||||
can now start adding hosts.
|
||||
|
||||
Adding a host
|
||||
-------------
|
||||
|
||||
Bootstrapping a host is done using the 'addhost' command:
|
||||
|
||||
```
|
||||
# ./addhost [-b] $fqdn
|
||||
```
|
||||
|
||||
The -b flag causes addhost to attempt to bootstrap cosmos on the remote host using
|
||||
ssh as root. This requires that root key trust be established in advance. The addhost
|
||||
command creates and commits the necessary changes to the repository to add a host named
|
||||
$fqdn. Only fully qualified hostnames should ever be used in cosmos+puppet.
|
||||
|
||||
The boostrap process will create a cron-job on $fqdn that runs
|
||||
|
||||
```
|
||||
# cosmos update && cosmos apply
|
||||
```
|
||||
|
||||
every 15 minutes. This should be a good starting point for your domain. Now you may
|
||||
want to add some 'naming rules'.
|
||||
|
||||
Defining naming rules
|
||||
---------------------
|
||||
|
||||
A naming rule is a mapping from a name to a set of puppet classes. These are defined in
|
||||
the file 'global/overlay/etc/puppet/cosmos-rules.yaml' (linked to the toplevel directory
|
||||
in multiverse). This is a YAML format file whose keys are regular expressions and whose
|
||||
values are lists of puppet class definitions. Here is an example that assigns all hosts
|
||||
with names on the form ns\<number\>.example.com to the 'nameserver' class.
|
||||
|
||||
```
|
||||
'ns[0-9]?.example.com$':
|
||||
nameserver:
|
||||
```
|
||||
|
||||
Note that the value is a hash with an empty value ('namserver:') and not just a string
|
||||
value.
|
||||
|
||||
Since regular expressions can also match on whole strings so the following is also
|
||||
valid:
|
||||
|
||||
```
|
||||
smtp.example.com:
|
||||
mailserver:
|
||||
relay: smtp.upstream.example.com
|
||||
```
|
||||
|
||||
In this example the mailserver puppet class is given the relay argument (cf puppet
|
||||
documentation).
|
||||
|
||||
Fabric integration
|
||||
------------------
|
||||
|
||||
Given the above example the following command would reload all nameservers:
|
||||
|
||||
```
|
||||
# fab --roles=nameservers -- rndc reload
|
||||
```
|
||||
|
||||
|
||||
Creating a change-request
|
||||
-------------------------
|
||||
|
||||
After performing whatever changes you want to the reqpository, commit the changes as usual
|
||||
and then sign an appropriately formatted tag. This last operation is wrapped in the 'bump-tag' command:
|
||||
|
||||
```
|
||||
# git commit -m "some changes" global/overlay/somethig or/other/files
|
||||
# ./bump-tag
|
||||
```
|
||||
|
||||
The bump-tag command will ask for confirmation before signing and will rely on the git and
|
||||
gpg commands to create, sign and push the correct tag.
|
||||
|
||||
Puppet modules
|
||||
--------------
|
||||
|
||||
Puppet modules can be maintained using a designated cosmos pre-task that reads a file
|
||||
global/overlay/etc/puppet/cosmos-modules.conf. This file is a simple text-format file
|
||||
with 3 columns:
|
||||
|
||||
```
|
||||
#
|
||||
# name source (puppetlabs fq name or git url) upgrade (yes/no)
|
||||
#
|
||||
concat puppetlabs/concat no
|
||||
stdlib puppetlabs/stdlib no
|
||||
cosmos git://github.com/leifj/puppet-cosmos.git yes
|
||||
ufw git://github.com/fredrikt/puppet-module-ufw.git yes
|
||||
apt puppetlabs/apt no
|
||||
vcsrepo puppetlabs/vcsrepo no
|
||||
xinetd puppetlabs/xinetd no
|
||||
#golang elithrar/golang yes
|
||||
python git://github.com/fredrikt/puppet-python.git yes
|
||||
hiera-gpg git://github.com/fredrikt/hiera-gpg.git no
|
||||
```
|
||||
|
||||
This is an example file - the first field is the name of the module, the second is
|
||||
the source: either a puppetlabs path or a git URL. The final field is 'yes' if the
|
||||
module should be automatically updated or 'no' if it should only be installed. As usual
|
||||
lines beginning with '#' are silently ignored.
|
||||
|
||||
This file is processed in a cosmos pre-hook so the modules should be available for
|
||||
use in the puppet post-hook. By default the file contains several lines that are
|
||||
commented out so review this file as you start a new multiverse setup.
|
||||
|
||||
In order to add a new module, the best way is to commit a change to this file and
|
||||
tag this change, allowing time for the module to get installed everywhere before
|
||||
adding a change that relies on this module.
|
||||
|
||||
HOWTO and Common Tasks
|
||||
======================
|
||||
|
||||
Adding a new operator
|
||||
---------------------
|
||||
|
||||
Add the ascii-armoured key in a file in `global/overlay/etc/cosmos/keys` with a `.pub` extension
|
||||
|
||||
```
|
||||
# git add global/overlay/etc/cosmos/keys/thenewoperator.pub
|
||||
# git commit -m "the new operator" \
|
||||
global/overlay/etc/cosmos/keys/thenewoperator.pub
|
||||
# ./bump-tag
|
||||
```
|
||||
|
||||
Removing an operator
|
||||
--------------------
|
||||
|
||||
Identitfy the public key file in `global/overlay/etc/cosmos/keys`
|
||||
|
||||
```
|
||||
# git rm global/overlay/etc/cosmos/keys/X.pub
|
||||
# git commit -m "remove operator X" \
|
||||
global/overlay/etc/cosmos/keys/X.pub
|
||||
# ./bump-tag
|
||||
```
|
||||
|
||||
Changing administrative domain for a host
|
||||
-----------------------------------------
|
||||
|
||||
In the `$new` repository:
|
||||
|
||||
```
|
||||
# ./addhost -b $hostname
|
||||
# fab -H $hostname chrepo repository:git://other/repo.git
|
||||
```
|
||||
|
||||
In the `$old` repository:
|
||||
|
||||
```
|
||||
# rsync -avz $hostname/ $new/$hostname/
|
||||
```
|
||||
|
||||
In the `$new` repository:
|
||||
|
||||
```
|
||||
# git add $hostname/
|
||||
# git commit -m "transfer from $old" $hostname
|
||||
# ./bump-tag
|
||||
```
|
||||
|
||||
In the `$old` repository:
|
||||
|
||||
```
|
||||
# git rm -rf $hostname
|
||||
# git commit -m "remove $hostname"
|
||||
# ./bump-tag
|
||||
```
|
||||
|
||||
Reviewing all changes
|
||||
---------------------
|
||||
|
||||
Running a command on multiple hosts
|
||||
-----------------------------------
|
||||
|
||||
On a single host:
|
||||
|
||||
```
|
||||
# fab -H $hostname -- command -a one -b another -c
|
||||
```
|
||||
|
||||
On multiple hosts based on category:
|
||||
|
||||
```
|
||||
# fab --roles=webserver -- ls /tmp
|
||||
```
|
||||
|
||||
On all hosts:
|
||||
|
||||
```
|
||||
# fab -- reboot # danger Will Robinsson!
|
||||
```
|
||||
Loading…
Reference in a new issue