While we already support setting a healthcheck timeout it probably
makes sense to be able to control how long we wait for a
fleetlock_lock() or fleetlock_unlock() call. This becomes important if
only running cosmos once a night or something like that. In that case we
you probably want to give a physical machine more than than 1 minute to
complete a reboot etc.
This can now be controlled by setting fleetlock_lock_timeout and
fleetlock_unlock_timeout in /etc/run-cosmos-fleetlock-conf. Keep in mind
that while it can make sense to increase the time for taking a lock,
releasing a lock should always be fast (either you have it and release
it, or you dont have it and it is a no-op) so setting a long unlock
timeout should probably never be done.
Since we also potentially wait the unlock timeout at boot (if the
fleetlock server is broken etc) that is another reason to keep it
short. The default 1m is probably OK for most uses.
In order to handle upgrades of the fleetlock server when running only
one server we need to handle connection errors like connection refused
or timed out errors gracefully.
Because there are several different ways the connection can fail and it
is hard to keep track of them all, just catch everything. We then also
need special handling of our own timeout execption so we are not
accidentally stuck retrying forever.
Also fix so we actually use the request_timeout arg for individual HTTP
requests instead of the global timeout.
While here run isort to keep imports tidy.
This extends run-cosmos with a new argument that calls the unlock
function already included in the script as well as using the already
existing lock() function to make sure there is no race between the
bootup process and cron starting a normal run-cosmos process at the same
time.
The oexit() function is added to support exiting with a OK exit value
the same way eexit() is used to signal something is wrong.
This change also adds the systemd unit file that runs run-cosmos with the
new fleetlock-unlock argument at boot if fleetlock is configured.
While here fix indentation that was mixed between 3 and 4 spaces: it is
now 4 spaces everywhere.
Makes run-cosmos request a fleetlock lock before running cosmos "update"
and "apply" steps. This is helpful for making sure only one (or several)
machine out of some set of machines runs cosmos changes at a time. This
way if cosmos (or puppet) decides that a service needs to be restarted
this will only happen on a subset of machines at a time. When the cosmos
"apply" is done a fleetlock unlock request will be performed so the
other machines can progress.
The unlock code in run-cosmos will also run the new tool
sunet-machine-healthy to decide things are good before unlocking. This
way if a restarted service breaks this will stop the unlock attempt
and in turn make it so the others should not break their service as
well, giving an operator time to figure out what is wrong.
The db-file, essentially providing reverse lookup of classes to host
names, is only used by some Nagios configuration instances and causes
continuing operational headaches in those ops-repos.
It should be kept/refactored to only apply to the monitoring hosts in
the cases where it is used, but we don't want any new ops-repos to use
it hence it should be removed from upstream multiverse.
Before this change there was a need to keep addhost and
bootstrap-cosmos.sh in sync regarding what version of the cosmos deb to
scp over and later run.
Now we find the latest version as decided by `sort -V` in both addhost
and bootstrap-cosmos.sh.
Solution discussed with @fredrikt.
Good idea to fail when unexpected things go wrong. Additional fixes
added to the script to not stop where we can expect a non-zero return
code.
Requested by @fredrikt who also reviewed the patch before going in,
thanks!
Micke Nordin
Patrik Lundin
Patrik Holmqvist
Johan Wassberg
Fredrik Thulin
Leif Johansson