kris/puppet-rsnapshot
1
0
Fork 0
mirror of https://github.com/krislamo/puppet-rsnapshot synced 2024-11-14 01:00:34 +00:00
Code Issues Releases Wiki Activity

Merge branch 'feature/parameterize_backup_scripts' into develop

Browse Source 
- parameterize backup scripts
- add custom commands to misc script
- add parameters to mysql and psql scripts
- update documentation
This commit is contained in:
Norbert Varzariu 2016-01-22 15:59:02 +01:00
commit daba8a562a
7 changed files with 252 additions and 59 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
.bundle/config
View File

@ -1,3 +0,0 @@
---
BUNDLE_PATH: vendor/bundle
BUNDLE_DISABLE_SHARED_GEMS: '1'

184
README.md
View File

@ -133,10 +133,14 @@ rsnapshot::hosts:
backup: backup:
'/var': './var' '/var': './var'
cron: cron:
'daily': mailto: 'bar1@example.com'
'minute': '0-10' daily:
'hour': '1..5' minute: '0-10'
hour: '1..5'
db1:
backup_scripts:
mysql:
misc:
``` ```
@ -243,10 +247,14 @@ The range notation is '$start..$end', so to pick a random hour from 8 pm to 2 am
For the range feature to work, hours >0 and <10 must not have a preceding zero. For the range feature to work, hours >0 and <10 must not have a preceding zero.
Wrong: `00.09` Wrong: `00.09`
Correct: `0..9` Correct: `0..9`
Also, you can set a mailto for each host, or globally now. The settings will be merged bottom to top, so if you override a setting in a hosts cron, it will have precedence over the global setting,
which in turn has precedence over the default.
Example: Example:
```puppet ```puppet
$cron = { $cron = {
mailto => 'admin@example.com',
hourly => { hourly => {
minute => '0..59', minute => '0..59',
hour => [ '20..23','0..2' ], hour => [ '20..23','0..2' ],
@ -259,6 +267,7 @@ Or in hiera:
```yaml ```yaml
rsnapshot::cron: rsnapshot::cron:
mailto: 'admin@example.com'
daily: daily:
minute: '20' minute: '20'
weekly: weekly:
@ -270,16 +279,32 @@ rsnapshot::cron:
```yaml ```yaml
rsnapshot::hosts: rsnapshot::hosts:
webserver: webserver:
cron:
mailto: 'support@example.com'
daily: daily:
hour: [ '20..23','0..2' ] hour: [ '20..23','0..2' ]
weekly: weekly:
hour: [ '20..23','0..2' ] hour: [ '20..23','0..2' ]
webhost:
customervm.provider.com:
backup_user: 'customer'
``` ```
`webhost`: Mails will go to `admin@example.com` (from the global override).
`webserver`: Mails will go to `support@example.com`.
`customervm.provider.com`: The backup (and thus ssh) user will be `customer@customervm.provider.com`
Hash is of the form: Hash is of the form:
```puppet ```puppet
$cron =>{ $cron =>{
mailto => param,
daily => { daily => {
minute => param, minute => param,
hour => param, hour => param,
@ -297,6 +322,7 @@ Default is:
```puppet ```puppet
$cron = { $cron = {
mailto => 'admin@example.com',
hourly => { hourly => {
minute => '0..59', # random from 0 to 59 minute => '0..59', # random from 0 to 59
hour => '*', # you could also do: ['21..23','0..4','5'], hour => '*', # you could also do: ['21..23','0..4','5'],
@ -366,20 +392,65 @@ Default is:
#### `$backup_scripts` #### `$backup_scripts`
Additional scripts to create, possible values are: mysql, psql, misc Additional scripts to create, possible values are: mysql, psql, misc
`mysql`: used for mysql backups
`psql`: used for postgresql backups
`misc`: custom commands to run on the node
You can set
`$dbbackup_user`: backup user
`$dbbackup_password`: password for the backup user
`$dumper`: path to the dump bin you wish to use
`$dump_flags`: flags for your dump bin
`$ignore_dbs`: databases to be ignored (the psql script ignores template and postgres databases by default)
`$commands`: array of commands to run on the host (this has no effect on psql and mysql scripts and is intended for your custom needs, see misc script section)
See below for defaults
NOTE: the psql and mysql scripts will SSH into your host and try and use $dumper.
Make sure you have those tools installed on your DB hosts.
Also, this module will try and use pbzip to compress your databases. You can install pbzip2 (and additional packages you might need) by passing an array to [$rsnapshot::package_name](#package_name)
Default is: Default is:
```puppet ```puppet
$backup_scripts = { $backup_scripts = {
mysql => { mysql => {
dbbackup_user => 'root', dbbackup_user => 'root',
dbbackup_password => 'myPassWord', dbbackup_password => '',
dumper => 'mysqldump',
dump_flags => '--single-transaction --quick --routines --ignore-table=mysql.event',
ignore_dbs => [ 'information_schema', 'performance_schema' ],
}, },
psql => { psql => {
dbbackup_user => 'postgres', dbbackup_user => 'postgres',
dbbackup_password => '', dbbackup_password => '',
dumper => 'pg_dump',
dump_flags => '-Fc',
ignore_dbs => [],
},
misc => {
commands => $::osfamily ? {
'RedHat' => [
'rpm -qa --qf="%{name}," > packages.txt',
],
'Debian' => [
'dpkg --get-selections > packages.txt',
],
default => [],
}, },
misc => {},
} }
}
``` ```
Configuration example: Configuration example:
@ -398,21 +469,31 @@ rsnapshot::hosts:
backup_scripts: backup_scripts:
mysql: mysql:
psql: psql:
dbbackup_user: 'backupuser' dumper: '/usr/local/bin/pg_dump'
dbbackup_password: 'password' dump_flags: '-Fc'
ignore_dbs: [ 'db1', 'tmp_db' ]
misc:
bazqux:de: bazqux:de:
backup_scripts: backup_scripts:
mysql: mysql:
dbbackup_user: 'myuser' dbbackup_user: 'myuser'
dbbackup_password: 'mypassword' dbbackup_password: 'mypassword'
misc:
commands:
- 'cat /etc/hostname > hostname.txt'
- 'date > date.txt'
``` ```
This creates This creates
- a mysql and a psql backup script for `foobar.com` using the credentials `dbbackup:hunter2` for mysql and `dbbackup:yeshorsebatterystaple` for psql - a mysql and a psql backup script for `foobar.com` using the credentials `dbbackup:hunter2` for mysql and `dbbackup:yeshorsebatterystaple` for psql
- the psql script will use `/usr/local/bin/pg_dump` as the dump program with flags `-Fc`
- it will ignore the postgres databases `db1` and `tmp_db` for postgres
- a mysql backup script for `bazqux.de` using the credentials `myuser:mypassword` - a mysql backup script for `bazqux.de` using the credentials `myuser:mypassword`
- a misc script for bazqux.de containing two commands to run on the node. the output will be redirected to hostname.txt and date.txt in the misc/ subfolder of the hosts backup directory (i.e. /snapshot_root/bazqux.de/daily.0/misc/hostname.txt)
The scripts look like this: The scripts look like this:
mysql:
##### `bazqux.de`
```bash ```bash
#!/bin/bash #!/bin/bash
@ -420,34 +501,103 @@ host=bazqux.de
user=myuser user=myuser
pass=mypassword pass=mypassword
dbs=( $(mysql -h "$host" -u "$user" -p"$pass" -e 'show databases' | sed '1d;/information_schema/d;/performance_schema/d') ) dbs=(
$(ssh -l root "$host" "mysql -u ${user} -p${pass} -e 'show databases' | sed '1d;/information_schema/d;/performance_schema/d'")
)
for db in "${dbs[@]}"; do for db in "${dbs[@]}"; do
mysqldump --host="$host" --user="$user" --password="$pass" --single-transaction --quick --routines --ignore-table=mysql.event "$db" > "$db".sql ssh -l root "$host" "mysqldump --user=${user} --password=${pass} --single-transaction --quick --routines --ignore-table=mysql.event ${db}" > "${db}.sql"
wait wait
pbzip2 -p3 "$db".sql pbzip2 "$db".sql
done done
``` ```
```bash
#!/bin/bash
ssh bazqux.de 'cat /etc/hostname > hostname.txt'
ssh bazqux.de 'date > date.txt'
```
##### `foobar.com`
psql: psql:
```bash ```bash
#!/bin/bash #!/bin/bash
host=foobar.com host=foobar.com
user=backupuser user=dbbackup
pass=password pass=yeshorsebatterystaple
PGPASSWORD="$pass" PGPASSWORD="$pass"
dbs=( $(psql -h "$host" -U "$user" -Atc "SELECT datname FROM pg_database WHERE NOT datistemplate AND datname <> 'postgres'") ) dbs=(
$(ssh -l root "$host" "psql -U ${user} -Atc \"SELECT datname FROM pg_database WHERE NOT datistemplate AND datname ~ 'postgres|db1|tmp_db'\"" )
)
for db in "${dbs[@]}"; do for db in "${dbs[@]}"; do
ssh -l root "$host" "pg_dump -U ${user} -Fc ${db}" > "$db".sql ssh -l root "$host" "pg_dump -U ${user} -Fc ${db}" > "$db".sql
wait wait
pbzip2 -p3 "$db".sql pbzip2 "$db".sql
done done
``` ```
mysql:
```bash
#!/bin/bash
host=foobar.com
user=dbbackup
pass=hunter2
dbs=(
$(ssh -l root "$host" "mysql -u ${user} -p${pass} -e 'show databases' | sed '1d;/information_schema/d;/performance_schema/d'")
)
for db in "${dbs[@]}"; do
ssh -l root "$host" "mysqldump --user=${user} --password=${pass} --single-transaction --quick --routines --ignore-table=mysql.event ${db}" > "${db}.sql"
wait
pbzip2 "$db".sql
done
```
misc (assuming foobar.com is a RedHat node):
```bash
#!/bin/bash
ssh foobar.com 'rpm -qa --qf "%{name}," > packages.txt'
```
##### another example with root user and empty password
mysql with root user:
```bash
#!/bin/bash
host=bazqux.de
user=root
password=
dbs=(
$(ssh -l root "$host" "mysql -e 'show databases' | sed '1d;/information_schema/d;/performance_schema/d'")
)
for db in "${dbs[@]}"; do
ssh -l root "$host" "mysqldump --single-transaction --quick --routines --ignore-table=mysql.event ${db}" > "${db}.sql"
wait
pbzip2 "$db".sql
done
```
### rsnapshot configuration variables ### rsnapshot configuration variables
Please read up on the following in the [rsnapshot manpage](http://linux.die.net/man/1/rsnapshot) Please read up on the following in the [rsnapshot manpage](http://linux.die.net/man/1/rsnapshot)
@ -596,4 +746,4 @@ Norbert Varzariu (loomsen)
## Contributors ## Contributors
Please see the [list of contributors.](https://github.com/loomsen/puppet-bloonix_agent/graphs/contributors) Please see the [list of contributors.](https://github.com/loomsen/puppet-bloonix_agent/graphs/contributors)
A big thank you to Hendrik Horeis <hendrik.horeis@gmail.com> for all his input and testing of this module.

30
manifests/config.pp
View File

@ -36,7 +36,7 @@ class rsnapshot::config (
$hosts_clean = assert_empty_hash($hosts) $hosts_clean = assert_empty_hash($hosts)
$hosts_clean.each |String $host, $hash | { $hosts_clean.each |String $host, $hash | {
$backup_user = pick($hash['backup_user'], $rsnapshot::params::config_backup_user) $backup_user = pick($hash['backup_user'], $rsnapshot::backup_user, $rsnapshot::params::config_backup_user, 'root')
$default_backup_dirs = pick($rsnapshot::default_backup, $rsnapshot::params::config_default_backup) $default_backup_dirs = pick($rsnapshot::default_backup, $rsnapshot::params::config_default_backup)
$backup_levels = pick($hash['backup_levels'], $rsnapshot::backup_levels, 'weekly') $backup_levels = pick($hash['backup_levels'], $rsnapshot::backup_levels, 'weekly')
$backup = $hash['backup'] $backup = $hash['backup']
@ -135,17 +135,18 @@ class rsnapshot::config (
content => template('rsnapshot/rsnapshot.erb'), content => template('rsnapshot/rsnapshot.erb'),
} }
if has_key($hash, backup_scripts) { if has_key($hash, backup_scripts) {
$hash[backup_scripts].each |$script, $scriptconf| {
$hash[backup_scripts].each |$script, $credentials| { $real_script = deep_merge($rsnapshot::params::backup_scripts[$script], $rsnapshot::backup_scripts[$script], $hash[backup_scripts][$script])
$dbbackup_user = $real_script[dbbackup_user]
if is_hash($credentials) { $dbbackup_password = $real_script[dbbackup_password]
$dbbackup_user = $credentials['dbbackup_user'] $dumper = $real_script[dumper]
$dbbackup_password = $credentials['dbbackup_password'] $dump_flags = $real_script[dump_flags]
} else { $ignore_dbs = $real_script[ignore_dbs]
$dbbackup_user = $rsnapshot::default_backup_scripts[$script]['dbbackup_user'] $compress = $real_script[compress]
$dbbackup_password = $rsnapshot::default_backup_scripts[$script]['dbbackup_password'] $commands = $real_script[commands]
}
concat::fragment { "${host}_${script}_backup": concat::fragment { "${host}_${script}_backup":
target => $config, target => $config,
@ -167,8 +168,14 @@ class rsnapshot::config (
# create cron files for each backup level # create cron files for each backup level
# merge possible cron definitions to one # merge possible cron definitions to one
$real_cron = deep_merge($rsnapshot::params::cron, $rsnapshot::cron, $hash[cron]) $real_cron = deep_merge($rsnapshot::params::cron, $rsnapshot::cron, $hash[cron])
concat::fragment { "mailto for $host":
content => "#This file is managed by puppet\nMAILTO=${real_cron[mailto]}\n\n",
target => $cronfile,
order => 1,
}
$backup_levels.each |String $level| { $backup_levels.each |String $level| {
$mailto = $real_cron[mailto]
$minute = rand_from_array($real_cron[$level][minute], "${host}.${level}.minute") $minute = rand_from_array($real_cron[$level][minute], "${host}.${level}.minute")
$hour = rand_from_array($real_cron[$level][hour], "${host}.${level}.hour") $hour = rand_from_array($real_cron[$level][hour], "${host}.${level}.hour")
$monthday = rand_from_array($real_cron[$level][monthday], "${host}.${level}.monthday") $monthday = rand_from_array($real_cron[$level][monthday], "${host}.${level}.monthday")
@ -178,6 +185,7 @@ class rsnapshot::config (
concat::fragment { "${host}.${level}": concat::fragment { "${host}.${level}":
target => $cronfile, target => $cronfile,
content => template('rsnapshot/cron.erb'), content => template('rsnapshot/cron.erb'),
order => 2,
} }
} }
} }

29
manifests/params.pp
View File

@ -7,7 +7,11 @@ class rsnapshot::params {
$config_backup_user = 'root' $config_backup_user = 'root'
$package_name = 'rsnapshot' $package_name = 'rsnapshot'
$package_ensure = 'present' $package_ensure = 'present'
$cron_service_name = 'crond' $cron_service_name = $::osfamily ? {
'RedHat' => 'crond',
'Debian' => 'cron',
default => '',
}
$cron_dir = '/etc/cron.d' $cron_dir = '/etc/cron.d'
$config_backup_levels = [ 'daily', 'weekly', 'monthly' ] $config_backup_levels = [ 'daily', 'weekly', 'monthly' ]
$config_backup_defaults = true $config_backup_defaults = true
@ -63,6 +67,7 @@ class rsnapshot::params {
} }
$config_backup_scripts = {} $config_backup_scripts = {}
$cron = { $cron = {
mailto => 'admin@example.com',
hourly => { hourly => {
minute => '0..59', minute => '0..59',
hour => '*', # you could also do: ['21..23','0..4','5'], hour => '*', # you could also do: ['21..23','0..4','5'],
@ -95,12 +100,30 @@ class rsnapshot::params {
$backup_scripts = { $backup_scripts = {
mysql => { mysql => {
dbbackup_user => 'root', dbbackup_user => 'root',
dbbackup_password => 'myFancyPassWord', dbbackup_password => '',
dumper => 'mysqldump',
dump_flags => '--single-transaction --quick --routines --ignore-table=mysql.event',
ignore_dbs => [ 'information_schema', 'performance_schema' ],
compress => 'pbzip2',
}, },
psql => { psql => {
dbbackup_user => 'postgres', dbbackup_user => 'postgres',
dbbackup_password => '', dbbackup_password => '',
dumper => 'pg_dump',
dump_flags => '-Fc',
ignore_dbs => [],
compress => 'pbzip2',
},
misc => {
commands => $::osfamily ? {
'RedHat' => [
'rpm -qa --qf="%{name}," > packages.txt',
],
'Debian' => [
'dpkg --get-selections > packages.txt',
],
default => [],
}, },
misc => {}, }
} }
} }

13
templates/misc.sh.erb
View File

@ -1,10 +1,7 @@
#!/bin/bash #!/bin/bash
# add custom stuff here #This file is managed by puppet
# FIXME: remember to concat for custom stuff after the last line of the puppet generated script #
<% if @osfamily == 'RedHat' -%> <%@commands.each do |command| -%>
ssh <%=@host-%> rpm -qa --qf='%{name},' > packages.txt ssh -l <%=@backup_user-%> <%=@host-%> '<%=command-%>'
<%end-%> <%end-%>
#
#
#
# ++++++++++++++++++++++++++++++++++++++++++++

22
templates/mysql.sh.erb
View File

@ -1,4 +1,5 @@
#!/bin/bash #!/bin/bash
# This file is managed by puppet
# written by Norbert Varzariu <loomsen@gmail.com> # written by Norbert Varzariu <loomsen@gmail.com>
# This Script is triggered by rsnapshot and dumps the specified dbs on the configured host. # This Script is triggered by rsnapshot and dumps the specified dbs on the configured host.
# #
@ -6,11 +7,24 @@ host=<%=@host%>
user=<%=@dbbackup_user%> user=<%=@dbbackup_user%>
pass=<%=@dbbackup_password%> pass=<%=@dbbackup_password%>
dbs=( $(mysql -h "$host" -u "$user" -p"$pass" -e 'show databases' | sed '1d;/information_schema/d;/performance_schema/d') ) <% if (@dbbackup_user == 'root' && @dbbackup_password == '') -%>
dbs=(
$(ssh -l <%=@backup_user-%> "$host" "mysql -e 'show databases' | sed '1d;<%@ignore_dbs.to_a.each do |db|-%>/<%=db-%>/d;<%end-%>'" )
)
<%else-%>
dbs=(
$(ssh -l <%=@backup_user-%> "$host" "mysql -u <%=@dbbackup_user-%> -p'<%=@dbbackup_password-%>' -e 'show databases' | sed '1d;<%@ignore_dbs.each do |db|-%>/<%=db-%>/d;<%end-%>'" )
)
<%end-%>
for db in "${dbs[@]}"; do for db in "${dbs[@]}"; do
mysqldump --host="$host" --user="$user" --password="$pass" --single-transaction --quick --routines --ignore-table=mysql.event "$db" > "$db".sql <% if (@dbbackup_user == 'root' && @dbbackup_password = '' )-%>
ssh -l <%=@backup_user-%> "$host" "<%=@dumper-%> <%=@dump_flags-%> ${db}" > "$db".sql
<%else-%>
ssh -l <%=@backup_user-%> "$host" "<%=@dumper-%> --user=<%=@dbbackup_user-%> --password='<%=@dbbackup_password-%>' <%=@dump_flags-%> ${db}" > "$db".sql
<%end-%>
wait wait
pbzip2 -p3 "$db".sql <% if @compress != '' -%>
<%=@compress-%> "$db".sql
<%end-%>
done done

12
templates/psql.sh.erb
View File

@ -1,4 +1,5 @@
#!/bin/bash #!/bin/bash
# This file is managed by puppet
# written by Norbert Varzariu <loomsen@gmail.com> # written by Norbert Varzariu <loomsen@gmail.com>
# This Script is triggered by rsnapshot and dumps the specified dbs on the configured host. # This Script is triggered by rsnapshot and dumps the specified dbs on the configured host.
# #
@ -7,11 +8,14 @@ user=<%=@dbbackup_user%>
pass=<%=@dbbackup_password%> pass=<%=@dbbackup_password%>
PGPASSWORD="$pass" PGPASSWORD="$pass"
dbs=( $(psql -h "$host" -U "$user" -Atc "SELECT datname FROM pg_database WHERE NOT datistemplate AND datname <> 'postgres'") ) dbs=(
$(ssh -l <%=@backup_user-%> "$host" "psql -U ${user} -Atc \"SELECT datname FROM pg_database WHERE NOT datistemplate AND NOT datname ~ '<%@ignore_dbs.each do |db|-%><%if db == @ignore_dbs.last-%><%=db-%><%else-%><%=db-%>|<%end-%><%end-%>'\"")
)
for db in "${dbs[@]}"; do for db in "${dbs[@]}"; do
ssh -l root "$host" "pg_dump -U ${user} -Fc ${db}" > "$db".sql ssh -l <%=@backup_user-%> "$host" "<%=@dumper-%> -U ${user} <%=@dump_flags-%> ${db}" > "$db".sql
wait wait
pbzip2 -p3 "$db".sql <% if @compress != '' -%>
<%=@compress-%> "$db".sql
<%end-%>
done done