Vault deploy fix is to put /usr/local/bin in PATH

- Added comprehensive documentation for all Puppet classes
- Applied bash linting and made minor improvements to check_cert.sh
- Updated domain issue cron to source the .bashrc file
- Implemented small changes to address Puppet linting issues

README.md

@@ -18,7 +18,7 @@ This module uses [acme.sh](https://github.com/Neilpang/acme.sh) to request
 letsencrypt certificates using the DNS-01 challenge.  Once valid certificates
 are recieved, they are stored in [Hashicorp
 vault](https://www.vaultproject.io/) where they can be retrieved by any
-appropriate machine.  
+appropriate machine.
 
 This module consists of a common class, a request class, and a deploy class.
 The request class is intended to be enabled on a single machine that will
@@ -30,17 +30,15 @@ enabled on any machine that requires the requested certificates.
 ### What acme_vault affects
 
 This module will create a new system user that is used to request and deploy
-certificates.  It uses [lexicon](https://github.com/AnalogJ/lexicon) to make
-api requests for dns changes.  We use namecheap, so the required namecheap
-python library is also included.  Both are installed via pip.
+certificates.
 
 This module also assumes a working installation of vault.
 
-### Beginning with acme_vault  
+### Beginning with acme_vault
 
 Just include the appropriate modules for the appropriate machines, and make
 sure the required values are provided via hiera or class.  Note: the common
-module must be included before either.  
+module must be included before either.
 
 Typically this would involve a profile like:
 
@@ -76,13 +74,17 @@ vault, vault vars, and cron mailto are needed for both request and deploy
 
 ##### `user`
 
-user to be created to request/deploy certs 
+user to be created to request/deploy certs
 
 Default value: `acme_vault`
 
 ##### `group`
 
-group that the user belongs to.  For deploy, this should probably be the webserver group 
+group that owns the created certificates
+
+##### `group_members`
+
+members of the above group that will have access to created certificates.  In most cases this will be the webserver group, or any other services that require reading the certs.
 
 Default value: `acme_vault`
 
@@ -102,7 +104,7 @@ Default value: `''`
 
 mapping of domains to be included in the cert.  The key is the "main" domain,
 and the value is the list of extra names to be requested.  Both the main domain
-and the list of domains are included. 
+and the list of domains are included.
 
 REQUIRED
 
@@ -124,18 +126,18 @@ REQUIRED
 
 Path to the vault binary.
 
-Default value: `${home_dir}/vault"`
+Default value: `/usr/local/bin/vault"`
 
 #### `vault_prefix`
 
-The path within vault where the certificates will be stored and retrieved. 
+The path within vault where the certificates will be stored and retrieved.
 
 Default value: `/secret/letsencrypt/`
 
 #### acme_vault::request
 
 This class uses acme.sh, and pulls down the git repo for it.  It uses the
-lexicon provider in acme.sh to do the dns updating for the dns-01 challenge.
+namecheap provider in acme.sh to do the dns updating for the dns-01 challenge.
 It configures a cron job to periodically check if a cert needs renewal.
 
 Note: it does not automatically trigger requesting certs, but relies on cron
@@ -147,13 +149,13 @@ can be triggered by running the cron job manually as needed.
 
 ##### `user`
 
-user to be created to request/deploy certs 
+user to be created to request/deploy certs
 
 Default value: `acme_vault`
 
 ##### `group`
 
-group that the user belongs to.  For deploy, this should probably be the webserver group 
+group that the user belongs to.  For deploy, this should probably be the webserver group
 
 Default value: `acme_vault`
 
@@ -173,7 +175,7 @@ Default value: `''`
 
 mapping of domains to be included in the cert.  The key is the "main" domain,
 and the value is the list of extra names to be requested.  Both the main domain
-and the list of domains are included. 
+and the list of domains are included.
 
 REQUIRED
 
@@ -202,37 +204,39 @@ Default value: `https://acme-v02.api.letsencrypt.org/directory`
 
 #### `acme_revision`
 
-git revision/tag to be used to checkout acme.sh repository.  
+git revision/tag to be used to checkout acme.sh repository.
 
 Default value: `HEAD`
 
 #### `acme_repo_path`
 
-where the repo should be checked out.  
+where the repo should be checked out.
 
 Default value: `$home_dir/acme.sh`
 
 #### `acme_script`
 
-path the the acme.sh script itself  
+path the the acme.sh script itself
 
 Default value: `$acme_repo_path/acme.sh`
 
-#### `lexicon_provider`
+#### `namecheap_sourceip`
 
-provider for lexicon to use for dns-01 challanges.  
+sourceip for namecheap requests (it is well known that this is ignored by their api)
+
+Default value: `127.0.0.1`
 
 REQUIRED
 
-#### `lexicon_username`
+#### `namecheap_username`
 
-username for lexicon dns.  
+username for namecheap dns api.
 
 REQUIRED
 
-#### `lexicon_token`
+#### `namecheap_api_key`
 
-token for lexicon user.  
+token for namecheap api user.
 
 REQUIRED
 
@@ -249,13 +253,13 @@ is appropriate to replace the existing one
 
 ##### `user`
 
-user to be created to request/deploy certs 
+user to be created to request/deploy certs
 
 Default value: `acme_vault`
 
 ##### `group`
 
-group that the user belongs to.  For deploy, this should probably be the webserver group 
+group that the user belongs to.  For deploy, this should probably be the webserver group
 
 Default value: `acme_vault`
 
@@ -269,31 +273,31 @@ Default value: `/home/$user`
 
 mapping of domains to be included in the cert.  The key is the "main" domain,
 and the value is the list of extra names to be requested.  Both the main domain
-and the list of domains are included. 
+and the list of domains are included.
 
 REQUIRED
 
-#### Parameters only for deploy: 
+#### Parameters only for deploy:
 
 ##### `cert_destination_path`
 
-where the cert should be deployed to.  cert will end up in $cert_destination_path/$domain/.  
+where the cert should be deployed to.  cert will end up in $cert_destination_path/$domain/.
 
-Default value: `/etc/acme-vault`
+Default value: `/etc/acme`
 
-##### `restart`
+##### `deploy_scripts`
 
-indicates if cron should include a restart after cert is deployed
+location for arbitrary scripts to reload certificates for applications. scripts will end up in $cert_destination_path/deploy.d/
 
-Valid values: `true` `false`
+Default value: `/etc/acme/deploy.d`
 
-##### `restart_command`
+##### `restart_method`
 
-The command used restart any service after cert is deployed
+shell that is run after successful deployment. runs scripts in $deploy_scripts
+
+Default value: `for f in /etc/acme/deploy.d/*.sh; do "$f"; done`
 
-Default value: `'echo restart!'`
 
- 
 ## Limitations
 
 Has only been tested on Centos 7

files/check_cert.sh

@@ -27,10 +27,10 @@ deploy_cert() {
     EXISTING_FULLCHAIN_PATH=$8
     
     echo "deploying cert to $EXISTING_CERT_PATH"
-    echo "$NEWCERT" > $EXISTING_CERT_PATH
-    echo "$NEWKEY"  > $EXISTING_KEY_PATH
-    echo "$NEWCHAIN"  > $EXISTING_CHAIN_PATH
-    echo "$NEWFULLCHAIN"  > $EXISTING_FULLCHAIN_PATH
+    echo "$NEWCERT" > "$EXISTING_CERT_PATH"
+    echo "$NEWKEY"  > "$EXISTING_KEY_PATH"
+    echo "$NEWCHAIN"  > "$EXISTING_CHAIN_PATH"
+    echo "$NEWFULLCHAIN"  > "$EXISTING_FULLCHAIN_PATH"
 
 }
 
@@ -44,8 +44,9 @@ EXISTING_CHAIN_PATH="${EXISTING_CERT_DIR}/chain.pem"
 EXISTING_FULLCHAIN_PATH="${EXISTING_CERT_DIR}/fullchain.pem"
 
 # variables
-ONE_WEEK=604800
 TODAY=$(date --iso-8601)
+# use VAULT_BIN if defined, otherwise, assume /usr/local/bin/vault
+: "${VAULT_BIN:=/usr/local/bin/vault}"
 
 
 NEWCERT_VAULT_PATH="/secret/letsencrypt/${DOMAIN}/cert.pem"
@@ -54,10 +55,10 @@ NEWCHAIN_VAULT_PATH="/secret/letsencrypt/${DOMAIN}/chain.pem"
 NEWFULLCHAIN_VAULT_PATH="/secret/letsencrypt/${DOMAIN}/fullchain.pem"
 
 # Get new cert info
-NEWCERT=$(vault read -field=value $NEWCERT_VAULT_PATH) || exit -1
-NEWKEY=$(vault read -field=value $NEWKEY_VAULT_PATH) || exit -1
-NEWCHAIN=$(vault read -field=value $NEWCHAIN_VAULT_PATH) || exit -1
-NEWFULLCHAIN=$(vault read -field=value $NEWFULLCHAIN_VAULT_PATH) || exit -1
+NEWCERT=$($VAULT_BIN kv get -field=value "$NEWCERT_VAULT_PATH") || exit 1
+NEWKEY=$($VAULT_BIN kv get -field=value "$NEWKEY_VAULT_PATH") || exit 1
+NEWCHAIN=$($VAULT_BIN kv get -field=value "$NEWCHAIN_VAULT_PATH") || exit 1
+NEWFULLCHAIN=$($VAULT_BIN kv get -field=value "$NEWFULLCHAIN_VAULT_PATH") || exit 1
 NEWCERT_FINGERPRINT=$(get_fingerprint "$NEWCERT")
 NEWCERT_ENDDATE=$(get_enddate "$NEWCERT")
 
@@ -65,7 +66,7 @@ NEWCERT_ENDDATE=$(get_enddate "$NEWCERT")
 if [ "$NEWCERT_FINGERPRINT" == "" ]
 then
     echo "no valid new cert found!"
-    exit -1
+    exit 1
 fi
 
 #echo "new fingerprint: $NEWCERT_FINGERPRINT"
@@ -74,16 +75,16 @@ fi
 # Get existing cert info if it exists.  if it doesn't exist, we don't need to
 # check it, we can just deploy.
 
-if [ -e $EXISTING_CERT_PATH ]
+if [ -e "$EXISTING_CERT_PATH" ]
 then 
-    EXISTING_CERT=$(cat $EXISTING_CERT_PATH)
+    EXISTING_CERT=$(cat "$EXISTING_CERT_PATH")
     EXISTING_CERT_FINGERPRINT=$(get_fingerprint "$EXISTING_CERT")
     EXISTING_CERT_ENDDATE=$(get_enddate "$EXISTING_CERT")
 else
     # create destination dir if needed
-    if [ ! -d $EXISTING_CERT_DIR ]
+    if [ ! -d "$EXISTING_CERT_DIR" ]
     then 
-        mkdir -p $EXISTING_CERT_DIR
+        mkdir -p "$EXISTING_CERT_DIR"
     fi
     deploy_cert "$NEWCERT" "$NEWKEY" "$NEWCHAIN" "$NEWFULLCHAIN" "$EXISTING_CERT_PATH" "$EXISTING_KEY_PATH" "$EXISTING_CHAIN_PATH" "$EXISTING_FULLCHAIN_PATH"
     exit 0
@@ -97,29 +98,23 @@ fi
 # if it is the same, exit normally, this will be the common case
 if [ "$NEWCERT_FINGERPRINT" == "$EXISTING_CERT_FINGERPRINT" ]
 then
-    exit -1
+    exit 1
 fi
 
 # check that new cert is newer than current cert
 if [ "$EXISTING_CERT_ENDDATE" \> "$NEWCERT_ENDDATE" ]
 then
     echo "existing cert expiration is older, exiting"
-    exit -1
+    exit 1
 fi
 
 # check that new cert is not expired
 if [ "$NEWCERT_ENDDATE" \< "$TODAY" ]
 then
     echo "new cert is expired, exiting"
-    exit -1
+    exit 1
 fi
 
 # if we made it this far, the cert looks good, replace it
 
 deploy_cert "$NEWCERT" "$NEWKEY" "$NEWCHAIN" "$NEWFULLCHAIN" "$EXISTING_CERT_PATH" "$EXISTING_KEY_PATH" "$EXISTING_CHAIN_PATH" "$EXISTING_FULLCHAIN_PATH"
-
-
-
-#openssl x509 -in <(vault read -field=value /secret/apidb.org/cert.pem) -noout -checkend 8640000
-
-

manifests/common.pp

@@ -1,9 +1,27 @@
-# Common configuration for acme_vault
+# acme_vault::common
+#
+# This class defines the common configuration for the acme_vault module.
+# It sets up the necessary user, group, home directory, and environment
+# variables for the acme_vault module to interact with Let's Encrypt and
+# HashiCorp Vault for certificate management.
+#
+# @param user The system user for the acme_vault module.
+# @param group The system group for the acme_vault module.
+# @param group_members Additional group members that require access to the certificates.
+# @param home_dir The home directory for the acme_vault user.
+# @param contact_email The email address for Let's Encrypt registration and notifications.
+# @param domains The list of domain names to request certificates for.
+# @param overrides A hash of domain-specific configuration overrides.
+#
+# @param vault_token The authentication token for accessing HashiCorp Vault.
+# @param vault_addr The address of the HashiCorp Vault server.
+# @param vault_bin The path to the Vault binary.
+# @param vault_prefix The prefix for storing certificates in Vault.
 #
-
 class acme_vault::common (
     $user               = $::acme_vault::params::user,
     $group              = $::acme_vault::params::group,
+    $group_members      = $::acme_vault::params::group_members,
     $home_dir           = $::acme_vault::params::home_dir,
     $contact_email      = $::acme_vault::params::contact_email,
     $domains            = $::acme_vault::params::domains,
@@ -17,7 +35,7 @@ class acme_vault::common (
 ) inherits acme_vault::params {
 
     $common_bashrc_template = @(END)
-    export PATH=$HOME:$PATH
+    export PATH=$HOME:$PATH:/usr/local/bin
     export VAULT_BIN=<%= @vault_bin %>
     export VAULT_TOKEN=<%= @vault_token %>
     export VAULT_ADDR=<%= @vault_addr %>
@@ -40,20 +58,22 @@ class acme_vault::common (
       mode   => '0750',
     }
 
-    # vault module isn't too flexible for install only, just copy in binary
-    # would be nice if this worked!
-    #class { '::vault::install':
-    #  manage_user => false,
-    #}
+    # group membership is handled through collected virtual resources.  This
+    # allows other modules/profiles to add members to the group, for services
+    # that require access to the certs
 
-    file { $vault_bin:
-        ensure => present,
-        owner  => 'root',
-        group  => 'root',
-        mode   => '0555',
-        source => 'puppet:///modules/acme_vault/vault',
+    @group { $group:
+      ensure => present,
+      system => true,
+      tag    => 'acme_vault_group',
     }
 
+    # include lines similar to this in your own modules to add members to the
+    # group.  We use this method here to add the group_members paramater, but
+    # it will work the same in any module.
+
+    Group <| tag == 'acme_vault_group' |> { members +> $group_members }
+
     # variables in bashrc
     concat { "${home_dir}/.bashrc":
       owner => $user,
@@ -79,7 +99,7 @@ class acme_vault::common (
 
     # renew vault token
     cron { 'renew vault token':
-      command => ". \$HOME/.bashrc && $vault_bin token-renew > /dev/null",
+      command => ". \$HOME/.bashrc && ${vault_bin} token renew > /dev/null",
       user    => $user,
       weekday => 1,
       hour    => 10,
@@ -87,4 +107,3 @@ class acme_vault::common (
     }
 
 }
-

manifests/deploy.pp

@@ -1,6 +1,19 @@
-# Configuration for deploying certs in vault to the filesystem
+# acme_vault::deploy
+#
+# This class configures the deployment of certificates from HashiCorp Vault
+# to the filesystem. It sets up the necessary directory structure, scripts,
+# and cron jobs to periodically check for and deploy updated certificates
+# for the specified domains.
+#
+# @param user The system user for the acme_vault module.
+# @param group The system group for the acme_vault module.
+# @param home_dir The home directory for the acme_vault user.
+# @param domains The list of domain names for which certificates will be deployed.
+#
+# @param cert_destination_path The directory where certificates will be deployed on the filesystem.
+# @param deploy_scripts The directory where deployment scripts will be stored.
+# @param restart_method The command to run after certificate deployment (e.g., to restart dependent services).
 #
-
 class acme_vault::deploy(
     $user                  = $::acme_vault::common::user,
     $group                 = $::acme_vault::common::group,
@@ -36,9 +49,9 @@ class acme_vault::deploy(
     cron { "${domain}_deploy":
       command => ". \$HOME/.bashrc && ${home_dir}/check_cert.sh ${domain} ${cert_destination_path} && ${restart_method}",
       user    => $user,
-      weekday => 2,
-      hour    => 11,
-      minute  => 17,
+      weekday => ['2-4'],
+      hour    => ['11-16'],
+      minute  => 30,
     }
 
     file {"${cert_destination_path}/${domain}":

manifests/params.pp

@@ -1,35 +1,70 @@
-# params for both common, request, and deploy
+# acme_vault::params
+#
+# This class defines the default parameters for the acme_vault module.
+# It includes settings for the acme user, authentication, staging and production URLs,
+# repository paths, Namecheap configuration, and deployment settings.
+#
+# @param user The system user for the acme_vault module.
+# @param group The system group for the acme_vault module.
+# @param group_members Members of the acme_vault group.
+# @param home_dir The home directory for the acme_vault user.
+# @param contact_email The contact email address for Let's Encrypt notifications.
+# @param domains The list of domain names for which certificates will be requested and managed.
+# @param overrides A list of challenge-alias overrides. Defaults to the domain itself.
+#
+# @param vault_token HashiCorp Vault authentication token.
+# @param vault_addr HashiCorp Vault server address.
+# @param vault_bin Path to the HashiCorp Vault binary.
+# @param vault_prefix The path prefix in Vault where Let's Encrypt secrets are stored.
+#
+# @param staging Whether to use the Let's Encrypt staging environment.
+# @param staging_url Let's Encrypt staging environment API URL.
+# @param prod_url Let's Encrypt production environment API URL.
+#
+# @param acme_revision The revision of the acme.sh script to use.
+# @param acme_repo_path The path to the acme.sh repository.
+# @param acme_script The path to the acme.sh script within the repository.
+#
+# @param namecheap_username Namecheap account username for DNS API.
+# @param namecheap_api_key Namecheap API key.
+# @param namecheap_sourceip The source IP address for Namecheap API requests.
+#
+# @param cert_destination_path The directory where certificates will be deployed on the filesystem.
+# @param deploy_scripts The directory where deployment scripts will be stored.
+# @param restart_method The command to run after certificate deployment (e.g., to restart dependent services).
+#
 class acme_vault::params {
     # settings for acme user
-    $user       = 'acme'
-    $group      = 'apache'
-    $home_dir   = '/home/acme_vault'
-    $contact_email = ''
-    $domains     = undef
+    $user           = 'acme'
+    $group          = 'acme'
+    $group_members  = []
+    $home_dir       = '/home/acme_vault'
+    $contact_email  = ''
+    $domains        = undef
     # overrides is a list of challenge-alias overrides.  It defaults to the domain itself.
     # see https://github.com/Neilpang/acme.sh/wiki/DNS-alias-mode
-    $overrides   = {}
+    $overrides      = {}
 
     # authentication
-    $vault_token = undef
-    $vault_addr  = undef
-    $vault_bin   = "${home_dir}/vault"
+    $vault_token    = undef
+    $vault_addr     = undef
+    $vault_bin      = '/usr/local/bin/vault'
 
     $vault_prefix   = '/secret/letsencrypt/'
 
     # whether to use the letsencrypt staging url, set those urls
-    $staging     = false
-    $staging_url = 'https://acme-staging-v02.api.letsencrypt.org/directory'
-    $prod_url    = 'https://acme-v02.api.letsencrypt.org/directory'
+    $staging        = false
+    $staging_url    = 'https://acme-staging-v02.api.letsencrypt.org/directory'
+    $prod_url       = 'https://acme-v02.api.letsencrypt.org/directory'
 
-    $acme_revision = 'HEAD'
+    $acme_revision  = 'HEAD'
     $acme_repo_path = "${home_dir}/acme.sh"
     $acme_script    = "${acme_repo_path}/acme.sh"
 
-    # lexicon
-    $lexicon_provider   = undef
-    $lexicon_username   = undef
-    $lexicon_token      = undef
+    # namecheap
+    $namecheap_username = undef
+    $namecheap_api_key  = undef
+    $namecheap_sourceip = '127.0.0.1'
 
     # settings for deploy
     $cert_destination_path = '/etc/acme'

manifests/request.pp

@@ -1,6 +1,27 @@
-# Configuration for requesting a cert from letsencrypt, and storing it in vault.
+# acme_vault::request
+#
+# This class configures the certificate request process from Let's Encrypt,
+# and stores the obtained certificates in HashiCorp Vault.
+#
+# @param user The system user for the acme_vault module.
+# @param group The system group for the acme_vault module.
+# @param home_dir The home directory for the acme_vault user.
+# @param contact_email The contact email address for Let's Encrypt notifications.
+# @param domains The list of domain names for which certificates will be requested and managed.
+# @param overrides A list of challenge-alias overrides. Defaults to the domain itself.
+#
+# @param staging Whether to use the Let's Encrypt staging environment.
+# @param staging_url Let's Encrypt staging environment API URL.
+# @param prod_url Let's Encrypt production environment API URL.
+#
+# @param acme_revision The revision of the acme.sh script to use.
+# @param acme_repo_path The path to the acme.sh repository.
+# @param acme_script The path to the acme.sh script within the repository.
+#
+# @param namecheap_username Namecheap account username for DNS API.
+# @param namecheap_api_key Namecheap API key.
+# @param namecheap_sourceip The source IP address for Namecheap API requests.
 #
-
 class acme_vault::request (
     $user               = $::acme_vault::common::user,
     $group              = $::acme_vault::common::group,
@@ -17,27 +38,20 @@ class acme_vault::request (
     $acme_repo_path     = $::acme_vault::params::acme_repo_path,
     $acme_script        = $::acme_vault::params::acme_script,
 
-    $lexicon_provider   = $::acme_vault::params::lexicon_provider,
-    $lexicon_username   = $::acme_vault::params::lexicon_username,
-    $lexicon_token      = $::acme_vault::params::lexicon_token,
-
+    $namecheap_username = $::acme_vault::params::namecheap_username,
+    $namecheap_api_key  = $::acme_vault::params::namecheap_api_key,
+    $namecheap_sourceip = $::acme_vault::params::namecheap_sourceip,
 
 ) inherits acme_vault::params {
 
     include acme_vault::common
 
     $request_bashrc_template = @(END)
-export TLDEXTRACT_CACHE=$HOME/.tld_set
-export PROVIDER=<%= @lexicon_provider %>
-export LEXICON_<%= @lexicon_provider.upcase %>_AUTH_USERNAME=<%= @lexicon_username %>
-export LEXICON_<%= @lexicon_provider.upcase %>_AUTH_TOKEN=<%= @lexicon_token %>
-END
-
-    # install lexicon
-    ensure_packages(['dns-lexicon', 'PyNamecheap'], {
-      ensure   => present,
-      provider => 'pip',
-    })
+      export TLDEXTRACT_CACHE=$HOME/.tld_set
+      export NAMECHEAP_USERNAME=<%= @namecheap_username %>
+      export NAMECHEAP_API_KEY=<%= @namecheap_api_key %>
+      export NAMECHEAP_SOURCEIP=<%= @namecheap_sourceip %>
+      | END
 
     # variables in bashrc
     concat::fragment { 'request_bashrc':
@@ -60,14 +74,14 @@ END
       owner  => $user,
       group  => $group,
       mode   => '0700',
-    } ->
-    file { "${home_dir}/.acme.sh/account.conf":
+    }
+    -> file { "${home_dir}/.acme.sh/account.conf":
       ensure => present,
       owner  => $user,
       group  => $group,
       mode   => '0600',
-    } ->
-    file_line { ' add email to acme conf':
+    }
+    -> file_line { ' add email to acme conf':
       path  => "${home_dir}/.acme.sh/account.conf",
       line  => "ACCOUNT_EMAIL='${contact_email}'",
       match => '^ACCOUNT_EMAIL=.*$',

metadata.json

@@ -13,6 +13,10 @@
     {
       "name": "puppetlabs-concat",
       "version_requirement": ">= 1.2.4"
+    },
+    {
+      "name": "onyxpoint-gpasswd",
+      "version_requirement": ">= 1.1.1"
     }
   ],
   "operatingsystem_support": [

templates/domain.epp

@@ -9,7 +9,7 @@
 <% } else { -%>
 --server <%= $prod_url %> \
 <% } -%>
---dns dns_lexicon \
+--dns dns_namecheap \
 --dnssleep 1800 \
 --domain "<%= $domain %>" --challenge-alias <%= "$domain" %> \
 <% $domains.each |$d| {