Puppet batch management more than 500 servers

Recommended for you: Get network issues from WhatsUp Gold. Not end users.


Puppet used for a long time, before writing hand Shun has not released to the blog, happen to have free time today, write a note. The company used in the server has more than 500 sets, for the basic CentOS version 6, 5 and two, management is not convenient, particularly the deployment of monitoring, including a large number of repetitive work, the use of puppet can be very convenient.


Before installation, introduction of course is a must, under the simple introduction. Puppet is a client and server based C/S architecture, based on Ruby development. So, you see, the installation of puppet, you need to install and configure ruby. The web management interface similar to the redmine installation, the integration of the passenger module using Apache.

The server deployment

1,Download and install packages
puppetlabs-release-5-6.noarch.rpm(puppet repo)

rpm –ivh puppetlabs-release-5-6.noarch.rpm
rpm –ivh puppet-dashboard-1.2.23-1.el5.noarch.rpm

yum install Puppet-server puppetdb puppetdb-termius (behind two can not be installed)

2,Install ruby,mysql,apache_passenger module

See the installation of redmine document.

3, Server configuration
For puppet.conf, which is divided into 3 parts [main], [master], [agent], outside of the document, some parameters are added to the [main], some are added to the [agent], more confused for beginners, which is correct. For agent, you can modify in agent. If your settings and [main] conflict, will retain the [agent] settings. So you don't need to [main] can be set up.
Do not modify the /etc/sysconfig/puppetmaster 
/etc/sysconfig/puppet              no changes
/etc/puppet/puppet.conf        no changes
Execution: /etc/init.d/puppetmaster start

4,Client configuration

rpm –ivh puppetlabs-release-5-6.noarch.rpm
yum install puppet
/etc/puppet/Puppet.conf is amended as follows contents, values for the puppet server hostname
Execution /etc/init.d/puppet start

Or does not modify the configuration file, puppet agent–server=server.example.com

Check the server certificate

puppet cert list –all

Check module position

puppet config print modulepath

View report

Puppet agent –t –summarize

6,Certificate Register

The client agent starts to send the certificate server for
the command to view the certificate on the server, and then the signature

puppet cert sign station3.example.com

7, Certificate Cancle

(1)Registration certificate

puppet cert revoke station3.example.com(Just let the certificate expires)
puppet cert clean station3.example.com (Delete certificate)

Restart puppetmaster

At this time, station3.exmaple.com was unable to connect to the puppet server
(2) re registration certificate in the client

rm -f /var/lib/puppet/ssl/certs/station3.example.com.pem
rm -f /var/lib/puppet/ssl/certificate_requests/station3.example.com.pem

Then restart puppet, execute puppet cert list will be able to see the re apply for the certificate on the server.

(3)Automatic registration certificate
can set master to automatically issue of all certificates, we only need to create the autosign.conf file in the /etc/puppet directory. (no need to modify the /etc/puppet/puppet.conf file, because my default autosign.conf file location without modification)

vim /etc/puppet/autosign.conf

     It all from the example.com domain on all clients will automatically register.

8, puppet dashboard

This is easy to see, see the report can quickly know the server puppet failed to update the application.
(1) modified my.cnf
add max_allowed_packet = 32M
in the [mysqld] module (2) create database

CREATE DATABASE dashboard_production CHARACTER SET utf8;
CREATE USER 'dashboard'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON dashboard.* TO 'dashboard'@'localhost';

(3)The editor/usr/share/puppet-dashboard/config/database.yml

(4)Change the time zone /usr/share/puppet-dashboard/config/environment.rb
#config.time_zone ='UTC'
          config.time_zone ='Beijing'
(5) Apache

vim /etc/httpd/conf.d/puppet.conf

LoadModule passenger_module /usr/local/ruby/lib/ruby/gems/1.8/gems/passenger-3.0.18/ext/apache2/mod_passenger.so
PassengerRoot /usr/local/ruby/lib/ruby/gems/1.8/gems/passenger-3.0.18
PassengerRuby /usr/local/ruby/bin/ruby

Listen 3001
<VirtualHost *:3001>
   ServerName server.example.com
# !!! Be sure to point DocumentRoot to 'public'!
   DocumentRoot /usr/share/puppet-dashboard/public   
  <Directory /usr/share/puppet-dashboard/public >
  # This relaxes Apache security settings.
    AllowOverride all
  # MultiViews must be turned off.
    Options -MultiViews

So puppet just like redmine with Apache way to start the.

(6)Initialize the database

cd /usr/share/puppet-dashboard/
rake RAILS_ENV=production db:migrate

(7) Introduction of reports (the default directory in the /var/lib/puppet/reports)

cd /usr/share/puppet-dashboard/
rake RAILS_ENV=production reports:import REPORT_DIR= /var/lib/puppet/reports

(8)Delayed Job Workers

cd /usr/share/puppet-dashboard/
env RAILS_ENV=production script/delayed_job -p dashboard -n 4 -m start(To analyze reports)
ps -ef|grep delayed_job|grep -v grep(To view the delayed_job process)
env RAILS_ENV=production script/delayed_job -p dashboard -n 4 -m stop(Stop the analysis)

Note that this process is not stopped, will always exist, so, sometimes to restart the server will forget, simply write to /etc/rc.local

 The installation process and that's the puppet server end, there are some skills added, such as client batch deployment, troubleshooting etc.

The Client Deployment 

Since puppet is a C/S architecture, so have to deploy the client on each server, but more than 500 servers, impossible to hand a to deploy, through the natural script mode.

The deployment of the premise

(1)No authentication

For 500 or more of the server cluster, the coupling degree between the application is very high, and in order to facilitate the management, generally operating system level of trust, it is SSH free certification.

Of course, some people will say that there will be security risks, if you control a server, then the whole network will fall. Absolutely. Safe and convenient itself is contradictory, I think to safety protection for IDC server cluster architecture is mainly from the firewall, access restrictions and access control, should not only ensure the normal operation of the business can also ensure that the server itself safety.

(2)The hosts file

Because it is a server cluster, all servers trust is generally through the /etc/hosts file records of other server hostname with IP mapping.

The deployment process

The authentication server based on free network, we can use the script will install script to all servers, and then run the installation script, it is realized the automatic installation of puppet client.

The installation script, install_puppet.sh

[ -f /etc/init.d/puppet ]&& /etc/init.d/puppet restart
[ -f /etc/sysconfig/puppet  ]&& exit

version5=`/bin/cat /etc/issue|head -1|grep '5.'|wc -l`
if [ $version5 = 1 ];then
    rpm -ivh http://yum.puppetlabs.com/el/5/products/i386/puppetlabs-release-5-6.noarch.rpm
    yum -y install puppet
    puppet agent --server server.example.com
    [ -f /sbin/chkconfig ]&&`chkconfig puppet on`
    #echo "centos 5"
    version6=`/bin/cat /etc/issue|head -1|grep '6.'|wc -l`
     if [ $version6 = 1 ];then
        rpm -ivh http://yum.puppetlabs.com/el/6/products/i386/puppetlabs-release-6-6.noarch.rpm
        yum -y install puppet
        puppet agent --server server.example.com
        [ -f /sbin/chkconfig ]&&`chkconfig puppet on`
        #echo "centos 6"

Push the script push.pl, /etc/hosts documents based on.

#!/usr/bin/perl -w

if (@ARGV)
    foreach (@ARGV)
        if ($ARGV[0] =~ "all")
            open(FILE,"</etc/hosts")||die"cannot open the file: $!\n";
            while (<FILE>)
                if ($_ =~ /^10/)
                            my @host=split;
                            print "########It's coping file to $host[1]########\n";
                            system("/usr/bin/rsync  install_puppet.sh $host[0]:/");
                            system("/usr/bin/ssh $host[0] /install_puppet.sh");
            close FILE;
                    print "########It's coping file to $_########\n";
                    system("/usr/bin/rsync install_puppet.sh $_:/");
                 system("/usr/bin/ssh $_ /install_puppet.sh");
    print "1.Usage: $0 hostname1 hostname2 ... \n";
    print "2.Usage: $0 all\n";

This will be the two scripts on the same directory, then the implementation of./push.pl all, and then leave it alone, all servers are deployed automatically, the corresponding version of the puppet client.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download

Posted by Harold at November 09, 2013 - 7:41 AM