Title

class: center, middle, front

# Puppet @ Gandi

DevOps Taiwan Meetup  
November 2016

https://mose.github.io/slides/puppet_at_gandi_2016

---

# mose ( 莫測 )

- DevOps at Gandi Asia since 2014

- Taiwanese since 2008

- http://mose.com

- https://github.com/mose

???
more on:

- http://blog.mose.com/
- http://www.linkedin.com/in/ozmose
- also Editor in Chief http://greenruby.org

---
![gandi.net](img/Puppet-Logo-Amber-Black-sm.png)  
- Configuration Management Framework
- extensible and modular
- not orchestrator
- not lightweight

- written in Ruby (and some clojure)
- https://puppet.com

???
and of course totally open-source
- https://github.com/puppetlabs

Puppet Company facts:
- founded in 2005
- 400+ employees in 5 office
- HQ in Portland, Oregon
- 30,000 organizations use puppet
  - NASA, Redhat, Intel, Sony, etc ...
- *source https://puppet.com/company*

---
![gandi.net](img/Gandi_logo_black.jpg)  
- Domain Names Registrar, 2M domains
- Hosting VPS, PaaS, Mail
- 1,600 servers in 4 Datacenters
- Debian, Nexenta and FreeBSD
- 15 devops in charge

- Created in 2001 in France. Taipei in 2014
- https://gandi.net

???
more Gandi facts
- offices in Paris, Luxembourg, San Francisco and Taipei
- 150 employees
- support 24/24, 6 days per week in english, french and chinese
- **no bullshit** - as a way of life

---
## Why Puppet at Gandi

- Gandi started using puppet in 2009
- Gandi is more a python shop
- but puppet is (almost) language agnostic
- .. with a complete DSL

- Debian packaging (debian/nexenta)
- or FreeBSD pkg-ng
- deployment is a done by system update
- ... with Puppet

---
## Puppet specificities

- nodes pull configuration and send facts to a central puppetmaster
- DSL is simple
- Descriptive and Idempotent
- Popular and widely supported

???
- Gandi is a python shop, no ruby ninjas available (or, only one)
- we got 4 puppetmasters for redundancy
- custom flask application to get gitlab hooks and pull new code on puppetmasters

---
## How Puppet at Gandi

- usage of Hiera
- inventory in ENC
- custom modules
- puppet 3.6 with puppetdb

- puppet code is in internal Gitlab
- automated validation with jenkins
- arbitrary branches (environment)
- gitlab workflow with MRs
- monitoring with Nagios

???
This is the result of 7 years of evolution
- 2009 -> only few people use it because lack of clear conventions
- 2010 -> it's a mess
- 2011 -> simple module template
- 2014 -> integration of the monitoring inside puppet
- 2015 -> use of yaml ENC files, normalization of modules, CI validations

---
## How it works
```
.-------------.
| environment |---------------------------.
'-------------'                           |
  | .----------.        .--------------.  |
  | | ENC file |--------| farm_modules |  |
  | '----------'        '--------------'  |
  |       |                    |          |
  |       |                    |          |
  | .------------.        .---------.     |
  | | hiera file |        | module1 |     |    .------------.
  | | hiera file |        | module2 |--------> | puppet run |
  | | hiera file |        | module3 |     |    '------------'
  | '------------'        '---------'     |
  |       |                    |          |
  |       '------> params -----'          |
  '---------------------------------------'
```
- most runs are still manual

???
- Environment is not dev/stage/prod, it is about branches and MRs
- each node/server has one ENC
- we have 1600 ENC files
- we begin to cron puppet runs for microservices servers
- we begin to use Ansible for orchestration and run puppet on several hosts
as an alternative to mcollective (as Ansible can perform arbitrary actions along the way)

---
## External Nodes Classifier

- declaration of what farm the node is in
- definition of what params apply

```bash
$ cat enc/consul1...gandi.net.yaml
---
classes:
  - consulnode       # <-- farm_modules/consulnode/manifests/init.pp
parameters:
  country: lu        # <-- params/country/lu.yaml
  datacenter: dc2    # <-- params/datacenter/dc2.yaml
  farm: consulnode   # <-- params/farm/consulnode.yaml
```

???
- we use real FQDN for the name of each ENC file, so they are easy to find

---
## Hiera Config

```yaml
:backends:
  - yaml
:hierarchy:
  - "nodes/%{fqdn}"
  - "farm_datacenter_flavor/%{farm}_%{datacenter}_%{farm_flavor}"
  - "farm_datacenter_step/%{farm}_%{datacenter}_%{step}"
  - "farm_datacenter/%{farm}_%{datacenter}"
  - "farm_flavor_step/%{farm}_%{farm_flavor}_%{step}"
  - "farm_flavor/%{farm}_%{farm_flavor}"
  - "farm_step/%{farm}_%{step}"
  - "farm/%{farm}"
  - "room/%{room}"
  - "datacenter_virtualdc/%{datacenter}_%{virtualdc}"
  - "datacenter/%{datacenter}"
  - "country/%{country}"
  - "os-lsb/%{operatingsystem}-%{lsbdistcodename}"
  - "os/%{operatingsystem}"
  - "common/common"
:merge_behavior: deeper

```

???
- if a parameter is not set, the params file is ignored
- if the params file is not there, it is just not used, without blocking anything
- the 'common' params file is the base all-hosts params file

---
## Hiera file example

```yaml
apt_config:
  experimental:
    enabled: "true"

monitoring_config:
  default:
    host:
      contact_groups: "stage_platform"
    service:
      contact_groups: "stage_platform"
```
???
- this is an example of a node params file
- on 1600 ENC files only 843 have node params files, the others just follow default farms configuration

---
## Hiera(rchy)

```bash
[-] (merged)
[0] params/farm_datacenter/monworker_dc2.yaml
[1] params/farm/monworker.yaml
[2] params/datacenter/dc2.yaml
[3] params/os/Debian.yaml
[4] params/common/common.yaml

[1] collectd_config.additional_plugins ["tcpconns"]
[3] collectd_config.generic_plugins ["vmem"]
[3] collectd_config.plugin_disk ["/^sd/"]
[1] monitoring_config.dependencies ["libsnmp-ruby1.8", "rpcbind"]
[0] monitoring_config.services_nrpe.proc_worker_bissen-master-pra.check_command check_procs -u nagios -c 3: -a worker_bissen-master-pra
[0] monitoring_config.services_nrpe.proc_worker_bissen-master-pra.use generic-service
[-] monworker_classes ["gandi_hosting_vm"]
    [0] monworker_classes ["gandi_hosting_vm"]
    [1] monworker_classes []
...
```
???
Sample of output of [Hieracles](https://github.com/Gandi/hieracles)
- it shows where what param is defined, in which file
- it also reveals how the params override is handled from which files

---
## Farm_modules

- 310 modules (all custom)

```bash
# consulnode farm
class consulnode {
    include base_modules
    include ssmtp
    include sysctl
    include snmpd
    include collectd
    include monitoring

include consul   # <-- modules/consul/
}
```
???
- Farms are like a family or a category of servers
- the name of the farm is (usually) the name of the classes file listing modules
- we have 2 subcategories that we call **flavor** and **step**

---
## Modules

```bash
modules/consul
├── files
│   └── default
├── manifests
│   ├── config.pp
│   ├── cron.pp
│   ├── files.pp
│   ├── init.pp
│   ├── install.pp
│   └── service.pp
├── README
└── templates
    └── default
        └── etc
            ├── consul.d
            │   └── 42-gandi.json.erb
            └── default
                └── consul.erb
```

???
- we have some scripts to create new modules according to a strict template
- all modules have same base manifest files
- `config.pp` contains definition of erb templates
- `cron.pp` defines cron jobs
- `files.pp` lists files and directory creations that are not templated (no variables)
- `init.pp` initializes the params from Hiera for usage in templates
- `install.pp` lists packages that need to be installed (or removed)
- `service.pp` manages start and stop of services
---
## Manifests

```ruby
# consul::files handles static/blob files
class consul::files inherits globals::files {
    motd::register{ $name: }

# default options for file provider
    File {
        require => Class['consul::install'],
        notify  => Class['consul::service']
    }

file {
        '/srv/consul':
            ensure => directory,
            owner  => 'consul',
            group  => 'consul';
    }
}
```

???
- this is not really ruby code, but the Puppet DSL
- the `motd::register` will add this module name in the motd
- that way when we ssh on the server we see what is its configuration

---
## Extra Puppet Tools

- Facter (gets facts)
- Puppetdb (can be curl'ed)
- Mcollective (batch operations)
- Hieracles (CLI to find hierarchy overrides)
- Hieraviz (hieracles in a web browser)
- Ansible (yeah, for provisioning and orchestration)

???
- facter https://docs.puppet.com/facter/
- puppetdb https://docs.puppet.com/puppetdb/
- mcollective https://docs.puppet.com/mcollective/
- hieracles https://github.com/Gandi/hieracles
- hieraviz https://github.com/Gandi/hieraviz

- we also tried Foreman https://theforeman.org/ but we abandoned it

- see http://spencerkrum.com/talks/better-together-cfgmgmtcamp2016/slides/#26
for another case of using Ansible alongside Puppet
---
class: center, middle, front

Thanks!

# Any question?

???
- Deeper presentation with more advanced details on http://pres.gandi.net/sysadmin6/
- this presentation is made with Remark https://github.com/gnab/remark