The Ruby side of Puppet - Part 3 - Custom Types and Providers

This is the final post in a three-part serie, covering the concepts and best practices for extending Puppet using custom facts, custom functions and custom types and providers.

Part 1 explores how to create custom facts, which allow nodes to send information to the Puppet Server.

Part 2 discusses building custom functions to process data or execute specific tasks.

Part 3 (this post) focuses on custom types and providers, allowing you to extend Puppet's DSL and control system resources.

---

Types are at the core of Puppet’s declarative DSL. Types describe the desired state of the system, targeting specific, configurable parts (sometimes OS-specific). Puppet provides a set of core types, available with any Puppet agent installation.

Some of the core types include:

• file
• user
• group
• package
• service

Why Create Custom Types and Providers:

There are several reasons to develop custom types and providers:

• Avoid relying on the often unreliable or hard-to-maintain exec resources
• Manage an application that requires CLI commands for configuration.
• Handle configurations with highly specific syntax that existing Puppet types cannot manage (like file or concat).

Content:

1. General concepts
2. Types and providers in modules
3. Type description
4. Provider implementation
5. Using custom types in Puppet DSL
6. Resource API
7. Summary

General concepts

A custom type consists of two main parts:

• Type definiton: This defines how the type will be used in the Puppet DSL.
• Provider implementation(s): One or more providers per type define how to interact with the system to manage resources.

Type definition

The type describes how Puppet resources are declared in the DSL. For example, to manage an application’s configuration:

app_config { <setting>:
  ensure   => present,
  property => <value>,
  param    => <app_args>,
  provider => <auth_method>,
}

In the type definition, you typically have a namevar (a key identifier) and mutiple parameters and properties.

• Namevar: The key identifying the resource instance in the type declaration.
• Properties: These represent something measurable on the target system, such as a user’s UID or GID, or an application’s config value.
• Parameters: Parameters influence how Puppet manages a resource but don’t directly map to something measurable on the system. For example, manage_home in the user type is a parameter that affects Puppet’s behavior but isn’t a property of the user account.

The difference between parameters and properties is nicely described on the Puppet Type/Provider development page:

Properties:

"Properties correspond to something measurable on the target system. For example, the UID and GID of a user account are properties, because their current state can be queried or changed. In practical terms, setting a value for a property causes a method to be called on the provider."

Parameters:

"Parameters change how Puppet manages a resource, but do not necessarily map directly to something measurable. For example, the user type’s managehome attribute is a parameter — its value affects what Puppet does, but the question of whether Puppet is managing a home directory isn’t an innate property of the user account."

In our example, the property is the value of the config setting, whereas the parameter specifies application attributes.

Provider implementations

Providers define the mechanisms for managing the state of the resources described by types. They handle:

• Determining whether the resource already exists.
• Creating or removing resources.
• Modifying resources.
• Optionally, listing all existing resources of the type.

Types and providers in modules

Custom types and providers are placed within the lib/puppet directory of a module.

• The type file is located in lib/puppet/type and named after the resource type (app_config.rb in this case).
• Provider implementations go into the lib/puppet/provider directory, inside a subdirectory named after the resource type. Each provider is named according to its provider implementation (e.g., ruby.rb, cli.rb, cert.rb, token.rb, user.rb)

For example:

1. Type: app_config
2. Providers:
   • cert
   • token
   • user

# <modulepath>/<modulename>
modules/application/
    \- lib/
        \- puppet/
            |- type/
            |    \- app_config.rb
            \- provider/
                \- app_config/
                    |- cert.rb
                    |- token.rb
                    \- user.rb

Type description

New custom types can be crated using two different API versions.
APIv1 is the old, classic way using getters and setters within the providers.
APIv2 is a new implementation which is integrated into PDK - this implementation is also called Resource API.

In the next section we introduce the APIv1 implementation. The Resource API implementation is handled later in this document.

APIv1

The traditional method uses Puppet::Type.newtype, which defines the type.
It is recommended to add the type documentation into the code.
This allows users to run puppet describe <type> on their system to display the documentation.

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  # ... the code ...
end

Management

The most important feature of a type is to add or remove something from the system.
This is usually handled with the ensure property.
To enable ensure, a single line is needed: ensurable

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  ensurable
end

Namevar

When declaring the type, one must specify a title - one could refer to this as a type instance identifier.
Usually this reflects for something the type is managing.

user { 'betadots': # <- title
}

The most simple implementation is to add a parameter with the name :name.

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  ensurable
  newparam(:name) do
    desc "The app config setting to manage. see app_cli conf --help"
  end
end

Passing the namevar: true key to the parameter is another way to identify a namevar:

newparam(:key, namevar: true) do
  desc 'The app config setting key to manage. see app_cli conf --help'
end

Properties

Next we add the other property. Each property can be validated by its content.
In our demo case we expect a string value.

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  ensurable
  newparam(:name) do
    desc "The app config setting to manage. see app_cli conf --help"
  end
  newproperty(:value)do
    desc "The config value to set."
    validate do |value|
      unless value =~ /^\w+/
        raise ArgumentError, "%s is not a valid value" % value
      end
    end
  end
end

Besides this one can provide specific valid values which are validated automatically:

newproperty(:enable) do
  newvalue(:true)
  newvalue(:false)
end

Please note that arrays as property values are validated in a different way:

From Puppet custom types website:

By default, if a property is assigned multiple values in an array:

• It is considered in sync if any of those values matches the current value.
• If none of those values match, the first one is used when syncing the property.

If all array values should be matched, the property needs the
array_matchingto be set to :all. The default value is :first

newproperty(:flags, :array_matching => :all) do
  # ...
end

Accessing values can be done in two ways:

1. should for properties, value for parameters
2. value for both, properties and parameters

We prefer to use the explizit methods as this makes it more clear, whether we deal with a property or a parameter.

Parameters

Parameters are defined in a similar way:

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
        cli_args => ['-p'], # persistency
      }
  }
  ensurable
  newparam(:name) do
    desc "The app config setting to manage. see app_cli conf --help"
  end
  newproperty(:value) do
    desc "The config value to set."
    validate do |value|
      unless value =~ /^\w+/
        raise ArgumentError, "%s is not a valid value" % value
      end
    end
  end
  newparam(:cli_args, :array_matching => :all) do
    desc "CLI options to use during command execution."
    validate do |value|
      unless value.class == Array
        raise ArgumentError, "%s is not a an array" % value
      end
    end
    defaultto ['-p']
  end
end

Boolean parameters

Parameter which get a bool value should be handled in a different way to avoid code repetition

require 'puppet/parameter/boolean'
# ...
newparam(:force, :boolean => true, :parent => Puppet::Parameter::Boolean)

Automatic relationships

Within the type we can specify soft dependencies between different types.

E.g. the app_cli should use a user which must be available on the system

autorequire(:user) do
  ['app']
end

From now on the new custom type can already be used in Puppet DSL, the compiler will create a catalog but the agent will produce an error, as there are no functional providers.

Agent side prerun evaluation

It is possible to have the agent first check some things prior the catalog will be applied.
This is what the :pre_run_check method can be used for

def pre_run_check
  File.exist?('/opt/app/bin/app.exe') && raise Puppet::Error, "App not installed"
end

Features

When using multiple providers (similar to the package resource) we want to be sure that the provider has all required implementations (features).
A type can require a feature:

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
        file     => '/opt/app/etc/app.cfg',
        cli_args => ['-p'], # persistency
      }
  }
  ensurable
  # global feature
  # feature :cli, "The cli feature requires some parameters", :methods => [:cli]
  newparam(:name) do
    desc "The app config setting to manage. see app_cli conf --help"
  end
  newproperty(:value) do
    desc "The config value to set."
    validate do |value|
      unless value =~ /^\w+/
        raise ArgumentError, "%s is not a valid value" % value
      end
    end
  end
  # The property config_file must be set when using the cli 
  #   option to configure app.
  # The cli provider will check for a feature.
  newproperty(:file, :required_features => %w{cli}) do
    desc "The config file to use."
    validate do |value|
      unless (File.expand_path(value) == value)
        raise ArgumentError, "%s is not an absolute path" % value
      end
    end
    defaultto: '/opt/app/etc/app.cfg'
  end
  newparam(:cli_args, :array_matching => :all) do
    desc "CLI options to use during command execution."
    validate do |value|
      unless value.class == Array
        raise ArgumentError, "%s is not a an array" % value
      end
    end
    defaultto: ['-p']
  end
end

Provider implementation

Once the type is defined, the provider must handle how the resource is managed. Providers typically implement methods to:

• Check if the resource exists (exists?).
• Create a new resource (create).
• Read existing resource properties (prefetch or a getter)
• Modify a resource (flush or a setter).
• Delete a resource (destroy).

# lib/puppet/provider/app_config/cli.rb
Puppet::Type.type(:app_config).provide(:cli) do
  desc "The app_config types cli provider."
end

It is also possible to re-use and extend existing providers classes.
Common code can be placed inside a generic provider (lib/puppet/provider/app_config.rb).

# lib/puppet/provider/app_config/cli.rb
Puppet::Type.type(:app_config).provide(:cli, :parent => Puppet::Provider::App_config) do
  desc "The app_config types cli provider reuses shared/common code from app_config.rb."
end

If you want to add shared functionality to multiple providers, you can place your code into the PuppetX module directory: lib/puppet_x/<company_name>/<unique class name>.

# lib/puppet/provider/app_config/cli.rb
require_relative '../../puppet_x/betadots/app_api.rb'
Puppet::Type.type(:app_config).provide(:cli) do
  desc "The app_config types cli provider reuses shared/common code from app_config.rb."
end

A new provider can be created by inheriting from and extending an existing provider:

# lib/puppet/provider/app_config/token.rb
Puppet::Type.type(:app_config).provide(:token, :parent => :api, :source => :api) do
  desc "The app_config types token provideris an extension to the api provider."
end

Additionally one can reuse any provider from any type:

# lib/puppet/provider/app_config/file.rb
Puppet::Type.type(:app_config).provide(:file, :parent => Puppet::Type.type(:ini_setting).provider(:ruby)) do
  desc "The app_config types token provideris an extension to the api provider."
end

Selection of provider

The Puppet Agent must know which provider to use, if multiple providers are present.
You can use of the provider meta parameterr or let providers perform checks to determine if they are valid for the system. The options include:

Check	Example	Description
commands	commands :app => "/opt/app/bin/app.exe"	Valid if the command /opt/app/bin/app.exe exists. The command can be later used using :app.
confine - exists	confine :exists => "/opt/app/etc/app.conf"	Valid if the file exists.
confine - bool	confine :true => /^10\./.match(%x{/opt/app/bin/app.exec version})	Valid if Version is 10.x
confine - Fact	confine 'os.family' => :debian	Valid if the fact value matches
confing - feature	confine :feature => :cli	Valid if the provider has the feature
defaultfor	defaultfor 'os.family' => :debian	Use this provider as default for Debian-based systems.

Reading and applying configuration

The provider needs the ability to create, read, update and delete individual configuration states (CRUD API).

Each configuration property needs a getter (read) and a setter (modify) method.

# lib/puppet/provider/app_config/cli.rb
#   app_config: { 'enable_logging':
#     ensure   => present,
#     value    => true,
#   }
#
Puppet::Type.type(:app_config).provide(:cli) do
  desc "The app_config types cli provider."

  # confine to existing command
  commands :app => "/opt/app/bin/app.exe"

  def exists?
    @result = app_cli('list').split(%r{\n}).grep(%r{^#{resource[:key]}})
    if @result.length > 1
      raise ParserError, 'found multiple config items found, please fix this'
    end
    return false if @result.empty?
    return true unless @result.empty?
  end

  def create
    app(['set', resource[:name], resource[:value]])
  end

  def destroy
    app(['rm', resource[:name]])
  end

  # getter
  def value()
    @result[0].split[1]
  end

  # setter
  def value=(value)
    app(['set', resource[:name], resource[:value]])
  end
end

It is recommended to also create the instances class method, which can collect all instances of a resource type into a hash.
The namevar is the hash key, which has a hash of paramters and properties as value.

require 'yaml'
def self.instances
  instances = []
  YAML.load(app_cli('list')).each do |key, value|
    attributes_hash = { key: key, ensure: :present, value: value, name: key}
    instances << new(attributes_hash)
  end
  instances
end

Sometimes it is not possible to collect all instances, such as with the file resource. In such cases no instance method is defined.

The instance method is used indirectly be each type when calling prefetch to get the acutal configuration and returns the@property_hash instance variable.

def self.prefetch(resources)
  resources.each_key do |name|
    provider = instances.find { |instance| instance.name == name }
    resources[name].provider = provider if provider
  end
end

This allows the type getter and setter to read an manipulate the instance variable, instead of writing getter and setter methods for each type.
This behavior is added by declaring the mk_resource_methods class method.

Once this is implemented you can run the command puppet resource app_config to retrieve all existing configurations.

Refresh events

In some cases it is required to refresh a resource, for example, to restart a service, remount a mount, or rerun an exec resource. To allow a type/provider to react to a refresh event, it needs special handling.

Within the type the refreshable feature must be activated and a refresh definition is added.

The following examples are taken from puppet service type and provider. The feature describes, which provider definition should be executed upon refresh event.

# lib/puppet/type/services.rb
# ...
  feature :refreshable, "The provider can restart the service.",
    :methods => [:restart]
# ...
  def refresh
    # Only restart if we're actually running
    if (@parameters[:ensure] || newattr(:ensure)).retrieve == :running
      provider.restart
    else
      debug "Skipping restart; service is not running"
    end
  end
# ...

Using custom types in Puppet DSL

Once the custom type and provider are implemented, you can use them in your manifests as follows:

# Type declaration
app_config { 'enable_logging':
  ensure    => present,
  value     => true,
  require   => File['/opt/app/etc/app.cfg'],
  subscribe => Service|'app'|,
}
# Type reference
service { 'app':
  ensure    => running,
  subscribe => App_config['enable_logging'],
}
# Type declaration using lambda and splat operator
$config_hash.each |String $key, Hash $hash| {
  app_config { $key:
    * => $hash,
  }
}
# Virtual or exported resource
@@app_config { 'app_mount':
  ensure => present,
  value  => "${facts['networking']['fqdn']}/srv/app_mount",
  tag    => 'blog',
}
# Resource collector
App_config <<| tag == 'blog' |>>

Resource API

The modern resource API has one limitation: one can not refresh types!
Besides this it also consists of types and providers which must be placed at the same location as the existing implementation.

Resource API Type

The resource API type uses an attributes hash to list all parameters and properties. Within the type key one can use any of the built-in Puppet data types.

# lib/puppet/type/app_config.rb
require 'puppet/resource_api'

Puppet::ResourceApi.register_type(
  name: 'app_config',
  docs: <<-EOS,
      @summary The type to manage app config settings
      @example
      app_config { 'key':
        ensure => present,
        value  => 'value',
      }

      This type provides Puppet with the capabilities to manage our application config
    EOS
  features: []
  attributes: {
    ensure: {
      type:    'Enum[present, absent]',
      desc:    'Whether the setting should be added or removed',
      default: 'present',
    },
    name: {
      type:     'String',
      desc:     'The setting to manage',
      behavior: :namevar,
    },
    value: {
      type: 'Variant[String, Integer, Array]',
      desc: 'The value to set'
    }
  }
)

Respource API Provider

The most simple solution is to make use of the SimpleProvider class.
This provider needs up to 4 defines:

• get
• create
• update
• delete

# lib/puppet/provider/app_config/cli.rb
require 'puppet/resource_api/simple_provider'
require 'open3'

class Puppet::Provider::AppConfig::Cli < Puppet::ResourceApi::SimpleProvider
  $app_command = '/opt/app/bin/app.exe'

  # Get: fetching all readable config settings into a hash
  def get(context)
    context.debug('Returning data from command')

    @result = []
    stdout, stderr, status = Open3.capture3('/opt/app/bin/app.exe list')
    raise ParseError "Error running command: \n#{stderr}" unless status.success?
    # Missing error handling. This is just for demo
    @command_result = stdout.split(%r{\n})
    @command_result.each do |line|
      next if line == '---'
      key = line.split(':')[0]
      value = line.split(':')[1].strip
      hash = {
        'ensure': 'present',
        'name': key,
        'value': value,
      }
      @result += [ hash ]
    end
    @result
  end

  # Create: add a new config setting with name and should value
  def create(context, name, should)
    context.notice("Creating '#{name}' with #{should[:value]}")
    _stdout, _stderr, status = Open3.capture3("/opt/app/bin/app.exe set #{name} #{should[:value]}")
    # Missing error handling. This is just for demo
    return true unless status.success?
  end

  # Update: correct an existing setting with name and replace with should value
  def update(context, name, should)
    context.notice("Updating '#{name}' with #{should[:value]}")
    _stdout, _stderr, status = Open3.capture3("/opt/app/bin/app.exe set #{name} #{should[:value]}")
    # Missing error handling. This is just for demo
    return true unless status.success?
  end

  # Delete: remove a config setting
  def delete(context, name)
    context.notice("Deleting '#{name}'")
    _stdout, _stderr, status = Open3.capture3("/opt/app/bin/app.exe rm #{name}")
    # Missing error handling. This is just for demo
    return true unless status.success?
  end
end

Summary

Types and providers are not complex. They basically describe the CRUD behavior of Puppet.
The type defines how to use the Puppet DSL, while the provider controls how to check and handle specific settings.

Please stop using exec for almost anything that is not super simple. In most cases, a simple type and provider will allow better analysis, error handling and control over what is happening. Besides this: custom types and providers execute faster compared to exec type usage - far more faster!

The example application and the working types and providers are available on GitHub:

• The app: workshop demo app
• The Puppet module with types and providers for the app: workshop demo module

App usage:

# Install APP:
git clone https://github.com/betadots/workshop-demo-app /opt/app
/opt/app/bin/app.exe

# Install Puppet module:
git clone https://github.com/betadots/workshop-demo-module -b ruby_workshop modules/app
# Type API V1
puppet resource app_config --modulepath modules
# Type Resource API
puppet resource app_config2 --modulepath modules

Happy puppetizing,
Martin