Towards Effortless Python Configuration Files Version 1

Introduction

As noted in the previous article the simplistic version is rife with problems including scalability, maintainability and extensibility.

A simple extension from Version Ø is to try and hide the Python configuration details behind a property class. That is implement a pseudo data class that exposes a set of properties that allow a developer to simply do property set and get calls to retrieve and persist property values.

From the maintainers point of view this implementation should support the following capabilities.

1. Allow auto creation of configuration sections if they are missing
2. Allow auto creation of property values if they are missing
3. The properties should be implemented as both read-through and write-through.
4. In order to avoid the startup costs for the above as an application instantiates this class throughout the application this class should be a Singleton.

Class Representation

The following UML Class diagram describes a class that would meet the requirements in the introduction. The ConfiguratonProperties class meets requirements 1 & 2 with the protected methods .createMissingSections and .createMissingKeys

Create Implementations

Create Missing Sections

The following code shows the implementation. Notice that additional sections require code updates to this method

SECTION_GENERAL:  str = 'General'
SECTION_DATABASE: str = 'Database'

def _createMissingSections(self):
    """
    Create missing sections.  Add additional calls for
    each defined section
    """
    self._createMissingSection(SECTION_GENERAL)
    self._createMissingSection(SECTION_DATABASE)

The missing section code is as follows.

def _createMissingSection(self, sectionName: str):
    """
    Only gets created if it is missing
    Args:
        sectionName: The potential section to create
    """
    hasSection: bool = self._configParser.has_section(sectionName)
    self.logger.info(f'hasSection: {hasSection} - {sectionName}')
    if hasSection is False:
        self._configParser.add_section(sectionName)

Create Missing Keys

The following code shows the implementation. Again note if we add an additional section the developer must add an additional loop for the new section.

GENERAL_PREFERENCES: Dict[str, str] = {
    'debug':    'False',
    'logLevel': 'Info'
}

DATABASE_PREFERENCES: Dict[str, str] = {
    'dbName': 'example_db',
    'dbHost': 'localhost',
    'dbPort': '5432'
}

def _createMissingKeys(self):
    """
    Create missing keys and their values.  Add additional calls for
    each defined section.
    """
    for keyName, keyValue in GENERAL_PREFERENCES.items():
        self._createMissingKey(sectionName=SECTION_GENERAL, keyName=keyName, defaultValue=keyValue)
    for keyName, keyValue in DATABASE_PREFERENCES.items():
        self._createMissingKey(sectionName=SECTION_DATABASE, keyName=keyName, defaultValue=keyValue)

The missing key code is as follows. Notice any missing keys are immediately persisted.

def _createMissingKey(self, sectionName: str, keyName: str, defaultValue: str):
    """
    Only gets created if it is missing.  The configuration file is updated
    immediately for each missing key and its value

    Args:
        sectionName:   The section name where the key resides
        keyName:       The key name
        defaultValue:  Itsß value
    """
    if self._configParser.has_option(sectionName, keyName) is False:
        self._configParser.set(sectionName, keyName, defaultValue)
        self._saveConfiguration()

Class Properties

Sample implementations for requirement 3 follow.

String Properties

Notice that setting a property writes-through to the configuration file by setting the property and immediately persisting it. Reading properties are effectively read-through because of how we immediately write set properties.

@property
def dbName(self) -> str:
    return self._configParser.get(SECTION_DATABASE, 'dbName')

@dbName.setter
def dbName(self, newValue: str):
    self._configParser.set(SECTION_DATABASE, 'dbName', newValue)
    self._saveConfiguration()

Integer Properties

Integer properties use the .getint method to retrieve the value. When setting the property the developer must manually convert it to a string.

@property
def dbPort(self) -> int:
    return self._configParser.getint(SECTION_DATABASE, 'dbPort')

@dbPort.setter
def dbPort(self, newValue: int):
    self._configParser.set(SECTION_DATABASE, 'dbPort', str(newValue))
    self._saveConfiguration()

Boolean Properties

Boolean properties use the .getboolean method to retrieve their value. When setting the property the developer must manually convert it to a string.

SECTION_GENERAL:  str = 'General'

@property
def debug(self) -> bool:
    return self._configParser.getboolean(SECTION_GENERAL, 'debug')

@debug.setter
def debug(self, newValue: bool):
    self._configParser.set(SECTION_GENERAL, 'debug', str(newValue))
    self._saveConfiguration()

Enumeration Properties

I will not cover enumeration properties in this article. There are two ways to persist them, by their name or by their value. Each mechanism requires a slightly different way to deserialize the values back to an enumeration type.

Accessing and Modifying Properties

The following code snippet demonstrates how to access and modify the properties.

from logging import Logger
from logging import getLogger
from logging import basicConfig
from logging import INFO

LOGGER_NAME: str = 'Tutorial'

basicConfig(level=INFO)

config: ConfigurationProperties = ConfigurationProperties()

logger: Logger = getLogger(LOGGER_NAME)

logger.info(f'{config.debug=}')
logger.info(f'{config.logLevel=}')
#
logger.info('Change the values and show them')
#
config.debug = True
logger.info(f'{config.debug=}')
config.logLevel = 'Warning'
logger.info(f'{config.logLevel=}')
#
logger.info('**** DataBase Section ****')
logger.info(f'{config.dbName=}')
logger.info(f'{config.dbHost=}')
logger.info(f'{config.dbPort=}')
#
logger.info('Change db values and print them')
config.dbName = 'ozzeeDb'
config.dbHost = 'ozzeeHost'
config.dbPort = 6666

logger.info(f'{config.dbName=}')
logger.info(f'{config.dbHost=}')
logger.info(f'{config.dbPort=}')

The above snippet produces the following output

INFO:Tutorial:Configuration file existed
INFO:Tutorial:hasSection: True - General
INFO:Tutorial:hasSection: True - Database
INFO:Tutorial:config.debug=True
INFO:Tutorial:config.logLevel='Warning'
INFO:Tutorial:Change the values and show them
INFO:Tutorial:config.debug=True
INFO:Tutorial:config.logLevel='Warning'
INFO:Tutorial:**** DataBase Section ****
INFO:Tutorial:config.dbName='ozzeeDb'
INFO:Tutorial:config.dbHost='ozzeeHost'
INFO:Tutorial:config.dbPort=6666
INFO:Tutorial:Change db values and print them
INFO:Tutorial:config.dbName='ozzeeDb'
INFO:Tutorial:config.dbHost='ozzeeHost'
INFO:Tutorial:config.dbPort=6666

Conclusion

The source code for this article is here. The support class SingletonV3 is here

The result of the implementation initially left me satisfied as a consumer of the code. I was able to get and set typed properties. However, as the maintainer of the code I had to manually update code data structures and code loops whenever I added new sections and new properties. Additionally, all I really got from this is a mechanism/pattern to use whenever I needed new configuration properties in different applications.

Advantages

• Easy type safe access to application properties
• Invoking the singleton in different parts of my application provided consistent and reliable access to properties regardless which part of the application modified values

Disadvantages

• Updates to add new properties was tedious
• Lots of boiler plate code
• No reusability across various applications. Essentially, I just had a template

See my next post that documents an alternate implementation to address the disadvantages I listed, while keeping the advantages.