January 2018 - Sasa Buklijas

Published on: 01.01.2018

Conclusion

Sometimes perfect is the enemy of the good.

Why use configuration file

I had Python program, that needed to access device via IP address and to do some diagnostic and commands on it.

At that time, I was accessing only one device (only one IP address).

But could see that in future (in few months to one year), I will need to do the same set of command on more devices.

One solution is to add IP address as a parameter to CLI program.

In my use case number of IP address that needs to be accessed will never be bigger than 34.

And writing 34 IP addresses as CLI parameter, that is around 373 letters, is not a nice solution.

When you need to read it, to see are all IP addressee included, it is not easy to read.

Python code as configuration was not possible

I was distributing my Python code as EXE, so use of Python code as configuration was not possible.

Altho, I think that Python code as the configuration is a good solution if you are executing source code, and only developer (not average user who does not know what Notepad is) will edit it.

Sometimes perfect is enemy of good

Sometimes I do have a problem that I tend to unnecessarily complicate things.

Because I think about all possible edge cases and all possible future uses.

And from these two, all possible future uses are the real problem.

Edge cases can happen, but is there a positive cost benefit to solve them and how?

This need to be determined on a case by case basis.

But “all possible future use” is trying to anticipate the future.

That is impossible.

From my experience, whenever I add code for future use cases, usually it was waste of time.

Even when I am the only user, so I can argue to my self that I know will need it.

Usually, I do not need it.

So, I have decided to eliminate waste when developing software, starting from this project.

For example, here I needed a configuration file that will have a list of IP addresses, which I will iterate in for loop.

This was the smallest requirement that I needed for my problem.

But immediately I was thinking that it would be nice to have:

checking if IP address is in the valid format
if I have 5 address that I need to write all 5, but I could add special syntax for that. Like 192.168.1.1 --- 192.168.1.5, and this can really get more complicated if you want to cover all edge cases.
I only need IPv4, but can I also ad IPv6
if there is some error in the configuration file, there can be a useful error message and suggested a solution
another configuration file can be added where I would define what are valid IP addresses, and then I can validate requested IP against valid IP

And the list can go on.

But I said to myself, NO.

You will just make code that can read a list of items as a list in Python from the configuration file.

Nothing more.

If when you have the real need, you will add additional features when needed.

When I think about this, maybe this is an example of premature design?

Premature design is deciding too early what a program should do.

Why JSON is better than Configparser for list datatype

After investigating possible solution, decision was to try ConfigParser.

ConfigParser configuration file was:

[other]
list_queue1 = [ "192.168.1.34", "192.168.1.35",
               192.168.1.36,
               "192.168.1.34" ]

[other]

list_queue1 = [ "192.168.1.34", "192.168.1.35",

192.168.1.36,

"192.168.1.34" ]

Code was:

import ConfigParser
import io

def get_as_list(text):
    _ = text.replace('[', '')
    _ = _.replace(']', '')
    _ = _.replace('"', '')
    tmp_list = [ item.strip() for item in _.split(',') ]
    return tmp_list

def main():
    # Load the configuration file
    with open("config_ini.ini") as f:
        sample_config = f.read()
    config = ConfigParser.RawConfigParser(allow_no_value=True)
    config.readfp(io.BytesIO(sample_config))

    # access them
    testing_1 = config.get('other', 'list_queue1')
    for item in get_as_list(testing_1):
        print item

if __name__ == '__main__':
    main()

import ConfigParser

import io

def get_as_list(text):

_ = text.replace('[', '')

_ = _.replace(']', '')

_ = _.replace('"', '')

tmp_list = [ item.strip() for item in _.split(',') ]

return tmp_list

def main():

# Load the configuration file

with open("config_ini.ini") as f:

sample_config = f.read()

config = ConfigParser.RawConfigParser(allow_no_value=True)

config.readfp(io.BytesIO(sample_config))

# access them

testing_1 = config.get('other', 'list_queue1')

for item in get_as_list(testing_1):

print item

if __name__ == '__main__':

main()

I found few problems with ConfigParser:

It did not have build in functionality to read data as Python List
- so even for this simple example I had to write custom code (this is why I have get_as_list() function)
There were no rules in format, eg. "192.168.1.36" and 192.168.1.36 was both valid.
- this was not big problem, but I just did not like it

Let try JSON (as configuration files in Python)

JSON configuration file was:

{
    "test_list":[
        "test_1",
        "test_2",
        "test_3"
    ]
}

{

"test_list":[

"test_1",

"test_2",

"test_3"

]

}

Code was:

import json

def main():
    with open('config_json.json') as json_data_file:
            data = json.load(json_data_file)

    for item in data['test_list']:
        print item

if __name__ == '__main__':
    main()

import json

def main():

with open('config_json.json') as json_data_file:

data = json.load(json_data_file)

for item in data['test_list']:

print item

if __name__ == '__main__':

main()

There is much less code, and line count is important.

One thing that I did not like about JSON is "ValueError: No JSON object could be decoded", this is error message that you get if you JSON is invalid.

I was hoping to get some more details, eg. what token in which line.

But you can not have it all, and ConfigParser was no bettter.

Decision was to use JSON as configuration file for Python, because code was less complicated (less lines of code).

I did not tried YAML.

Syntax, at least to me, is less readable than JSON.

Questions and remarks please leave in comments.

Sasa Buklijas

Month: January 2018

Always start with simple solution (ConfigParser vs. JSON for Python configuration file)