Always start with simple solution (ConfigParser vs. JSON for Python configuration file)

Published on: 01.01.2018

Conclusion

Sometimes perfect is the enemy of the good.

Why use configuration file

I had Python program, that needed to access device via IP address and to do some diagnostic and commands on it.

At that time, I was accessing only one device (only one IP address).

But could see that in future (in few months to one year), I will need to do the same set of command on more devices.

One solution is to add IP address as a parameter to CLI program.

In my use case number of IP address that needs to be accessed will never be bigger than 34.

And writing 34 IP addresses as CLI parameter, that is around 373 letters, is not a nice solution.

When you need to read it, to see are all IP addressee included, it is not easy to read.

Python code as configuration was not possible

I was distributing my Python code as EXE, so use of Python code as configuration was not possible.

Altho, I think that Python code as the configuration is a good solution if you are executing source code, and only developer (not average user who does not know what Notepad is) will edit it.

Sometimes perfect is enemy of good

Sometimes I do have a problem that I tend to unnecessarily complicate things.

Because I think about all possible edge cases and all possible future uses.

And from these two, all possible future uses are the real problem.

Edge cases can happen, but is there a positive cost benefit to solve them and how?

This need to be determined on a case by case basis.

But “all possible future use” is trying to anticipate the future.

That is impossible.

From my experience, whenever I add code for future use cases, usually it was waste of time.

Even when I am the only user, so I can argue to my self that I know will need it.

Usually, I do not need it.

So, I have decided to eliminate waste when developing software, starting from this project.

For example, here I needed a configuration file that will have a list of IP addresses, which I will iterate in for loop.

This was the smallest requirement that I needed for my problem.

But immediately I was thinking that it would be nice to have:

  • checking if IP address is in the valid format
  • if I have 5 address that I need to write all 5, but I could add special syntax for that. Like 192.168.1.1 --- 192.168.1.5, and this can really get more complicated if you want to cover all edge cases.
  • I only need IPv4, but can I also ad IPv6
  • if there is some error in the configuration file, there can be a useful error message and suggested a solution
  • another configuration file can be added where I would define what are valid IP addresses, and then I can validate requested IP against valid IP

And the list can go on.

But I said to myself, NO.

You will just make code that can read a list of items as a list in Python from the configuration file.

Nothing more.

If when you have the real need, you will add additional features when needed.

When I think about this, maybe this is an example of premature design?

Premature design is deciding too early what a program should do.

Why JSON is better than Configparser for list datatype

After investigating possible solution, decision was to try ConfigParser.

ConfigParser configuration file was:

Code was:

I found few problems with ConfigParser:

  1. It did not have build in functionality to read data as Python List
    • so even for this simple example I had to write custom code (this is why I have get_as_list() function)
  2. There were no rules in format, eg. "192.168.1.36" and 192.168.1.36 was both valid.
    • this was not big problem, but I just did not like it

Let try JSON (as configuration files in Python)

JSON configuration file was:

Code was:

There is much less code, and line count is important.

One thing that I did not like about JSON is "ValueError: No JSON object could be decoded", this is error message that you get if you JSON is invalid.

I was hoping to get some more details, eg. what token in which line.

But you can not have it all, and ConfigParser was no bettter.

Decision was to use JSON as configuration file for Python, because code was less complicated (less lines of code).

I did not tried YAML.

Syntax, at least to me, is less readable than JSON.

Questions and remarks please leave in comments.

7 comments

  1. “I did not tried YAML.”

    A shame, since it requires less punctuation than the JSON version *and* allows comments. There’s also stuff like anchors & references for when you need several structures with common parts.

  2. how about just a file with one IP per line and use this to load them:

    with open(‘config’) as data:
    lines = data.readlines()
    items = [i.strip() for i in lines if not i.isspace()]

    1. You are correct, that is the simplest way.

      What did I not disclose in the article is that I also wanted somehow to name data.

      I wanted this because I also know that in the configuration file, expect IP addressed I will also need to store sleep duration.

      I just find it more articulated if I also have a name for data.

      Now I use https://github.com/pasztorpisti/json-cfg for storing configuration as JSON. json-cfg has the ability to add comments to JSON file.

  3. Yes, I guess one doesn’t need the complexity until you do. I always used a small function for parsing command line arguments until recently when I folded and started using argparse because arguments kept getting more complex.

    Another configuration library worth checking out is Universal Configuration Library: https://github.com/vstakhov/libucl

  4. Hi,
    Your ConfigParser example seems overly complex.

    You’re making life more difficult by trying to parse something that looks like a python list, just accept ips seperated by whitespace – like this:

    [other]
    list_queue1 = 192.168.1.34 192.168.1.35 192.168.1.36 192.168.1.34

    It’s more ergnomic for your users and parsing is now super simple. If you use shlex.split you can support any whitespace that is expected in the shell too.

    Maybe you have reasons for using the raw reading in ConfigParser and converting from bytes, but if you don’t ConfigParser can handle opening the file and give you back a string.

    Here is a whole working version for the file you provided:

    Notebook –
    https://gist.github.com/stuaxo/a6af1cda64e7ae04a214efc8e10ff877

    import ConfigParser
    import io
    import shlex

    def main():
    config = Config = ConfigParser.ConfigParser(allow_no_value=True)
    config.read(‘config_ini.ini’)
    testing_1 = shlex.split(config.get(‘other’, ‘list_queue1’))

    if __name__ == ‘__main__’:
    main()

    1. I prefer to have each IP address on the separate line, it is easier to read them for me.

      If somebody is fine with having an IP address separated by whitespace, your example is perfect.

Join the Conversation

Your email address will not be published. Required fields are marked *