Python - Dynamic Multidimensional Dictionary Keys

Published on Feb. 20, 2020, 4:29 a.m.

This morning I was writing a reset data script for a Django project I am working on. This is a fairly common script that I usually copy and paste from another project. But this time, I wanted to do something different. Here's the way it normally starts off:

#!/usr/bin/env bash

DBUSER='admin'
DBPASS='admin'
DBNAME='repositext'

...

I fill this info out at the top of my script and then the next routines are basically dropping the database, recreating it, running migrations and then creating a superuser. I usually do this reset data routine until I have something that basically just works. After that, I'll allow the migrations to accumulate like one normally would.

But anyways, the subject here is that I'm duplicating variables that I could extract from my Django's project settings.py file. So, I wrote a Python script that would pull in the all of the settings.py properties as a dict. I also wrote some functionality that would allow my reset-data.sh script to run like this to get the database info:

./project_info.py DATABASES default USER
./project_info.py DATABASES default PASSWORD 
./project_info.py DATABASES default NAME

Each call to this script will extract the database info from the dict of settings.py:

...

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'HOST': 'localhost',
        'NAME': 'repositext',
        'USER': 'repositext',
        'PASSWORD': 'admin',
        'PORT': '3306',
    }
}

...

I wrote the following code to extract this (not all of it but just the parts that are relevant):

#!/usr/bin/env python

import os
import sys

os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings'

import myproject.settings as settings # noqa E402

info = settings.__dict__

for arg in sys.argv[1:]:
    info = info[arg]
print(info)

The settings.dict should return everything from this file as a dictionary. I would then call what I needed with:

info[DATABASES]  # returns the whole DATABASES dictionary
info[DATABASES][default]  # returns the default (which is usually all I have in a project)
info[DATABASES][default][NAME]

and I would call the script from bash like so:

./project_info.py DATABASES default NAME

I was able to capture "DATBASES default NAME" as arguments to the script with sys.argv. So, at the beginning I had to assign these selections not so dyanamically like so:

key1 = sys.argv[1]
key2 = sys.argv[2]
key3 = sys.argv[3]

This was kind of ugly. What if my tree goes deeper than 3 levels? What if I only give one level as an argument. I then had to write all kinds of code-smelling handlers for what if statements:

key1 = sys.argv[1] if len(sys.argv) > 1 else None
key1 = sys.argv[2] if len(sys.argv) > 2 else None
key1 = sys.argv[3] if len(sys.argv) > 3 else None

This was actually the best I could come up with for a bit and I wasn't too satisfied with it. So, I continued to refactor this until I came to the realization I was also going to need to dynamically allocate keys for the info dictionary. I looked over Stackoverflow and other parts of the web to see if anyone had encountered this before. I really didn't find anything that I thought fit this case perfectly.

Eventually I found this:

for arg in sys.argv[1:]:
    info = info[arg]
print(info)

I would iterate through all sys.argv from the 2nd element on (remember that with sys.argv, the first element refers to the script itself) and then whittle down the dict (which is the variable "info" above) until I got the value I was looking for. The lone issue though is that info holds the full dict value and then on subsequent iterations becomes smaller. This may sound good but realize that every time you set a value for a string in Python, behind the scenes a new string variable is being created. Garbage collection should take care of this but I do wonder if there's not a more efficient way to do it. For now, it does the trick and since it's really a small script, it's not doing any harm. But, I do wonder for much larger data sets if there would be an issue. I guess I'll have to try it out on some really large json object to see how it does.

Anyways, if you happen to be looking for a way to dynamically increase your dictonary call's exactness, this may be a way to do it. If you have a better idea, feel free to contact me and let me know. I'll be happy to update this article with your solution and give you full credit.

Hope this was fun! - H.S.

If this blog is helpful, please consider helping me pay it backward with a coffee.

Buy Me a Coffee at ko-fi.com