Radio Shows As Podcasts: Part 3 – Daemonisation

Note: this post was original published before I’d had a chance to write up Part 2, which discussed how I generated the XML for the podcast feed, hence the out of order numbering. It also got way bigger than I initially intended for it to be, hence it being broken up into two parts!

The hardest part in making this programme has been finding out how to run a Python script as a service on the Raspberry Pi. This is known as making the programme a daemon. Achieving this required three steps:

  1. Writing an init.d script to launch the Python script, allow it to be controlled via service service name start|stop|status commands, and to allow the script to be ran at start up. My explanation of how to modify the skeleton init.d file can be found here, and the init.d script itself can be found here
  2. Creating a way to run the Python code every x minutes, a time period set by the user (and stored in the configuration file). My (rather long) explanation of how I figured out how to achieve this with the threading Python module can be found here, and the Python scripts I ended up using can be found here.
  3. Re-arranging and editing the code so that the Python files could be imported properly into the new file. Some of the important steps I undertook are outlined below, below the fold.

All told, achieving the daemonisation of my programme took about 10 hours. If it wasn’t for missing obvious mistakes it would probably have taken less time than that. Perhaps the most important lesson I’ve learnt is to NOT USE RELATIVE PATHS, especially not when writing scripts which might be ran by any user.

I’m going to leave the feature-demonise fork open on GitHub for the time being, to allow some other people to look at, comment, edit and contribute to the code. I’ve now closed & deleted the feature-demonise fork on GitHub, as I’m happy with how everything is working.

Modifying the Python Scripts

I had to edit my Python scripts in xxx main ways: I had to introduce the code to schedule the existing functions, and write a new function to run the scheduler; I had to introduce an if __name__ == "__main__" block to the imported files to ensure the code ran properly; I had to attempt to find out what caused the sudo service ipodcasts start command to fail and fix it; and I had to fix various other errors. Of these, the middle two edits are of the most interesting, and I’m still looking for ways in which the last might have been found easier.

The code which introduced the scheduling was pretty much just an editing of the Test Scheduler I’ve linked to else where.

The __name__ Variable

__name__ is a special variable which takes one of two values:

  • __main__ if the module is being ran as a script (i.e. python
  • The module name if it is being imported, i.e. import myscript results in __name__ having the value of “myscript”

This means that one can place code within the if __name__ == "__main__" block that should only be executed if the module is ran as a script, and any code outside of this block will be ran if and when the module is imported.

I was not using this block on the earlier versions of my programme, which meant that all of the code was being ran when the modules were imported, rather than when I actually wanted the functions included in the modules to be executed. A rearrange of when the functions were called (and consequently which variables were passed to the functions), meant that the code would be executed in the correct order.[1]

Failure to Launch

I knew that my init.d script worked properly, and that the code to schedule the Python code worked properly, as I had written these separately and they only required minor modifications in order to point to the correct files and functions, respectively. Yet I could not get the programme to launch properly as a service. It would provide me with a positive starting message, yet would not stay active. i.e. It was crashing straight away. Unfortunately I was not receiving any log output, nor any errors on the stdout of Terminal.

In order to figure out what was causing this, I copied a limited amount of code into a toy test file, and slowly introduced more and more of the full programme’s features. What struct me as odd, and caused a large part of the problems, was that the script would run fine when launched through the Python interpreter, i.e. python yet the daemon would fail to launch.

I had narrowed it down to something relating to the settings.conf file, and believed it to be some kind of permissions issue (as I had received permission issues when first trying to run the script from the command line). This, however, turned out to be a red herring. What I eventually realised (and should have much sooner), was that I was still using a relative path to the settings file. This meant that when the daemon was initiated via the sudo service ipodcasts start command, the programme was looking for the settings file at /settings.conf, rather than /ipodcasts/settings.conf.

More Learning to be Done

What annoyed me the most about the relative path error above, was that I had no way of identifying what was causing the error. My google-fu was sadly lacking when searching for a way to produce the errors, which might be caused by any part of the programme, when the script is being ran either by a different user, or by “from” a different location. At best, I found this nice little explanation of how to log the traceback information due to an exception, but without knowing where or when to place the try: ... except: block, this wasn’t much use.

More frustratingly, I had identified that the opening of the settings file was causing some problem before I realised it was the relative path specifically which was the problem, yet could not figure out how to change the code from a while open() as f block, to a try: .. except: block, which probably added another hour onto my diagnosis and solving of the problem.

Really the problem seems to be that when running a Python script as a daemon, the stdout and stderr are hidden, which means that errors, warnings and traceback out output which would usually appear in the Terminal window do not. I could not find a way to catch this output quickly enough, before identifying the cause of my issues.

Any guidance or help in how to address this sort of problem in the future would be great.

[1] I learnt how to use the __name__ variable from this Stack Overflow answer, and by writing a couple of toy example scripts to see when variables would be printed, and how best to pass variables between them.

This entry was posted in Programming, Python. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *