Seamless Restarts With Multi-mode WSGI

When running many KARLs behind one Apache, we need a way to update and restart one specific site without doing a graceful restart on all the rest. The restart might require coming up in a different “mode”.

The Problem

We are moving to a model where all the KARLs run on one box, behind one Apache, in different WSGI app daemon processes. In a model like this, we need a way to update the software for one particular site, possibly displaying the maintenance page if it isn’t a “zero downtime” update.

In theory, we could simply do a graceful restart and the single site would change, as intended, while the other sites popped right back up. But we’d prefer not to do that. First, it would mean a lot more graceful restarts impacting each KARL site. This shouldn’t be a problem, but it’s still a lot of stress. Second, the first request to a view is slow, and it would be nice not to have those hit.

Of course, we’re able to make just one WSGI app reload. But the maintenance page is served by an Apache conf setting. We need a way to get a specific WSGI app into, and out of, various modes without restarting Apache.

Fortunately modwsgi gives us plenty of choices on this.

Proposal

• Change the modwsgi app to be “multi-mode”. Based on some flag, the WSGI process will behave differently on startup.
• Instead of graceful restarts, simply touch the .wsgi file (changing the timestamp) to force an application reload.
• The .wsgi app will either, inside itself or in a companion module, detect which mode to restart into.
• One mode will be, serve up the maintenance page. This mode will have access to the karl.ini file, thus knowing how to set the path to the static resources. (Those resources will have to be served up same as in the KARL app.)
• Another mode might be “read-only mode”. Allow the site to continue operating, but not accept changes.
• Design this in a buildout-friendly way. For example, if the .wsgi app is auto-generated by buildout and blown away at the start of running buildout, that’s a problem, except if we switch to Tres’s proposed deployment approach where buildout is run in a new spot.