Tips & Tricks

Put your tips and tricks in this page with your credits

Retrieving a stack trace on request

AUTHOR: Charles Duffy

Touch the referenced file when you want to dump a stack trace for all running threads to stderr

# add in WSGI handler run by uwsgi

import sys
import os.path

stack_dump_file = '/tmp/stack-dump-trigger' ## given as an example; put this somewhere better than /tmp
uwsgi_signal_number = 17                    ## nothing magic about this number; just can't conflict with
                                            ## other uwsgi signals you define

try:
    import uwsgi
    import threading

    if not os.path.exists(stack_dump_file):
        open(stack_dump_file, 'w')

    def thread_dump(dummy_signum):
        print >>sys.stderr, "### BEGIN THREAD DUMP"
        thread_names = dict([(t.ident, t.name) for t in threading.enumerate()])
        for threadId, stack in sys._current_frames().items():
            print >>sys.stderr, "\n# ThreadID: %s (%s)" % (threadId, thread_names.get(threadId, 'NO_NAME_AVAILABLE'))
            for filename, lineno, name, line in traceback.extract_stack(stack):
                print >>sys.stderr, 'File: "%s", line %d, in %s' % (filename, lineno, name)
                if line:
                    print >>sys.stderr, " %s" % (line.strip())
        print >>sys.stderr, "### END THREAD DUMP"

    uwsgi.register_signal(uwsgi_signal_number, 'workers', thread_dump)
    uwsgi.add_file_monitor(uwsgi_signal_number, stack_dump_file)

    print >>sys.stderr, "Listening for stack dump requests via %r" % (stack_dump_file,)
except ImportError:
    print >>sys.stderr, "Not running under uwsgi; unable to configure stack dump trigger"
except IOError:
    print >>sys.stderr, "IOError creating stack dump trigger %r" % (stack_dump_file,)

Attaching a python shell to a running instance

AUTHOR: Yaniv Aknin

Call INTERACT() in your code whenever you want to interrupt execution and working in a python shell

class RestoredStandardInputContext(object):
    def __enter__(self):
        self.backup_stdin = os.dup(sys.stdin.fileno())
        os.dup2(sys.stdout.fileno(), sys.stdin.fileno())
    def __exit__(self, error_type, error, traceback):
        os.dup2(self.backup_stdin, sys.stdin.fileno())

def interact(locals=None, plain=False):
    with RestoredStandardInputContext():
        code.interact(local=locals or inspect.currentframe().f_back.f_locals)

__builtins__['INTERACT'] = interact

uWSGI + django autoreload mode

AUTHOR: Simone Federici

if you don't want lose the django auto reload by runserver command register this simple reload method

import uwsgi
from uwsgidecorators import timer
from django.utils import autoreload

@timer(3)
def change_code_gracefull_reload(sig):
    if autoreload.code_changed():
        uwsgi.reload()

now if you change your code it will be reloaded

Multi site system on single django project with uwsgi

AUTHOR:  Roman Koblov

Here is uWSGI Upstart script for ubuntu server:

description "uWSGI starter"

start on (local-filesystems
and runlevel [2345])
stop on runlevel [016]

respawn

exec /usr/local/bin/uwsgi --uid penpen \
-s 127.0.0.1:9010 -M -p 2 --reload-os-env \
--logto /home/penpen/logs/biribiri/uwsgi_log \
-H /home/penpen/biribiri/virtualenv \
--pythonpath /home/penpen --vhost --vhost-host

This script uses uWSGI virtual hosting mode, which allows multiple wsgi apps in single interpreter (unlike emperor mode, which uses interpreter per app (or site)). It is necessary to use “—reload-os-env” option, which passes environment variable through UWSGI_SETENV parameter. In this installation nginx passes current site name in uwsgi, which loads settings from module named as biribiri.hosts.SITE_NAME. For production use, it would be a nice idea to check if file exists and use default settings for non-existant sites. I use DJANGO_HOST instead of DJANGO_SETTINGS_MODULE because in current installation it overrides somewhere. Nginx config:

server {
    listen 80 default;
    server_name biribiri;
    root /home/penpen/biribiri;

    location / {
        uwsgi_pass 127.0.0.1:9010;
        uwsgi_param UWSGI_MODULE biribiri.production.wsgi;
        uwsgi_param UWSGI_SETENV DJANGO_HOST=biribiri.hosts.$http_host;
        include uwsgi_params;	
    }
}

wsgi.py connector for django (in biribiri.production.wsgi):

import os
import sys

path = '/home/penpen/biribiri'

if path not in sys.path:
    sys.path.append(path)

os.environ['DJANGO_SETTINGS_MODULE'] = os.environ['DJANGO_HOST'].replace('.ru', '')

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

In the above code snippet we are going to remove TLD from site name, because python modules don’t allow dot in their names.

A middleware for rewriting django urls for app mounted under subdirs in cherokee

from django.core.handlers.wsgi import WSGIHandler

djangoapp = WSGIHandler()

def application(e, sr):
    e['PATH_INFO'] = "%s%s" % (e['SCRIPT_NAME'], e['PATH_INFO'])
    e['SCRIPT_NAME'] = ''

    return djangoapp(e, sr)

Monitor multiple files and send a reload on changes

from uwsgidecorators import *

filemon('/tmp/foo')(uwsgi.reload)
filemon('/tmp/foo2')(uwsgi.reload)
filemon('/tmp/foo3')(uwsgi.reload)
filemon('/tmp/foo4')(uwsgi.reload)
filemon('/tmp/foo5')(uwsgi.reload)

Using screen to re-attach to a running uWSGI pyshell (1.0-dev)

create a session named uwsgi000 with a psyshell in it

screen -dmS uwsgi000 ./uwsgi --pyshell -p 8 -M --module welcome -s :3031

get the list of sessions

screen -ls

re-attach to a session:

screen -r uwsgi000

Zerg dance to test new code and fallback to the old one in case of emergency (uWSGI 1.0)

ZergMode is born to allows easy tuning of the number of workers, but you will find it very useful for other scopes too, like testing:

Start a uwsgi instance with the zerg server enabled

[uwsgi]
socket = 127.0.0.1:3031
master = true
processes = 8
wsgi-file = /var/www/mywsgiapp.wsgi
zerg-server = /tmp/mainzerg-server.socket
pidfile = maininstance.pid

After a bit of time, you would want to put new code in production. Even if you are a guru of test-driven development, shit will always happens, so you will leave the old instance running.

The first step is running a second instance as a zerg client

[uwsgi]
master = true
processes = 8
wsgi-file = /var/www/mywsgiapp.wsgi
zerg = /tmp/mainzerg-server.socket
pidfile = secondinstance.pid
zerg-server = /tmp/secondzerg-server.socket

(check the zerg-server option available even in the second instance. This is needed when the new instance will became the old one [that's the Zerg dance]) now the instances will both respond to requests. This is not what we want, so the next step is "pausing" the old instance:

uwsgi --pause maininstance.pid

Now the new instance is the only active one but....

SHIT a bug showed on our production site !!!

do not worry...

simply resume the old app

uwsgi --resume maininstance.pid

and stop the new one

uwsgi --stop secondinstance.pid

Now you are ready for bug-hunting again...

Troubleshooting

As usual, the first things to do is to check the logs. This implies:

• the web server log, which will indicate if it couldn't connect to the uWSGI process,
• the uWSGI log, which will indicate if an exception was thrown.

Typical gotchas:

• If the socket is a file, the Web server process should have read, write and execute permissions on the socket file. The --chmod-socket option can do it.
• In some cases, for instance if uWSGI was started without --vacuum or killed with SIGKILL, it won't remove the socket and pidfile when it is interrupted. It is safe to remove them manually (by the way, uWSGI will overwrite them on

next spawn)

• uWSGI can start the process on the foreground, this will make errors easily visible to the system administrator.

FastRouter performance tips

AUTHOR: Łukasz Mierzwa

If you are using nginx->fastrouter setup and both nginx and fastrouter are running on the same host you should connect nginx to fastrouter using file socket instead of tcp, using tcp will add more unnecessary load to tcp stack. So in nginx configuration use

uwsgi_pass  unix:///var/run/uwsgi.socket;

instead of IP:PORT

uwsgi_pass   127.0.0.1:3031;

and edit fastrouter config tu use:

fastrouter = /var/run/uwsgi.socket

When you want to handle high number of connections/requests you may need to tweak some linux sysctl settings, most important seems to be

• net.ipv4.tcp_max_syn_backlog
• net.core.somaxconn

look at  http://agiletesting.blogspot.com/2009/03/haproxy-and-apache-performance-tuning.html for details.

Running the Sentry server via uWSGI

By default the sentry server is executed in a dedicated http server (normally gunicorn). You can run it direcly in uWSGI too:

pip install sentry
sentry init /foo/bar/sentry.conf

now edit /foo/bar/sentry.conf for your db setup and add this line on the top:

from sentry.conf.server import *

finally use this config file

[uwsgi]
; choose the socket and protocol you need
; here we use the http protocol, but you can proxy it via uwsgi protocol as well
http-socket = :9090
; map the sentry config to a virtual module
pymodule-alias = my_sentry_conf=/foo/bar/sentry.conf
; use the virtual module as the django settings module
env = DJANGO_SETTINGS_MODULE=my_sentry_conf
; load sentry
module = sentry.wsgi

; spawn the master and 4 processes
master = true
processes = 4

pay attention to the pymodue-alias directive, it will allows you (via module aliasing) to map a raw file to a python module.

Always use absolute path names for the sentry config files (it is not strictly required, but will avoid mess with system files)

Profiling/tracing python apps

There are (starting from 1.2-dev) two profilers/tracers for python embedded in uWSGI. To enable one of them use

--profiler pycall

to enable tracing of function (c and python) calls

and

--profiler pyline

to enable tracing of each line of your code

The reported time deltas are in microseconds

Sharing virtualenv with your app and cron tasks

This  blog posts made me think about it

[uwsgi]
; our socket and basic concurrency setup ...
socket = :3031
master = true
processes = 4

; our django app
chdir = /var/www/app0
module = wsgi

; our virtualenv
virtualenv = /var/www/envs/001
; and our cron task every day at 2:59 using our virtualenv
; the cron tasks run from the specified chdir (the django app dir)
cron = 59 2 -1 -1 -1 %(virtualenv)/bin/python manage.py runmyfunnytask

Testing Python Modules That Use uwsgidecorators

When you execute code outside the uwsgi (for testing perhaps) importing the uwsgidecorator modules fails because uwsgi can't be imported. So, define your own decorators in that case. For example:

try:
    from uwsgidecorators import *
except:
    def postfork(f): return f

@postfork
def initialize():
    ...

Author: Marco in  http://osdir.com/ml/python-wsgi-uwsgi-general/2012-05/msg00064.html