At work I've created a number of Twisted applications for handling various internal services. Unlike my TurboGears applications, which I package as eggs to install using easy_install (provided by setuptools) I have no nice way to deploy my Twisted apps.

Until now.

Twisted provides a nice plugin system that allows an application to plug itself into the "twistd" command-line application starter. When properly packaged a Twisted application can be automatically plugged into the Twisted world at installation time and started by using twistd.

The only trouble is that there is no documentation for how to package a Twisted application so it can be deployed in this way.

Here I try to provide some documentation by showing an example of what is required to package a simple Twisted application. In fact, I will take the Twisted finger tutorial and write what I consider to be Step 12: "How to package the finger service as an installable Twisted application plugin for twistd" (aka "The missing step").

Step 12: How to package the finger service as an installable Twisted application plugin for twistd

Create a directory structure like this:

finger
finger/__init__.py
finger/finger.py
MANIFEST.in
setup.py
twisted
twisted/plugins
twisted/plugins/finger_plugin.py

finger/finger.py is the finger application from http://twistedmatrix.com/projects/core/documentation/howto/tutorial/index.html packaged as finger.

twisted/plugins is a directory structure containing the finger_plugin.py file that will be described below. Note that there must be no __init__.py files within twisted and twisted/plugins.

finger_plugin.py provides a class implementing the IServiceMaker and IPlugin interfaces. Basically, this is the plugin point that defines the services the application will provide and any command-line options that it supports.

# ==== twisted/plugins/finger_plugin.py ====
# - Zope modules -
from zope.interface import implements

# - Twisted modules -
from twisted.python import usage
from twisted.application.service import IServiceMaker
from twisted.plugin import IPlugin

# - Finger modules -
from finger import finger

class Options(usage.Options):
    synopsis = "[options]"
    longdesc = "Make a finger server."
    optParameters = [
        ['file', 'f', '/etc/users'],
        ['templates', 't', '/usr/share/finger/templates'],
        ['ircnick', 'n', 'fingerbot'],
        ['ircserver', None, 'irc.freenode.net'],
        ['pbport', 'p', 8889],
    ]
    
    optFlags = [['ssl', 's']]

class MyServiceMaker(object):
    implements(IServiceMaker, IPlugin)
    
    tapname = "finger"
    description = "Finger server."
    options = Options
    
    def makeService(self, config):
        return finger.makeService(config)

serviceMaker = MyServiceMaker()

setup.py is the standard distutils setup.py file. Take note of the "packages" and "package_data" arguments to setup(). Also note the refresh_plugin_cache() function which is called after setup() completes. This forces a refresh of the Twisted plugins cache (twisted/plugins/dropin.cache).

# ==== twisted/plugins/finger_plugin.py ====
'''setup.py for finger.

This is an extension of the Twisted finger tutorial demonstrating how
to package the Twisted application as an installable Python package and
twistd plugin (consider it "Step 12" if you like).

Uses twisted.python.dist.setup() to make this package installable as
a Twisted Application Plugin.

After installation the application should be manageable as a twistd
command.

For example, to start it in the foreground enter:
$ twistd -n finger

To view the options for finger enter:
$ twistd finger --help
'''

__author__ = 'Chris Miles'


import sys

try:
    import twisted
except ImportError:
    raise SystemExit("twisted not found.  Make sure you "
                     "have installed the Twisted core package.")

from distutils.core import setup

def refresh_plugin_cache():
    from twisted.plugin import IPlugin, getPlugins
    list(getPlugins(IPlugin))

if __name__ == '__main__':
    
    if sys.version_info[:2] >= (2, 4):
        extraMeta = dict(
            classifiers=[
                "Development Status :: 4 - Beta",
                "Environment :: No Input/Output (Daemon)",
                "Programming Language :: Python",
            ])
    else:
        extraMeta = {}

    setup(
        name="finger",
        version='0.1',
        description="Finger server.",
        author=__author__,
        author_email="you@email.address",
        url="http://twistedmatrix.com/projects/core/documentation/howto/tutorial/index.html",
        packages=[
            "finger",
            "twisted.plugins",
        ],
        package_data={
            'twisted': ['plugins/finger_plugin.py'],
        },
        **extraMeta)
    
    refresh_plugin_cache()

MANIFEST.in contains one line, which I assume tells distutils to modify the existing Twisted package (to install twisted/plugin/finger_plugin.py) or something like that.

graft twisted

With all that in place you can install the package the usual way,

$ python setup.py install

Then you should be able to run twistd to see and control the application. See the twistd options and installed Twisted applications with:

$ twistd --help
Usage: twistd [options]
 ...
Commands:
    athena-widget      Create a service which starts a NevowSite with a single
                       page with a single widget.
    ftp                An FTP server.
    telnet             A simple, telnet-based remote debugging service.
    socks              A SOCKSv4 proxy service.
    manhole-old        An interactive remote debugger service.
    portforward        A simple port-forwarder.
    web                A general-purpose web server which can serve from a
                       filesystem or application resource.
    inetd              An inetd(8) replacement.
    vencoderd          Locayta Media Farm vencoderd video encoding server.
    news               A news server.
    words              A modern words server
    toc                An AIM TOC service.
    finger             Finger server.
    dns                A domain name server.
    mail               An email service
    manhole            An interactive remote debugger service accessible via
                       telnet and ssh and providing syntax coloring and basic
                       line editing functionality.
    conch              A Conch SSH service.

View the options specific to the finger server:

$ twistd finger --help
Usage: twistd [options] finger [options]
Options:
  -s, --ssl         
  -f, --file=       [default: /etc/users]
  -t, --templates=  [default: /usr/share/finger/templates]
  -n, --ircnick=    [default: fingerbot]
      --ircserver=  [default: irc.freenode.net]
  -p, --pbport=     [default: 8889]
      --version     
      --help        Display this help and exit.

Make a finger server.

Start the finger server (in the foreground) with:

$ sudo twistd -n finger --file=users
2007/12/23 22:12 +1100 [-] Log opened.
2007/12/23 22:12 +1100 [-] twistd 2.5.0 (/Library/Frameworks/Python.framework/
Versions/2.5/Resources/Python.app/Contents/MacOS/Python 2.5.0) starting up
2007/12/23 22:12 +1100 [-] reactor class: <class 'twisted.internet.selectreactor.SelectReactor'>
2007/12/23 22:12 +1100 [-] finger.finger.FingerFactoryFromService starting on 79
2007/12/23 22:12 +1100 [-] Starting factory <finger.finger.FingerFactoryFromService instance at 0x1d0a4e0>
2007/12/23 22:12 +1100 [-] twisted.web.server.Site starting on 8000
2007/12/23 22:12 +1100 [-] Starting factory <twisted.web.server.Site instance at 0x1d0a558>
2007/12/23 22:12 +1100 [-] twisted.spread.pb.PBServerFactory starting on 8889
2007/12/23 22:12 +1100 [-] Starting factory <twisted.spread.pb.PBServerFactory instance at 0x1d0a670>
2007/12/23 22:12 +1100 [-] Starting factory <finger.finger.IRCClientFactoryFromService instance at 0x1d0a5f8>

twistd provides many useful options, such as daemonizing the application, specifying the logfile and pidfile locations, etc.

Unfortunately Twisted and setuptools don't play nicely together, so I'm not able to package my Twisted app as an egg, take advantage of the setuptools package dependency resolution system, or install it using easy_install.

References:

http://twistedmatrix.com/projects/core/documentation/howto/plugin.html

http://twistedmatrix.com/projects/core/documentation/howto/tap.html

http://twistedmatrix.com/projects/core/documentation/howto/tutorial/index.html