An arguable drawback with SMF is that you have to define your service configuration with an XML file, called a service manifest. I think it would be fair to say that most people do what I used to do: copy an existing manifest and change the relevant bits. A simple but practical method, admittedly, but I decided it could be improved upon.
For that reason I recently put together a little tool called Manifold. It is a simple command-line tool, written in Python, that creates the SMF manifest for you after asking you some questions about the service.
The best way to explain what it does is with a demonstration. Here I will use Manifold to create an SMF manifest for memcached, showing how to validate the result and create the service with it.
Using manifold to create an SMF manifest for memcached is easy. Give it an output filename, then it will prompt for all the answers it needs to create the manifest.
$ manifold memcached.xml
The service category (example: 'site' or '/application/database') [site]
The name of the service, which follows the service category
(example: 'myapp') [] memcached
The version of the service manifest (example: '1') [1]
The human readable name of the service
(example: 'My service.') [] Memcached
Can this service run multiple instances (yes/no) [no] ? yes
Enter value for instance_name (example: default) [default]
Full path to a config file; leave blank if no config file
required (example: '/etc/myservice.conf') []
The full command to start the service; may contain
'%{config_file}' to substitute the configuration file
(example: '/usr/bin/myservice %{config_file}') [] /opt/memcached/bin/memcached -d
The full command to stop the service; may specify ':kill' to let
SMF kill the service processes automatically
(example: '/usr/bin/myservice_ctl stop' or ':kill' to let SMF kill
the service processes automatically) [:kill]
Choose a process management model:
'wait' : long-running process that runs in the foreground (default)
'contract' : long-running process that daemonizes or forks itself
(i.e. start command returns immediately)
'transient' : short-lived process, performs an action and ends quickly
[wait] contract
Does this service depend on the network being ready (yes/no) [yes] ?
Should the service be enabled by default (yes/no) [no] ?
The user to change to when executing the
start/stop/refresh methods (example: 'webservd') [] webservd
The group to change to when executing the
start/stop/refresh methods (example: 'webservd') [] webservd
Manifest written to memcached.xml
You can validate the XML file with "svccfg validate memcached.xml"
And create the SMF service with "svccfg import memcached.xml"View the resulting manifest:
$ cat memcached.xml
<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">
<!--
Created by Manifold
--><service_bundle type="manifest" name="memcached">
<service name="site/memcached" type="service" version="1">
<dependency name="network" grouping="require_all" restart_on="error" type="service">
<service_fmri value="svc:/milestone/network:default"/>
</dependency>
<instance name="default" enabled="false">
<method_context>
<method_credential user="webservd" group="webservd"/>
</method_context>
<exec_method type="method" name="start" exec="/opt/memcached/bin/memcached -d" timeout_seconds="60"/>
<exec_method type="method" name="stop" exec=":kill" timeout_seconds="60"/>
<property_group name="startd" type="framework">
<propval name="duration" type="astring" value="contract"/>
<propval name="ignore_error" type="astring" value="core,signal"/>
</property_group>
<property_group name="application" type="application">
</property_group>
</instance>
<stability value="Evolving"/>
<template>
<common_name>
<loctext xml:lang="C">
Memcached
</loctext>
</common_name>
</template>
</service>
</service_bundle>Now validate the manifest and use it to create the SMF service:
$ svccfg validate memcached.xml $ sudo svccfg import memcached.xml $ svcs memcached STATE STIME FMRI disabled 9:52:18 svc:/site/memcached:default
The service can be started and controlled using svcadm:
$ sudo svcadm enable memcached $ svcs memcached STATE STIME FMRI online 9:52:53 svc:/site/memcached:default $ ps auxw | grep memcached webservd 16098 0.0 0.1 2528 1248 ? S 09:52:53 0:00 /opt/memcached/bin/memcached -d
Find more information at the Manifold project page or download Manifold from pypi.
It is called "egg:Paste#watch_threads" (if you can call that a name) and obviously it is a feature of Paster (so if you serve your Pylons app via mod_wsgi, for example, you wouldn't be able to use it; not that it is recommended to enable it in a production environment given the information/power it exposes).
Enabling it for a Pylons app is simply a matter of modifying the config file (development.ini). It took me a bit of scanning of the Paster docs to work out how to get the config correct, so I'll share the simple magic here.
You need to replace this part of the Pylons config (e.g. development.ini):
[app:main] use = egg:Myapp full_stack = true
with this:
[composite:main] use = egg:Paste#urlmap / = myapp /.tracker = watch_threads [app:watch_threads] use = egg:Paste#watch_threads allow_kill = true [app:myapp] use = egg:Myapp full_stack = true #... rest of app config ...
What we are doing is replacing the main app with a composite app. The composite app uses "egg:Paste#urlmap" to mount the Pylons app at "/" while also mounting the "watch_threads" app at "/.tracker" (use whatever path you like; I borrowed from the examples I found).
So now if you fire up the Pylons application it should behave like normal, but you should also be able to browse to "/.tracker" (e.g. http://127.0.0.1:5000/.tracker) to see the active request thread debugger.
Below is a screenshot demonstrating watch_threads examining a Pylons app I was working on. Two threads are active; the request/WSGI environment is being shown for one of them.
Luckily, this is easily fixed so keep reading.
You can tell when readline isn't going to be included by examining the end of the make output. You will see something like this:
Failed to find the necessary bits to build these modules: _gdbm ossaudiodev readline spwd To find the necessary bits, look in setup.py in detect_modules() for the module's name.
The steps below detail my method for adding readline (and gdbm which you can skip if you don't want it) support to Python 3.0 (this probably works with other Python versions too).
Firstly, install the readline and gdbm libraries. One of the easiest ways to do that is to use MacPorts (aka DarwinPorts). If you don't have it already you can download the MacPorts installer to set things up. Once that is done then open Terminal/iTerm and enter:
$ sudo port install readline $ sudo port install gdbm
If that works, then you are ready to build Python. Get the Python 3.0 source code and unpack it. You need to tell setup.py where to find the libraries you installed. MacPorts (usually) installs all of the software it manages in /opt/local/ so in setup.py find the two lines:
add_dir_to_list(self.compiler.library_dirs, '/usr/local/lib') add_dir_to_list(self.compiler.include_dirs, '/usr/local/include')
and add two similar lines before them that point to /opt/local/lib and /opt/local/include, like:
add_dir_to_list(self.compiler.library_dirs, '/opt/local/lib') add_dir_to_list(self.compiler.include_dirs, '/opt/local/include') add_dir_to_list(self.compiler.library_dirs, '/usr/local/lib') add_dir_to_list(self.compiler.include_dirs, '/usr/local/include')
Now you can configure and build Python.
$ ./configure --enable-framework MACOSX_DEPLOYMENT_TARGET=10.5 --with-universal-archs=all $ make $ make test $ sudo make frameworkinstall
Note that if you've got any other non-Apple distributed versions of Python installed and want to keep the default version as it was, use (for example, to revert default back to 2.5):
$ cd /Library/Frameworks/Python.framework/Versions/ $ sudo rm Current && sudo ln -s 2.5 Current
Finally, so that the command "python3.0" works from the command-line, you need to either add /Library/Frameworks/Python.framework/Ver
$ sudo ln -s /Library/Frameworks/Python.framework/Versions/3.0/bin/python3.0 /usr/local/bin/ $ sudo ln -s /Library/Frameworks/Python.framework/Versions/3.0/bin/2to3 /usr/local/bin/
It is called Zoner and differs from many DNS management interfaces in that it works directly with live zone files. The zone files remain the master copy of domain details and can still be edited manually without effecting Zoner, as opposed to storing the domain structure in a database and generating zone files when needed (or reconfiguring bind to read directly from SQL). It also stores an audit trail for all changes (made through Zoner) and zones can be rolled back to any previous version.
Zoner might also be a useful reference app for anyone learning TurboGears 1.0. It is relatively simple, uses SQLAlchemy and Kid with Paginate and Form widgets.
Until now.
Twisted provides a nice plugin system that allows an application to plug itself into the "twistd" command-line application starter. When properly packaged a Twisted application can be automatically plugged into the Twisted world at installation time and started by using twistd.
The only trouble is that there is no documentation for how to package a Twisted application so it can be deployed in this way.
Here I try to provide some documentation by showing an example of what is required to package a simple Twisted application. In fact, I will take the Twisted finger tutorial and write what I consider to be Step 12: "How to package the finger service as an installable Twisted application plugin for twistd" (aka "The missing step").
Step 12: How to package the finger service as an installable Twisted application plugin for twistd
Create a directory structure like this:
finger finger/__init__.py finger/finger.py MANIFEST.in setup.py twisted twisted/plugins twisted/plugins/finger_plugin.py
finger/finger.py is the finger application from http://twistedmatrix.com/projects/c
twisted/plugins is a directory structure containing the finger_plugin.py file that will be described below. Note that there must be no __init__.py files within twisted and twisted/plugins.
finger_plugin.py provides a class implementing the IServiceMaker and IPlugin interfaces. Basically, this is the plugin point that defines the services the application will provide and any command-line options that it supports.
# ==== twisted/plugins/finger_plugin.py ====
# - Zope modules -
from zope.interface import implements
# - Twisted modules -
from twisted.python import usage
from twisted.application.service import IServiceMaker
from twisted.plugin import IPlugin
# - Finger modules -
from finger import finger
class Options(usage.Options):
synopsis = "[options]"
longdesc = "Make a finger server."
optParameters = [
['file', 'f', '/etc/users'],
['templates', 't', '/usr/share/finger/templates'],
['ircnick', 'n', 'fingerbot'],
['ircserver', None, 'irc.freenode.net'],
['pbport', 'p', 8889],
]
optFlags = [['ssl', 's']]
class MyServiceMaker(object):
implements(IServiceMaker, IPlugin)
tapname = "finger"
description = "Finger server."
options = Options
def makeService(self, config):
return finger.makeService(config)
serviceMaker = MyServiceMaker()setup.py is the standard distutils setup.py file. Take note of the "packages" and "package_data" arguments to setup(). Also note the refresh_plugin_cache() function which is called after setup() completes. This forces a refresh of the Twisted plugins cache (twisted/plugins/dropin.cache).
# ==== twisted/plugins/finger_plugin.py ====
'''setup.py for finger.
This is an extension of the Twisted finger tutorial demonstrating how
to package the Twisted application as an installable Python package and
twistd plugin (consider it "Step 12" if you like).
Uses twisted.python.dist.setup() to make this package installable as
a Twisted Application Plugin.
After installation the application should be manageable as a twistd
command.
For example, to start it in the foreground enter:
$ twistd -n finger
To view the options for finger enter:
$ twistd finger --help
'''
__author__ = 'Chris Miles'
import sys
try:
import twisted
except ImportError:
raise SystemExit("twisted not found. Make sure you "
"have installed the Twisted core package.")
from distutils.core import setup
def refresh_plugin_cache():
from twisted.plugin import IPlugin, getPlugins
list(getPlugins(IPlugin))
if __name__ == '__main__':
if sys.version_info[:2] >= (2, 4):
extraMeta = dict(
classifiers=[
"Development Status :: 4 - Beta",
"Environment :: No Input/Output (Daemon)",
"Programming Language :: Python",
])
else:
extraMeta = {}
setup(
name="finger",
version='0.1',
description="Finger server.",
author=__author__,
author_email="you@email.address",
url="http://twistedmatrix.com/projects/core/documentation/howto/tutorial/index.html",
packages=[
"finger",
"twisted.plugins",
],
package_data={
'twisted': ['plugins/finger_plugin.py'],
},
**extraMeta)
refresh_plugin_cache()
MANIFEST.in contains one line, which I assume tells distutils to modify the existing Twisted package (to install twisted/plugin/finger_plugin.py) or something like that.
graft twisted
With all that in place you can install the package the usual way,
$ python setup.py install
Then you should be able to run twistd to see and control the application. See the twistd options and installed Twisted applications with:
$ twistd --help
Usage: twistd [options]
...
Commands:
athena-widget Create a service which starts a NevowSite with a single
page with a single widget.
ftp An FTP server.
telnet A simple, telnet-based remote debugging service.
socks A SOCKSv4 proxy service.
manhole-old An interactive remote debugger service.
portforward A simple port-forwarder.
web A general-purpose web server which can serve from a
filesystem or application resource.
inetd An inetd(8) replacement.
vencoderd Locayta Media Farm vencoderd video encoding server.
news A news server.
words A modern words server
toc An AIM TOC service.
finger Finger server.
dns A domain name server.
mail An email service
manhole An interactive remote debugger service accessible via
telnet and ssh and providing syntax coloring and basic
line editing functionality.
conch A Conch SSH service.
View the options specific to the finger server:
$ twistd finger --help
Usage: twistd [options] finger [options]
Options:
-s, --ssl
-f, --file= [default: /etc/users]
-t, --templates= [default: /usr/share/finger/templates]
-n, --ircnick= [default: fingerbot]
--ircserver= [default: irc.freenode.net]
-p, --pbport= [default: 8889]
--version
--help Display this help and exit.
Make a finger server.Start the finger server (in the foreground) with:
$ sudo twistd -n finger --file=users 2007/12/23 22:12 +1100 [-] Log opened. 2007/12/23 22:12 +1100 [-] twistd 2.5.0 (/Library/Frameworks/Python.framework/ Versions/2.5/Resources/Python.app/Contents/MacOS/Python 2.5.0) starting up 2007/12/23 22:12 +1100 [-] reactor class: <class 'twisted.internet.selectreactor.SelectReactor'> 2007/12/23 22:12 +1100 [-] finger.finger.FingerFactoryFromService starting on 79 2007/12/23 22:12 +1100 [-] Starting factory <finger.finger.FingerFactoryFromService instance at 0x1d0a4e0> 2007/12/23 22:12 +1100 [-] twisted.web.server.Site starting on 8000 2007/12/23 22:12 +1100 [-] Starting factory <twisted.web.server.Site instance at 0x1d0a558> 2007/12/23 22:12 +1100 [-] twisted.spread.pb.PBServerFactory starting on 8889 2007/12/23 22:12 +1100 [-] Starting factory <twisted.spread.pb.PBServerFactory instance at 0x1d0a670> 2007/12/23 22:12 +1100 [-] Starting factory <finger.finger.IRCClientFactoryFromService instance at 0x1d0a5f8>
twistd provides many useful options, such as daemonizing the application, specifying the logfile and pidfile locations, etc.
Unfortunately Twisted and setuptools don't play nicely together, so I'm not able to package my Twisted app as an egg, take advantage of the setuptools package dependency resolution system, or install it using easy_install.
References:
http://twistedmatrix.com/projects/c
http://twistedmatrix.com/projects/c
http://twistedmatrix.com/projects/c
This version has been a long time coming, but has been well tested over that time. This version features many enhancements and bugfixes, some of them listed below. A special thanks to Zac Stevens and Mark Taylor for their contributions.
- Added support for Spread messaging as an alternative to Elvin.
- Implemented a DiskStatistics data collector for Linux.
- More command-line options and support for running as daemon.
- Added a "log" action. Use it to append to a log file, log via syslog, or print on the eddie tty.
- Variables can be set in directives, which can then be used in rule evaluation. For example, if the directive has "maxcpu=30", then the rule can address this as "rule='pcpu > _maxcpu'".
- HTTP checks support cookie persistence.
- Added "DBI" directive, for database query checking.
- Added Solaris SMF method/manifest files to contrib.
- Many more enhancements and bugfixes - see http://dev.eddie-tool.net/trac/brow
ser/eddie/trunk/doc/CHANGES.txt
$ ssh root@10.0.1.14
root@10.0.1.14's password:
Last login: Fri Sep 21 21:53:30 2007 from 10.0.1.2
# uname -a
Darwin CM iPhone 9.0.0d1 Darwin Kernel Version 9.0.0d1: Fri Jun 22 00:38:56 PDT 2007; root:xnu-933.0.1.178.obj~1/RELEASE_ARM_S5L8900XRB iPhone1,1 Darwin
# python
Python 2.5.1 (r251:54863, Jul 27 2007, 12:05:57)
[GCC 4.0.1 LLVM (Apple Computer, Inc. build 2.0)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import os
>>> os.uname()
('Darwin', 'CM iPhone', '9.0.0d1', 'Darwin Kernel Version 9.0.0d1: Fri Jun 22 00:38:56 PDT 2007; root:xnu-933.0.1.178.obj~1/RELEASE_ARM_S5L8900XRB', 'iPhone1,1')
>>>
Like I did at EuroPython, I gave a lightning talk on PSI, although this time I was better prepared with real slides, instead of using vim as a presentation tool and attempting to give a real-time demo (which ran me out of time too quickly).
I have even made the slides available, for anyone who may be curious.
PSI is a C extension that gives Python direct access to run-time system information by querying the relevant system calls. This version provides information about run-time process details. A Python program can take a snapshot of a process or all currently active processes on a system and inspect process details to its heart's content. PSI provides a consistent interface across all supported architectures, so programs written for one should (mostly) work on others. Where a particular architecture cannot supply the requested information that others can it will raise an appropriate exception.
This release supports 3 popular architectures: Solaris, Mac OS X and Linux. Hopefully more are on the way if I can round up volunteers.
If you want to have a play just: download it; svn checkout the source; or easy_install psi.
Here's some examples of it in action:
>>> import psi >>> a = psi.arch.arch_type() >>> a <psi.arch.ArchMacOSX object type='Darwin'> >>> isinstance(a, psi.arch.ArchMacOSX) True >>> isinstance(a, psi.arch.ArchDarwin) True >>> a.sysname 'Darwin' >>> a.nodename 'laptop' >>> a.release '8.9.1' >>> a.version 'Darwin Kernel Version 8.9.1: Thu Feb 22 20:55:00 PST 2007; root:xnu-792.18.15~1/RELEASE_I386' >>> a.machine 'i386' >>> psi.loadavg() (0.705078125, 0.73046875, 0.7626953125) >>> import os >>> mypid = os.getpid() >>> mypid 13903 >>> p = psi.process.Process(mypid) >>> p.command 'Python' >>> p.command_path '/Library/Frameworks/Python.framework/Versions/2.5/ Resources/Python.app/Contents/MacOS/Python' >>> p.user 'chris' >>> p.start_datetime datetime.datetime(2007, 9, 1, 10, 58, 51) >>> p.parent <psi.process.Process object pid=13860> >>> p.parent.command 'bash' >>> "%0.1f MB" % (p.resident_size/1024.0/1024.0) '9.7 MB' >>> "%0.1f MB" % (p.virtual_size/1024.0/1024.0) '43.5 MB' >>> ps = psi.process.ProcessTable() >>> ps.count 115 >>> ps.pids (0, 1, 27, 31, 39, 40, 41, 42, 43, 44, 45, 46, 47, 49, 50, 51, 56, 59, 63, 66, 67, 69, 71, 72, 89, 117, 122, 134, 136, 149, 155, 156, 159, 162, 172, 175, 176, 177, 179, 180, 182, 183, 190, 194, 214, 229, 238, 242, 245, 246, 248, 251, 256, 257, 264, 265, 267, 268, 270, 271, 272, 273, 274, 286, 392, 401, 402, 403, 1135, 1258, 1442, 1589, 1703, 1704, 1705, 1706, 1707, 1708, 1709, 1710, 1712, 1713, 1714, 1715, 1716, 1717, 1718, 1719, 1720, 1721, 2575, 2577, 2578, 2616, 2631, 2632, 9118, 9903, 10159, 10990, 12444, 12596, 13122, 13582, 13840, 13904, 13973, 13974, 13976, 14404, 14579, 14580, 14587, 14627, 14719) >>> p = ps.processes[114] >>> p.command 'TextMate'
So, if a user does not have write access to its $HOME directory then the temporary unzip will fail and so will the import. Very disappointing.
This whole zipped egg thing feels too much like a hack. At the very least shouldn't it attempt to unzip to the system tmp directory so it can still import the package and continue?
Anyway, the lesson to learn is always install eggs unzipped (which I was starting to do anyway, as I often need to examine the insides of an installed package when debugging and having to unzip the eggs first is a bit of a pain).
I am impressed by their offering of free wifi. The hotel (where the conference is also located) offers free wifi throughout, and I've just sat down at a coffee shop in a big shopping centre and was surprised to find another free wifi signal. Given the low costs of wifi infrastructure and broadband, more cities should encourage free wifi. I can't really see London doing it though... (nothing is free, or even cheap, in London).
Anyway, with any luck I'll walk around the "old town" today, which dates back to the 13th century and try and see more of the culture than shopping centres and wifi hotspots.
The conference starts tomorrow, so not much time for seeing sights after that. No doubt that will be when the rain stops and the sun comes out.
- Location:Vilnius, Lithuania
The first step is to make sure you've got a Python build that contains debugging symbols. Build Python manually using "make OPT=-g".
The nice Python guys have even supplied some handy gdb macros. Grab the Misc/gdbinit file from the Python source tree and make it your ~/.gdbinit file.
$ cd Python-2.5/Misc
$ cp gdbinit ~/.gdbinit
Now let's play with gdb. Fire it up and point it at the interpreter.
$ gdb
(gdb) file /opt/python-2.4.4-debug/bin/python
Reading symbols for shared libraries .... done
Reading symbols from /opt/python-2.4.4-debug/bin/python...done.
A very useful feature with gdb is the ability to set breakpoints on files that haven't been loaded yet, such as shared libraries. Let's set one in the source of a module I've been playing with. The shared library won't be loaded until Python processes the import statement, but gdb will still let us set it.
(gdb) b processtable.c:654
No source file named processtable.c.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (processtable.c:654) pending.
Now let's fire up the unit tests, to get something happening. You can see the pending breakpoint is automatically resolved when the relevant library is loaded.
(gdb) run setup.py test
Starting program: /opt/python-2.4.4-debug/bin/python setup.py test
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
running test
Reading symbols for shared libraries . done
Reading symbols for shared libraries . done
Breakpoint 1 at 0x627338: file processtable.c, line 654.
Pending breakpoint 1 - "processtable.c:654" resolved
test_args (tests.process_test.ProcessCommandTest) ... ok
test_command (tests.process_test.ProcessCommandTest) ... ok
test_command_path (tests.process_test.ProcessCommandTest) ... ok
test_env (tests.process_test.ProcessCommandTest) ... ok
test_nice (tests.process_test.ProcessPriorityTest)... ok
test_priority (tests.process_test.ProcessPriorityTest)... ok
test_resident_size (tests.process_test.ProcessSizeTest) ... ok
test_virtual_size (tests.process_test.ProcessSizeTest) ... ok
test_flags (tests.process_test.ProcessTimeTest) ... ok
test_parent_pid (tests.process_test.ProcessTimeTest) ... ok
test_status (tests.process_test.ProcessTimeTest) ... ok
test_terminal (tests.process_test.ProcessTimeTest) ... ok
test_threads (tests.process_test.ProcessTimeTest) ... ok
test_current_gid (tests.process_test.ProcessUserTest) ... ok
test_current_group (tests.process_test.ProcessUserTest) ... ok
test_current_uid (tests.process_test.ProcessUserTest) ... ok
test_current_user (tests.process_test.ProcessUserTest) ... ok
test_real_gid (tests.process_test.ProcessUserTest) ... ok
test_real_group (tests.process_test.ProcessUserTest) ... ok
test_real_uid (tests.process_test.ProcessUserTest) ... ok
test_real_user (tests.process_test.ProcessUserTest) ... ok
test_bad_arg (tests.process_test.SimplestProcessTest)... ok
test_pid (tests.process_test.SimplestProcessTest)... ok
test_type (tests.process_test.SimplestProcessTest)... ok
test_args (tests.processtable_test.ProcessTableProcessTests) ...
Breakpoint 1, ProcessTable_init (self=0x4410e0, args=0x405030, kwds=0x0) at processtable.c:654
654 if (PyList_Insert(self->processes, 0, (PyObject*)proc_obj)) {
Python ran some tests until it hit our breakpoint, inside the C extension module. We can view the source, of course.
(gdb) list
649
650
651 /* Add processes to list in reverse order, which ends up ordering
652 * them by ascending PID value.
653 */
654 if (PyList_Insert(self->processes, 0, (PyObject*)proc_obj)) {
655 return -1; /* failure */
656 }
657 Py_DECREF(proc_obj);
658 }
We are inside the __init__ function of a class. So there's the usual Python self object. In C extension modules, self is a pointer to a struct representing the internal attributes of the class. Let's take a look at self->processes.
(gdb) p self
$1 = (ProcessTableObject *) 0x4410e0
(gdb) p self->processes
$2 = (PyObject *) 0x4b5940
In this case, self is a pointer to our custom class. self->processes is a pointer to a PyObject, which could be any Python object type. The .gdbinit we borrowed from the Python source defines a very useful macro for inspecting the target of PyObject pointers.
(gdb) pyo self->processes
object : []
type : list
refcount: 1
address : 0x4b5940
$3 = void
Cool, so self->processes is a list type, and its current value is an empty list. Our breakpoint is located within a loop, so let's iterate around and get an object added to this list.
(gdb) cont
Continuing.
Breakpoint 1, ProcessTable_init (self=0x4410e0, args=0x405030, kwds=0x0) at processtable.c:654
654 if (PyList_Insert(self->processes, 0, (PyObject*)proc_obj)) {
(gdb) pyo self->processes
object : [<psi.process.process object="object" pid="16543">]
type : list
refcount: 1
address : 0x4b5940
$4 = void
Cool, the list now contains an object. Let's add another by looping again.
(gdb) cont
Continuing.
Breakpoint 1, ProcessTable_init (self=0x4410e0, args=0x405030, kwds=0x0) at processtable.c:654
654 if (PyList_Insert(self->processes, 0, (PyObject*)proc_obj)) {
(gdb) pyo self->processes
object : [<psi.process.process object="object" pid="16536">, <psi.process.process object="object" pid="16543">]
type : list
refcount: 1
address : 0x4b5940
$5 = void
So, self->processes is a list and currently contains 2 objects. Is it possible to fetch an element from the list and examine it? Sure is. We need to call the Python C functions that know how to deal with Python objects. gdb will allow us to do this.
(gdb) pyo PyObject_GetItem(self->processes,Py_BuildValue("i",0))
object : <psi.process.process object="object" pid="16536">
type : psi.process.Process
refcount: 3
address : 0x4dbf28
$6 = void
PyObject_GetItem(obj, y) is the C equivalent of obj[y] or obj.__getitem__(y)). The "y" must also be a Python object, you cannot just give it a C int. So we use Py_BuildValue() to build a Python integer object. The above is the equivalent of
self.processes[0]. (Note that you cannot have any spaces within the argument given to pyo, as arguments to gdb macros are split by white space and pyo will only use the first one ($arg0).)So, how do we examine the Process object itself? We can easily look at an attribute of the object, which might be handy. Let's look at the "command" attribute of the Process object.
(gdb) pyo PyObject_GetAttr(PyObject_GetItem(self->processes,Py_BuildValue("i",0)),Py_BuildV alue("s","command"))
object : 'gdb-i386-apple-d'
type : str
refcount: 3
address : 0x640bb0
$7 = void
and same for the other object in the list.
(gdb) pyo PyObject_GetAttr(PyObject_GetItem(self->processes,Py_BuildValue("i",1)),Py_BuildV alue("s","command"))
object : 'python'
type : str
refcount: 3
address : 0x63cf60
$8 = void
Cool, so even though we are deep within a C extension module, we can still introspect our objects with relative ease.
When building for Solaris you often run into issues as Solaris is not a GNU environment. Although, these days Sun provides a lot of GNU software with Solaris, just most of it is not installed in standard locations (certainly not standard from a Linux POV).
This is the case with GNU readline. It is an optional Solaris package (SFWrline), that is distributed by Sun on the "Companion" disc that can be downloaded along with the Solaris install discs. It gets installed into /opt/sfw (instead of /usr or /usr/local in a Linuxy world) but is otherwise "normal".
Unfortunately I have to jump through hoops of various sizes to get readline linked with some of my favorite software. Fair enough, the configure/build tools of these projects does not automatically look in /opt/sfw for optional GNU libraries (be nice if they did though, when they were configured on a Solaris box) but in some cases teaching the software that the libraries are in custom locations is more difficult than it should be. I'm mainly pointing the finger at Python here.
In the cases of Python and SQLite, I have documented how to build these with readline support on Solaris 10, follow the links below if you'd like to know how.
Building Python with readline on Solaris 10
Building SQLite with readline on Solaris 10
Here's a brief walk-through which demonstrates how to build RRDTool Python bindings on OS X.
(Tested with Python 2.4.1 and RRDTool 1.2.15 by me, and some other minor versions by others.)
Tell configure to disable Python bindings (we'll build them ourselves). I disable all bindings because I don't need them. --prefix can be anywhere you like.
$ cd rrdtool-1.2.15/ $ ./configure --prefix=/opt/rrdtool-1.2.15 --disable-perl --disable-tcl --disable-python $ make $ make install $ cd bindings/python
Copy my setup.py to this directory (download it or copy & paste from below)
Find & replace all occurrences of "rrdtoolmodule" to "rrdtool" in rrdtoolmodule.c
$ perl -p -i -e 's/rrdtoolmodule/rrdtool/g' rrdtoolmodule.c
Now distutils can build and install the module.
$ python setup.py build $ python setup.py install
Test with:
$ python >>> import rrdtool >>>
If the import produces no errors, you are ready to go.
Alternatively, if you prefer to use rrdtool and Python from DarwinPorts, try this (thanks
![[info]](http://l-stat.livejournal.com/img/userinfo.gif){kind=small}
burritob):Use DarwinPorts to build (but not install) rrdtool
$ sudo port build rrdtool
Change into binding/python under the work directory, eg
$ cd /opt/local/var/db/dports/build/*_net_rrdtool/work/rrdtool-1.2.12/bindings/python
Replace all occurrences of "rrdtoolmodule" with "rrdtool"
$ sudo perl -p -i -e 's/rrdtoolmodule/rrdtool/g' rrdtoolmodule.c
Build and install the extension
$ sudo /opt/local/bin/python setup.py build $ sudo /opt/local/bin/python setup.py install
Move out of the build directory and install the port
$ cd $ sudo port install rrdtool
---- setup.py ----
from distutils.core import setup, Extension
rrdtool = Extension(
'rrdtool',
sources = ['rrdtoolmodule.c'],
include_dirs = ['../../src'],
library_dirs = ['../../src/.libs'],
extra_link_args = ['-lrrd'],
)
setup(
name = 'rrdtool',
version = '1.2.15',
description = 'rrdtool',
ext_modules = [rrdtool],
)---- ----
Here's a quick demo.
>>> import psi >>> psi.loadavg() (0.751953125, 1.35400390625, 1.16259765625)
The 3-tuple of load averages as available on most UNIX platforms.
>>> ps = psi.process.ProcessList() >>> ps.count 119
ProcessList() creates an object representing a snapshot of the processes on the system. In this example there are 119 processes. Let's see all the process IDs:
>>> ps.pids (0, 1, 27, 31, 39, 40, 41, 42, 43, 44, 45, 46, 47, 49, 50, 51, 56, 59, 63, 66, 67, 69, 71, 72, 89, 117, 122, 134, 149, 155, 156, 159, 162, 172, 175, 176, 177, 179, 180, 182, 183, 190, 194, 214, 229, 238, 242, 245, 246, 248, 251, 256, 257, 264, 265, 267, 268, 270, 271, 272, 273, 286, 392, 401, 402, 403, 1135, 1258, 1442, 1589, 1703, 1704, 1705, 1706, 1707, 1708, 1709, 1710, 1712, 1713, 1714, 1715, 1716, 1717, 1718, 1719, 1720, 1721, 2575, 2577, 2578, 2616, 2631, 2632, 9118, 9903, 10159, 10990, 12444, 12596, 13582, 13840, 13904, 13976, 14404, 15093, 15443, 15498, 15814, 15845, 15947, 16092, 16093, 16238, 16270, 16272, 18040, 18088, 18183)
Each process is represented by a Process object. ps.processes will return a list of all Process objects, or we can fetch a dict of Process objects keyed by pid using ps.processes_dict.
Let's examine a process. We'll take a quick glance at what each process is called so we can pick one.
>>> [(p.pid,p.command) for p in ps.processes] [(0, 'kernel_task'), (1, 'launchd'), (27, 'dynamic_pager'), (31, 'kextd'), (39, 'bfobserver'), (40, 'KernelEventAgent'), (41, 'mDNSResponder'), (42, 'netinfod'), (43, 'syslogd'), (44, 'cron'), (45, 'configd'), (46, 'coreaudiod'), (47, 'diskarbitrationd'), (49, 'memberd'), (50, 'notifyd'), (51, 'securityd'), (56, 'update'), (59, 'distnoted'), (63, 'DirectoryService'), (66, 'WindowServer'), (67, 'blued'), (69, 'coreservicesd'), (71, 'ATSServer'), (72, 'loginwindow'), (89, 'crashreporterd'), (117, 'mds'), (122, 'AppleFileServer'), (134, 'httpd'), (149, 'ARDHelper'), (155, 'nmpppstatsd'), (156, 'NMPCCardDemonVMC'), (159, 'iScroll2Daemon'), (162, 'hpusbmond'), (172, 'HPIO Trap Monito'), (175, 'ARDAgent'), (176, 'AppleVNCServer'), (177, 'rmdb'), (179, 'rmdb'), (180, 'rmdb'), (182, 'python'), (183, 'pbs'), (190, 'Dock'), (194, 'SystemUIServer'), (214, 'ntpd'), (229, 'nfsiod'), (238, 'rpc.lockd'), (242, 'automount'), (245, 'iTunesHelper'), (246, 'iCalAlarmSchedul'), (248, 'System Events'), (251, 'automount'), (256, 'Quicksilver'), (257, 'UniversalAccess'), (264, 'Synergy'), (265, 'teleportd'), (267, 'LCCDaemon'), (268, 'HPEventHandler'), (270, 'Print Daemon'), (271, 'HP IO Classic Pr'), (272, 'HP IO Classic Pr'), (273, 'HP Scheduler'), (286, 'dmnotifyd'), (392, 'cupsd'), (401, 'Activity Monitor'), (402, 'pmTool'), (403, 'Memory Monitor'), (1135, 'AppleSpell'), (1258, 'iChatAgent'), (1442, 'GrowlHelperApp'), (1589, 'NetNewsWire'), (1703, 'DashboardClient'), (1704, 'DashboardClient'), (1705, 'DashboardClient'), (1706, 'DashboardClient'), (1707, 'DashboardClient'), (1708, 'DashboardClient'), (1709, 'DashboardClient'), (1710, 'DashboardClient'), (1712, 'DashboardClient'), (1713, 'DashboardClient'), (1714, 'DashboardClient'), (1715, 'DashboardClient'), (1716, 'DashboardClient'), (1717, 'DashboardClient'), (1718, 'DashboardClient'), (1719, 'DashboardClient'), (1720, 'DashboardClient'), (1721, 'DashboardClient'), (2575, 'iTerm'), (2577, 'login'), (2578, 'bash'), (2616, 'TextMate'), (2631, 'login'), (2632, 'bash'), (9118, 'Adium'), (9903, 'free-1 USB Phone'), (10159, 'httpd'), (10990, 'Skype'), (12444, 'iChat'), (12596, 'ssh-agent'), (13582, 'Finder'), (13840, 'Safari'), (13904, 'iCal'), (13976, 'ssh-agent'), (14404, 'check_afp'), (15093, 'ssh-agent'), (15443, 'Preview'), (15498, 'ssh-agent'), (15814, 'vim'), (15845, 'Mail'), (15947, 'DashboardClient'), (16092, 'login'), (16093, 'bash'), (16238, 'iTunes'), (16270, 'lookupd'), (16272, 'slpd'), (18040, 'mdimport'), (18088, 'mdimport'), (18183, 'python')]
I'm curious about process 'ntpd' (pid 214):
>>> p = ps.processForPid(214) >>> p <psi.process.Process object pid=214> >>> dir(p) ['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', 'args', 'command', 'command_path', 'cpu_time', 'env', 'flags', 'gid', 'group', 'nice', 'pcpu', 'pid', 'priority', 'resident_size', 'start_time', 'status', 'system_time', 'terminal', 'threads', 'uid', 'user', 'user_time', 'virtual_size'] >>> p.pid 214 >>> p.command 'ntpd' >>> p.user 'root' >>> p.command_path >>> p.args >>> p.env >>> p.resident_size >>>
hmmm, many of the interesting attributes return None. That's because this process is owned by root and we don't have privileges to access all the details about it. Let's give ourselves some privileges and try again:
$ sudo python2.4
Python 2.4.1 (#2, Mar 31 2005, 00:05:10)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psi
>>> ps = psi.process.ProcessList()
>>> p = ps.processForPid(214)
>>> p.command
'ntpd'
>>> p.command_path
'/usr/sbin/ntpd'
>>> p.args
['ntpd', '-f', '/var/run/ntp.drift', '-p', '/var/run/ntpd.pid']
>>> p.env
{'SHLVL': '2', 'PATH':
'/bin:/sbin:/usr/bin:/usr/sbin:/usr/libexec:/System/Library/CoreServices',
'PWD': '/', '_': '/usr/sbin/ntpd'}How much memory is the process using:
>>> p.resident_size 221184 >>> print "%d KB" % (p.virtual_size / 1024) 27504 KB
When did the process start?
>>> from datetime import datetime >>> print datetime.fromtimestamp(p.start_time) 2006-12-26 10:50:30
How much elapsed CPU time has this process used?
>>> p.cpu_time 36.497602000000001 >>> p.system_time 26.527072 >>> p.user_time 9.9705300000000001
What percentage of CPU is this process currently using?
>>> p.pcpu 0.0
ok, it is idle. Pretty boring. Let's grab an active process, say iTunes.
>>> [(p.pid,p.command) for p in ps.processes if 'itunes' in p.command.lower()] [(245, 'iTunesHelper'), (16238, 'iTunes')] >>> p = ps.processForPid(16238) >>> p.command 'iTunes' >>> p.command_path '/Applications/iTunes.app/Contents/MacOS/iTunes' >>> p.pcpu 9.0999999999999996
Damn iTunes chews up some CPU on my old PowerBook.
That's the end of the demo. Mac-savvy readers may have noticed that I performed this demo on an OS X box. Currently PSI only fetches information for OS X coz that's what I've been developing on. Next I'll be adding support for Solaris, Linux, maybe FreeBSD. I may need some help with Win32 support, but my goal will be to support as many systems as possible (everything that Python runs on?). The module is a C extension that fetches information directly from the relevant system calls, so it aims to be as efficient as possible (i.e. rather than parsing the output of commands like 'ps' as is commonly done).
Another goal of PSI is to make available as much information about the system as possible. That will include memory & swap usage details, filesystem usage & I/O information, network interface activity, and whatever else is useful.
This is my first attempt at a Python C extension module, so it is an interesting challenge and I'll certainly be open to suggestions, advice & criticism. I'm still getting used to the nuances of reference counting, so don't expect the first release to be absolutely perfect. I aim to release the first version after I add support for a few more systems and add plenty more unit tests.
- Mood:creative
What does this mean? Quoting John Levon: "This means Python modules can make use of 64-bit versions of libraries, as well as 64-bit plugins."
64-bit Python in Nevada build 53
I found it interesting because I hadn't really had any first-hand dealings with anyone in "the real world" talking about applications using these technologies (hey, I don't get out much). Not outside of technical circles, I mean. It was refreshing to not hear mention of the technology that starts with P and ends in HP, or the other that starts with J and ends in an overdue and over budget management nightmare.
We, by the way, are getting right into TurboGears for some of our new projects. It appealed to me most when I was shopping around for a new web framework many moons ago, and I haven't been disappointed so far. I mainly chose TG because it is a glue between many technologies and you (mostly) get a choice of what those technologies will be. For the record, we are using SQLAlchemy, Kid (probably moving to Genshi as it matures, and I have time to play with it) and MochiKit (1.4, which has loads of nice goodies).
So anyway, my "real world" survey so far declares Django 1 vs Rails 1.
- Mood:
thirsty - Music:Dj Fexomat
Here are the steps required to get Python 2.4.3 built with readline support on Solaris 10, when you only want to use Sun's supplied SFWrline package (rather than independent packages provided by Blastwave, Sunfreeware.com, etc).
$ ./configure --prefix=/opt/python-2.4.3 --enable-shared $ vi Modules/Setup.local
add one line:
readline readline.c -I/opt/sfw/include -L/opt/sfw/lib -lreadline -ltermcap
Then build:
$ make
the build will sadly fail with an ld error:
Undefined first referenced symbol in file initreadline ./libpython2.4.so ld: fatal: Symbol referencing errors. No output written to python collect2: ld returned 1 exit status *** Error code 1 make: Fatal error: Command failed for target `python'
however, just running make again ends up building it fine ...
Update: use GNU make (gmake) instead of Sun make to avoid this problem. The build actually regenerates the Makefile but Sun make doesn't recognize the change. (Thanks to Martin v. Löwis for pointing this out.)
$ make $ make install # might need root access (sudo/su)
I hope this process can be improved, and if I have time I'll look at why make fails half way but works fine when kicked off a second time.
I'd like to get /opt/sfw/ added to the standard lib/include path for Solaris (10 at least) as this is where Sun's freeware packages get installed. I hope this is just a matter of raising a ticket.
- Mood:awake
Python's built-in unittest module handles most of the grunt work, making it dead-easy to write tests for some code. How easy is this:
class FooTests(unittest.TestCase):
def setUp(self):
self.foo = Foo(5)
def testType(self):
self.failUnlessEqual(type(self.foo), Foo)
def testValue(self):
self.failUnlessEqual(self.foo.value, 5)
then simply call unittest.main() and unittest handles the rest. Nice.
So that is probably old news for most decent Python coders, but hey I'm slow to catch on.
What is more impressive is the tool I just started using to automate running tests across a whole project. It's called nose! It works like a heat-seeker and tracks down all the Python packages and modules that contain runnable test cases and executes them. At the end it produces a simple summary describing how many tests passed and failed. Of course, it can produce a much more detailed report if you ask it to. Very nice and simple to use.
But it gets cooler than that. If you install Ned Batchelder's coverage module (how's that for an inventive module name?) and fire off "nosetests --with-coverage" you end up with a report telling you, of all the executable lines in your modules that were just tested, how many of them were actually executed (and hence tested). Do your unittests cover all of your code lines? Well, now you know. I think this is very cool.
- Mood:
tired
The double-edged sword of the Python world is currently the abundance of web frameworks. All of them are good in their own right, but which one is right for your particular job? Here's what I tried for my simple project described above.
try:
Nevow / Twisted
except:
I really like what I've seen of Nevow so far. I've been playing with Twisted for a few months now, and I've used Nevow to build a few test/prototype projects. Along with Formal it is a nice web framework.
What is questionable (at least in my eyes) is how ready for production it is. I haven't put it under any real load, and don't know anyone who has, so I put the question to #twisted.web, "Any performance issues to be concerned about? Let's say, access rate of 100 hits/sec ..." and got mixed responses, none of them giving me confidence that Nevow was ready for busy production site deployment, some of them sarcastically unhelpful, a few of them pointing out the performance bottlenecks to avoid (like the built-in Nevow sequence renderer) and recommendations to use a lot of caching (both within Twisted and in front of) as much as possible.
I was left with the feeling that Twisted/Nevow probably would do the job, as those guys do seem to know what they are doing, and I was keen to continue using a framework I enjoy, but really there were too many doubts and I didn't want to use this customer project as a guinea pig for Nevow benchmarking.
try:
Zope (specifically 2.9+Five)
except:
I have implemented some simple Zope websites in the past (all TTW - "Through The Web") and done some Plone/ArchGenXML development, and I like some elements of Zope, especially the Zope Page Templates.
However, I decided that if I am to use Zope for any new projects (and certainly customer projects) that they would all be filesystem based, not TTW. Fair enough, you say? I quizzed #zope on the best current methods for implementing filesystem based sites in Zope and the response was to use Zope 2.9 + Five (which takes advantage of some of Zope 3's architecture to further separate the implementation from the view from the configuration).
In theory it sounded right. In practice, I grabbed two Zope 2.9+Five tutorials and went through them both. My needs were much simpler than the tutorial examples, so I attempted to strip the examples down to produce a simple dynamic web page which was all I needed to start with. After half a day of screwing around with Interface definitions, XML config files and obscure classes just to output a simple dynamic page, all without any success, I gave up. Admittedly I didn't try to read any of the documentation to understand it properly, but frankly, that is the point. I shouldn't have to. I really wasn't trying to implement anything complex.
I am sure Zope is great for much larger scale projects (as Plone proves) and I'm sure I'd catch on after reading the docs in more detail along with one of the Zope 3 books, but as a lightweight web app environment it ain't even worth considering.
try:
CherryPy
except:
I had considered CherryPy to be a bit mickey mouse in the past, and had never actually used it. However, for the simple needs of this project, I soon realised, mickey mouse is all I needed. I grabbed CherryPy, grabbed the tutorial, and within minutes, literally, I had some dynamic pages up & running. Nice.
The good thing about CherryPy is that it comes with no templating engine - the user is left to choose his own. The bad thing about CherryPy is that it comes with no templating engine... so the next few hours were spent reviewing the current bunch of recommended templating engines. My preference was an XML compliant templating system, along the lines of Zope Page Templates or Nevow's templating system. I had a quick look at Kid but was left unsure about the health of the project - comments along the lines of the project being abandoned, or at least, not kept up-to-date, made me step back with caution.
In the end, because time had really dragged on, and I needed to get this implementation up & running quick smart for the customer, I grabbed Cheetah and ran with it. It has so far has proven itself to be quite flexible and insanely easy to use. I really hate the look of the templates that come out of a Cheetah project, they are ugly as sin. Arguably more ugly than PHP... But I can't deny that it works nicely and is trivial to pick up and get instant results.
finally:
When you need to quickly implement a solution for a customer using reliable technology, stick with the simplest possible technologies. In this case, the CherryPy/Cheetah framework has produced results extremely quickly, has an easy learning curve and good community support.
When you need to build a much more complex solution, then the likes of Zope or the Plone CMS may become more practical. I am certainly keen to learn Zope 3 properly one day, but fear that a good excuse to do so may still be far away.
When you want the best of both worlds, with top-notch technology from a bunch of smarties, Nevow appears to be the solution, and it is my personal favourite so far. However, the Nevow & Twisted community need to learn how to step down and speak to the normal folk once in a while and help promote the quality and reliability of their technologies.
- Music:TROPICALVIBE2006 - set by Lucio K
If you want the same excitement look here for my bookmarks to Python on PSP.
I found the asynchronous Twisted concepts easy to pick up, but applying them in practice was sometimes tricky, as my mind kept wanting to think of implementations in terms of threading. The examples mentioned above helped shake this out of me, and my project is coming along really nicely. It is a Twisted app that uses Twisted deferreds to perform "background" operations, with a web based UI that uses Nevow and Pollenation Forms to provide an interface.
So far I have found Nevow a really nice web framework to work with. It is relatively low level, at least compared with Zope/TAL, so you find yourself writing a fair amount of code, although part of that difference is that Nevow absolutely prevents you from implementing logic or Python in XHTML/XML, whereas Zope/TAL allows this - something that the Zope guys argue over endlessly. I don't feel that this will disadvantage Nevow as common template operations can be grouped into Nevow page classes and shared around as recipes or high-level implementations of the framework (which is what Pollenation Forms offers for HTML form handling - very nicely too).
All in all I am very happy with Twisted and Nevow so far. Twisted was easier to learn than the stories made out. They say it is abstraction upon abstraction, almost to a ridiculous degree - but have they tried to develop with Plone and Archetypes??
I can also see Twisted as a promising framework for an Eddie monitoring tool re-architecture in the near future...
Simon has a habit of keeping me out drinking till 4am after these events, but last night we agreed we would leave at a sensible hour and actually catch the trains. Well, our will powers being as poor as they are, we were convinced to introduce a newbie to the Python London drinking culture, so we ended up going to our favourite haunt, the Cubana Bar, and enjoying cocktails till about 12:30am, which is still a relatively sensible hour, however I did miss my train and had to taxi it. All in the name of fun.
- Mood:hungover
I have also been invited to join the Plone Multimedia Sprint, which directly follows the conference on Thursday & Friday. That should be a lot of fun, but should also be an excellent opportunity to work with some Plone experts and learn what I can. Not to mention be involved in a cool project to help expand Plone's capabilities in the multimedia area.
- Mood:
pleased - Music:Morning Lemon
If you are interested in beta testing this, take a look at the latest snapshot: eddie-0.34f.zip
Read doc/README.win32.txt before reading the other docs as usual.
- Mood:accomplished
- Music:starbuckswhitenoise
