Previously I talked about using custom authentication for a web application, supporting two methods of authentication simultaneously. Browser-based users would be challenged with a typical styled login form, while applications integrating with the service would be challenged with HTTP Basic Authentication. The choice of which authentication to use would be based on how clients are classified. Taking advantage of content negotiation, clients preferring a HTML response would be classified as a "browser" and would be challenged with a login form; any other clients would be classified as an "app" and would be challenged with HTTP Basic Auth.

Implementing this custom authentication and classification scheme for Pylons was quite simple using repoze.who. Here I describe how I implemented it.

I'll start from a skeleton Pylons app.

$ paster create -t pylons CustomAuth

The defaults of 'mako' for templates and no SQLAlchemy are fine for this example.

I based my repoze.who configuration off this recipe from the Pylons cookbook but I'll quickly repeat the necessary steps here, so that the example is complete.

If you haven't done so already, install the repoze.who package with:

$ easy_install repoze.who

The next step is to add repoze.who to the WSGI middleware of your Pylons app. Edit config/middleware.py and add an import:

from repoze.who.config import make_middleware_with_config as make_who_with_config

Then after the comment "CUSTOM MIDDLEWARE HERE" add the following line:

app = make_who_with_config(app, global_conf, app_conf['who.config_file'], app_conf['who.log_file'], app_conf['who.log_level'])

Now edit development.ini and add to the [app:main] section:

who.config_file = %(here)s/who.ini
who.log_level = debug
who.log_file = stdout

Now create a who.ini file in the same location as development.ini containing:

[plugin:form]
use = repoze.who.plugins.form:make_redirecting_plugin
login_form_url = /account/login
login_handler_path = /account/dologin
logout_handler_path = /account/logout
rememberer_name = auth_tkt

[plugin:auth_tkt]
use = repoze.who.plugins.auth_tkt:make_plugin
secret = yoursecret

[plugin:basicauth]
# identification and challenge
use = repoze.who.plugins.basicauth:make_plugin
realm = CustomAuth

[general]
request_classifier = customauth.lib.auth:custom_request_classifier
challenge_decider = repoze.who.classifiers:default_challenge_decider

[identifiers]
plugins =
    form;browser
    auth_tkt;browser
    basicauth

[authenticators]
plugins =
    customauth.lib.auth:UserModelPlugin

[challengers]
plugins =
    form;browser
    basicauth

[mdproviders]
plugins =
    customauth.lib.auth:UserModelPlugin

You would replace "customauth" with the package name of your Pylons app.

Take note of the request_classifier in [general]. It specifies a custom classifier function "custom_request_classifier" located in the lib.auth module of your application. This function is called for each request and returns a classification that, for this application, will be either "browser" or "app" (some other classifications are possible, like "dav", but we're not worrying about them in this application; they'll be treated like "app").

You can see that in the [identifiers] and [challengers] sections there are multiple plugins listed. The choice of plugin to use in each case is based on the value returned by the classifier. If the classifier is "browser" then the "form" challenger will be used, otherwise the "basicauth" challenger is chosen. This is the key to the custom authentication, and as you can see it is all handled by repoze.who and extremely simple to configure.

Create an auth.py file in the lib directory of the Pylons app containing:

from webob import Request

import zope.interface
from repoze.who.classifiers import default_request_classifier
from repoze.who.interfaces import IRequestClassifier

class UserModelPlugin(object):
    
    def authenticate(self, environ, identity):
        """Return username or None.
        """
        try:
            username = identity['login']
            password = identity['password']
        except KeyError:
            return None
        
        if (username,password) == ('foo', 'bar'):
            return username
        else:
            return None
    
    def add_metadata(self, environ, identity):
        username = identity.get('repoze.who.userid')
        if username is not None:
            identity['user'] = dict(
                username = username,
                name = 'Mr Foo',
            )
    

def custom_request_classifier(environ):
    """ Returns one of the classifiers 'app', 'browser' or any
    standard classifiers returned by
    repoze.who.classifiers:default_request_classifier
    """
    classifier = default_request_classifier(environ)
    if classifier == 'browser':
        # Decide if the client is a (user-driven) browser or an application
        request = Request(environ)
        if not request.accept.best_match(['application/xhtml+xml', 'text/html']):
            # In our view, any client who doesn't support HTML/XHTML is an "app",
            #   not a (user-driven) "browser".
            classifier = 'app'
    
    return classifier
zope.interface.directlyProvides(custom_request_classifier, IRequestClassifier)

This is where the custom_request_classifier function is defined. It first calls the default_request_classifier provided by repoze.who, which attempts to classify the request as one of a few basic types: 'dav', 'xmlpost', or 'browser'. If the default classification results in 'browser' then we try to classify it further based on content negotiation. If the client prefers a HTML or XHTML response then we leave the classification as 'browser', otherwise we classify it as 'app'.

The other part of the auth module is the UserModelPlugin class. This class provides "authenticator" and "mdprovider" plugins. The job of the authenticate method is to authenticate the request, typically by verifying the username and password provided, but of course that depends on the type of authentication used. In this example, we simply provide a stub authenticator that compares authentication details against a hard-coded username/password pair. In a real app you would authenticate against data in a database or LDAP service, or whatever you decided to use.

The add_metadata method of UserModelPlugin is called to supply metadata about the authenticated user. In this example we simply supply a hard-coded name, but in a real app you would fetch details from a database or LDAP or whatever.

The final bit of code needed is the login form. Create an account controller:

$ paster controller account

Then edit controllers/account.py and add a login method to AccountController:

    def login(self):
        identity = request.environ.get('repoze.who.identity')
        if identity is not None:
            came_from = request.params.get('came_from', None)
            if came_from:
                redirect_to(str(came_from))
        
        return render('/login.mako')

Also add a test method to the same controller so that we can verify authentication works:

    def test(self):
        identity = request.environ.get('repoze.who.identity')
        if identity is None:
            # Force skip the StatusCodeRedirect middleware; it was stripping
            #   the WWW-Authenticate header from the 401 response
            request.environ['pylons.status_code_redirect'] = True
            # Return a 401 (Unauthorized) response and signal the repoze.who
            #   basicauth plugin to set the WWW-Authenticate header.
            abort(401, 'You are not authenticated')
        
        return """
<body>
Hello %(name)s, you are logged in as %(username)s.
<a href="/account/logout">logout</a>
</body>
</html>
""" %identity['user']

The test action checks whether a user has been authenticated for the current request. If not, it forces a 401 response which will have a different effect depending on which classification was chosen. If the request was classified as "browser" then, due to the repoze.who config specifying "form" as the challenger plugin for this classification, the repoze.who middleware will intercept the 401 response and replace it with a 302 redirect to the login form page. For any other classification, the "basicauth" challenger will be chosen which will return the 401 response with an appropriate "WWW-Authenticate" header.

Note that we needed to suppress the StatusCodeRedirect middleware for the 401 response to prevent Pylons from returning a custom error document and messing with our 401 error.

In a real application you may want to move the identity check into the __before__ method of the controller (or BaseController class) or into a custom decorator. Or you could use repoze.what.

In the templates directory create login.mako containing a simple form such as:

<html>
<body>
  <p>
    <form action="/account/dologin" method="POST">
      Username: <input type="text" name="login" value="" />
      <br />
      Password: <input type="password" name="password" value ="" />
      <br />
      <input type="submit" value="Login" />
    </form>
  </p>
</body>
</html>

Now you should be ready to run the application and test authentication.

$ paster serve --reload development.ini

Using your favourite web browser, go to http://127.0.0.1:5000/account/test

You should immediately be redirected to /account/login (with a came_from parameter) with your login form displayed. Enter bogus details and you shouldn't make it pass the form. Now enter the hard-coded login details ("foo", "bar") and you should be authenticated and see the text from /account/test.

Now we can test whether basic auth works. Using curl, try to fetch /account/test

$ curl -i http://127.0.0.1:5000/account/test
HTTP/1.0 302 Found
Server: PasteWSGIServer/0.5 Python/2.5.1
Date: Tue, 03 Mar 2009 08:57:59 GMT
Location: /account/login?came_from=http%3A%2F%2F127.0.0.1%3A5000%2Faccount%2Ftest
content-type: text/html
Connection: close

<html>
  <head><title>Found</title></head>
  <body>
    <h1>Found</h1>
    <p>The resource was found at <a href="/account/login?came_from=http%3A%2F%2F127.0.0.1%3A5000%2Faccount%2Ftest">/account/login?came_from=http%3A%2F%2F127.0.0.1%3A5000%2Faccount%2Ftest</a>;
you should be redirected automatically.
/account/login?came_from=http%3A%2F%2F127.0.0.1%3A5000%2Faccount%2Ftest
<!--  --></p>
    <hr noshade>
    <div align="right">WSGI Server</div>
  </body>
</html>

You can see that, by default, the request is classified as 'browser' and so a 302 redirect to the login form was returned. Note that if no Accept header field is present, then it is assumed that the client accepts all media types, which is why the request was classified as "browser".

Now let's specify a preference for 'application/json' (using the Accept header) and see what we get.

$ curl -i -H "Accept:application/json" http://127.0.0.1:5000/account/test
HTTP/1.0 401 Unauthorized
Server: PasteWSGIServer/0.5 Python/2.5.1
Date: Tue, 03 Mar 2009 09:21:09 GMT
WWW-Authenticate: Basic realm="CustomAuth"
content-type: text/plain; charset=utf8
Connection: close

401 Unauthorized
This server could not verify that you are authorized to
access the document you requested.  Either you supplied the
wrong credentials (e.g., bad password), or your browser
does not understand how to supply the credentials required.

Perfect. We get a 401 response with a WWW-Authenticate header specifying "Basic" authentication is required. (Note that ideally we should return a JSON response body as that is what the client requested.)

Now we can repeat the request, including our authentication details.

$ curl -i -H "Accept:application/json" -u foo:bar http://127.0.0.1:5000/account/test
HTTP/1.0 200 OK
Server: PasteWSGIServer/0.5 Python/2.5.1
Date: Tue, 03 Mar 2009 11:39:43 GMT
Content-Type: text/html; charset=utf-8
Pragma: no-cache
Cache-Control: no-cache
Content-Length: 107

<html>
<body>
Hello Mr Foo, you are logged in as foo.
<a href="/account/logout">logout</a>
</body>
</html>

And there we have it. Dual authentication on the same controller.

For a recent web service project I wanted to make it as RESTful as possible. It needed to provide both a user interface (for interactive users) as well as exposing an API for programmatic integration. So I implemented both, but the two are not separate. Every applicable resource is exposed under only one URI each, usable by both interactive users (with web browsers) and by applications.

The "magic" of HTTP content negotiation is what makes this work. Clients that prefer HTML will get a rich HTML UI to interact with the application and data. Clients that prefer JSON will get back a JSON representation of the resource and, similarly, those that prefer XML will get back an XML representation. So most URIs provide 3 representations of themselves: HTML, JSON and XML.

When web browsers make a HTTP request they send an "Accept" header indicating their preference for HTML, so interactive users get the rich HTML UI, all styled and pretty looking. However, they are still viewing exactly the same resource as those fetching the JSON or XML representation, just that it is pleasing to the eye and is surrounded by navigation and other UI niceties.

All this should be pretty familiar to those who already play with RESTful HTTP. The part of the implementation that may not be familiar is how I handled authentication.

To keep with the typical "web experience" for interactive users, I wanted to provide the conventional login form/cookies method of authentication. This method is all but useless for applications, so I wanted to provide HTTP Basic Auth for them.

Now, given that a resource lives on a single URI, how do we support both types of authentication at once? Or perhaps the question is: should we? I decided the answer was "yes", as I didn't want to force interactive users to have to use HTTP Auth (login forms are intrusive and unstyled [1]; most users aren't used to them; and, perhaps worse of all, you can't logout with most browsers without plugins or hackary [2]).

So how did I support two forms of authentication simultaneously without separating web UI URIs from "API" URIs? I relied on our old friend, content negotiation. I decided that: any client who negotiates to receive a HTML representation is classified as a "browser" and will be challenged for authentication with a login form (redirected to the login page) and remembered with cookies. Any other client will be classified as an "app" and will be challenged with HTTP Basic Auth (with a 401 response).

I tossed this idea around for a while, deciding if it was too much hackary, but decided to implement it and see how it faired in practice. My conclusion is that it does the job well, allowing a resource to not only provide multiple representations of itself, but to allow the authentication method to be chosen that best fits the client.

I share this because I am interested in comments from the RESTful community as to how others tackle this kind of problem. Is this a suitable use of content negotiation or am I pushing the whole RESTful ideology too far?

Is it better practice to separate the "UI" from the "API", in effect exposing a resource in two places (doesn't sound very RESTful to me)? Is it better practice to enforce only one type of authentication, making users accept the awkward way that browsers handle HTTP Auth?

On a final (implementation-related) note, I built the application in question using Pylons and for the custom authentication I used repoze.who which ended up being the perfect tool for the job. repoze.who is very pluggable and so with minimal code I was able to configure it to handle authentication in exactly the way I wanted. If I get a chance later I'll write about how I configured repoze.who with Pylons to handle dual authentication.

[1] When will the W3C improve HTTP authentication so that it can be optionally styled, doing away with the need for custom form/cookie auth for most web sites?

[2] When will browser makers add a simple logout option for HTTP Auth?

When designing the FLVio RESTful HTTP API I ended up choosing XHTML as the data representation format. My natural instinct was to use XML and invent my own schema, but RESTful Web Services convinced me otherwise.

While explaining to a customer today about simply using a web browser to help debug the API I said,

"It is no coincidence that we use XHTML to represent data as it is not only a well-understood XML format but also makes life much easier when debugging."

Which has proven itself true so far. Any browser becomes a debugging tool for the API. Although, until browsers support all the HTTP verbs (or XHTML5 / Web Forms 2.0) you'll need an addon like Poster for Firefox to test commands like PUT and DELETE.

One of my web applications is a CherryPy server that serves large files. I wanted to enable HTTP 1.1 byte range requests so I expected to have to get my hands dirty modifying my app to look for the right headers and do the dirty work.

Not so! I was already taking advantage of CherryPy's built-in helper function serveFile (cherrypy.lib.cptools.serveFile in CP 2) to efficiently serve static files back to the client. Glancing at the code for serveFile revealed that support for HTTP 1.1 byte ranges was already supported. But why were HTTP 1.1 range requests being ignored by my app?

The answer was simply that I had to tell CherryPy to enable HTTP 1.1 features. A quick change to the application config file to add:

server.protocol_version = "HTTP/1.1"

and a restart and success!

$ telnet media.serve.flvio.com 80
Trying 82.118.75.220...
Connected to media.serve.flvio.com.
Escape character is '^]'.
GET /media/mediakit/thumb/moovoob/2.jpg HTTP/1.1
host:media.serve.flvio.com
Range: bytes=10-20

HTTP/1.1 206 Partial Content
Date: Thu, 03 Jul 2008 10:10:15 GMT
Server: CherryPy/2.3.0
Accept-Ranges: bytes
Content-Length: 11
Content-Range: bytes 10-20/12438
Content-Type: image/jpeg
Last-Modified: Wed, 02 Jul 2008 05:47:28 GMT

51.57.1^]
telnet> cl
Connection closed.