Common Lisp and „Web 2.0“

The whole „Web 2.0„-Hyped Stuff has brought a lot of new good ideas and possibilities to communicate (and a lot of bad ones, too). I used to think that most new „platforms“ are useless, because they are mostly just a remix of old stuff which has been there 30 years ago already. And well, yes, the most „platforms“ like Facebook, 4chan or Twitter are not really innovative – anything they can do could have been done 30 years ago, maybe it would have looked a little less good, but it would have worked. There is no really new technology required.

But meanwhile I noticed, that being innovative is not really necessary to be „good“. For example, I was very sceptical about Twitter, but meanwhile I use it and really like it. Its simply more than the „sum of its parts“. I am addicted to blogging, and for anything which is too short to create a whole blogpost, twitter is all right. And meanwhile I accept other people’s sympathy to platforms like Facebook or 4chan or StudiVZ. Well, I dont really use them, because I dont share this feeling. But I can understand why Facebook is more than a „Wiki with User-Pages“. To the users, its simply more than the sum of its parts.

A huge problem with Twitter is, that I couldnt really find any Twitter-Client I really like. I.e. I couldnt find any Client thats really worth installing and using, compared to the web-interface. Not even closed-source ones.

Therefore, the first thing a nerd thinks of: Write your own with your favourite Language. Well, my favorite Language is Common Lisp. And there is already a simple Implementation of the Twitter-API, called cl-twitter – which I didnt try yet, but it doesnt support OAuth, which is recommended by the API, while the rest of the API is rather trivial. The hard part is to implement OAuth.

Ok, OAuth is interesting in general, so I consider to at least read the Spec to some degree myself, and maybe implement parts of it. But in „Web 2.0“ there are a lot of standards which are huge and which are hard to implement and for which there is no simple embeddable C-Library. This really sucks. Because embedding C-Thingies is mostly trivial, while embedding any taller Scripting- or Language-Environment can really cause headaching. Common Lisp is no exception in that case. Embedding Common Lisp into other Applications is really a pain! Which decreases the motivation of implementing a binding for some huge protocol – noone, except Lispers, could use it, noone else could be interested in improving it.

Well, it would be great if there were not that much of different „standards“ (which are mostly only used by one or two other platforms), but on the other hand, thats how the „Web 2.0“ works – someone has a problem, creates a solution, specifies it (hopefully), implements it, and then … it becomes a standard … or not. If he is interested in spreading his idea fast he will make efforts to have many popular languages supporting it. If not, he maybe just implements it in PHP or Perl. However a lot of work has to be done – implementing a huge protocol involves a lot of debugging, testing, and expierience with the protocol itself, and a deep understanding of the protocol you implement – which you can only gain by reading the specifications and tutorials and understand applications working with it – which you simply dont want to do when you quickly want to code up a small webpage. So most people will simply choose the language with the most libraries. And thats not common lisp.

(This is not only a common lisp problem. If you look at libgmail, which is a python library, using it you are bound to python, no matter if you want it or not)

What alternatives are there? Well, there are special cl-implementations like Armed Bear Common Lisp, there are trials of embedding perl, python, java, etc., in Lisp. There is clojure – which I dont like that much (soon or later I will have to get used to it, but not now), but also clojure is just „embedding“ Java-Stuff.

As far as I see there are only two solutions: Embedding or Reimplementing. Reimplementing is a lot of work. Embedding can make things slower due to calling-overhead, but once you can embed one language into another, you can use every of its libraries with only comparably little work. So I think, in most cases concerning „Web 2.0“, the calling-overhead isnt a problem – retrieving a document via HTTP should take much longer than parsing it into some appropriate data-structure and passing it from one language to another. Mostly you call one function which returns a List of some elements to you once – you dont call it thousands of times, so the overhead can be ignored.

Anyway, embedding can be complicated or even impossible. Highly engineered systems like Perl and Common Lisp need highly optimized systems that do a lot of „Magic“ – which can lead to the problem that they cannot be run in the same process anyway, if they are implemented to run on the PC’s native environment. There are virtual machines giving a common environment to them – most commonly, there is the Java VM and the CLI, and other approaches like LLVM and Parrot. These are made by people who – unlike myself – know what they are doing, and try to do it in a clean way, as far as I think.

I myself would think of simpler solutions to connect languages. Actually, I am trying to talk to PHP from CL using XMLRPC via stdin and stdout – because both do have XMLRPC-Libraries. This creates a lot of overhead (and maybe I’ll turn away from XMLRPC to an easier protocol), but I think it is basically the best solution to make sure to have access to the newest brainfarts of the web-2.0-engineers.

And well, I would not be the first one using such a solution.


7 Responses to Common Lisp and „Web 2.0“

  1. Err… so what’s the problem with using your Lisp’s FFI (or CFFI) to use the functions of a C library?

  2. dasuxullebt sagt:

    Nothing. As I said. Embedding a C-Library is mostly trivial, as long as it doesnt do too much „magic“. As far as I read, there are some problems of embedding Perl in Common Lisp, see for example. And embedding PHP via FFI is a pain even directly from C without Common Lisp (I didnt even get php-embed work, and it seems to be deprecated), and even if I get it embedded, I am pretty sure others wont (want to) get my FFI-Bindings work, because – well, no Linux-Distro has packages containing php-embed, and compiling it oneself is … hard – you have to completely rebuild it, if it was built without –enable-embed. And the embedding SAPI is considered deprecated.

    The easiest way I see is simply using some IPC/RPC-Mechanism. LTk does well. Why shouldnt some other API do so, too? (And PHP is made for this kind of access.) It is much less work. And it should be sufficient for most purposes.

  3. Zach Beane sagt:

    Wait, what’s the hard part about implementing OAuth? That seems like the *easiest* thing, to me.

  4. dasuxullebt sagt:

    Nearly anything from Twitter is available in JSON-Format. JSON-Format is easy to parse (and there is a good cl-library for that). The Requests are not hard to understand, and there are not much of them. This part is easy.
    Have you read the OAuth-Specification? You have to add additional Headers to your HTTP-Requests, use some Cipher- and Hash-Algorithms to sign your credentials, have a model of connecting different service-providers – well, its straightforward to implement, but good luck when you make a mistake and have to debug.
    If it really was that easy, why do so many new Twitter-APIs not implement it?

  5. Zach Beane sagt:

    Hey, Leslie Polzer heard your pleas and is working on cl-oauth at

  6. dasuxullebt sagt:

    Ah. Thank you. As soon as I find time I will look at it. Maybe I can help him (since I have read and experimented with that standard a lot).

  7. Actually I didn’t read the part about OAuth in the OP but started implementing it for another project some days ago.

    Feel free to join in the effort. I’m currently focusing on the Service Provider side and the common base. It should be easy to add the Consumer parts using Drakma.

Kommentar verfassen

Trage deine Daten unten ein oder klicke ein Icon um dich einzuloggen:

Du kommentierst mit Deinem Abmelden /  Ändern )

Google+ Foto

Du kommentierst mit Deinem Google+-Konto. Abmelden /  Ändern )


Du kommentierst mit Deinem Twitter-Konto. Abmelden /  Ändern )


Du kommentierst mit Deinem Facebook-Konto. Abmelden /  Ändern )


Verbinde mit %s

%d Bloggern gefällt das: