Tagged: python

Turbogears rocks


So I’ve finally started working on my first serious (serious might be too strong a word, but certainly non-toy) web app using [[http://turbogears.org|Turbogears]]. I don’t want to make this yet another Turbogears vs. X,Y,Z ([[http://rubyonrails.com|Rails]], [[http://www.djangoproject.com/|Django]], put-your-favorite-framework) post — after much reading (and a fairly low signal-to-noise ratio in a lot of such comparisons), I’ve decided that it doesn’t really matter. Functionally, almost all frameworks are more or less equal. Personally, I feel that for small projects, it boils down to personal taste.

For me, Rails was out from the start because I’m a Python guy. I’ve coded extensively in Ruby at one point, but I didn’t enjoy it as much as I enjoy coding Python and thats that. Besides, I’m heavily tied into some tools (such as matplotlib) that just don’t have comparable couterparts in Ruby (yet). To be honest I haven’t looked at Django in any depth, so I can’t make a fair comparison. On the other hand, I //have// been following Turbogears pretty much from its beginnings, although I haven’t tried it in any great depth either.

What it all boiled down to was that I had to pick one of them, any one, and dive in. Stick with it till I run into the meat, and then only I would figure out the strenghs and weaknesses. I chose Turbogears, because I really like the philosophy behind the project. Each of the component projects it itself quite mature. I’ve been tracking the mailing list of TG, and the community is great! There are a lot of smart people who evidently know what they’re doing.

I spent some hours with TG today and I’m really enjoying it. Here are some of the things that I thought were great, but which are not really highlighted as TG features. I think they should be emphasized a bit more, because they’re all pretty significant for any project:

* Extending Jeff Watkin’s [[http://nerd.newburyportion.com/2005/11/updated-identity-framework|identity framework]] was really easy and intuitive. I just derived my own User class from TG_User and thats it! I was all set for using identity!
* I really **loved** [[http://checkandshare.com/catwalk/|Catwalk]]. I mean, I cannot stress how useful it was for me. And how well it has been done. It is the perfect tool when you’re bootstrapping your database and testing your application. The interface is beautiful and intuitive, and even support SQLObject inheritance and joins!
* The entire [[http://turbogears.org/docs/toolbox/|toolbox]] is actually very useful, specially the WidgetBrowser. The things that I didn’t find of much use were the web based Python console and [[http://www.checkandshare.com/modelDesigner/|ModelDesigner]]. I think the latter is actually useful, I just haven’t gotten around to using it yet.
* Widgets: the new widgets are a great way for quickly putting together interactive elements on a page and even displaying data programatically.

So far I haven’t touched [[http://mochikit.com/|Mochikit]] at all, and barely scratched the surface of Kid and CherryPy, but with my experiences so far, I’m really looking forward to digging deeper. A comforting thought is that TG is highly flexible in terms of the components — people have used SQLAlchemy instead of SQLObjects, there are [[http://www.blueskyonmars.com/2006/01/06/template-plugins-for-everyone/|template plugins]] for Cheetah, Stan etc. So if I don’t enjoy this particular combination for some reason, I can atleast hope that my efforts into learning TG will not go to waste — something or the other will work out :-)

Help with Twisted!


If any of you out there have used [[http://twistedmatrix.com/|Twisted]], I need some help! The basic problem is this: I have a server, that needs to talk to another server before it can serve a client request (think of something like a recursive DNS query). Let me throw some pseudo-code to make the problem clear:


def client:
# this will actually be done via a callback, but I simplify for demonstration
print server1.getdata()

def server1:
# server1 has to talk to server2 to get the result
# further, server1 is an XMLRPC server written in twisted
# so the return value of the function is actually returned to the client
# therefore, I _cannot_ use deferreds here because if we jump
# into a callback, we lose context
def getdata():
# the call I make to server2 returns a deferred
# how do I block on it? I tried waitForDeffered, but couldn't get it to work
# examples much appreciated
return server2.callRemote().addCalback(really?)

I don’t know if the code/comments help. Basically my client is talking to an XML-RPC server, which needs to talk to another XML-RPC server to serve the requets. In ”twisted.web.xmlrpc”, calling remote methods via a Proxy object returns a deferred. However, server1 needs to wait for this deferred to fire/finish before it can return to the client. How do I do this? Any ideas?

**Update**: On [[http://www.cs.ucsd.edu/~mvrable/|Mike's]] suggestion, I tried simply returning the Deferred in server1, and guess what, it worked! It seems that when I try to return a Deferred from Server-1, Twisted automatically waits for the Deferred to fire, and returns //its// result instead, which is exactly what I wanted. Ah, the mysteries of asynchronous programming.

Tools I use: matplotlib


I get a lot of questions along the lines of “Hey Diwaker, what do you use for blah?” (insert your requirement there). Apparently, I seem to have a talent for finding “smart” tools that people like using. So I figured I should blog about some of the tools I used. Maybe others can benefit.

I’ll skip a couple of obvious ones here: my editor of choice is [[http://floatingsun.net/blog/tags/vim|vim]], and I used [[http://floatingsun.net/blog/tags/wordpress|wordpress]] for my blogs.

Let me instead come to something that //every// grad student ends up doing a lot of — making graphs (well, almost every. My friends in theory hardly draw graphs). And frequently, even people outside of research need to make pretty looking graphs and plots. Within academia and researchers, [[http://www.gnuplot.info|gnuplot]] has been the defacto plotting tool for as long as I can remember (I’m pretty sure it goes back atleast a decade, if not more). Outside the research community, most people tend to use the plotting tools that come with Office software — M$ Excel or Powerpoint and the likes.

Don’t even get me started on the Excel/Powerpoint crap. Maybe they are good enough for a quick and dirty work. But for anything more than that, for doing any //real// analysis/visualization, they are pretty much useless. For me, a good tool must meet the following requirements:

* It must be scriptable: I don’t want to have to open a bloated GUI and click a 100 buttons and drag-and-select columns to get a plot out. When dealing with large amounts of data stored in myriads of files, it is **critical** to be able to script/automate the process.
* It must support multiple output formats: EPS, PS, PDF, PNG, JPG, SVG are the ones I usually need. (E)PS/PDF for embedding in papers. PNG/JPG for viewining/emailing. SVG is just cool :-)
* It must support a variety of graph types: bar charts, pie charts, histograms, error bars.
* It should be **highly** customizable: tick size, label fonts, colors, line styles, thicknesses, positioning, subplots, grids, log scales, transparency, marker styles — EVERYTHING.
* Easy things should be **really** easy, and complicated things must be possible.

GNUPlot has served us quite well over the years. Atleast in CSE, I can confidently say that close to 80% of all graphs in papers are done using GNUPlot. In rare cases its OpenOffice/Excel. But GNUPlot is showing its age now: it can only deal with very simplistic input formats, its not very customizable, it supports a very limited number of graph types (AFAIK, it //still// doesn’t support bar charts natively). But for me, the biggest gripe is that it forces me to break my data analysis phase in two steps: in the first phase I write some scripts (typically in Python) to process the raw data into a form that can be consumed by GNUPlot; in the second phase I write another script (in GNUPlot) to do the actual plotting.

Enter [[http://matplotlib.sourceforge.net/|Matplotlib]]: this is easily the **best** plotting library I have ever used. Endlessly customizable, Matplotlib can do almost [[http://matplotlib.sourceforge.net/screenshots.html|any kind of plot you can imagine]] and some more. Apart from the traditional object oriented interface, Matplotlib also gives a very simple MATLAB (R) like interface for easy plotting. The API maintains a high degree of compatibility with MATLAB API, so MATLAB users will feel right at home.

Furthermore, since it is written in Python, it means that I can unify my data analysis — the possibilties are endless. I can feed all kinds of data directly to Matplotlib. I can process, analyze and plot in the same script. I can make my scripts highly generic (since they are in Python, I can pass command line parameters and what not — none of this is possible with GNUPlot).

Here’s a code fragment to make a really simple plot (stolen from [[http://matplotlib.sourceforge.net/tutorial.html|the excellent tutorial]]):


from pylab import *
plot([1,2,3])
xlabel('time')
ylabel('volts')
title('A line')
show()

And, best of all, you get a fabulous, interactive user interface for free! Yes, I know GNUPlot has an interactive mode too, but this is beyond comparison. You can pan, zoom, go back-forward in view history, save and what not. Here’s a screenshot:
{{ http://matplotlib.sourceforge.net/tut/navcontrols2.png?300×200|Toolbar2}}

Finally, since its Python, its very portable. It supports a variety of [[http://matplotlib.sourceforge.net/backends.html|backends]], [[http://matplotlib.sourceforge.net/interactive.html|interfaces with ipython]] and is under active development (at version 0.86 currently). So next time you have to do a plot, consider doing it in style — do it with matplotlib!

Projects

Most of my new code can be found on Github.
  • gaestebin: a secure, private pastebin for Google App Engine
  • uBoggle: the best way to play Boggle online! (taken offline because of.. ahem..)
  • ucsd-thesis: A LaTeX class to typeset thesis for the University of California, San Diego

WordPress related

Python

Random

Old/Abandoned Projects

  • Starfish: a Java based password manager.
  • JUnC: a Java compiler written in Java.