nonzen journal

Shiny yak: shave all HTML email!

2012-02-06T00:00:00-05:00

The year is 2012. Some curmudgeonly persons that missed the memo still refuse to embrace web 2.0 and hold on to their beloved, but old-fashioned, fetchmail + procmail + mutt + emacs + msmtp setups for email processing. Your humble correspondent is one of these persons. This blog thing (how quaint) is obviously an evidence.

At some point in the past, I was so curmudgeonly that my email signature was this:

sigs waste bandwidth.

If you're in the above-stated curmudgeonly group, you might even recall this blast from the past:

:    /"\
:    \ /    ASCII Ribbon Campaign
:     X   against HTML email & vCards
:    / \

That war was valiantly fought, and lost. (It seems that the old warriors have silently admitted defeat and fatigue. Maybe winning the war against vCards was a reasonable compromise.) One reason would be that most email clients (notably Gmail's web interface and Outlook) chose to ignore the outcry and their default is to compose HTML email. Another reason would be that bandwidth and disk space are not expensive and scarce anymore. Yet another reason would be that, unlike days of the olde, majority of email users are not nerds that care about this stuff.

This however bothers our curmudgeonly person, who has faithfully subscribed to a number of old-fashioned mailing lists that are (of course!) is affected by the the "problem" of HTML email. (Dealing with HTML email from friends and associates was enough, but you'd expect better from fellow nerds in bloody mailing lists. But alas.) On confronting this nasty problem everywhere, our curmudgeonly person would think: "I know, I will unleash the might of procmail on these bastards!"

Now he has a buttload of butt-ugly problems. You've got to pay attention to this anyway, because this is a kinda-sorta shiny yak.

What didn't work

One trouble with HTML email is that it is usually really composed of two parts: a text/plain part, and a text/html part. With procmail, it should be fairly straightforward to add a filter to get rid of the text/html part, but it wasn't. I tried a couple of things that didn't work:

Use pm-jamime-kill.rc from procmail-lib: this didn't work since it depends on mimencode which is not available in Debian any more. See, even those curmudgeonly Debian people have given upon you!
mimefilter would suit old-fashioned curmudgeonly mailing list admins. It scrubs the offending email clean of text/html part before forwarding it to a list, and sends a stern warning to the originator of the email. I might be curmudgeonly but I'm not a mailing list admin, so I don't want to warn anyone. I just want "cleaner" email. There's no configuration parameter to change this behavior, and I wasn't inclined to hack on the script.

What did work

The solution was finally found in a little script found in an old Perlmonks.org post. (The title of that post, Strip Brain-Damaged Mails of "HTML Alternative" Evilness!, is perhaps reflective of global nerd community's sentiments of the time.) I saved the script, and stuck this to my ~/.procmailrc:

CLEANMAIL=$HOME/bin/clean-mail.pl

:0
* ^((List-Id|X-(Mailing-)?List):(.*[<]\/[^>]*))
{
    LISTID=$MATCH

    :0:
    * LISTID ?? ^\/[^@\.]*
    | $CLEANMAIL >> $MATCH
}

This will "automagically" sort mailing list mail into folders by name of the list, while cleaning the suckers of any evil text/html. (There's a bit of Org-mode fail in there: please ignore the ORG-LIST-END-MARKER line if you see it.)

Conclusion

You might be inclined to make fun of Perl (and procmail) because (a) you're too cool to use this old ugly hippie proletarian stuff, or (b) your complain that Perl and the such it ignores advancements made in programming language research over the last twenty or so years, or (c) you tend to compare ParrotVM to Duke Nukem Forever, or (d) all of the above.

But I've got to advocate: it works, and it works flawlessly, for certain values of "works". It's a shiny enough yak.

Building Copperhead

2011-10-16T00:00:00-04:00

This semester, I've signed up for three courses:

All of them have been great so far. The last two are graduate seminar classes and both classes have a project. I'm happy about this setup since there is some overlap in what I'm doing - the DSL class specifically focuses on various forms of parallelism, so I get to study more of the same thing. Projects for both these classes involve some amount of GPU computation, so it's more of more of the same thing, or something.

Copperhead

(The image is from nvidia's site.)

For the DSL class, the project I've signed up for is a comparative study of Copperhead and Accelerate, which are GPGPU DSLs for Python and Haskell respectively. There is some literature available: mainly, the PPoPP paper, Compiling an Embedded Data Parallel Language and Brian Catanzaro's PhD thesis, Compilation Techniques for Embedded Data Parallel Languages on Copperhead; and Accelerating Haskell Array Codes with Multicore GPUs on Accelerate.

I'd been having some trouble with building Copperhead – mainly because I haven't done any Python in a long time, and because I'm installing stuff into non-default directories, and because I've been lax in reading documentation. So I thought of writing down the exercise here.

The machine happens to run Gentoo 2.0.3; and I do not have admin access to this machine. So some of the dependencies are installed in my home directory, under ~/software. Therefore I also need to set some environment variables:

export PYTHONPATH=\
        $HOME/software/lib64/python2.7/site-packages:\
        $HOME/software/lib/python2.7/site-packages:\
        $PYTHONPATH
export PATH=~/software/bin:$PATH

The dependencies are:

Python 2.7. We have Python 2.7.1 on this machine.
CUDA 3.0. We have 4.0.
numpy 1.3. We have 1.6.0.
Boost 1.38, though only boost-python and boost-thread are really needed. We have boost 1.42.
PyCUDA - this was a pain point. PyCUDA's build.py doesn't seem to support installing with non-root permissions ("python setup.py" would say: "--prefix? We don't know what that is!"). Finally I actually read Python distutils documentation and figured out this should have been actually really painless, and felt quite embarrassed:

git clone http://git.tiker.net/trees/pycuda.git
cd pycuda
easy_install --prefix=~/software .

This installs PyCUDA, including the dependencies. (decorator, py, pytest, pytools etc.)

(This doesn't quite work – see the January 3 update below.)

CodePy, which is a C/C++ metaprogramming toolkit in Python! It's "meta" in the sense that it supports:
- Generating C/C++ code;
- And then compiling the generated code and, then loading it dynamically into Python runtime.

Installing CodePy turned out to be straightforward:

git clone http://git.tiker.net/trees/codepy.git
cd codepy
easy_install --prefix=~/software .

There's some noise about a missing cgen dependency; I chose to ignore it for now.

Thrust, a C++ CUDA library that resembles STL. Since this is a template library, there's really nothing to "build" and install; furthermore, CUDA 4.0 ships with Thrust. Squee!

(As a side note, the last three projects appear to be very interesting on their own and deserve further investigation. This I will hopefully get to do before the end of the semester.)

Finally, installed copperhead itself:

hg clone https://code.google.com/p/copperhead/
cd copperhead
easy_install --prefix=~/software .

No trouble there either. However:

$ python -c "import copperhead;"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "[...] copperhead/__init__.py", line 20, in <module>
    from prelude import *
  File "[...] copperhead/prelude.py", line 46, in <module>
    import copperhead.runtime.places as PL
  File "[...] copperhead/runtime/__init__.py", line 29, in <module>
    cuda.init()
pycuda._driver.RuntimeError: cuInit failed: no device

This probably means we're in trouble:

$ python -c "import pycuda.driver as cuda; cuda.init()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
pycuda._driver.RuntimeError: cuInit failed: no device

Bummer. It looks the machine doesn't have a CUDA driver installed yet:

$ /opt/cuda/sdk/C/bin/linux/release/deviceQuery
[deviceQuery] starting...
/opt/cuda/sdk/C/bin/linux/release/deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 38
-> no CUDA-capable device is detected
[deviceQuery] test results...
FAILED

Press ENTER to exit...

Double bummer!

Turns out that the nVidia GeForce 7300 GT isn't listed as a CUDA device in nVidia's website. This is surprising because: two weeks back I wrote and executed a simple OpenCL program just fine with CL_DEVICE_TYPE_GPU with nVidia's SDK. The other option, CL_DEVICE_TYPE_CPU did not work – my guess is that nVidia's OpenCL doesn't do CPUs yet, despite the "heterogeneous" moniker. The machine has Intel CPU, but no Intel OpenCL SDK installed.

On the other hand, my Thinkpad, which is a recent Sandy Bridge machine, could do CL_DEVICE_TYPE_CPU with Intel OpenCL SDK, but they don't do CL_DEVICE_TYPE_GPU yet. But of course, no CUDA since this is Intel.

(Yes, there are several vendors in this space: nVidia, which pushes both their own CUDA as well as OpenCL; Apple, Intel, AMD, PowerVR, IBM et al. which work with Khronos industry consortium and has support for OpenCL in varying degrees.)

Now to go look for a machine with an actual CUDA device. The perils of bleeding edge…

Updates

(Updated on 22 Oct 2011)

It turned out that I was wrong, which is really awesome, because: GeForce 7300 GT is actually a CUDA device! The actual problem was that I wasn't in the video group, and once that was solved, I could run some CUDA demo programs (such as deviceQuery), and they agreed that there indeed is some CUDA device. (Which is: Device 0: Tesla C1060.)

It didn't end here though.

The next trouble was with Python package cgen, which was mentioned earlier. Cgen used to be a part of codepy, but these days it is a separate project with a separate Python package and everything. And this package turned out to be uninstallable via the usual means:

$ pip install cgen
Downloading/unpacking cgen
  Could not find any downloads that satisfy the requirement cgen
No distributions at all found for cgen

$ easy_install cgen 
Checking existing site.py in $PYTHONPATH
Searching for cgen
Reading http://pypi.python.org/simple/cgen/
Reading http://mathema.tician.de/software/codepy
No local packages or download links found for cgen
error: Could not find suitable distribution for Requirement.parse('cgen')

(I suppose I should alert the maintainer. I've been busy.)

So I installed cgen like so:

git clone http://git.tiker.net/trees/cgen.git
cd cgen
easy_install --prefix=$HOME/software cgen

I should be almost ready to run some simple test programs, but wait! Now it's CUDA turn to complain: gcc-4.5 is not supported, and this is presumbaly because of gcc's move to DWARF. gcc-4.4 is supported and was available, so I created a symlink ~/bin/gcc to /usr/bin/gcc-4.4.5, and placed $HOME/bin ahead in the $PATH.

Next problem was a missing libboost_python-gcc4.3-mt. The system has boost_python installed; however codepy's default is to use the former. This was solved by having an ~/.aksetup-defaults.py, with the following:

BOOST_PYTHON_LIBNAME=["boost_python"]
BOOST_THREAD_LIBNAME=["boost_thread"]

This obscure piece of information came from here. I have no idea what does aksetup mean; nor could I find an explanation in codepy where they look for the said file.

We have one more environment variable to set up:

export THRUST_PATH=/opt/cuda/include

At this point one of the sample programs bundles with copperhead run. Yay!

$ python simple_tests.py 

---- Simple INTEGER tests ----
Procedure 'incr'                                   ... PASSED
   copperhead : [1, 2, 3, 4, 5, 6, 7]
Procedure 'incrList'                               ... PASSED
   copperhead : [1, 2, 3, 4, 5, 6, 7]
Procedure 'as_ones'                                ... PASSED
   copperhead : [1, 1, 1, 1, 1, 1, 1]
Procedure 'idm'                                    ... PASSED
   copperhead : [0, 1, 2, 3, 4, 5, 6]
Procedure 'idx'                                    ... PASSED
   copperhead : [0, 1, 2, 3, 4, 5, 6]
Procedure 'saxpy'                                  ... PASSED
   copperhead : [1, 3, 5, 7, 9, 11, 13]
Procedure 'saxpy2'                                 ... PASSED
   copperhead : [1, 3, 5, 7, 9, 11, 13]
Procedure 'saxpy3'                                 ... PASSED
   copperhead : [1, 3, 5, 7, 9, 11, 13]
Procedure 'sxpy'                                   ... PASSED
   copperhead : [0, 2, 4, 6, 8, 10, 12]

---- Simple FLOAT tests ----
Procedure 'as_ones'                                ... PASSED
   copperhead : [1, 1, 1, 1, 1, 1, 1]
Procedure 'idm'                                    ... PASSED
   copperhead : [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0]
Procedure 'idx'                                    ... PASSED
   copperhead : [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0]
Procedure 'sxpy'                                   ... PASSED
   copperhead : [0.0, 2.0, 4.0, 6.0, 8.0, 10.0, 12.0]

Sadly that is the only sample I have managed to run. None of the other sample code worked, despite trying to get their dependencies also (matplotlib, plac, scipy…) to work, and grinding teeth harder. They all try hard, spew out some compiler messages and a Python backtrace, and finally fail with the same error:

codepy.CompileError: module compilation failed

This is, in fact, consistent with the disclaimer, so I guess I should not complain too much. :-)

Copperhead is currently under development. Many valid Copperhead programs do not yet compile, and the compiler does not produce helpful error messages. Code that does compile and run may execute inefficiently, compared to hand-coded CUDA. Join the mailing list and let us know of your experiences, but don't expect things to work right out of the box.

Alright, off to join the mailing list then.

(Update: eventually rest of the tests also worked after rebuilding PyCUDA with system-installed version of Boost – see the January 3 update below.)

Afterword

Needless to say, overall I found this whole experience very… tedious. I documented the whole process nevertheless so that (hopefully!) I or my suffering partner in this project would not have to figure all this stuff out again if there is another time. I understand this is the natural price one pays for using software that has gone through very limited field testing, and that Copperhead did not had a chance to gather a community around it, at least yet, and that it's been a one-man project. Still, I'd have liked the setup part to be less work than this.

Further, sadly, little work has been done after Copperhead's author graduated from Berkeley. There are precisely three commit messages in the repository and all of them are from September 2010 and the project hardly seems to have made any progress after that. There's some talk about a roadmap in the mailing list though.

What is really interesting, however, is that Accelerate is nicely chugging along, in spite of Haskell's seemingly smaller community. (You'd agree that it's way smaller as opposed to Python community, yes?) Freshly checked out Accelerate builds just fine on my Thinkpad T420 (with the alternate CPU backend, not CUDA) without any of the bothersome setup that Copperhead demanded; there are hundreds of commits in the "main" github repository; and ten other forks not counting mine. (Which is yet to see some activity. If only the semester would let me!)

Another round of updates

(Updated on 3 January 2012)

I was wrong in assuming that work on Copperhead has come to a stop after Bryan's graduation and move to nVidia. Bryan is still working on this project, and his updates are present on a cloned repository. Hopefully we'll get to see a release soon; and I sincerely hope that Copperhead, just like Thrust, will eventually become part of nVidia's official CUDA SDK release. That would seriously help in putting an end to the era of boilerplate-ridden, highly error-prone, highly frustrating process of writing GPGPU code.

Secondly, PyCUDA as installed as above does not quite work, and after asking in the mailing list it turned out that this is because of PyCUDA shipping with a version of Boost. The solution is to use system-installed Boost. These settings, from from PyCUDA's siteconf.py, are what worked for me:

BOOST_INC_DIR=["/usr/include/boost-1_42/"]
BOOST_LIB_DIR=["/usr/lib"]
BOOST_COMPILER = 'gcc'
USE_SHIPPED_BOOST = False
BOOST_PYTHON_LIBNAME = ['boost_python']
BOOST_THREAD_LIBNAME = ['boost_thread']

That made rest of the tests also work.

My impressions with Copperhead, in spite of the rather painful installation procedure, are extremely positive: for code that is practically developed by one person, it works quite well; and in our benchmarks Copperhead code outperformed Accelerate. This is not quite surprising if you consider the fact that Copperhead-decorated Python code in reality is loaded and run as native code, and not interpreted. Admittedly our benchmarks are by no means exhaustive or even complete; we merely benchmarked what's found to be the "common surface area" of code that we could get working. Which was, hopefully, Good Enough (TM) for a course project.

Bulb Song Man

2011-09-25T00:00:00-04:00

Bulb Song Man.

This I drew using Gimp and a Wacom Bamboo, on request from Shiv. I would claim that this drawing bears some resemblance to Shiv, but let us hear from our man first!

Meow!

2011-08-21T00:00:00-04:00

Meow!

I have been slowly slowly moving stuff into my new apartment. It is technically mine now since I have signed the lease and paid the deposit and everything, but not quite mine since I haven't moved in entirely. It is kinda nuanced like that.

I took over the lease of this apartment from Roshan, and he has left a ton of stuff behind. The furniture and kitchen appliances and utensils are certainly useful and saves me a great deal of hassle with piling up all that stuff again. Some of the stuff will be shipped to wherever Rosh is finally settling down. He has done quite a job of leaving a clean apartment behind.

But there are certain kind of stuff, like computer cables, that just refuse to be clean. While clearing a dusty cable mess from beneath the computer table, I found a Wacom Bamboo tablet.

Awesome. I've been meaning to get one of these for many years now. Finders ought to be keepers, yes? Especially when it works in my favor, yes?

Even if I don't get to keep this one, I'm convinced that I should eventually get one. This tablet just worked with Debian wheezy/sid, and Debian has mypaint, gimp, inkscape etc. which works quite well with this Bamboo. That is a really sweet deal.

I do need some practice with my wobbly hands though. I'm really not keeping my hopes up since the acrylic paint and sketchbook I bought for using in my summer skills development plan haven't been touched yet.

Hello world!

2011-08-18T00:00:00-04:00

Bloomington, August 2011.

Text files are pretty awesome. So are distributed version control systems, such as git and mercurial. Not everyone would agree, but I think Emacs is really awesome. I've been seriously using Emacs Org-mode for a while now, as a todo-list and calendar/agenda software, and for keeping notes and stuff, and for putting a quick webpage together. Org-mode is pretty neat.

Weblogs are nice too. That people can publish their thoughts and ideas and stories, and have them read by friends are strangers in the Internet, and sometimes hear back from them, is pretty nice. They're sort of out of fashion these days, since the kids are all now using the new shiny social network platforms. Weblogs still serve some useful function, in my opinion.

But do you know what's really awesome? Version controlled text files, written using Org-mode, published in weblog form. Those are pretty grand. You don't need pesky database connections and pesky PHP code which runs on the server, which really slow down things and make things annoying. Really nifty.

I ~~have~~ had been a regular user or livejournal for some years, which had been a pretty decent deal in its days; but poor old livejournal is practically a ghost town these days. The servers are kind of shaky, and are often bought to their knees by random DDOS attackers, and there's fly-by advertising everywhere. Furthermore, most of the people that really made the place, they're no longer using livejournal, and a large number of those journals are practically dead for lack of updates. It's not the kind of place you would want to invite your friends anymore. This appears to be the fate of social networks of all fashion: they rise and fall in popularity. I do however respect and admire the few friends who continue using livejournal; I just have lost the resolve and desire to do so.

In the meanwhile, I have been missing out on telling the story of my life. I might have done a few interesting things in the last few years; but I have no written memory of those events. For example, I left my career of several years as a humble working class programmer and came to graduate school at Indiana University. It's been a fairly big change, and I have been having a whale of a time. I would have liked to keep a record of what I've been doing and how much of a doofus I have been, but alas.

In the past I have learned much as a result of those attempts, and I would like to continue doing so. This is that attempt. I want to keep a record of my graduate school experience, as well as other life experiences here; and I want to stay relatively unaffected by the shenanigans of platforms run by other people. Yup, that is the goal.

This is a test post made using Emacs, Org-mode and Jekyll site generator.