Senko’s corner of the Web

Imagine you’ve seen a blog post titled Free linux-like kernel sources for x86-64 PC’s with the following contents:

Do you pine for the nice days of linux 2.4, when men were men and wrote
their own device drivers? Are you without a nice project and just dying
to cut your teeth on a OS you can try to modify for your needs? Are you
finding it frustrating when everything works on linux? No more all-
nighters to get a nifty program working? Then this post might be just
for you :-)

As I tweeted a month(?) ago, I’m working on a free version of a
linux-lookalike for x86-64 computers. It has finally reached the stage
where it’s even usable (though may not be depending on what you want),
and I am willing to put out the sources for wider distribution. It is
just version 0.02 (+1 (very small) patch already), but I’ve successfully
run bash/gcc/gnu-make/gnu-sed/compress etc under it.

Sources for this pet project of mine can be found on github. The repo also
contains some README-file and a couple of binaries to work under it
(bash, update and gcc, what more can you ask for :-). Full kernel
source is provided, as no linux code has been used. Library sources are
only partially free, so that cannot be distributed currently. The
system is able to compile “as-is” and has been known to work. Heh.
Sources to the binaries (bash and gcc) can be found at the same place.

ALERT! WARNING! NOTE! These sources still need linux to be compiled
(and gcc, possibly gcc4, haven’t tested), and you need linux to
set it up if you want to run it, so it is not yet a standalone system
for those of you without linux. I’m working on it. You also need to be
something of a hacker to set it up (?), so for those hoping for an
alternative to linux, please ignore me. It is currently meant for
hackers interested in operating systems and x86-64’s with access to linux.

The system needs an IDE harddisk (sorry) and any VGA card. If
you are still interested, please read my blog post about it, and/or ask me
on Twitter for additinal info.

I can (well, almost) hear you asking yourselves “why?”. Next year (or the
one after it who knows) will be The Year Of Linux Desktop, and I’ve already got
Ubuntu. This is a program for hackers by a hacker. I’ve enjouyed doing
it, and somebody might enjoy looking at it and even modifying it for
their own needs. It is still small enough to understand, use and
modify, and I’m looking forward to any comments you might have.

Does this sound familiar to you? If you know your Linux history, is should - it’s a ripoff of the original Linux announcement post that Linus posted to the comp.os.minix newsgroup.

A few points to consider:

• look what a crazy (in a positive way) guy’s hobby did to the computer industry
• is kernel programming still fun? should it be? is it serious work now?
• is there still fun to be had in non-web related stuff?

N900 is a device with a lot of connectivity options and a very capable browser. With that, it’s a good Wikipedia reader out of the box. But not so if your connectivity is limited (you’re on top of a mountain, or roaming, or don’t have a data plan alltogether and there are no open wifi hotspots).

Since I got a great Christmas gift from Collabora I’ve been poking at a small toy application that could store and read Wikipedia articles offline. Since Wikipedia hosts a huge number of articles, and device capabilities are limited compared to a desktop PC, this posed an interesting (but not unsolvable) challenge for the weekend hack sessions.

The result is Mawire (Maemo Wikipedia Reader):

Mawire 0.1 - home screen

Mawire 0.1 - search results

Mawire 0.1 - Article view

The application

Having worked on bits and pieces in Maemo 5, I knew my way around the Maemo 5 SDK and some of the APIs, but nevertheless the Developer’s Guide was of great help. I’ve also perused examples, code (such as browser launcher) and packaging from marnanel’s raeddit application.

The application is a lightweight reader, so the aim is not to display complete article, but rather to show enough information for a quick check, and provide convenient way for the user to find more on the Wikipedia itself.

At the moment you can download the application from here, or browse or download the source code from github. I hope to upload it to maemo-extras soon. If you’re reading this from your N900, you can install mawire automatically by clicking on the install icon. Warning: It’s an early development (alpha) version, only tested on my device so far, so proceed with caution and only if you know what you’re doing.

Since writing this blog post (but before I hit the Publish button), I’ve played around with portrait mode functionality, and released version 0.2 which has portrait mode support. If your keyboard is closed, mawire will switch to portrait mode. When you slide it open at any time (e.g. for typing search queries, or when you want to copy/paste part of the article), mawire will switch to landscape mode.

Data handling

Since Wikipedia (especially the English edition) has a huge number of articles, amount of text for a complete and comprehensive copy is just too large to conveniently use on the device. Recent enwiki dump contains more than 3 million articles and is almost 6GB of bzip2′d XML.

To minimise database size, and since I wasn’t trying to replicate complete Wikipedia functionality, I decided to strip the articles as much as possible, not only of markup (so only bold and emphasis are preserved), but also to only include content of a few first paragraphs - up to the first heading. The idea is that topic overview is probably outlined first, and then each paragraph expands the coverage (often also having a complete article of its own).

So, the reader only includes the overview and provides “Read more…” button that connects to Wikipedia proper. So the app can be used not only as offline reader, but as quick Wikipedia launcher by itself.

The other problem is number of articles and search performance. If database contains up to a few million articles, sequential searching through the database is extremely slow. Unfortunately, the version of SQlite3 shipped on N900 (and indeed in many Linux distros ATM) doesn’t support fast fulltext search (FTS3), which is ideal for mawire.

So, the application currently ships with its own copy of SQLite library with enabled FTS3 module. It’s installed in a private lib directory so it doesn’t clash with the OS version (similar to what Firefox on Linux distros does), and is only ever used by mawire.

The data itself was prepared by a Python program that:

1. parses the XML dump (using expat),
2. extracts, parses and strips Wikipedia markup (using a bunch of hacked regexps, as I haven’t found a suitable wikimedia markup parser - this is the weakest part of the program and I hope to improve it in the future),
3. filters articles we want to exclude (special pages, lists of things, too short articles),
4. compresses the article text (using zlib) and finally stores them to a SQlite3 database (using SQLalchemy).

The final step is manual, building of FTS3 index, and consists of two SQL statements. This is so it could be done separately, on a machine having SQLite with FTS3.

The program is included in mawire source code, so you can use it to create custom Wikimedia databases (or, indeed, database for any MediaWiki powered wiki).

A database of selected articles (3000 most visited + featured + good + vital; about 14 thousand articles in total) from English Wikipedia (13.6MB). Database is installed in /opt so it doesn’t fill up your rootfs.

Complete English edition, as well as several other major language editions are also available, but I haven’t created Maemo packages for them. You can download them directly, put them on your device or MMC card and select from the application menu.

• English edition, complete (1.4GB)
• French edition, complete (313MB)
• German edition, complete (479MB)
• Russian edition, complete (289MB)
• Spanish edition, complete (288MB)
• Croatian edition, complete (31MB)

Automatic statistical translation tools like Google Translate are getting better and better. They still often produce nonsensical and grammatically incorrect translations (depending on available language corpus, I guess), but the translations can be used to understand the crux of the text.

So, what happens if we try to apply these tools to translating software (in specific, translating gettext potfiles, which is the most widespread way of providing i18n in free software I know of)? Might be good, might be bad, and it will probably be hilarious ;-)

So, that’s my new pet project, googtext, does. It’s a small wrapper around Google AJAX Language API that allows users to upload their potfiles, extracts the message ids, passes them through the language api, and saves the resulting message to a new potfile the user gets to download.

Translations generated by googtext should certainly be redacted by a real person, but I think it’s sometimes easier to improve upon a bad translation than start from scratch (depending on the situation), so this might have some use (or just might be funny :) So, try it and let me know what you think in the comments.

The entire code for this is in the public domain, so if you want to play with the code, you can download it here. Since I couldn’t find any python module for Language API, I’ve made a thin wrapper myself (gtranslate module). There’s also the googtext utility itself (commandline), and the accompanying PHP upload script and HTML pages.

Have fun!

In the Telepathy stack there are a lot of components, so often it gets very confusing for people hearing about it for the first time. So, here’s a very basic breakdown:

Telepathy

A framework for doing IM and VOIP. Supports Jabber/GTalk, SIP, IRC, MSN, ICQ,… (not all of those support VOIP at the moment). It has no UI and the end user doesn’t really need to be concerned about it.

Empathy

A IM/VOIP client for GNOME that is using the Telepathy framework.

Farsight

An extension to GStreamer that supports VOIP. (Farsight devs, mea culpa for this dumbing down so much). Farsight is used by Telepathy for actually moving around audio and video in calls.

That’s it. Easy, no? :-)

What are you doing?

Microblogging is now much more (well, for some) than just answering the above question, but some people do want to publish what they’re doing at the moment. If they also use an IM client (and who isn’t?), it makes sense to make the IM status message and “what I’m doing” tweets the same.

As my technology of choice when it comes to IM is Telepathy, I started thinking about creating a connection manager (essentially, a protocol support driver) for Twitter, but then it dawned to me there’s much simpler and better way to do it, if you just want to retweet the status messages.

My preferred Twitter client is gwibber, written in Python, and it was very easy to extend it to listen to presence status change messages from Telepathy’s Mission Control and repost them to the configured microblogging services. Together with Davyd’s work on improved status message widget for Empathy, I think this is going to be useful for people changing their presence often (and not using canned statuses).

To keep the requirements to the minimum (and since it’s a very simple client), I’ve used dbus-python directly, and also wrapped the whole thing so that patched gwibber continues to run even if there’s no dbus-python available. The big ugly chunk of the change in the diff is preferences.glade, where for some reason glade reindented a whole lot of lines that I never touched (another table row with a single checkbox is the only UI change).

The code is in bzr branch in Launchpad. Go, grab it and play with it! Keep in mind it’s fresh and not thoroughly tested. Also beware, ye rhythmbox (or other music player) users that update your IM status with every new song - this will spam your microblog if you let it :)

Does this mean I think Empathy should become microblogging client? Absolutely not. There’s more to microblogging than sharing statuses, and existing clients (e.g. gwibber) already do it very well. This patch is just a way to integrate previously disconnected parts of your digital life (ie. IM and microblogging). To publish your status (both on IM and Twitter), use Empathy. To microblog, use gwibber.

Yesterday Google began rolling out video support for GMail. The support is in form of a browser plugin, which is currently available for Mac and Windows platform.

While it’s nice that GMail gets audio and video capabilities, having to download and install plugin voids the webapp benefts (available anywhere), and raises a question on why videochat wasn’t done as an upgrade to desktop Google GTalk client.

The devil is in the (protocol) details

What’s more problematic is the way Google decided to implement these features in the protocol. As GTalk is actually XMPP (with a couple of Google extensions), the good news is that Google continues using XMPP for the new features. The bad news is that Google completely ignored XMPP standard for voice and video (a.k.a Jingle RTP) sessions, and invented their own signalling.

To make matters worse, the way they decided to implement it is really screwed up. In a video call, there’s typically two media streams - one for video and one for audio (if you don’t want audio, you can set up just the video one). Google’s videochat also uses two streams - but, instead of signalling them as two jingle contents, they bundle all the information into one stream, and then have to do ugly hacks with transport candidate naming to distinguish between ports for each of those, and so on.)

All this can be worked around with, and while it will make everyone’s client into shpagetthi code for supporting such a twisted implementation, it can (and will) be solved. The real problem is that,they’ve decided on supporting only one codec, H.264/SVC that practically nobody uses (well, up until now), and that has no free implementations (apparently they’re using the codecs from Vidyo). They could’ve added additional support for any other codec (e.g. H.263, or Theora), that could be used when calling from/to non-gmail client) .. but they didn’t.

While they might have the best of intentions, this all looks very different from the “spirit of open communications” they’re mentioning. While it’s too late for them to change the signalling protocol, I hope they’ll be more sensible about the codecs, support more open (or at least more widely used) ones, and try to play nicer with the rest of us.

A few days ago (actually two weeks ago, but those were pretty hectic weeks) I attended Croatian Linux User’s Conference [DORS/CLUC'08], and gave a presentation about Telepathy there.

Although the slides are in Croatian [well, a few words, anyways, because I didn't force translations for untranslatable terms], there were some english-speaking people in the crowd so I presented in English (and was a bit anxious about it and rushed up a bit, but I think it came through fairly well in the end).

The talks were filmed so I hope there’s going to be a video available at some point, but in the meantime, you can check out the presentation slides (PDF here):

| View | Upload your own

Update: fixed the broken PDF version of the slides.

…is possible starting from today, because AOL seems to have implemented official XMPP<->AIM bridge!

I don’t have the link to the official anouncement (if there is any), and got the good news via Florian Jensen’s Weblog. It’s great to see XMPP getting more and more acknowledge as the de-facto standard IM protocol (today already 4 big “networks” use it - Jabber, GTalk, AIM and ICQ).

Of course, Telepathy users have already had the ability to connect to their AIM/ICQ accounts using great telepathy-haze which provides suppor for a host of protocols using Pidgin’s libpurple.

But, for this excercise, we’ll be using telepathy-gabble and the XMPP protocol. You’ll need a recent telepathy-gabble (0.7.2 is fresh from the oven, 0.7.1 should also work) and loudmouth (1.3.3 is the latest, 1.3.2 will probably work too). If you’re using Ubuntu Hardy Heron or Debian Unstable, you may already have the required packages available in your nearby repos.

The reason newer gabble is needed is because AOL XMPP doesn’t do old-style SSL (on port 5223), but TLS is required. Sadly, they don’t seem to have a valid certificate installed, so for now you’ll have to set ignore-ssl-errors flag (or, alternatively, not set require-encryption because it defaults to not ignoring ssl errors). Also, it seems their server are getting quite overloaded (or are still pretty buggy), so it might take several attempts to successfully establish connection. Also, from my experience, AOL really hates it if you try to reconnect too fast, so keep that in mind.

Params for the connection are:

• server: xmpp.oscar.aol.com
• port: 5222
• username: uin@aol.com (or your aim username, presumably)

Now we wait to see what MSN does. (If you don’t want to wait, there’s telepathy-butterfly and pymsn, and, of course, again telepathy-haze :)

Since my original Epiphany plugin hack, Empathy has matured and now sports new and improved version of Python bindings, including working bindings for Empathy-GTK, so we can reuse its widgets in other apps instead of reinventing half-assed ones.

So, I’ve updated the hack to work with the new bindings, and to use fancy Empathy’s ContactListView widget with contacts’ aliases, avatars and status messages, instead of plain ol’ boring JID list. I think it’s now usable (and pretty) enough to actually be used everyday, and not just an example of what could be done - so if you’ve got working Empathy setup already, give it a try! You can download the new version here: Empathy Link Share

Obligatory screenshot

Empathy and the bindings are still in heavy development, so it’s no surprise if some things don’t work as expected. If you get assertions like CRITICAL **: empathy_contact_list_get_members: assertion `EMPATHY_IS_CONTACT_LIST (list)' when you try to run the script although you are online, you might want to try and patch the bindings with this. Just go into empathy/python/pyempathygtk/ directory, patch it with patch -p0 < path_to_diff_file and rerun make and make install in the project directory.

Update: I’ve updated the extension to a newer version of Empathy (0.12+), you can read about it here.

There’s lots of exciting things going on in Telepathy and Empathy projects. Telepathy has been quite stable and usable for some time now, but since it’s really a backend service, it wasn’t very visible on the desktop, and the Empathy client (a fork of Gossip) is still mainly used as a standalone client.

Better way to use Empathy is actually to use the provided API and widgets and incorporate them into other applications and desktop components, so that the instant communication (both text-based and audio/video) really becomes a first-class object in GNOME and free desktops in general. There is some work already on the way - Soylent can use contact information provided by Telepathy to display information about the contacts, and there is a Google SoC project to enable Jokosher to use audio calls (using Telepathy) as Jokosher instruments.

Empathy provides a rich set of reusable components which can be used to easily incorporate im-related stuff into other application, and consists of two libraries, libempathy, which is the core API not dependant on GNOME or GTK+ (glib is required, though), and libempathy-gtk, a set of readily-available widget for things such as contact lists, account creation, chat windows, etc. Empathy is written in C, but one of the benefits of the Jokosher+Telepathy projects (which aims to reuse Empathy widgets) is that we now have Python bindings for Empathy! It’s a work in progress and some things are missing, but it is already usable.

To show how easy it is to integrate IM components into other application, and why it would be nice to do it in the first place, I’ve spent a couple of hours writing an Epiphany extension that allows the user to share the links on a web page with their IM contacts. Most of the time was spent figuring out how to make an Epiphany extension, and installing
the required Empathy libraries and bindings.

The result:

Hover over the interesting link and right click


Select one of your contacts


The contact gets the message with the link

The code for the extension is here. As you can see, the actual code for getting the contacts and initiating the chat is the smallest part of the extension :) The stuff is currently just a hack and needs to be cleaned up and properly packaged, which I do intend to do in the near future, by which time hopefully the pyempathy-gtk will be stable enough to sport real contact list chooser widgets instead of a hacked-up version I did here.

Update: I’ve confused Sabayon with Soylent. Thanks to cassidy for the correction