Jun 06

Urban Airship 1.0 integration with an Android Phonegap app

Since I wrote my previous post, UrbanAirship has released the final version of their android library. Oh well... the new version comes with some new shiny features like analytics for your app, the means to customize the look and feel of notifications (including sounds) and the possibility of using C2DM as your transport medium instead of helium. (It seems this feature was already present in the version I was using before but the docs back didn't mention it). Here you can find an updated howto of how to integrate it into your phonegap project:

AndroidManifest.xml

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
      package="com.example"
      android:versionCode="1"
      android:versionName="1.0">
    <uses-sdk android:minSdkVersion="8" />

    <supports-screens
        android:largeScreens="true"
        android:normalScreens="true"
        android:smallScreens="true"
        android:resizeable="true"
        android:anyDensity="true"
    />

    <application android:debuggable="true" android:icon="@drawable/icon" android:label="@string/app_name" android:name=".MainApplication">
        <activity android:configChanges="orientation|keyboardHidden" android:name="com.example.MainActivity"
                  android:label="@string/app_name">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>

        <!-- REQUIRED -->
        <receiver android:name="com.urbanairship.CoreReceiver">
            <!-- REQUIRED IntentFilter - For Helium and Hybrid -->
            <intent-filter>
              <action android:name="android.intent.action.BOOT_COMPLETED" />
              <action android:name="android.intent.action.ACTION_SHUTDOWN" />
            </intent-filter>
        </receiver>

        <!-- REQUIRED for C2DM and Hybrid -->
        <receiver android:name="com.urbanairship.push.c2dm.C2DMPushReceiver"
                android:permission="com.google.android.c2dm.permission.SEND">
          <!-- Receive the actual message -->
          <intent-filter>
              <action android:name="com.google.android.c2dm.intent.RECEIVE" />
              <category android:name="com.example" />
          </intent-filter>
          <!-- Receive the registration id -->
          <intent-filter>
              <action android:name="com.google.android.c2dm.intent.REGISTRATION" />
              <category android:name="com.example" />
          </intent-filter>
        </receiver>

        <!-- REQUIRED -->
        <!-- The 'android:process' parameter is optional. Set it to a value starting
            with a colon (:) to make it run in a separate, private process -->
        <service android:name="com.urbanairship.push.PushService"
                android:process=":com.urbanairship.push.process"/>

        <!-- OPTIONAL, if you want to receive push, push opened and registration completed intents -->
        <receiver android:name="com.example.IntentReceiver" />
    </application>
    <!-- REQUIRED -->
    <uses-permission android:name="android.permission.INTERNET"/>
    <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE"/>
    <uses-permission android:name="android.permission.RECEIVE_BOOT_COMPLETED"/>
    <uses-permission android:name="android.permission.READ_PHONE_STATE"/>
    <uses-permission android:name="android.permission.VIBRATE"/>
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

    <!-- REQUIRED for C2DM  -->
    <!-- Only this application can receive the messages and registration result -->
    <permission android:name="com.example.permission.C2D_MESSAGE" android:protectionLevel="signature" />
    <uses-permission android:name="com.example.permission.C2D_MESSAGE" />

    <!-- This app has permission to register and receive message -->
    <uses-permission android:name="com.google.android.c2dm.permission.RECEIVE" />
</manifest>

PushNotification.js

function PushNotification() {
}

PushNotification.prototype.registerCallback = function(successCallback, failureCallback) {
     return PhoneGap.exec(
            successCallback,           // called when signature capture is successful
            failureCallback,           // called when signature capture encounters an error
            'PushNotificationPlugin',  // Tell PhoneGap that we want to run "PushNotificationPlugin"
            'registerCallback',        // Tell the plugin the action we want to perform
            []);                       // List of arguments to the plugin
};

PushNotification.prototype.notificationCallback = function (json) {
    var data = Ext.util.JSON.decode(json);
    Ext.Msg.alert("Success", data.msg);
};

PhoneGap.addConstructor(function() {
    if (typeof navigator.pushNotification == "undefined")
        navigator.pushNotification = new PushNotification();
});

PushNotificationPlugin.java

package com.example;

import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;

import android.util.Log;

import com.phonegap.api.Plugin;
import com.phonegap.api.PluginResult;
import com.phonegap.api.PluginResult.Status;

public class PushNotificationPlugin extends Plugin {
    final static String TAG = PushNotificationPlugin.class.getSimpleName();

    static PushNotificationPlugin instance = null;

    public static final String ACTION = "registerCallback";

    public static PushNotificationPlugin getInstance() {
        return instance;
    }

    public void sendResultBack(String msg, String payload) {
        JSONObject data = new JSONObject();
        try {
            data.put("msg", msg);
            data.put("payload", payload);
        } catch (JSONException e) {
            Log.e(TAG, e.getMessage());
        }
        String js = String.format("navigator.pushNotification.notificationCallback('%s');", data.toString());
        //Log.d(TAG, "Sending javascript " + js);
        this.sendJavascript(js);
    }

    @Override
    public PluginResult execute(String action, JSONArray data,
            String callbackId) {

        instance = this;

        PluginResult result = null;
        if (ACTION.equals(action)) {
            result = new PluginResult(Status.NO_RESULT);
            result.setKeepCallback(false);
        } else {
            Log.d(TAG, "Invalid action: " + action + " passed");
            result = new PluginResult(Status.INVALID_ACTION);
        }
        return result;
    }
}

MainApplication.java

package com.example;

import android.app.Application;

import com.urbanairship.UAirship;
import com.urbanairship.push.PushManager;

public class MainApplication extends Application {

    final static String TAG = MainApplication.class.getSimpleName();

    @Override
    public void onCreate() {
        super.onCreate();

        UAirship.takeOff(this);
        PushManager.enablePush();
        PushManager.shared().setIntentReceiver(IntentReceiver.class);
    }

    public void onStop() {
        UAirship.land();
    }
}

MainActivity.java

package com.example;

import android.os.Bundle;

import com.phonegap.DroidGap;
import com.urbanairship.UAirship;

public class MainActivity extends DroidGap {

    /** Called when the activity is first created. */
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        super.loadUrl("file:///android_asset/www/index.html");
        // phonegap plugins
        super.addService("PushNotificationPlugin", "com.example.PushNotificationPlugin");
    }

    @Override
    public void onStart() {
        super.onStart();
        UAirship.shared().getAnalytics().activityStarted(this);
    }

    @Override
    public void onStop() {
        super.onStop();
        UAirship.shared().getAnalytics().activityStopped(this);
    }
}

IntentReceiver.java

package com.example;

import android.content.BroadcastReceiver;
import android.content.Context;
import android.content.Intent;
import android.util.Log;

import com.urbanairship.UAirship;
import com.urbanairship.push.PushManager;

public class IntentReceiver extends BroadcastReceiver {

    private static final String TAG = IntentReceiver.class.getSimpleName();

    @Override
    public void onReceive(Context context, Intent intent) {
        Log.i(TAG, "Received intent: " + intent.toString());
        String action = intent.getAction();

        if (action.equals(PushManager.ACTION_PUSH_RECEIVED)) {
            int id = intent.getIntExtra(PushManager.EXTRA_NOTIFICATION_ID, 0);

            Log.i(TAG, "Received push notification. Alert: " + intent.getStringExtra(PushManager.EXTRA_ALERT)
                + ". Payload: " + intent.getStringExtra(PushManager.EXTRA_STRING_EXTRA) + ". NotificationID="+id);

            String alert = intent.getStringExtra(PushManager.EXTRA_ALERT);
            String extra = intent.getStringExtra(PushManager.EXTRA_STRING_EXTRA);

            PushNotificationPlugin plugin = PushNotificationPlugin.getInstance();
            plugin.sendResultBack(alert, extra);

        } else if (action.equals(PushManager.ACTION_NOTIFICATION_OPENED)) {
            Log.i(TAG, "User clicked notification. Message: " + intent.getStringExtra(PushManager.EXTRA_ALERT)
                    + ". Payload: " + intent.getStringExtra(PushManager.EXTRA_STRING_EXTRA));

            Intent launch = new Intent(Intent.ACTION_MAIN);
            launch.setClass(UAirship.shared().getApplicationContext(), MainActivity.class);
            launch.setFlags(Intent.FLAG_ACTIVITY_NEW_TASK);

            UAirship.shared().getApplicationContext().startActivity(launch);

        } else if (action.equals(PushManager.ACTION_REGISTRATION_FINISHED)) {
            Log.i(TAG, "Registration complete. APID:" + intent.getStringExtra(PushManager.EXTRA_APID)
                    + ". Valid: " + intent.getBooleanExtra(PushManager.EXTRA_REGISTRATION_VALID, false));
        }

    }
}

Follow the rest of the instructions regarding airshipconfig.properties and you will be set. The folks from UrbanAirship suggested me that I should mention that most of the people will want to use the c2dm transport instead of helium. Check out their plans and decide by yourself what are your requisites and whether the basic plan fits your needs or go premium instead.

This is an example airshipconfig.properties:

developmentAppKey = yourDevelopmentAppKey
developmentAppSecret = yourDevelopmentAppSecret
productionAppKey = yourProductionAppKey
productionAppSecret = yourProductionAppSecret

#transport is "helium", "c2dm" or "hybrid"
transport = c2dm

c2dmSender = [email protected]
inProduction = false

You just need now to generate an auth token executing python ua-android-lib-latest/tools/clientauth.py and paste the result in the "C2DM Authorization Token" of your urbainairship app (webpage).

I've uploaded to my github account a sample project that you can use as a base for your app. Please have a look and let me know what you think.

UPDATE: Added UAirship.land() in onStop()

UPDATE2: Mention that c2dm should be used instead of helium, show how to configure a C2DM based app. Add a sample project that summarizes this post.

UPDATE3: UrbanAirship is officially supporting PhoneGap 2.0, check out https://github.com/urbanairship/phonegap-ua-push

Jun 03

Urban Airship integration with an Android Phonegap app

For my new job I had to integrate Urban Airship with an Android Phonegap application. UA's documentation only provides an example for iPhone apps, so if you are looking for a way to integrate an Android phonegap with UA look no further!

The task is going to require you to write your own phonegap plugin with a javascript and a java part, without further ado:

PushNotification.js

function PushNotification() {
}

PushNotification.prototype.registerCallback = function(successCallback, failureCallback) {
     return PhoneGap.exec(
            successCallback,           // called when signature capture is successful
            failureCallback,           // called when signature capture encounters an error
            'PushNotificationPlugin',  // Tell PhoneGap that we want to run "PushNotificationPlugin"
            'registerCallback',        // Tell the plugin the action we want to perform
            []);                       // List of arguments to the plugin
};

PushNotification.prototype.notificationCallback = function (json) {
    var data = Ext.util.JSON.decode(json);
    Ext.Msg.alert("Success", data.msg);
};

PhoneGap.addConstructor(function() {
    if (typeof navigator.pushNotification == "undefined")
        navigator.pushNotification = new PushNotification();
});

Note that I'm also using Sencha Touch in this project, you will need a json decoder to decode the json message sent from java.

PushNotificationPlugin.java

package com.example;

import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;

import android.util.Log;

import com.phonegap.api.Plugin;
import com.phonegap.api.PluginResult;
import com.phonegap.api.PluginResult.Status;

public class PushNotificationPlugin extends Plugin {
    final static String TAG = PushNotificationPlugin.class.getSimpleName();

    static PushNotificationPlugin instance = null;

    public static final String ACTION = "registerCallback";

    public static PushNotificationPlugin getInstance() {
        return instance;
    }

    public void sendResultBack(String msg, String payload) {
        JSONObject data = new JSONObject();
        try {
            data.put("msg", msg);
            data.put("payload", payload);
        } catch (JSONException e) {
            Log.e(TAG, e.getMessage());
        }
        String js = String.format("navigator.pushNotification.notificationCallback('%s');", data.toString());
        //Log.d(TAG, "Sending javascript " + js);
        this.sendJavascript(js);
    }

    @Override
    public PluginResult execute(String action, JSONArray data,
            String callbackId) {
        Log.d(TAG, "Plugin Called");

        instance = this;

        PluginResult result = null;
        if (ACTION.equals(action)) {
            result = new PluginResult(Status.NO_RESULT);
            result.setKeepCallback(false);
        } else {
            Log.d(TAG, "Invalid action: " + action + " passed");
            result = new PluginResult(Status.INVALID_ACTION);
        }
        return result;
    }
}

MainApplication.java

package com.example;

import android.app.Application;
import android.content.Intent;
import android.util.Log;

import com.urbanairship.push.APIDReceiver;
import com.urbanairship.push.AirMail;
import com.urbanairship.push.PushReceiver;

public class MainApplication extends Application {

    final static String TAG = MainApplication.class.getSimpleName();

    public void onCreate() {
        AirMail am = AirMail.getInstance();
        am.acceptPush(this, new PushReceiver() {

            @Override
            public void onClick(String msg, String payload) {
                /*
                 * Fire up MainActivity when the user clicks the Status Bar
                 * Notification
                 */
                Intent intent = new Intent("android.intent.action.MAIN");
                intent.setClass(MainApplication.this,
                        MainActivity.class);
                intent.setFlags(Intent.FLAG_ACTIVITY_NEW_TASK);
                MainApplication.this.startActivity(intent);
            }

            @Override
            public void onReceive(String msg, String payload) {
                // Get a reference to the plugin and send result to js
                PushNotificationPlugin plugin = PushNotificationPlugin.getInstance();
                plugin.sendResultBack(msg, payload);
            }

        });

        am.setAPIDReceiver(this, new APIDReceiver() {
            @Override
            public void onReceive(String apid, boolean valid) {
                if (valid) {
                    Log.d(TAG, "Got apid: " + apid);
                } else {
                    Log.d(TAG, "Application registration invalid!");
                }
            }

            @Override
            public void onAirMailInstallRefusal() {
                MainActivity.register = false;
                Log.d(TAG, "AirMail Install Refused!");
            }
        });
    }
}

You will need to set up your AndroidManifest.xml and the rest of the settings as described in the Android docs.

Feb 12

Sitemaps for Django static pages

Django sitemaps is a great framework to generate sitemaps for your site. The only problem is that it's not quite ready for static pages -pages where you use render_to_response because the flatpages app does not meet your needs.

How can we add them to our sitemap and provide a sound last modification date? Some approaches I saw on the net returned just datetime.datetime.now() but that's inaccurate. I ended up coding the following solution:

# Given the following urls.py
from foo.catalog.sitemaps import CategorySiteMap, ProductSiteMap
from foo.static.sitemaps import StaticSiteMap

...

urlpatterns += patterns('foo.static.views',
    url(r'^about/$', 'about', {'template': 'static/about.html'}, name='static_about'),
    url(r'^faq/$', 'faq', {'template': 'static/faq.html'}, name='static_faq'),
    url(r'^mission/$', 'mission', {'template': 'static/mission.html'}, name='static_mission'),
    url(r'^terms/$', 'terms', {'template': 'static/terms.html'}, name='static_terms'),
)

sitemaps = {
    'categories': CategorySiteMap,
    'products': ProductSiteMap,
    'static': StaticSitemap(urlpatterns)
}

urlpatterns += patterns('django.contrib.sitemaps.views',
    (r'^sitemap\.xml$', 'sitemap', {'sitemaps': sitemaps}),
)

I created the following class that I include in my sitemaps dictionary:

import datetime
import os

from django.contrib import sitemaps
from django.core import urlresolvers
from django.conf import settings

class StaticSitemap(sitemaps.Sitemap):
    """Return the static sitemap items"""
    priority = 0.5

    def __init__(self, patterns):
        self.patterns = patterns
        self._items = {}
        self._initialize()

    def _initialize(self):
        for p in self.patterns:
            if getattr(p, 'name', None) is not None and p.name.startswith('static_'):
                self._items[p.name] = self._get_modification_date(p)

    def _get_modification_date(self, p):
        template = p.default_args['template']
        template_path = self._get_template_path(template)
        mtime = os.stat(template_path).st_mtime
        return datetime.datetime.fromtimestamp(mtime)

    def _get_template_path(self, template_path):
        for template_dir in settings.TEMPLATE_DIRS:
            path = os.path.join(template_dir, template_path)
            if os.path.exists(path):
                return path

        return None

    def items(self):
        return self._items.keys()

    def changefreq(self, obj):
        return 'monthly'

    def lastmod(self, obj):
        return self._items[obj]

    def location(self, obj):
        return urlresolvers.reverse(obj)

Now my sitemap includes the static pages with an accurate last modified date.

UPDATE: Fixed typo, thanks Soviut

Feb 01

What I've been up to

After a long hiatus (9 months since my last post) I'm back! Loads of things have happened till my last post... I quit my job and went on a travel around SE Asia, and I'm now -temporary- living in the Philippines.

I've been mostly working on Django and Android projects the last months, so expect some posts about this. Won't bother you with more details, just a quick post to let you know that this is not dead.

Apr 18

python-messaging: SMS encoder/decoder for the masses

A pure python SMS encoder/decoder

python-messaging is a pure python SMS encoder decoder. Some of its features:

  • 7bit/8bit/UCS2 encoding: Now you can text your Chinese friend
  • Multipart SMS: Because sometimes 140 chars are not enough
  • SMS status report: Be sure they read it
  • Stuff you take for granted: Alphanumeric addresses, timezone awareness, good test coverage
  • Used in production by several projects: Wader, BCM and MobileManager

As far as we know, is the most advanced SMS encoder/decoder written in pure Python (otherwise we would be using it ;) If you need a PDU encoder/decoder for your project, python-messaging should fit your needs.

API walkthrough

>>> from messaging import PDU
>>> p = PDU()
>>> p.encode_pdu('+342323312', 'hello')
[(18, '001100099143323213F20000AA05E8329BFD06')]
>>> p.encode_pdu('+342323312', 'hello', csca='+345343234')
[(18, '069143353432F41101099143323213F20000AA05E8329BFD06')]
>>> p.encode_pdu('+342323312', 'hello', csca='+345343234', request_status=True)
[(18, '069143353432F43102099143323213F20000AA05E8329BFD06')]

The only class you need to deal with is PDU. It is able to encode and decode PDUs with two simple functions: encode_pdu and decode_pdu. encode_pdu receives a number and the text you want to send, plus a number of optional parameters where you can specify smsc, validity, reference number, whether you want a confirmation or not, etc. The function returns a list of tuples with the pdu length and the pdu as a string ready to be sent via the serial port. Why a list of tuples? Well, you might want to send a SMS with more than 140 chars. If so, every fragment of the SMS will be a tuple in the list.

python-messaging can not send the SMS, is just an encoder! This is what you need to do in order to send a SMS with pyserial and python-messaging:

import messaging
import serial

def send_text(number, text, path='/dev/ttyUSB0'):
    # encode the SMS
    p = messaging.PDU()
    # notice how I get the first returned element, this does
    # not deal with concatenated SMS.
    pdu_length, pdu = p.encode_pdu(number, text)[0]
    # open the modem port (assumes Linux)
    ser = serial.Serial(path, timeout=1)
    # write the PDU length and wait 1 second till the
    # prompt appears (a more robust implementation
    # would wait till the prompt appeared)
    ser.write('AT+CMGS=%d\r' % pdu_length)
    print ser.readlines()
    # write the PDU and send a Ctrl+z escape
    ser.write('%s\x1a' % pdu)
    ser.close()

send_text('+3223223', 'hey how are you?')

Updated: The above snippet had an error, it was trying to send the SMS with GMCS instead of CMGS, thanks Jakob for spotting it.

It was not too hard was it? This code does not deal with PIN authentication by the way, so be sure your device is already authenticated, or auth is disabled. Oh by the way, sending a UCS2 encoded message is easy, just pass a unicode string and the PDU object will do the right thing.

Decoding a PDU is a no brainer:

>>> p.decode_pdu('07911326040000F0040B911346610089F60000208062917314080CC8F71D14969741F977FD07')
{'cnt': 0, 'seq': 0, 'text': u'How are you?', 'fmt': 0, 'pid': 0, 'csca': '+31624000000', 'number': '+31641600986', 'type': 0, 'date': '02/08/26 19:37:41', 'ref': 0}

This PDU string was taken from redxanela's online decoder default text, you can compare python-messaging's output with redxanela. Check out the method signatures for what options accepts. I need to write a thorough documentation one of this days, this is an initial effort in that way.

Hope you liked this walkthrough and that python-messaging fits your project needs. I didn't want to bother you with the project's history, you can check it out in the README. Looking forward to any questions/contributions ;-)

Apr 03

reStructuredText directive for google-code-prettify

A custom rst directive for code pretty printing

reStructuredText is teh light markup format, if you never heard about it go check it out. If you are still living in a Docbook world, you would be amazed of what rst and Sphinx can do for your productivity.

Anyway, for the last articles I needed a prettify solution to publish several code listings. I use django-mingus as my blog engine, it comes with not one but two solutions for this problem. The first one is Pygments, which is very nice but no matter what I tried could not make it work. The second is prettify.js, a javascript module plus some css, that prettifies in the client side at runtime.

The only problem of this approach is that the code list will not show up prettified in RSS and such. But you suffered the same problem with Pygments anyway unless you inlined the styles.

The reStructuredText directive

The code is deadly simple:

from cgi import escape

from docutils import nodes
from docutils.parsers.rst import directives, Directive


class CodeDirective(Directive):
    """
    reStructuredText directive to show code listings with google-code-prettify
    """

    has_content = True

    def run(self):
        self.assert_has_content()
        text = '<pre class="prettyprint">%s</pre>' % self._get_escaped_content()
        return [nodes.raw('', text, format='html')]

    def _get_escaped_content(self):
        return '\n'.join(map(escape, self.content))

directives.register_directive("code-list", CodeDirective)
directives.register_directive("sourcecode", CodeDirective)

It just wraps the code in <pre class="prettyprint"> and </pre>, and that's it. Oh well, it also wanted the content to be escaped, fortunately Python comes with a handy function for it: cgi.escape. I will probably write an article about rst directives soon, there is not that much documentation about it.

Mar 31

Playing with QML and DBus (Part 2)

second part of the QML/DBus integration tutorial

In part 1 I showed how to modify the recipes.qml example in order to show contacts instead of recipes, and a basic skeleton to provide a ListModel with contact data from QT/C++ space.

In part 2, I am going to show how to retrieve this contact data from a DBus-backed application, Wader in this case, and the required steps to make it work.

Wader, ModemManager and Betavine Connection Manager

Wader is a 3G daemon accessible via DBus, written in Python and released under the GPLv2. Wader runs on Linux and OSX and supports more than fifty heterogeneous devices. Wader also happens to implement the ModemManager API.

ModemManager is a joint effort between the NetworkManager and Wader projects to produce a modern 3G DBus API that can be reused by other applications of the Unix Desktop. You can have a look at the ModemManager specification to get a glimpse of it.

Betavine Connection Manager is the third generation of Vodafone's open source mobile connect project, hosted at Betavine's site. BCM 3.0 is built upon Wader core, with a brand new user interface and completely integrated in the Linux desktop.

ModemManager internals

ModemManager is a heavy user of DBus interfaces, we are going to interact with just four of the ModemManager specification:

First we need to get a list with the object paths of the devices registered in the system. To do so we will call EnumerateDevices, this will return an array of object paths. We will use the first element in this array for the rest of the operations.

Once we have the object path, we need to setup the device with the Enable method. This method accepts a boolean indicating whether to enable or disable the device. This method may be interrupted if PIN authentication is enabled, for simplicity we are not going to handle possible SimPinRequired errors. So make sure that authentication is disabled, or else send the PIN with minicom beforehand.

With the device enabled, we are just two calls away from finishing! We are going to get the list of contacts and messages, and we will correlate them before populating our classes. We tried to provide a set of common operations in ModemManager, and the Contacts and Sms APIs are the best examples. Both provide a List method that will return the contacts or messages list.

How to wrap a DBus daemon with QT

Wrapping a DBus daemon with QT could not be any easier thanks to qdbusxml2cpp. This tool will parse the ModemManager interfaces that we are going to use and will generate the necessary boilerplate to make calls to remote objects. First things first, create the following files in ~/devel/git/contacts:

org.freedesktop.ModemManager.xml

<!DOCTYPE node PUBLIC "-//freedesktop//DTD D-BUS Object Introspection 1.0//EN"
        "http://www.freedesktop.org/standards/dbus/1.0/introspect.dtd">
<node name="/org/freedesktop/ModemManager">
    <interface name="org.freedesktop.ModemManager">
        <method name="EnumerateDevices">
            <annotation name="com.trolltech.QtDBus.QtTypeName.Out0" value="QList<QDBusObjectPath>"/>
            <arg name="devicesList" type="ao" direction="out" />
        </method>
    </interface>
</node>

org.freedesktop.ModemManager.Modem.xml

<!DOCTYPE node PUBLIC "-//freedesktop//DTD D-BUS Object Introspection 1.0//EN"
        "http://www.freedesktop.org/standards/dbus/1.0/introspect.dtd">
<node name="/">
    <interface name="org.freedesktop.ModemManager.Modem">
        <method name="Enable">
            <arg name="enable" type="b" direction="in" />
        </method>
    </interface>
</node>

org.freedesktop.ModemManager.Modem.Gsm.Contacts.xml

<!DOCTYPE node PUBLIC "-//freedesktop//DTD D-BUS Object Introspection 1.0//EN"
        "http://www.freedesktop.org/standards/dbus/1.0/introspect.dtd">
<node name="/">
    <interface name="org.freedesktop.ModemManager.Modem.Gsm.Contacts">
        <method name="List">
            <annotation name="com.trolltech.QtDBus.QtTypeName.Out0" value="QList<ContactStruct>"/>
            <arg name="contactList" type="a(uss)" direction="out" />
        </method>
    </interface>
</node>

org.freedesktop.ModemManager.Modem.Gsm.Sms.xml

<!DOCTYPE node PUBLIC "-//freedesktop//DTD D-BUS Object Introspection 1.0//EN"
        "http://www.freedesktop.org/standards/dbus/1.0/introspect.dtd">
<node name="/">
    <interface name="org.freedesktop.ModemManager.Modem.Gsm.Sms">
        <method name="List">
            <annotation name="com.trolltech.QtDBus.QtTypeName.Out0" value="QList<QVariantMap>"/>
            <arg name="smsList" type="a{sv}" direction="out" />
        </method>
    </interface>
</node>

This files follow the DBus introspection format and describe the methods available in each interface (only the ones we are going to use actually) and their arguments. If the type is a generic one, we can just use the standard DBus syntax -i.e. 'b' for bool. If on the other hand the response has a custom type, we need to provide an annotation for it with its type plus some boilerplate code so that the QT type system is able to transform in runtime a DBus response to a QT struct/class. We will revisit this point later on.

Now we are going to create a script that will generate the necessary code to access the ModemManager service. For each of the four interfaces that we are going to interact with, we need to create a proxy to it. The code bellow generates the static code necessary to access them.

generate-interfaces.sh

#!/bin/sh

set -e

qdbusxml2cpp -N -c ModemManagerProxy \
             -p modemmanager_generic.h:modemmanager_generic.cpp \
             org.freedesktop.ModemManager.xml \
             org.freedesktop.ModemManager

qdbusxml2cpp -N -c ModemManagerModemProxy \
             -p modemmanager_modem.h:modemmanager_modem.cpp \
             org.freedesktop.ModemManager.Modem.xml \
             org.freedesktop.ModemManager.Modem

qdbusxml2cpp -N -c ModemManagerContactsProxy \
             -p modemmanager_contacts.h:modemmanager_contacts.cpp \
             org.freedesktop.ModemManager.Modem.Gsm.Contacts.xml \
             org.freedesktop.ModemManager.Modem.Gsm.Contacts

qdbusxml2cpp -N -c ModemManagerSmsProxy \
             -p modemmanager_sms.h:modemmanager_sms.cpp \
             org.freedesktop.ModemManager.Modem.Gsm.Sms.xml \
             org.freedesktop.ModemManager.Modem.Gsm.Sms

This commands generate a class with the -c switch (first argument), contained in a .cpp and .h files (second argument), by reading the given interface description (third argument) and finally the name of the accessed interface as fourth argument. Note the initial switch -N that produces classes with no namespaces, we had to use it as it would not work otherwise. I think it has to do with the different "depths" of the interfaces used here. Check out qdbusxml2cpp documentation for more details.

Add this new files to contacts.pro:

TEMPLATE = app
TARGET = contacts
DEPENDPATH += .
INCLUDEPATH += .
CONFIG += qdbus
QT += declarative

# Input
SOURCES += contact.cpp modemmanager_generic.cpp \
           modemmanager_modem.cpp modemmanager_contacts.cpp \
           modemmanager_sms.cpp main.cpp
HEADERS += contact.h modemmanager_generic.h modemmanager_modem.h \
           modemmanager_contacts.h modemmanager_sms.h
RESOURCES += contacts.qrc
sources.files = $$SOURCES $$HEADERS $$RESOURCES contacts.pro contacts.qml
sources.path = .
target.path = .

INSTALLS += sources target

Now add the modemmanager_* headers to main.cpp, generate the interfaces and make sure it compiles after this last changes:

chmod +x generate-interfaces.sh
./generate-interfaces.sh
/usr/local/Trolltech/Qt-4.7.0/bin/qmake contacts.pro
make

We are about to access a DBus daemon written in Python, from a small C++ binary that will in turn show the results in a QML application. Interesting mix isn't it?

We need to obtain the object path of the device first, to do so add this in your main.cpp (right after the QApplication declaration):

ModemManagerProxy *proxy = new ModemManagerProxy("org.freedesktop.ModemManager",
                                                 "/org/freedesktop/ModemManager",
                                                 QDBusConnection::systemBus(),
                                                 static_cast<QObject*>(&app));

QDBusObjectPath opath;
QDBusPendingReply<QList<QDBusObjectPath> > reply = proxy->EnumerateDevices();
reply.waitForFinished();
if (reply.isError()) {
    qDebug() << reply.error();
    app.exit(1);
} else {
    opath = reply.value().first();
    qDebug() << opath.path();
}

ModemManagerModemProxy *modem_proxy = new ModemManagerModemProxy(
                                                 "org.freedesktop.ModemManager",
                                                 opath.path(),
                                                 QDBusConnection::systemBus(),
                                                 proxy);
QDBusPendingReply<> modem_reply = modem_proxy->Enable(true);
modem_reply.waitForFinished();
if (modem_reply.isError()) {
    qDebug() << "Error enabling device: " << modem_reply.error();
    app.exit(1);
}

This code instantiates a new object, proxy which is a pointer to the ModemManagerProxy class we've just created in generate-interfaces.sh. The constructor receives the service and object path of the object as first and second arguments, and the system bus is used. Ignore the fourth argument for now.

We call EnumerateDevices, check that the reply has no errors, and we store the first value of the response in the opath variable. This object path identifies our device in ModemManager and thus will be used in the rest of the tutorial.

The next step is to enable the device with Enable, ModemManagerModemProxy is instantiated and receives the object path of the device this time. Enable does not return anything, so we only check for errors in the reply object.

At this point we have an enabled device ready to receive operations, guess what's next? Yep, retrieving the list of contacts and messages, correlate them and populate the data model with it. Add the following code before the main function:

QList<QObject*> GetContacts(QDBusObjectPath opath)
{
    ModemManagerContactsProxy *cts_proxy = new ModemManagerContactsProxy(
                                                 "org.freedesktop.ModemManager",
                                                 opath.path(),
                                                 QDBusConnection::systemBus(),
                                                 0);
    ModemManagerSmsProxy *sms_proxy = new ModemManagerSmsProxy(
                                                 "org.freedesktop.ModemManager",
                                                 opath.path(),
                                                 QDBusConnection::systemBus(),
                                                 0);

    QList<QObject*> ret;

    QList<ContactStruct> contactList;
    QDBusPendingReply<QList<ContactStruct> > cts_reply = cts_proxy->List();
    cts_reply.waitForFinished();
    if (cts_reply.isError()) {
        qDebug() << cts_reply.error();
        return ret;
    } else {
        contactList = cts_reply.value();
    }

    QList<QVariantMap> messagesList;
    QDBusPendingReply<QList<QVariantMap> > sms_reply = sms_proxy->List();
    sms_reply.waitForFinished();
    if (sms_reply.isError()) {
        qDebug() << sms_reply.error();
        return ret;
    } else {
        messagesList = sms_reply.value();
    }

    const QString picture = QString("qrc:/content/pics/person.png");

    foreach(ContactStruct contact, contactList) {
        QList<QString> smsList;
        foreach(QVariantMap sms, messagesList) {
            if (sms.value("number").toString().endsWith(contact.number)) {
                smsList.append(sms.value("text").toString());
            }
        }
        QString messages = "<html><ul>";
        if (!smsList.size()) {
            messages.append("<li>No messages!</li>");
        } else {
            foreach(QString sms, smsList) {
                messages.append(QString("<li>%1</li>").arg(sms));
            }
        }
        messages.append("</ul></html>");
        ret.append(new Contact(contact.name, contact.number, messages, picture));
    }

    delete cts_proxy;
    delete sms_proxy;

    return ret;
}

This method accepts the object path of a device, obtains the list of contacts and messages, correlates them and returns a QList<QObject *> object with the data. If you try to compile it as it is, it will fail by several reasons: First ContactStruct (Gsm.Contacts.List return value) is undefined, second we need to register it with the QT type system, and third we need to provide marshalling functions to serialize/deserialize ContactStruct from DBus.

Create the following file dbustypes.h and import it in main.cpp before the modemmanager_* ones:

#include <QtDBus>

struct ContactStruct {
     uint index;
     QString name;
     QString number;
};

Q_DECLARE_METATYPE(ContactStruct);
Q_DECLARE_METATYPE(QList<ContactStruct>);
Q_DECLARE_METATYPE(QList<QVariantMap>);

// Marshall the ContactStruct data into a D-BUS argument
QDBusArgument &operator<<(QDBusArgument &argument, const ContactStruct &mystruct)
{
     argument.beginStructure();
     argument << mystruct.index << mystruct.name << mystruct.number;
     argument.endStructure();
     return argument;
}

// Retrieve the ContactStruct data from the D-BUS argument
const QDBusArgument &operator>>(const QDBusArgument &argument, ContactStruct &mystruct)
{
     argument.beginStructure();
     argument >> mystruct.index >> mystruct.name >> mystruct.number;
     argument.endStructure();
     return argument;
}

Now, before ModemManagerProxy's instantiation, add this lines in main.cpp:

qDBusRegisterMetaType<ContactStruct>();
qDBusRegisterMetaType<QList<ContactStruct> >();
qDBusRegisterMetaType<QList<QVariantMap> >();

This will register the compound types that we are going to use in the DBus calls, without this lines the project will not compile.

At this point we are ready to hook GetContacts in main.cpp. Add after the modem_reply block the following lines:

QList<QObject*> dataList;
foreach(QObject* contact, GetContacts(opath)) {
    dataList.append(contact);
}

This will populate our data model with the DBus data, and we are ready to go:

make clean
rm -rf modemmanager_*
./generate-interfaces.sh
/usr/local/Trolltech/Qt-4.7.0/bin/qmake contacts.pro
make
./contacts

Here you can see the end result of part 2, best viewed with Chrome/Firefox 3.6:

This is the current source of the tutorial. Next (and last) part of the tutorial will look at retrieving this data at runtime from QML.

Mar 30

Playing with QML and DBus (Part 1)

a QML/DBus integration tutorial

How to integrate your DBus app and a QML UI

In this tutorials I am going to explain the necessary steps to integrate an existing codebase with DBus IPC and a QML (a.k.a. QtQuick) user interface. In this case, I am going to write a QML interface that will query a ModemManager instance running in the background. It will be possible to see the contacts in the SIM card and each contact will have a detail page where you will be able to see all the SMS sent to that contact.

Preparation

You need to download QT 4.7 -at the time of writing still an alpha version- and compile it by yourself. Bear in mind that this process might take several hours. Also I recommend you to start using the wonderful QT Creator 2.0 -an alpha version too at the time of writing- as IDE. It has a decent VIM integration and several goodies, like a visual QML editor. Do not even think about it start downloading them now! Note that the QT 4.7 API might have changed since I wrote this, and some parts might need minor fixes. Also make sure to download the most recent version of both QT and QT Creator instead of the alpha versions I had to use.

Start the project

First I created a new "Qt Gui" project. I've used the ~/devel/git/contacts path. This will create a bunch of files, some of them need some editing. But first... design is not one of my strong points, so I started by reusing one of the examples included in the documentation, the recipes example:

cp -R ~/software/qt-everywhere-opensource-src-4.7.0-tp/examples/declarative/listview/* ~/devel/git/contacts

Now remove the following files, as we won't need them:

cd ~/devel/git/contacts
rm dynamic.qml highlight.qml itemlist.qml listview.qml sections.qml
rm -rf dummydata content/ClickAutoRepeating.qml

Move MediaButton to the top level:

mv content/MediaButton.qml .

Edit the lines starting with "source:" and add content/ to the path to reflect that we just moved it to the top level.

Download the default image for our contacts:

wget http://minimoesfuerzo.org/media/photos/person.png -O content/pics/person.png

Rename recipes.qml to contacts.qml:

mv recipes.qml contacts.qml

Understand what is going on under the hood

Before we go on, I think it will be good to provide a minimal overview of how recipes.qml works, and what our goal is. Note that this tutorial assumes certain familiarity with QML/QT and related concepts, if you never heard of this technologies, it is a good moment to check out the docs.

recipes.qml imports "content" at the beginning of the script, and that folder contains a bunch of images and a MediaButton.qml script. A QML script can import other QML scripts and refer to other files. In addition, "dummydata" is also imported and that is where the hardcoded data for recipes resides, check out dummydata/Recipes.qml and look at its structure and attributes.

So when you run recipes.qml, it imports in turn a bunch of other scripts and one of them happens to be a ListModel that contains the initial data that you get to see. How is that data referenced in recipes.qml then? Very easy, in recipes.qml line 137 you can see that a ListView is defined and has a model named Recipes. A ListView will always have a delegate for painting and showing the model's data. The delegate for this ListView is recipeDelegate.

recipeDelegate is a Component declared in line 13, and inside the block if you want to show/paint the model's data, you just use the name of the attribute in the model. For example, if with the current model I want to show the title of the recipe, I need to use 'text: title'. Similarly, if I want to show the details of a recipe, I would use 'text: details'.

So our goal is to get rid of that hardcoded data model, and use a custom one populated by ModemManager via DBus.

Initial steps

Instead save the following lines in ~/devel/git/contacts/contacts.pro:
TEMPLATE = app
TARGET = contacts
DEPENDPATH += .
INCLUDEPATH += .
CONFIG += qdbus
QT += declarative

# Input
SOURCES += main.cpp contact.cpp
HEADERS += contact.h
RESOURCES += contacts.qrc
sources.files = $$SOURCES $$HEADERS $$RESOURCES contacts.pro contacts.qml
sources.path = .
target.path = .

INSTALLS += sources target

This is a minimal QT project template with all the parts that we, initially, need for the task. The target will be a binary named "contacts", for now the project will use main.cpp and contact.cpp as sources, contact.h as headers and will use the QtDBus and QDeclarative libraries for obvious reasons.

Initial main.cpp implementation:

#include <QApplication>

#include <qdeclarativeengine.h>
#include <qdeclarativecontext.h>
#include <qdeclarative.h>
#include <qdeclarativeitem.h>
#include <qdeclarativeview.h>

#include "contact.h"

int main(int argc, char ** argv)
{
    QApplication app(argc, argv);

    const QString picture = QString("qrc:/content/pics/person.png");
    const QString messages = QString("<html><ul><li>No SMS!</li></ul></html>");

    QList<QObject*> dataList;
    dataList.append(new Contact("John", "+34243234232", messages, picture));

    QDeclarativeView view;
    QDeclarativeContext *ctxt = view.rootContext();
    ctxt->setContextProperty("myModel", QVariant::fromValue(dataList));

    view.setSource(QUrl("qrc:contacts.qml"));
    view.show();

    return app.exec();
}

Initial contact.h:

#ifndef CONTACT_H
#define CONTACT_H

#include <QObject>

class Contact : public QObject
{
    Q_OBJECT

    Q_PROPERTY(QString name READ name WRITE setName)
    Q_PROPERTY(QString number READ number WRITE setNumber)
    Q_PROPERTY(QString picture READ picture WRITE setPicture)
    Q_PROPERTY(QString messages READ messages)

public:
    Contact(QObject *parent=0);
    Contact(const QString &name, const QString &number,
            const QString &messages, const QString &picture,
            QObject *parent=0);

    QString name() const;
    void setName(const QString &name);

    QString number() const;
    void setNumber(const QString &number);

    QString picture() const;
    void setPicture(const QString &picture);

    QString messages() const;

private:
    QString m_name;
    QString m_number;
    QString m_messages;
    QString m_picture;
};

#endif // CONTACT_H

Initial contact.cpp implementation:

#include "contact.h"

Contact::Contact(QObject *parent)
    : QObject(parent)
{
}

Contact::Contact(const QString &name, const QString &number,
                 const QString &messages, const QString &picture,
                 QObject *parent)
    : QObject(parent), m_name(name), m_number(number),
      m_messages(messages), m_picture(picture)
{
}

QString Contact::name() const
{
    return m_name;
}

void Contact::setName(const QString &name)
{
    m_name = name;
}

QString Contact::number() const
{
    return m_number;
}

void Contact::setNumber(const QString &number)
{
    m_number = number;
}

QString Contact::picture() const
{
    return m_picture;
}

void Contact::setPicture(const QString &picture)
{
    m_picture = picture;
}

QString Contact::messages() const
{
    return m_messages;
}

What did we just do? We have declared a QObject class, Person, with four properties (name, number, messages, picture) that will be accessed from QML space transparently.

Modify contacts.qml

At this point, contacts.qml still has the contents of recipes.qml, and needs to be changed. We are going to change the references to old attributes, to the new ones that we will use.

  • In line 57: Change 'text: title' to 'text: model.name'
  • In line 59: Change 'text: "Ingredients"' to 'text: "Details"'
  • In line 63: Change 'text: ingredients' to 'text: model.name'
  • In line 78: Change 'text: "Method"' to 'text: "List of Messages"'
  • In line 85: Change 'text: method' to 'text: model.messages'
  • In line 137: Change 'model: Recipes' to 'model: myModel'

To sum up, we have changed the old attributes in the model (title, picture, ingredients and method) to (name, number, messages and picture). And the model name (from Recipes to myModel as that's the name we've used in main.cpp). A saavy reader will have noticed by now that there's no mention of the number attribute in the current contacts.qml file and that's true, to include it we just need this last modification to contacts.qml, line 44 defines the topLayout Row and should look like this after the changes:

Row {
    id: topLayout
    x: 10; y: 10; height: recipePic.height; width: parent.width
    spacing: 10

    Image {
        id: recipePic
        source: model.picture; width: 48; height: 48
    }

    Column {
        height: recipePic.height; width: background.width-recipePic.width-20
        spacing: 5

        Column {
            Text { id: title; text: model.name; font.bold: true; font.pointSize: 15 }
            Text { id: phone; text: model.number; font.bold: true; font.pointSize: 9 }
        }

        Text {
            text: "Details"; font.pointSize: 12; font.bold: true
            opacity: wrapper.detailsOpacity
        }
        Text {
            text: model.name; wrap: true; width: parent.width
            opacity: wrapper.detailsOpacity
        }
    }
}

Now is a good time to remove all the outstanding references to the old attributes in the code, I've done the following substitutions in contacts.qml:

sed -i 's/recipeDelegate/contactDelegate/g' contacts.qml
sed -i 's/recipePic/contactPic/g' contacts.qml
sed -i 's/methodTitle/smsListTitle/g' contacts.qml

The last thing we need to do is provide an initial contacts.qrc:

<!DOCTYPE RCC><RCC version="1.0">
<qresource>
    <file>contacts.qml</file>
    <file>MediaButton.qml</file>
    <file>content/pics/button.png</file>
    <file>content/pics/button-pressed.png</file>
    <file>content/pics/person.png</file>
    <file>content/pics/moreDown.png</file>
    <file>content/pics/moreUp.png</file>
</qresource>
</RCC>

This file references every QML component/asset that is going to be used in the project. If they are not referenced here, you will get an error message for each time you try to access from QML an asset not included in contacts.qrc.

At this point, we are ready to compile our project and run our project:

qmake contacts.pro
make

Make sure that the qmake you have used is the 4.7 one, otherwise some of the required libraries, such as QtDeclarative* will not be found. Now launch the contacts binary and rejoy!

Make sure to read part 2 of this post where I will show how to retrieve the contacts and messages from the SIM via ModemManager . This is the current source for the project contacts-part1.tar.gz.

Updated: I have added a small video showing the current state of the project, so you can have a quick look of what we are building. Thanks Andrew!

Mar 12

Testing ntrack

For the last years, applications that needed to be online have been patched to ask NetworkManager for connectivity status. This neat idea turned out to be a disaster for other apps that provide connectivity...

Background

Until NetworkManager appeared in scene, network management was painful on Linux. Connecting to a wireless network or a VPN involved several command line scripts, editing files with alien syntax, etc. Seriously, it was a joke when compared to what other operating systems had available. So when NetworkManager was released it filled a nice gap in the Linux Desktop, and everyone rejoiced.

The problem

In early 2008, some key apps of the Linux desktop were patched to listen to NetworkManager for connectivity status. Applications such as Firefox or pidgin would not work if NetworkManager considered the system wasn't online.

As you might know, I have been involved in Linux connection managers since late 2006. The software were I have spent most of my energies, Wader, suffered from this. As the connection was established without NetworkManager, the applications that listened to its signals thought they were offline and thus did not work. Some apps had a workaround (like Firefox) while others didn't (pidgin).

Say you want to connect via a competing connection manager, such as WICD, Firefox & co will not work. Say you are a token ring user, well you are out of luck, NetworkManager has no plans to support token ring. As you can see the current status is not that good if you are not a NetworkManager developer/user.

The solution

Thankfully, Alexander Sack stepped up and started working on ntrack after some talks we had in a UDS.

ntrack is described on its homepage as:

"ntrack aims to be a lightweight and easy to use library for application developers that want to get events on network online status changes such as online, offline or route changes."

ntrack does one thing, and excels at it. It just listens for Netlink messages regarding route changes, and will infer if it is connected from them. As you can see, this approach will work regardless of the connection manager or underlying technology used. In order to be online, you must create a route. Thus the responsibility of determining whether we are connected or not resides on a small daemon completely independent of the connection manager. This is a good thing.

Today I have tested the last ntrack release (0.6) and a patched empathy that listens to ntrack rather than NetworkManager and it worked wonderfully! I tested it with both PPP and NDIS (HSO) dialup and ntrack detected immediately if I was connected or not. Nice job Alexander :)

Future steps

We are going to "lobby" for its inclusion on Debian/Ubuntu, and patch those applications listening to NetworkManager to listen to ntrack instead. ntrack has already been accepted in Debian and it will automatically appear in the next Ubuntu version (Lucid+1). Hopefully the rest of the distros will follow, and in 1-2 years we will have recovered from this error.

Dec 29

minimoesfuerzo.org site details

Building a minimal blog in Django is so easy that is not even funny, the problem lies in the definition of "minimal".

I got interested in Django around mid 2007 (0.96 timeline), I quickly built a django blog following a known early tutorial in the matter. It was lean and fast, but some basic features were missing.

Around that time Nat released django-basic-blog (now a part of django-basic-apps), it offered pretty much what you need from a basic blog, but still I wanted to do something on my own.

I started a couple of half-baked attempts before I lost all interest, it was fun but I couldn't help but thinking that I was reimplementing other people's apps. That's when django-mingus appeared on my radar.

django-mingus

Mingus is a blog engine leveraging reusable apps for all its features, its author explained the apps that power django-mingus in great detail.

I quickly knew that this is what I intended to build in first place, so I just forked it and started customizing it.

Customizations

Apart from some CSS tweaks, I was generally pleased with the feature set provided by Mingus. The only grip that I initially had is that Mingus (as Wordpress IIRC) sports both "Categories" and "Tags", I didn't quite like the overlap between them so I removed all the categories uses by tags'. I added restructuredtext support too as that's my favourite lightweight markup. I also removed the quoteme related functionality, as I didn't picture myself using it much.

Deployment

This site is deployed with mod_wsgi and it was quite easy, mod_wsgi should be the blessed deployment method rather than mod_python by a mile!

Dec 29

Hello everyone

minimoesfuerzo's society presentation

I finally did it, got my site working! In a not so distant post will explain the nitty-gritty details of the site.

I plan to write about computer science, science and warfare, pretty much my favourite topics, what a combination... Knowing myself I'll probably blog from time to time, so this will be a low-bandwidth site.

Having said this, welcome to my site :)