Hy (lisp) and Kivy

I was recently reminded of the super cool Hy project. Hy is a lisp that compiles to python’s own abstract syntax tree, so it works perfectly with existing Python code (including with Cython etc.) but also exposes all the power of lisp.

For instance, here’s a simple Kivy application that simply displays a Label with the obligatory Hy pun, but written in Hy. I’ve included the normal Python code as comments so you can see exactly what the code is doing. If you’re new to Kivy and want to understand what the code actually does, check out my Kivy crash course:

(import [kivy.app [App]]
        [kivy.uix.label [Label]])
;; from kivy.app import App
;; from kivy.uix.label import Label


(defclass HyApp [App]
  [[build
    (fn [self]
      (apply Label [] {"text" "Hy world!"
                       "font_size" 100
                       "color" (, 0 1 0 1)}))]])

;; class HyApp(App):
;;     def build(self):
;;         return Label(text="Hy world!",
;;                      font_size=100,
;;                      color=(0, 1, 0, 1))

(.run (HyApp))

;; HyApp().run()

This works great, though only with python3 due to a small bug in Kivy - the kwargs of Label are eventually read in cython with a variable typed as str, which in python2 excludes the unicode Hy passes. Still, that’s not surprising even if it’s cool - part of the point of Hy is to interoperate perfectly with Python.

A tougher problem is how to use Kivy’s kv language with Hy. kv is a simple domain-specific language for declaring widget trees, making it easy to define event-driven interactions between the different properties of widgets. It’s really useful and we tend to recommend using it as much as possible, so it’d be great to have it work with Hy. I won’t explain the language here (you can see the Kivy doc or my own tutorials), but the key point is that much of it consists of interpreting normal python code, which I’d like to replace with Hy code.

It turns out making this work is actually really easy. Here’s the relevant part of lang.py in Kivy’s source, the file containing the code for the kv parser:

self.co_value = compile(value,
                        self.ctx.filename or '<string>',
                        mode)

value is the string of Python code whose output will set a property of a widget or be run when an event is registered. For instance, a line of kv code might be color: (1, 0, 0, some_function_of(self.alpha)), in which case value would be "(1, 0, 0, some_function_of(self.alpha))".

To make a line of Hy code work instead of Python, we can do an awful hack, replacing the above line with:

if value[-3:] == '#hy':
    from hy.importer import (ast_compile,
                             import_buffer_to_ast)
    from hy.compiler import hy_compile
    import ast
    ast_part = import_buffer_to_ast(value[:-3], '<stdin>')
    if mode == 'eval':
        ast_part = ast.Expression(ast_part.body[0].value)
    self.co_value = ast_compile(ast_part,
                                self.ctx.filename or '<string>',
                                mode)
else:
    self.co_value = compile(value,
                            self.ctx.filename or '<string>',
                            mode)

This new code checks if the line of Python ends with #hy, and if so runs the code through Hy’s own equivalent of compile (effectively parsing the Hy code to ast before doing the same thing as the normal Python code). I also have the extra muckiness of taking apart this ast if the compilation is in eval mode, because I couldn’t get Hy to return an ast.Expression in the first place. This is probably very easily and neatly fixed, but I’ve left it like this because a silly hack is good enough for a proof of concept. All credit for this part goes to the friendly Hy people on their irc channel, #hy on Freenode.

With this in place, we can write a new Python program, but this time use our Hy+kv language to define the widget tree. Here’s the new code on the Python (now Hy) side:

(import [kivy.app [App]]
        [kivy.lang [Builder]])

;; from kivy.app import App
;; from kivy.lang import Builder

(setv root (Builder.load_file "hy.kv"))

;; root = Builder.load_file("hy.kv")

(defclass HyApp [App]
  [[build
    (fn [self]
      root)]])

;; class HyApp(App):
;;     def build(self):
;;         return root

(.run (HyApp))

;; HyApp().run()

This obviously depends on our new kv file, "hy.kv", whose contents are as below. Kivy users will notice this file would normally be loaded automatically because the app name starts with Hy, but something about Hy seems to have broken this so I manually loaded it with the Builder.

BoxLayout:
    orientation: "vertical"
    Label:
        id: label
        text: "What is your name?"
    TextInput:
        id: ti
        text: ""
    Button:
        text: (.format "Greet me as {}" ti.text) #hy
        on_press: (setv label.text (.format "Hy there {}" ti.text)) #hy

# as normal kv, except the final 2 rules would normally be:
# text: "Greet me as {}".format(ti.text)
# on_press: label.text = "Hy there {}".format(ti.text)

Running the code…it works perfectly! Here’s a picture after typing my name and clicking the button:

Image of Kivy program after running Hy code

For those not familiar with kv, one of its features is that it automatically detects property changes and updates dependent properties - in this case, the text of the button should change every time ti.text changes (i.e. every time a letter is typed in the TextInput). This works too with the new Hy interface, because the parser detects the dependency by searching the string for substrings like ti.text, and these have been unmodified by the move to Hy. Hy does support syntax that would break this relationship, but it’s quite convenient as it is.

So…there we go, Hy support in Kivy! The hack to make kv language work is pretty terrible, but it looks like a proper solution with this basis would work fine - we could subclass the kv parsing Builder to support a Hy loading option, removing the need for the #hy at the end of each Hy line.

Hy (lisp) and Kivy

I was recently reminded of the super cool Hy project. Hy is a lisp that compiles to python’s own abstract syntax tree, so it works perfectly with existing Python code (including with Cython etc.) but also exposes all the power of lisp.

For instance, here’s a simple Kivy application that simply displays a Label with the obligatory Hy pun, but written in Hy. I’ve included the normal Python code as comments so you can see exactly what the code is doing. If you’re new to Kivy and want to understand what the code actually does, check out my Kivy crash course:

(import [kivy.app [App]]
        [kivy.uix.label [Label]])
;; from kivy.app import App
;; from kivy.uix.label import Label


(defclass HyApp [App]
  [[build
    (fn [self]
      (apply Label [] {"text" "Hy world!"
                       "font_size" 100
                       "color" (, 0 1 0 1)}))]])

;; class HyApp(App):
;;     def build(self):
;;         return Label(text="Hy world!",
;;                      font_size=100,
;;                      color=(0, 1, 0, 1))

(.run (HyApp))

;; HyApp().run()

This works great, though only with python3 due to a small bug in Kivy - the kwargs of Label are eventually read in cython with a variable typed as str, which in python2 excludes the unicode Hy passes. Still, that’s not surprising even if it’s cool - part of the point of Hy is to interoperate perfectly with Python.

A tougher problem is how to use Kivy’s kv language with Hy. kv is a simple domain-specific language for declaring widget trees, making it easy to define event-driven interactions between the different properties of widgets. It’s really useful and we tend to recommend using it as much as possible, so it’d be great to have it work with Hy. I won’t explain the language here (you can see the Kivy doc or my own tutorials), but the key point is that much of it consists of interpreting normal python code, which I’d like to replace with Hy code.

It turns out making this work is actually really easy. Here’s the relevant part of lang.py in Kivy’s source, the file containing the code for the kv parser:

self.co_value = compile(value,
                        self.ctx.filename or '<string>',
                        mode)

value is the string of Python code whose output will set a property of a widget or be run when an event is registered. For instance, a line of kv code might be color: (1, 0, 0, some_function_of(self.alpha)), in which case value would be "(1, 0, 0, some_function_of(self.alpha))".

To make a line of Hy code work instead of Python, we can do an awful hack, replacing the above line with:

if value[-3:] == '#hy':
    from hy.importer import (ast_compile,
                             import_buffer_to_ast)
    from hy.compiler import hy_compile
    import ast
    ast_part = import_buffer_to_ast(value[:-3], '<stdin>')
    if mode == 'eval':
        ast_part = ast.Expression(ast_part.body[0].value)
    self.co_value = ast_compile(ast_part,
                                self.ctx.filename or '<string>',
                                mode)
else:
    self.co_value = compile(value,
                            self.ctx.filename or '<string>',
                            mode)

This new code checks if the line of Python ends with #hy, and if so runs the code through Hy’s own equivalent of compile (effectively parsing the Hy code to ast before doing the same thing as the normal Python code). I also have the extra muckiness of taking apart this ast if the compilation is in eval mode, because I couldn’t get Hy to return an ast.Expression in the first place. This is probably very easily and neatly fixed, but I’ve left it like this because a silly hack is good enough for a proof of concept. All credit for this part goes to the friendly Hy people on their irc channel, #hy on Freenode.

With this in place, we can write a new Python program, but this time use our Hy+kv language to define the widget tree. Here’s the new code on the Python (now Hy) side:

(import [kivy.app [App]]
        [kivy.lang [Builder]])

;; from kivy.app import App
;; from kivy.lang import Builder

(setv root (Builder.load_file "hy.kv"))

;; root = Builder.load_file("hy.kv")

(defclass HyApp [App]
  [[build
    (fn [self]
      root)]])

;; class HyApp(App):
;;     def build(self):
;;         return root

(.run (HyApp))

;; HyApp().run()

This obviously depends on our new kv file, “hy.kv”, whose contents are as below. Kivy users will notice this file would normally be loaded automatically because the app name starts with Hy, but something about Hy seems to have broken this so I manually loaded it with the Builder.

BoxLayout:
    orientation: "vertical"
    Label:
        id: label
        text: "What is your name?"
    TextInput:
        id: ti
        text: ""
    Button:
        text: (.format "Greet me as {}" ti.text) #hy
        on_press: (setv label.text (.format "Hy there {}" ti.text)) #hy

# as normal kv, except the final 2 rules would normally be:
# text: "Greet me as {}".format(ti.text)
# on_press: label.text = "Hy there {}".format(ti.text)

Running the code…it works perfectly! Here’s a picture after typing my name and clicking the button:

Image of Kivy program after running Hy code

For those not familiar with kv, one of its features is that it automatically detects property changes and updates dependent properties - in this case, the text of the button should change every time ti.text changes (i.e. every time a letter is typed in the TextInput). This works too with the new Hy interface, because the parser detects the dependency by searching the string for substrings like ti.text, and these have been unmodified by the move to Hy. Hy does support syntax that would break this relationship, but it’s quite convenient as it is.

So…there we go, Hy support in Kivy! The hack to make kv language work is pretty terrible, but it looks like a proper solution with this basis would work fine - we could subclass the kv parsing Builder to support a Hy loading option, removing the need for the #hy at the end of each Hy line.

Using Facebook SDK with Python-for-Android / Kivy

For another museum project, Arnaud asked me to see if we could integrate Facebook in an app on Android. The usual libraries are made for desktop, and manually open a webbrowser to handle the user authorization. But that’s not really nice for us, it would be nicer to have a native integration instead. Let’s see how we can use the official Facebook Android SDK.

Prerequisites

You’ll need to have:
– Kivy’s python-for-android
– A Facebook account (obviously)
Facebook SDK 3.0

Get the example

Clone the Kivy/Facebook example:

$ git clone https://github.com/tito/p4a-facebook-example

Register a Facebook application

  1. Go to the Facebook developers interface
  2. Click on Sélection_107
  3. Fill the information “App Name”. Don’t put “FB” or “Facebook” in the title, or your application name will be considered as Invalid :)
    Sélection_108
  4. Pass the captcha
  5. Write down your application id somewhereSélection_111
  6. Now, you need to activate Native Android App. The package name is the same name as the one we will pass in --package when building the APK. The Class Name is the packagename/activity. Activity class is always the same for all the applications compiled with python-for-android.
    Sélection_109
  7. Right now, the Facebook Android SDK will not authorize our android application to be used, because Facebook want to know the signature hash that will be used by our generated APK. To generate the key hashes, you need to use the keytool. Here is an example for the android “debug” key:
    $ keytool -exportcert -alias androiddebugkey \
       -keystore ~/.android/debug.keystore | \
       openssl sha1 -binary | openssl base64
    Enter keystore password:  android
    u+bzQmG87L298C4KGM8yODi3W/4=

    Copy the generated key hash, and paste it to the field:

    Sélection_110

Setup the our application

  1. Go into p4a-facebook-example, and edit the main.py
  2. Search for FACEBOOK_APP_ID, and replace with your own Application Id
  3. Go into python-for-android, and create a distribution:

    $ ./distribute.sh -m 'kivy'

  4. Go into the dist/default
  5. Generate the APK one time (for generating all the file we need to customize). Replace /path/to/p4a-facebook-example:

    ./build.py --name "Facebook test" --package org.test.facebook \
      --version 1 --private /path/to/p4a-facebook-example/ \
      --window --permission INTERNET debug

  6. Add a new reference to project.properties to include the Facebook SDK. The path to the Facebook SDK MUST be a relative path, otherwise you’ll get issues during the next build.

    android.library.reference.1=../../facebook-android-sdk-3.0.2/facebook

  7. Edit the templates/AndroidManifest.tmpl.xml to include the Facebook login activity, just under the <application>:

    <application android:label="@string/appName" android:icon="@drawable/icon">
    <activity android:name="com.facebook.LoginActivity"/>

  8. Rebuild your application… and you’re ready to test :)
  9. Install the application on your device:

    adb install -r bin/Facebooktest-1-debug.apk

Test your application

The Facebook SDK 3.0 require to separate read and publish permissions. To the user, it means you’ll have 2 dialog to accept. Even if you just want to publish to the application stream, you need to have the basic permission (read) accepted by the user before publish permissions. This is already implemented in the example app.

When you start the application the first time, you’ll have:

device-2013-08-08-110647

Depending if you have the Native Facebook application or not, you’ll have 2 kinds of authentification boxes.

Without Facebook installed on the device:
device-2013-08-08-110834

And with Facebook installed on the device:

device-2013-08-08-111002

After authentication, the user will need to accept our permissions:

device-2013-08-08-111111

It’s done!

The application have now the right to post :) The example post in the user stream as the application itself, not as the user. It’s called “publish_actions”.

Getting further

Obviously, when you use the Facebook SDK itself, you feel the pain of an API designed for Java. For every callback that Facebook want to call, you need to implement a Java class and define the callback method. Python is really simpler and fun to use.

See for yourself: we want to make a call of “Request.newStatusUpdateRequest“. Se weed first to implement a GraphUserCallback class, in Java. Thanks to Pyjnius, we can do it directly in Python:

class _FacebookGraphUserCallback(PythonJavaClass):

    __javainterfaces__ = ['com.facebook.Request$GraphUserCallback']
    __javacontext__ = 'app'

    def __init__(self, callback):
        self.callback = callback
        super(_FacebookGraphUserCallback, self).__init__()

    @java_method('(Lcom/facebook/model/GraphUser;Lcom/facebook/Response;)V')
    def onCompleted(self, user, response):
        self.callback(user, response)

This Python/Java class will call our own python callback when the Java callback onCompleted will be called. Then:

@run_on_ui_thread
    def post(self, text, callback):
        req = Request.newStatusUpdateRequest(
                self._session, text, _FacebookRequestCallback(callback))
        req.executeAsync()

All you have to do at the end, is to call the post method:

def fb_post(self, text):
        def callback(*args):
            from time import time
            self.post_status = 'message posted at {}'.format(time())
        self.facebook.post(text, callback=callback)

I don’t provide a wrapper around all the possible Request method you can do with Facebook. I just have an example for “post” and “me”. Both wrap “Request.newStatusUpdateRequest” and “Request.newMeRequest“. Please note that every request call must happen in the UI thread. Use the python-for-android/runnable module for that, with @run_on_ui_thread decorator.

The end

It was not easy to get it right, and it still complex to make all the pieces together. I’ve tried to use their own LoginButton (because they said that’s how they want it, everywhere), but i’ve not be able to use it in our app. Mostly because the LoginButton is a pure android widget, and because it doesn’t want to bring back the Activity after login. I was stuck too much time on it, and preferred to go in another way.
Please note that you should not share an application with a fake Facebook button, the design / look-and-feel must be the same as the original one.

I hope that will help some of you to get started !

Using Facebook SDK with Python-for-Android / Kivy

For another museum project, Arnaud asked me to see if we could integrate Facebook in an app on Android. The usual libraries are made for desktop, and manually open a webbrowser to handle the user authorization. But that’s not really nice for us, it would be nicer to have a native integration instead. Let’s see how we can use the official Facebook Android SDK.

Prerequisites

You’ll need to have:
– Kivy’s python-for-android
– A Facebook account (obviously)
Facebook SDK 3.0

Get the example

Clone the Kivy/Facebook example:

$ git clone https://github.com/tito/p4a-facebook-example

Register a Facebook application

  1. Go to the Facebook developers interface
  2. Click on Sélection_107
  3. Fill the information “App Name”. Don’t put “FB” or “Facebook” in the title, or your application name will be considered as Invalid 🙂
    Sélection_108
  4. Pass the captcha
  5. Write down your application id somewhereSélection_111
  6. Now, you need to activate Native Android App. The package name is the same name as the one we will pass in --package when building the APK. The Class Name is the packagename/activity. Activity class is always the same for all the applications compiled with python-for-android.
    Sélection_109
  7. Right now, the Facebook Android SDK will not authorize our android application to be used, because Facebook want to know the signature hash that will be used by our generated APK. To generate the key hashes, you need to use the keytool. Here is an example for the android “debug” key:
    $ keytool -exportcert -alias androiddebugkey \    -keystore ~/.android/debug.keystore | \    openssl sha1 -binary | openssl base64 Enter keystore password:  android u+bzQmG87L298C4KGM8yODi3W/4=

    Copy the generated key hash, and paste it to the field:

    Sélection_110
  8. Setup the our application
  1. Go into p4a-facebook-example, and edit the main.py
  2. Search for FACEBOOK_APP_ID, and replace with your own Application Id
  3. Go into python-for-android, and create a distribution:
    $ ./distribute.sh -m 'kivy'
  4. Go into the dist/default
  5. Generate the APK one time (for generating all the file we need to customize). Replace /path/to/p4a-facebook-example:
    ./build.py --name "Facebook test" --package org.test.facebook \
      --version 1 --private /path/to/p4a-facebook-example/ \
      --window --permission INTERNET debug
  6. Add a new reference to project.properties to include the Facebook SDK. The path to the Facebook SDK MUST be a relative path, otherwise you’ll get issues during the next build.
    android.library.reference.1=../../facebook-android-sdk-3.0.2/facebook
  7. Edit the templates/AndroidManifest.tmpl.xml to include the Facebook login activity, just under the <application>:
     
  8. Rebuild your application… and you’re ready to test 🙂
  9. Install the application on your device:
    adb install -r bin/Facebooktest-1-debug.apk

Test your application

The Facebook SDK 3.0 require to separate read and publish permissions. To the user, it means you’ll have 2 dialog to accept. Even if you just want to publish to the application stream, you need to have the basic permission (read) accepted by the user before publish permissions. This is already implemented in the example app.

When you start the application the first time, you’ll have:

device-2013-08-08-110647

Depending if you have the Native Facebook application or not, you’ll have 2 kinds of authentification boxes.

device-2013-08-08-110834

Without Facebook installed on the device:

And with Facebook installed on the device:

device-2013-08-08-111002

After authentication, the user will need to accept our permissions:

device-2013-08-08-111111

It’s done!

The application have now the right to post 🙂 The example post in the user stream as the application itself, not as the user. It’s called “publish_actions”.

Getting further

Obviously, when you use the Facebook SDK itself, you feel the pain of an API designed for Java. For every callback that Facebook want to call, you need to implement a Java class and define the callback method. Python is really simpler and fun to use.

See for yourself: we want to make a call of “Request.newStatusUpdateRequest“. Se weed first to implement a GraphUserCallback class, in Java. Thanks to Pyjnius, we can do it directly in Python:

Python
class _FacebookGraphUserCallback(PythonJavaClass):

    __javainterfaces__ = ['com.facebook.Request$GraphUserCallback']
    __javacontext__ = 'app'

    def __init__(self, callback):
        self.callback = callback
        super(_FacebookGraphUserCallback, self).__init__()

    @java_method('(Lcom/facebook/model/GraphUser;Lcom/facebook/Response;)V')
    def onCompleted(self, user, response):
        self.callback(user, response)

This Python/Java class will call our own python callback when the Java callback onCompleted will be called. Then:

Python
@run_on_ui_thread
def post(self, text, callback):
    req = Request.newStatusUpdateRequest(
            self._session, text, _FacebookRequestCallback(callback))
    req.executeAsync()

All you have to do at the end, is to call the post method:

Python
def fb_post(self, text):
    def callback(*args):
        from time import time
        self.post_status = 'message posted at {}'.format(time())
    self.facebook.post(text, callback=callback)

I don’t provide a wrapper around all the possible Request method you can do with Facebook. I just have an example for “post” and “me”. Both wrap “Request.newStatusUpdateRequest” and “Request.newMeRequest“. Please note that every request call must happen in the UI thread. Use the python-for-android/runnable module for that, with @run_on_ui_thread decorator.

The end

It was not easy to get it right, and it still complex to make all the pieces together. I’ve tried to use their own LoginButton (because they said that’s how they want it, everywhere), but i’ve not be able to use it in our app. Mostly because the LoginButton is a pure android widget, and because it doesn’t want to bring back the Activity after login. I was stuck too much time on it, and preferred to go in another way.
Please note that you should not share an application with a fake Facebook button, the design / look-and-feel must be the same as the original one.

I hope that will help some of you to get started !

GRDevDay 2013 and My Kivy Talk

I was lucky enough to give a talk about Kivy – a multi-touch, cross-platform Python GUI and application framework – at GRDevDay 2013. Although my talk was the least-attended talk I went to all day, I enjoyed giving it and I got to meet a few new Python folks from Michigan. I’m going to chalk up the attendance to the fact that I was in the same time slot as a lot of good talks, including three that I wanted to see myself. Ah well, such are conferences. At any rate, it gave me an opportunity to see what worked and what didn’t in this talk so I can make some changes before I give it at MobiDevDay Detroit in May.

Watching the

Watching the “So you think you know Javascript” talk at GRDevDay

GRDevDay was a great conference. The organizers really treat the speakers well with a semi-private speaker’s lounge and even special gifts. This is one of the few local events that I attend where I don’t do any organizing, and even though I feel like kind of a slacker, it also feels good to not have all of that pressure for the whole day. The GitHub-hosted drink-up at Kitchen76 was unbelievably crowded, but I still had fun and got to talk at length with some people I usually chat with infrequently online.

For those interested, I’ve posted the slides from my Kivy talk to Speaker Deck, and you can find the tutorial as it existed for the talk in this repository on GitHub.

[Edit 2013-03-05]
There were a few questions I was unable to answer at my GRdevDay talk. As I find answers, I will post them here.

Question: Can you make Android widgets using Kivy?
Answer: Probably not, but there has been a lot of work in pyjnius. No one has tried it recently.

[Edit 2013-03-07]
In my talk I said that you have to specify Android permissions in the Manifest.xml file. This is incorrect, you specify the permissions as part of the build command. See the Python for Android docs for more information.