Year: 2012

So i’ve been sinking a lot of time in trying to make python-for-android support multiprocessing, and for various reasons, it turns out to be a bad idea®, but in doing that, i had to go to great extents to understand more about python-for-android, about gdb, and about android itself. So, if it can save some trouble to others, i’ll try to document here the things i found how to do in this situation.

So the first thing to do is to get python-for-android compiled with debug symbols, so you can feed gdb with them, i found android-ndk-r8b to work much better after (well, to work at all), so it’s good to start with it, i put my changes in a branch of python-for-android repository, aptly called `debug` but they are not that big in fact, it’s mostly removing stripping in places, and adding -g in others.

With those changes, one can build a python-for-android distribution usable for debugging.

./distribute.sh -f -m 'kivy' -d debug

then, build your to-be-debugged application with this distribution, and install it, from then, you need a shell access to your android terminal (i use ssh, because adb over wifi is flacky but that’s another story, adb shell is good), you must then check if your terminal has the gdbserver command, if not, you can push it from the ndk to the terminal, and give it execution rights.

adb push ~/android-ndk-r8b/prebuilt/android-arm/gdbserver
adb shell
chmod 755 gdbserver

now, you can start your test program (from the UI), and tell gdbserver to attach to it,

ps | grep python
gdbserver :5050 --attach $pid

obviously using the pid found using the command, i use a shortcut though

./gdbserver :5050 --attach $(ps | grep python | tail -n 1 | tr -s " " " "| cut -d" " -f2)

This will tell gdbserver to listen on port 5050 (you can chose about any number between 1024 and 65536) and then the fun part begins

using the gdb in the ndk, load the pymodules.so file

$ ~/android-ndk-r8b/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-gdb\
 ~/python-for-android/dist/debug/private/libpymodules.so

of course, this path (and the following ones) will need to be adapted to your system!

once in gdb, you’ll want to attach to the gdbserver on your device, i assume you are on the same wifi network as your device, but if not, you can do a port redirection through adb link i think

(gdb) target remote transformer:5050

If everything went fine, you should get a message a bit like this

Remote debugging using transformer:5051
warning: while parsing target library list (at line 2): No segment defined for com.showgen.processcraft.freemium:python
warning: Could not load shared library symbols for 87 libraries, e.g. /system/bin/linker.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
0x400b9384 in ?? ()

as the helpful message put it, we need to tell gdb where it can load the shared libs with debug symbols. Personnaly i found this setting of `solib-search-path`to work well:

(gdb) set solib-search-path \
/home/gabriel/python-for-android/src/obj/local/armeabi/:\
/home/gabriel/python-for-android/dist/debug/private/lib/python2.7/lib-dynload/:\
/home/gabriel/python-for-android/dist/debug/libs/armeabi/

You can check the result of this by using the command `info sharedlibrary` (or `i sh` for the lazy) to see if symbols have been found, the interresting part for me is:

0x56d31348  0x56d5c148  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl.so
0x579b5400  0x579d5798  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl_image.so
0x579fa428  0x57a2c1f8  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl_ttf.so
0x56a5a6b0  0x56a69c20  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl_mixer.so
0x5b985570  0x5bbc9e90  Yes         /home/gabriel/python-for-android/dist/billing/libs/armeabi/libpython2.7.so
0x56a79a1c  0x56a79c04  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libapplication.so
0x56dee534  0x56dee5e8  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl_main.so
0x57604450  0x5761ed40  Yes         /home/gabriel/python-for-android/dist/billing/private/lib/python2.7/lib-dynload/_io.so
0x57a7426c  0x57a77630  Yes         /home/gabriel/python-for-android/dist/billing/private/lib/python2.7/lib-dynload/unicodedata.so
0x57b5b058  0x57b97d00  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsqlite3.so

Interrestingly, it seems the command is only useful after connecting to the target, doing it in reverse order doesn’t seem to bring any result, so the `search` is probably done only once, whet the search-patch is set.
Now, you can use the command `continue` or set breakpoints, or other rather common usage of gdb there. There are quite some resources out there about gdb usage :)

It’s not perfect, but i found it helping to be able to backtrace things on the tablet, in the nasty case you get a crash, or some weird “OSError”, with an unhelpful message, comming from the C side of things.

For the story, my error was likely caused by memory corruption due to the way we tried to reimplement the multiprocessing use of semaphore (sem_open being not implemented in android), and there is apparently no good way to do it. So i’ll look into android services instead when i have time, and try to give an API in python for that.

So i’ve been sinking a lot of time in trying to make python-for-android support multiprocessing, and for various reasons, it turns out to be a bad idea®, but in doing that, i had to go to great extents to understand more about python-for-android, about gdb, and about android itself. So, if it can save some trouble to others, i’ll try to document here the things i found how to do in this situation.

So the first thing to do is to get python-for-android compiled with debug symbols, so you can feed gdb with them, i found android-ndk-r8b to work much better after (well, to work at all), so it’s good to start with it, i put my changes in a branch of python-for-android repository, aptly called `debug` but they are not that big in fact, it’s mostly removing stripping in places, and adding -g in others.

With those changes, one can build a python-for-android distribution usable for debugging.

./distribute.sh -f -m 'kivy' -d debug

then, build your to-be-debugged application with this distribution, and install it, from then, you need a shell access to your android terminal (i use ssh, because adb over wifi is flacky but that’s another story, adb shell is good), you must then check if your terminal has the gdbserver command, if not, you can push it from the ndk to the terminal, and give it execution rights.

adb push ~/android-ndk-r8b/prebuilt/android-arm/gdbserver
adb shell
chmod 755 gdbserver

now, you can start your test program (from the UI), and tell gdbserver to attach to it,

ps | grep python
gdbserver :5050 --attach $pid

obviously using the pid found using the command, i use a shortcut though

./gdbserver :5050 --attach $(ps | grep python | tail -n 1 | tr -s " " " "| cut -d" " -f2)

This will tell gdbserver to listen on port 5050 (you can chose about any number between 1024 and 65536) and then the fun part begins

using the gdb in the ndk, load the pymodules.so file

$ ~/android-ndk-r8b/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-gdb\
 ~/python-for-android/dist/debug/private/libpymodules.so

of course, this path (and the following ones) will need to be adapted to your system!

once in gdb, you’ll want to attach to the gdbserver on your device, i assume you are on the same wifi network as your device, but if not, you can do a port redirection through adb link i think

(gdb) target remote transformer:5050

If everything went fine, you should get a message a bit like this

Remote debugging using transformer:5051
warning: while parsing target library list (at line 2): No segment defined for com.showgen.processcraft.freemium:python
warning: Could not load shared library symbols for 87 libraries, e.g. /system/bin/linker.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
0x400b9384 in ?? ()

as the helpful message put it, we need to tell gdb where it can load the shared libs with debug symbols. Personnaly i found this setting of `solib-search-path`to work well:

(gdb) set solib-search-path \
/home/gabriel/python-for-android/src/obj/local/armeabi/:\
/home/gabriel/python-for-android/dist/debug/private/lib/python2.7/lib-dynload/:\
/home/gabriel/python-for-android/dist/debug/libs/armeabi/

You can check the result of this by using the command `info sharedlibrary` (or `i sh` for the lazy) to see if symbols have been found, the interresting part for me is:

0x56d31348  0x56d5c148  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl.so
0x579b5400  0x579d5798  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl_image.so
0x579fa428  0x57a2c1f8  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl_ttf.so
0x56a5a6b0  0x56a69c20  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl_mixer.so
0x5b985570  0x5bbc9e90  Yes         /home/gabriel/python-for-android/dist/billing/libs/armeabi/libpython2.7.so
0x56a79a1c  0x56a79c04  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libapplication.so
0x56dee534  0x56dee5e8  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl_main.so
0x57604450  0x5761ed40  Yes         /home/gabriel/python-for-android/dist/billing/private/lib/python2.7/lib-dynload/_io.so
0x57a7426c  0x57a77630  Yes         /home/gabriel/python-for-android/dist/billing/private/lib/python2.7/lib-dynload/unicodedata.so
0x57b5b058  0x57b97d00  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsqlite3.so

Interrestingly, it seems the command is only useful after connecting to the target, doing it in reverse order doesn’t seem to bring any result, so the `search` is probably done only once, whet the search-patch is set.
Now, you can use the command `continue` or set breakpoints, or other rather common usage of gdb there. There are quite some resources out there about gdb usage :)

It’s not perfect, but i found it helping to be able to backtrace things on the tablet, in the nasty case you get a crash, or some weird “OSError”, with an unhelpful message, comming from the C side of things.

For the story, my error was likely caused by memory corruption due to the way we tried to reimplement the multiprocessing use of semaphore (sem_open being not implemented in android), and there is apparently no good way to do it. So i’ll look into android services instead when i have time, and try to give an API in python for that.

As someone asked how to do such thing in kivy, i spent some time writting it, and as i (sadly) don’t blog on kivy often (if ever?), and since i think this example is quite telling about how kivy make such things quite easy, let’s talk a bit about this code.

To put things in context, what we want is the bar at the top of the android phones, that one can pull down to see notifications, this one is semi transparent and goes over the normal screen.

So, here is the code. first the kv part:

FloatLayout:
    FloatLayout:
        # placeholder for the "normal screen"
        Label:
            center: root.center
            text: 'test content'
            size_hint: None, None
            size: self.texture_size

    ScrollView:
        # container for the "notifications"
        y: dg.top # let's stick it to the top
        x: root.x # and to the left
        size_hint_y: None
        do_scroll_x: False # only vertical scrolling
        do_scroll_y: True
        
        # most of the magic is there, auto adjust size to follow the draggable label
        height: root.top - dg.y

        # let's put a nearly opaque black background
        canvas:
            Color:
                rgba: 0, 0, 0, .8
            Rectangle:
                pos: self.pos
                size: self.size
        
        # the actual notification container, with placeholder content
        BoxLayout:
            size_hint_y: None
            height: 1000
            orientation: 'vertical'
            Label:
                text: 'test'
            Label:
                text: 'test'
            Label:
                text: 'test'
            Label:
                text: 'test'
            Label:
                text: 'test'
            Label:
                text: 'test'
            Label:
                text: 'test'
            Label:
                text: 'test'
            Label:
                text: 'test'

    # the draggable label, which behaviour is defined in python file
    DraggableLabel:
        # some decoration behind the text
        canvas.before:
            Color:
                rgba: 0, 0, 0, 1
            Rectangle:
                pos: self.pos
                size: self.size
            Color:
                rgba: .5, .5, .5, 1
            Rectangle:
                pos: self.pos
                size: self.width, 1

        size_hint_y: None
        top: root.top
        # assign its id to "dg" so we can reference it elsewhere
        id: dg
        height: '20pt'
        text: 'drag me'

then the python part

from kivy.app import App 
from kivy.uix.label import Label
from kivy.animation import Animation


class DraggableLabel(Label):
    '''A label you can drag upside-down'''
    def on_touch_down(self, touch):
        if self.collide_point(*touch.pos):
            # assure ourselves we will get the updates of this motion
            touch.grab(self)
            return True

        return super(DraggableLabel, self).on_touch_down(touch)

    def on_touch_move(self, touch):
        if touch.grab_current is self:
            # really straightforward...
            self.y = touch.y
            return True

        return super(DraggableLabel, self).on_touch_move(touch)

    def on_touch_up(self, touch):
        if touch.grab_current is self:
            # check if the movement direction was up or down
            if touch.dy < 0:
                a = Animation(y=0) # down? put the bar all the way down
            else:
                a = Animation(top=self.parent.top) # up? put it at the top

            a.start(self) # actually start the animation
            return True

        return super(DraggableLabel, self).on_touch_up(touch)


class TabApp(App):
    pass


TabApp().run()

 

I think it doesn’t really need more explanations, the DraggableLabel is looking for touch events that are for it, first if they are on it, then if they are grabbed by itself, and move accordingly, and the kv auto adjust the size of the ScrollView to take all the distance between top of the screen and top of the DraggableLabel.

Of course, if things are not clear, feel free to ask questions :)

As someone asked how to do such thing in kivy, i spent some time writting it, and as i (sadly) don’t blog on kivy often (if ever?), and since i think this example is quite telling about how kivy make such things quite easy, let’s talk a bit about this code.

To put things in context, what we want is the bar at the top of the android phones, that one can pull down to see notifications, this one is semi transparent and goes over the normal screen.

So, here is the code. first the kv part:

FloatLayout:
    FloatLayout:
        # placeholder for the "normal screen"
        Label:
            center: root.center
            text: 'test content'
            size_hint: None, None
            size: self.texture_size

    ScrollView:
        # container for the "notifications"
        y: dg.top # let's stick it to the top
        x: root.x # and to the left
        size_hint_y: None
        do_scroll_x: False # only vertical scrolling
        do_scroll_y: True
        
        # most of the magic is there, auto adjust size to follow the draggable label
        height: root.top - dg.y

        # let's put a nearly opaque black background
        canvas:
            Color:
                rgba: 0, 0, 0, .8
            Rectangle:
                pos: self.pos
                size: self.size
        
        # the actual notification container, with placeholder content
        BoxLayout:
            size_hint_y: None
            height: 1000
            orientation: 'vertical'
            Label:
                text: 'test'
            Label:
                text: 'test'
            Label:
                text: 'test'
            Label:
                text: 'test'
            Label:
                text: 'test'
            Label:
                text: 'test'
            Label:
                text: 'test'
            Label:
                text: 'test'
            Label:
                text: 'test'

    # the draggable label, which behaviour is defined in python file
    DraggableLabel:
        # some decoration behind the text
        canvas.before:
            Color:
                rgba: 0, 0, 0, 1
            Rectangle:
                pos: self.pos
                size: self.size
            Color:
                rgba: .5, .5, .5, 1
            Rectangle:
                pos: self.pos
                size: self.width, 1

        size_hint_y: None
        top: root.top
        # assign its id to "dg" so we can reference it elsewhere
        id: dg
        height: '20pt'
        text: 'drag me'

then the python part

from kivy.app import App 
from kivy.uix.label import Label
from kivy.animation import Animation


class DraggableLabel(Label):
    '''A label you can drag upside-down'''
    def on_touch_down(self, touch):
        if self.collide_point(*touch.pos):
            # assure ourselves we will get the updates of this motion
            touch.grab(self)
            return True

        return super(DraggableLabel, self).on_touch_down(touch)

    def on_touch_move(self, touch):
        if touch.grab_current is self:
            # really straightforward...
            self.y = touch.y
            return True

        return super(DraggableLabel, self).on_touch_move(touch)

    def on_touch_up(self, touch):
        if touch.grab_current is self:
            # check if the movement direction was up or down
            if touch.dy < 0:
                a = Animation(y=0) # down? put the bar all the way down
            else:
                a = Animation(top=self.parent.top) # up? put it at the top

            a.start(self) # actually start the animation
            return True

        return super(DraggableLabel, self).on_touch_up(touch)


class TabApp(App):
    pass


TabApp().run()

 

I think it doesn’t really need more explanations, the DraggableLabel is looking for touch events that are for it, first if they are on it, then if they are grabbed by itself, and move accordingly, and the kv auto adjust the size of the ScrollView to take all the distance between top of the screen and top of the DraggableLabel.

Of course, if things are not clear, feel free to ask questions :)