Debugging python-for-android with gdb

So i’ve been sinking a lot of time in trying to make python-for-android support multiprocessing, and for various reasons, it turns out to be a bad idea®, but in doing that, i had to go to great extents to understand more about python-for-android, about gdb, and about android itself. So, if it can save some trouble to others, i’ll try to document here the things i found how to do in this situation.

So the first thing to do is to get python-for-android compiled with debug symbols, so you can feed gdb with them, i found android-ndk-r8b to work much better after (well, to work at all), so it’s good to start with it, i put my changes in a branch of python-for-android repository, aptly called `debug` but they are not that big in fact, it’s mostly removing stripping in places, and adding -g in others.

With those changes, one can build a python-for-android distribution usable for debugging.

./distribute.sh -f -m 'kivy' -d debug

then, build your to-be-debugged application with this distribution, and install it, from then, you need a shell access to your android terminal (i use ssh, because adb over wifi is flacky but that’s another story, adb shell is good), you must then check if your terminal has the gdbserver command, if not, you can push it from the ndk to the terminal, and give it execution rights.

adb push ~/android-ndk-r8b/prebuilt/android-arm/gdbserver
adb shell
chmod 755 gdbserver

now, you can start your test program (from the UI), and tell gdbserver to attach to it,

ps | grep python
gdbserver :5050 --attach $pid

obviously using the pid found using the command, i use a shortcut though

./gdbserver :5050 --attach $(ps | grep python | tail -n 1 | tr -s " " " "| cut -d" " -f2)

This will tell gdbserver to listen on port 5050 (you can chose about any number between 1024 and 65536) and then the fun part begins

using the gdb in the ndk, load the pymodules.so file

$ ~/android-ndk-r8b/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-gdb\
 ~/python-for-android/dist/debug/private/libpymodules.so

of course, this path (and the following ones) will need to be adapted to your system!

once in gdb, you’ll want to attach to the gdbserver on your device, i assume you are on the same wifi network as your device, but if not, you can do a port redirection through adb link i think

(gdb) target remote transformer:5050

If everything went fine, you should get a message a bit like this

Remote debugging using transformer:5051
warning: while parsing target library list (at line 2): No segment defined for com.showgen.processcraft.freemium:python
warning: Could not load shared library symbols for 87 libraries, e.g. /system/bin/linker.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
0x400b9384 in ?? ()

as the helpful message put it, we need to tell gdb where it can load the shared libs with debug symbols. Personnaly i found this setting of `solib-search-path`to work well:

(gdb) set solib-search-path \
/home/gabriel/python-for-android/src/obj/local/armeabi/:\
/home/gabriel/python-for-android/dist/debug/private/lib/python2.7/lib-dynload/:\
/home/gabriel/python-for-android/dist/debug/libs/armeabi/

You can check the result of this by using the command `info sharedlibrary` (or `i sh` for the lazy) to see if symbols have been found, the interresting part for me is:

0x56d31348  0x56d5c148  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl.so
0x579b5400  0x579d5798  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl_image.so
0x579fa428  0x57a2c1f8  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl_ttf.so
0x56a5a6b0  0x56a69c20  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl_mixer.so
0x5b985570  0x5bbc9e90  Yes         /home/gabriel/python-for-android/dist/billing/libs/armeabi/libpython2.7.so
0x56a79a1c  0x56a79c04  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libapplication.so
0x56dee534  0x56dee5e8  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl_main.so
0x57604450  0x5761ed40  Yes         /home/gabriel/python-for-android/dist/billing/private/lib/python2.7/lib-dynload/_io.so
0x57a7426c  0x57a77630  Yes         /home/gabriel/python-for-android/dist/billing/private/lib/python2.7/lib-dynload/unicodedata.so
0x57b5b058  0x57b97d00  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsqlite3.so

Interrestingly, it seems the command is only useful after connecting to the target, doing it in reverse order doesn’t seem to bring any result, so the `search` is probably done only once, whet the search-patch is set.
Now, you can use the command `continue` or set breakpoints, or other rather common usage of gdb there. There are quite some resources out there about gdb usage :)

It’s not perfect, but i found it helping to be able to backtrace things on the tablet, in the nasty case you get a crash, or some weird “OSError”, with an unhelpful message, comming from the C side of things.

For the story, my error was likely caused by memory corruption due to the way we tried to reimplement the multiprocessing use of semaphore (sem_open being not implemented in android), and there is apparently no good way to do it. So i’ll look into android services instead when i have time, and try to give an API in python for that.

Debugging python-for-android with gdb

So i’ve been sinking a lot of time in trying to make python-for-android support multiprocessing, and for various reasons, it turns out to be a bad idea®, but in doing that, i had to go to great extents to understand more about python-for-android, about gdb, and about android itself. So, if it can save some trouble to others, i’ll try to document here the things i found how to do in this situation.

So the first thing to do is to get python-for-android compiled with debug symbols, so you can feed gdb with them, i found android-ndk-r8b to work much better after (well, to work at all), so it’s good to start with it, i put my changes in a branch of python-for-android repository, aptly called `debug` but they are not that big in fact, it’s mostly removing stripping in places, and adding -g in others.

With those changes, one can build a python-for-android distribution usable for debugging.

./distribute.sh -f -m 'kivy' -d debug

then, build your to-be-debugged application with this distribution, and install it, from then, you need a shell access to your android terminal (i use ssh, because adb over wifi is flacky but that’s another story, adb shell is good), you must then check if your terminal has the gdbserver command, if not, you can push it from the ndk to the terminal, and give it execution rights.

adb push ~/android-ndk-r8b/prebuilt/android-arm/gdbserver
adb shell
chmod 755 gdbserver

now, you can start your test program (from the UI), and tell gdbserver to attach to it,

ps | grep python
gdbserver :5050 --attach $pid

obviously using the pid found using the command, i use a shortcut though

./gdbserver :5050 --attach $(ps | grep python | tail -n 1 | tr -s " " " "| cut -d" " -f2)

This will tell gdbserver to listen on port 5050 (you can chose about any number between 1024 and 65536) and then the fun part begins

using the gdb in the ndk, load the pymodules.so file

$ ~/android-ndk-r8b/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-gdb\
 ~/python-for-android/dist/debug/private/libpymodules.so

of course, this path (and the following ones) will need to be adapted to your system!

once in gdb, you’ll want to attach to the gdbserver on your device, i assume you are on the same wifi network as your device, but if not, you can do a port redirection through adb link i think

(gdb) target remote transformer:5050

If everything went fine, you should get a message a bit like this

Remote debugging using transformer:5051
warning: while parsing target library list (at line 2): No segment defined for com.showgen.processcraft.freemium:python
warning: Could not load shared library symbols for 87 libraries, e.g. /system/bin/linker.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
0x400b9384 in ?? ()

as the helpful message put it, we need to tell gdb where it can load the shared libs with debug symbols. Personnaly i found this setting of `solib-search-path`to work well:

(gdb) set solib-search-path \
/home/gabriel/python-for-android/src/obj/local/armeabi/:\
/home/gabriel/python-for-android/dist/debug/private/lib/python2.7/lib-dynload/:\
/home/gabriel/python-for-android/dist/debug/libs/armeabi/

You can check the result of this by using the command `info sharedlibrary` (or `i sh` for the lazy) to see if symbols have been found, the interresting part for me is:

0x56d31348  0x56d5c148  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl.so
0x579b5400  0x579d5798  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl_image.so
0x579fa428  0x57a2c1f8  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl_ttf.so
0x56a5a6b0  0x56a69c20  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl_mixer.so
0x5b985570  0x5bbc9e90  Yes         /home/gabriel/python-for-android/dist/billing/libs/armeabi/libpython2.7.so
0x56a79a1c  0x56a79c04  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libapplication.so
0x56dee534  0x56dee5e8  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsdl_main.so
0x57604450  0x5761ed40  Yes         /home/gabriel/python-for-android/dist/billing/private/lib/python2.7/lib-dynload/_io.so
0x57a7426c  0x57a77630  Yes         /home/gabriel/python-for-android/dist/billing/private/lib/python2.7/lib-dynload/unicodedata.so
0x57b5b058  0x57b97d00  Yes         /home/gabriel/python-for-android/src/obj/local/armeabi/libsqlite3.so

Interrestingly, it seems the command is only useful after connecting to the target, doing it in reverse order doesn’t seem to bring any result, so the `search` is probably done only once, whet the search-patch is set.
Now, you can use the command `continue` or set breakpoints, or other rather common usage of gdb there. There are quite some resources out there about gdb usage :)

It’s not perfect, but i found it helping to be able to backtrace things on the tablet, in the nasty case you get a crash, or some weird “OSError”, with an unhelpful message, comming from the C side of things.

For the story, my error was likely caused by memory corruption due to the way we tried to reimplement the multiprocessing use of semaphore (sem_open being not implemented in android), and there is apparently no good way to do it. So i’ll look into android services instead when i have time, and try to give an API in python for that.