python-for-android revamp replaces master

This post is to announce that the revamped python-for-android toolchain, introduced in this previous post, has now been merged into python-for-android’s master branch. This is now the master branch going forward.

The revamp project is largely (but not quite) feature complete with the old toolchain, supporting almost all the same options when building pygame-based APKs. It also supports a new and much better SDL2 backend, which Kivy will move to in the future, but which also supports other kinds of python projects such as Vispy as described in the previous post.

We’ve done our best to minimise problems arising from this change. The old toolchain (with distribute.sh and build.py) is still available in the old_toolchain branch. Issues and PRs relating to this branch are still accepted, though existing PRs will need to be retargeted or merged manually (we’ll try to do what’s easiest case by case, if necessary).

If you use buildozer, this now pulls from the old_toolchain branch and so will work exactly as before. However, you will need to install the latest version from pypi or github (at least version 0.30) for this to work. Older versions of buildozer will continue to build APKs fine with existing projects, but if you create a new one they will download the new and incompatible python-for-android master. The revamp includes a fake distribute.sh executable giving these same instructions, so if this happens the problem and solution should be clearly displayed.

The new toolchain is currently documented (temporarily) here. We’ll push the new documentation to the normal readthedocs site as soon as possible, which will also include the legacy doc for the old toolchain so nothing is lost.

In the slightly longer term, the new toolchain will receive an official release and hopefully be released itself on pypi; unlike the old toolchain, it behaves as a fully installable python module with an improved command line interface.

python-for-android revamp replaces master

This post is to announce that the revamped python-for-android toolchain, introduced in this previous post, has now been merged into python-for-android’s master branch. This is now the master branch going forward.

The revamp project is largely (but not quite) feature complete with the old toolchain, supporting almost all the same options when building pygame-based APKs. It also supports a new and much better SDL2 backend, which Kivy will move to in the future, but which also supports other kinds of python projects such as Vispy as described in the previous post.

We’ve done our best to minimise problems arising from this change. The old toolchain (with distribute.sh and build.py) is still available in the old_toolchain branch. Issues and PRs relating to this branch are still accepted, though existing PRs will need to be retargeted or merged manually (we’ll try to do what’s easiest case by case, if necessary).

If you use buildozer, this now pulls from the old_toolchain branch and so will work exactly as before. However, you will need to install the latest version from pypi or github (at least version 0.30) for this to work. Older versions of buildozer will continue to build APKs fine with existing projects, but if you create a new one they will download the new and incompatible python-for-android master. The revamp includes a fake distribute.sh executable giving these same instructions, so if this happens the problem and solution should be clearly displayed.

The new toolchain is currently documented (temporarily) here. We’ll push the new documentation to the normal readthedocs site as soon as possible, which will also include the legacy doc for the old toolchain so nothing is lost.

In the slightly longer term, the new toolchain will receive an official release and hopefully be released itself on pypi; unlike the old toolchain, it behaves as a fully installable python module with an improved command line interface.

python-for-android revamp replaces master

This post is to announce that the revamped python-for-android toolchain, introduced in this previous post, has now been merged into python-for-android’s master branch. This is now the master branch going forward.

The revamp project is largely (but not quite) feature complete with the old toolchain, supporting almost all the same options when building pygame-based APKs. It also supports a new and much better SDL2 backend, which Kivy will move to in the future, but which also supports other kinds of python projects such as Vispy as described in the previous post.

We’ve done our best to minimise problems arising from this change. The old toolchain (with distribute.sh and build.py) is still available in the old_toolchain branch. Issues and PRs relating to this branch are still accepted, though existing PRs will need to be retargeted or merged manually (we’ll try to do what’s easiest case by case, if necessary).

If you use buildozer, this now pulls from the old_toolchain branch and so will work exactly as before. However, you will need to install the latest version from pypi or github (at least version 0.30) for this to work. Older versions of buildozer will continue to build APKs fine with existing projects, but if you create a new one they will download the new and incompatible python-for-android master. The revamp includes a fake distribute.sh executable giving these same instructions, so if this happens the problem and solution should be clearly displayed.

The new toolchain is currently documented (temporarily) here. We’ll push the new documentation to the normal readthedocs site as soon as possible, which will also include the legacy doc for the old toolchain so nothing is lost.

In the slightly longer term, the new toolchain will receive an official release and hopefully be released itself on pypi; unlike the old toolchain, it behaves as a fully installable python module with an improved command line interface.

python-for-android revamped

I’ve recently been working on a significantly revamped version of python-for-android, the Kivy-project tools that take a Python program and package it - along with any dependencies and the Python interpreter itself - into an Android APK that can be run and distributed just like a normal Android app. This rewrite is driven by the problem that although the current python-for-android is fairly robust as far as the build process goes, it’s quite set in its ways and hard to modify to make large changes such as to support backends other than Kivy.

To this end, the revamp project has several major goals:

  • Rewrite python-for-android to a fully Python toolchain that can be more easily modified and extended.
  • Support multiple bootstrap targets for different kinds of Python scrips to run on Android, starting with a new SDL2 backend for Kivy applications.
  • Support Python 3 on Android.
  • Support some kind of binary distribution, enabling easier and cross-platform distribution…this should work on Windows!
  • Be a standalone PyPI module with a more convenient and standard interface, potentially interfacing with setuptools.
  • Support (less painfully) multiple Android architectures, or fat APKs supporting more than one.

Each of these is individually a popularly requested feature, and this is a great opportunity to go for all of them at once!

I’m making this post to announce that the python-for-android revamp project has reached a usable state, with several of these goals either implemented or significantly advanced, and all of them at least made much more accessible. The core change is that all of the original toolchain has been rewritten in Python, with the initial structure based on the recent Kivy-iOS rewrite. It’s also much lighter, all heavy pygame stuff is downloaded on demand instead of built in, and is designed to be accessed by a single set of commands and the new python-for-android executable rather than by the separate invocation of different scripts in different places. I won’t go into the technical details here, but you can find the (WIP) documentation temporarily hosted here. If anyone would like to test it you can try the instructions there, but the project is in an experimental state right now and it’s likely you may encounter bugs or missing features, the current focus is ironing these out. I’m very happy to discuss these on the kivy-users mailing list or #kivy irc channel on irc.freenode.net.

Another of the core goals of this rewrite was to support multiple app backends; in Kivy’s case in particular we want to move from Pygame to a new SDL2 bootstrap, but this could also include support for other Python module backends. As of now, the new python-for-android supports the old Pygame bootstrap mostly as before, but also implements an SDL2 based PythonActivity that works very well with Kivy - highly anecdotal testing found, amongst other things, app start time reduced to 60% of what it was with Pygame. It also simplifies maintenance as SDL2’s java components fulfil the same role as those maintained in the Pygame backend, but no longer require much special treatment as functionality such as touch interaction and pausing are now accessed with the SDL2 api just as on desktop platforms.

Further, python-for-android can now build non-Kivy apps! The first project with this support built in is the Vispy scientific visualisation library. This uses the same new SDL2 backend but Kivy is not involved, and the build process does not use Cython (unlike with Kivy); instead, SDL2 and OpenGL are called entirely with ctypes, using pysdl2 and Vispy’s own gloo wrapper respectively. I didn’t even have to patch things much for this, Vispy is mostly self contained and already quite platform independent, barring a couple of possible small bugs and a hack to avoid the lack of a supported font provider on Android. Vispy also uses numpy heavily, but this was already supported by python-for-android and seems to work fine. The Vispy support is itself quite experimental and there are some performance issues that will need resolving, but it was very simple to set up with the new toolchain. Here’s a screenshot of one of Vispy’s 3D examples running on Android - there are a few small visual artifacts, but I think these are small bugs in Vispy’s OpenGL ES 2 support that the Vispy team are actively addressing:

Image of Vispy running on Android

Support for binary distribution and multiple architectures are both partially implemented but (at the time of writing) not yet working. However, the toolchain is built around them, so there should be no major issues. The initial idea with binary distribution will be to simply make available a number of prebuilt distributions (i.e. Android projects with the Python interpreter) that include common dependencies, so that when the user adds modules as requirements when calling python-for-android they can automatically be checked and an appropriate choice downloaded, with this process being transparent to the user and not requiring any special options. This should not only make many builds faster but also work on Windows, one of our most requested features but something that was not possible when the toolchain required that everything be locally compiled. Likewise, the toolchain has semi-implemented support for multiple architectures internally, but none other than armeabi are yet supported and there will be bugs to work out when more are enabled. Still, these will (fingers crossed) be things to look forward to in the relatively near future.

I should note here that this model of binary distribution is what I initially targeted as a natural extension of python-for-android’s distribution system - although we never made much of it, the first step is already to build a standalone android package which later can be distributed separately and populated with app details by a user, and the above just involves making such prebuilt distributions available to download and use automatically. I found more recently the method of the pybee project/Toga toolkit’s Python-Android-template. This is a similar idea (built by a modified python-for-android) but implemented much better as a standalone project with app details populated by cookiecutter and the packaging of the user’s Python modules taken care of using ant itself, the normal APK build and deployment interface - I didn’t know this was even possible! This means that the user can just write their Python code, drop it in place, and run the APK build, a very neat process. It should be quite easy to modify python-for-android’s dist output to easily create such templates, its dist system is functionally the same thing with a different and less standard-Android interface, and I hope that doing so could make it even more convenient for users to build and distribute different kinds of Python apps.

This leaves the elephant in the room, support for Python 3 on Android, which is perhaps the most requested feature for Kivy itself. The new toolchain makes significant progress on this in a couple of ways. The first is by removing hardcoded use of Python 2, so that now (barring remaining bugs) a Python 3 build should be well behaved once a recipe and appropriate Android patch set are added. The second is that the old Kivy Android bootstrap probably needed significant changes to support Python 3, but this is sidestepped entirely by moving to the new SDL2 backend which should have no issues supporting Python3 more or less the same as on the desktop. However, the missing link here is still actually being able to compile Python 3 for Android, and I don’t yet know how to do this. Some of Kivy’s main Python 2 patches come from this site, but this has Python 3 patches only up to 3.2.2 and it would be ideal to target 3.4 or 3.5 (and to be able to support new versions as they appear). I’ve looked around and seen a few different discussions of this, but I’m not sure what’s the best direction to try. If anyone has any information about places to find more up to date patch sets I would be very interested. I can’t make any other predictions about this as I don’t know much about the Python compilation process or what difficulties might be involved in doing the work we need.

That’s all for now. I’ll note again that this is an initial announcement of the new toolchain; I hope that people may be interested to look and try it, and it should support most of what the old toolchain does when it comes to compiling Pygame APKs, but there are likely to be bugs and missing features particularly surrounding (but not limited to) the new additions. If you’re interested in making this toolchain work with other modules or backends, or just have any questions, comments or criticisms, let us know! If you want to keep informed, watch this space, I’ll make further announcements as things develop. If there is developer interest and people start switching from the old toolchain, I hope development will speed up a lot and quickly approach proper feature parity.

tl;dr (I was told I should have one): Kivy’s python-for-android build tools have been revamped to have a better interface, build apps based on SDL2, build non-Kivy apps (currently Vispy apps), and be more extensible. Further semi-complete features include binary distribution, Windows support, and multiple architecture targets. Python 3 is brought closer but needs direct compilation work.

python-for-android revamped

I’ve recently been working on a significantly revamped version of python-for-android, the Kivy-project tools that take a Python program and package it - along with any dependencies and the Python interpreter itself - into an Android APK that can be run and distributed just like a normal Android app. This rewrite is driven by the problem that although the current python-for-android is fairly robust as far as the build process goes, it’s quite set in its ways and hard to modify to make large changes such as to support backends other than Kivy.

To this end, the revamp project has several major goals:

  • Rewrite python-for-android to a fully Python toolchain that can be more easily modified and extended.
  • Support multiple bootstrap targets for different kinds of Python scrips to run on Android, starting with a new SDL2 backend for Kivy applications.
  • Support Python 3 on Android.
  • Support some kind of binary distribution, enabling easier and cross-platform distribution…this should work on Windows!
  • Be a standalone PyPI module with a more convenient and standard interface, potentially interfacing with setuptools.
  • Support (less painfully) multiple Android architectures, or fat APKs supporting more than one.

Each of these is individually a popularly requested feature, and this is a great opportunity to go for all of them at once!

I’m making this post to announce that the python-for-android revamp project has reached a usable state, with several of these goals either implemented or significantly advanced, and all of them at least made much more accessible. The core change is that all of the original toolchain has been rewritten in Python, with the initial structure based on the recent Kivy-iOS rewrite. It’s also much lighter, all heavy pygame stuff is downloaded on demand instead of built in, and is designed to be accessed by a single set of commands and the new python-for-android executable rather than by the separate invocation of different scripts in different places. I won’t go into the technical details here, but you can find the (WIP) documentation temporarily hosted here. If anyone would like to test it you can try the instructions there, but the project is in an experimental state right now and it’s likely you may encounter bugs or missing features, the current focus is ironing these out. I’m very happy to discuss these on the kivy-users mailing list or #kivy irc channel on irc.freenode.net.

Another of the core goals of this rewrite was to support multiple app backends; in Kivy’s case in particular we want to move from Pygame to a new SDL2 bootstrap, but this could also include support for other Python module backends. As of now, the new python-for-android supports the old Pygame bootstrap mostly as before, but also implements an SDL2 based PythonActivity that works very well with Kivy - highly anecdotal testing found, amongst other things, app start time reduced to 60% of what it was with Pygame. It also simplifies maintenance as SDL2’s java components fulfil the same role as those maintained in the Pygame backend, but no longer require much special treatment as functionality such as touch interaction and pausing are now accessed with the SDL2 api just as on desktop platforms.

Further, python-for-android can now build non-Kivy apps! The first project with this support built in is the Vispy scientific visualisation library. This uses the same new SDL2 backend but Kivy is not involved, and the build process does not use Cython (unlike with Kivy); instead, SDL2 and OpenGL are called entirely with ctypes, using pysdl2 and Vispy’s own gloo wrapper respectively. I didn’t even have to patch things much for this, Vispy is mostly self contained and already quite platform independent, barring a couple of possible small bugs and a hack to avoid the lack of a supported font provider on Android. Vispy also uses numpy heavily, but this was already supported by python-for-android and seems to work fine. The Vispy support is itself quite experimental and there are some performance issues that will need resolving, but it was very simple to set up with the new toolchain. Here’s a screenshot of one of Vispy’s 3D examples running on Android - there are a few small visual artifacts, but I think these are small bugs in Vispy’s OpenGL ES 2 support that the Vispy team are actively addressing:

Image of Vispy running on Android

Support for binary distribution and multiple architectures are both partially implemented but (at the time of writing) not yet working. However, the toolchain is built around them, so there should be no major issues. The initial idea with binary distribution will be to simply make available a number of prebuilt distributions (i.e. Android projects with the Python interpreter) that include common dependencies, so that when the user adds modules as requirements when calling python-for-android they can automatically be checked and an appropriate choice downloaded, with this process being transparent to the user and not requiring any special options. This should not only make many builds faster but also work on Windows, one of our most requested features but something that was not possible when the toolchain required that everything be locally compiled. Likewise, the toolchain has semi-implemented support for multiple architectures internally, but none other than armeabi are yet supported and there will be bugs to work out when more are enabled. Still, these will (fingers crossed) be things to look forward to in the relatively near future.

I should note here that this model of binary distribution is what I initially targeted as a natural extension of python-for-android’s distribution system - although we never made much of it, the first step is already to build a standalone android package which later can be distributed separately and populated with app details by a user, and the above just involves making such prebuilt distributions available to download and use automatically. I found more recently the method of the pybee project/Toga toolkit’s Python-Android-template. This is a similar idea (built by a modified python-for-android) but implemented much better as a standalone project with app details populated by cookiecutter and the packaging of the user’s Python modules taken care of using ant itself, the normal APK build and deployment interface - I didn’t know this was even possible! This means that the user can just write their Python code, drop it in place, and run the APK build, a very neat process. It should be quite easy to modify python-for-android’s dist output to easily create such templates, its dist system is functionally the same thing with a different and less standard-Android interface, and I hope that doing so could make it even more convenient for users to build and distribute different kinds of Python apps.

This leaves the elephant in the room, support for Python 3 on Android, which is perhaps the most requested feature for Kivy itself. The new toolchain makes significant progress on this in a couple of ways. The first is by removing hardcoded use of Python 2, so that now (barring remaining bugs) a Python 3 build should be well behaved once a recipe and appropriate Android patch set are added. The second is that the old Kivy Android bootstrap probably needed significant changes to support Python 3, but this is sidestepped entirely by moving to the new SDL2 backend which should have no issues supporting Python3 more or less the same as on the desktop. However, the missing link here is still actually being able to compile Python 3 for Android, and I don’t yet know how to do this. Some of Kivy’s main Python 2 patches come from this site, but this has Python 3 patches only up to 3.2.2 and it would be ideal to target 3.4 or 3.5 (and to be able to support new versions as they appear). I’ve looked around and seen a few different discussions of this, but I’m not sure what’s the best direction to try. If anyone has any information about places to find more up to date patch sets I would be very interested. I can’t make any other predictions about this as I don’t know much about the Python compilation process or what difficulties might be involved in doing the work we need.

That’s all for now. I’ll note again that this is an initial announcement of the new toolchain; I hope that people may be interested to look and try it, and it should support most of what the old toolchain does when it comes to compiling Pygame APKs, but there are likely to be bugs and missing features particularly surrounding (but not limited to) the new additions. If you’re interested in making this toolchain work with other modules or backends, or just have any questions, comments or criticisms, let us know! If you want to keep informed, watch this space, I’ll make further announcements as things develop. If there is developer interest and people start switching from the old toolchain, I hope development will speed up a lot and quickly approach proper feature parity.

tl;dr (I was told I should have one): Kivy’s python-for-android build tools have been revamped to have a better interface, build apps based on SDL2, build non-Kivy apps (currently Vispy apps), and be more extensible. Further semi-complete features include binary distribution, Windows support, and multiple architecture targets. Python 3 is brought closer but needs direct compilation work.

python-for-android revamped

I’ve recently been working on a significantly revamped version of python-for-android, the Kivy-project tools that take a Python program and package it - along with any dependencies and the Python interpreter itself - into an Android APK that can be run and distributed just like a normal Android app. This rewrite is driven by the problem that although the current python-for-android is fairly robust as far as the build process goes, it’s quite set in its ways and hard to modify to make large changes such as to support backends other than Kivy.

To this end, the revamp project has several major goals:

  • Rewrite python-for-android to a fully Python toolchain that can be more easily modified and extended.
  • Support multiple bootstrap targets for different kinds of Python scrips to run on Android, starting with a new SDL2 backend for Kivy applications.
  • Support Python 3 on Android.
  • Support some kind of binary distribution, enabling easier and cross-platform distribution…this should work on Windows!
  • Be a standalone PyPI module with a more convenient and standard interface, potentially interfacing with setuptools.
  • Support (less painfully) multiple Android architectures, or fat APKs supporting more than one.

Each of these is individually a popularly requested feature, and this is a great opportunity to go for all of them at once!

I’m making this post to announce that the python-for-android revamp project has reached a usable state, with several of these goals either implemented or significantly advanced, and all of them at least made much more accessible. The core change is that all of the original toolchain has been rewritten in Python, with the initial structure based on the recent Kivy-iOS rewrite. It’s also much lighter, all heavy pygame stuff is downloaded on demand instead of built in, and is designed to be accessed by a single set of commands and the new python-for-android executable rather than by the separate invocation of different scripts in different places. I won’t go into the technical details here, but you can find the (WIP) documentation temporarily hosted here. If anyone would like to test it you can try the instructions there, but the project is in an experimental state right now and it’s likely you may encounter bugs or missing features, the current focus is ironing these out. I’m very happy to discuss these on the kivy-users mailing list or #kivy irc channel on irc.freenode.net.

Another of the core goals of this rewrite was to support multiple app backends; in Kivy’s case in particular we want to move from Pygame to a new SDL2 bootstrap, but this could also include support for other Python module backends. As of now, the new python-for-android supports the old Pygame bootstrap mostly as before, but also implements an SDL2 based PythonActivity that works very well with Kivy - highly anecdotal testing found, amongst other things, app start time reduced to 60% of what it was with Pygame. It also simplifies maintenance as SDL2’s java components fulfil the same role as those maintained in the Pygame backend, but no longer require much special treatment as functionality such as touch interaction and pausing are now accessed with the SDL2 api just as on desktop platforms.

Further, python-for-android can now build non-Kivy apps! The first project with this support built in is the Vispy scientific visualisation library. This uses the same new SDL2 backend but Kivy is not involved, and the build process does not use Cython (unlike with Kivy); instead, SDL2 and OpenGL are called entirely with ctypes, using pysdl2 and Vispy’s own gloo wrapper respectively. I didn’t even have to patch things much for this, Vispy is mostly self contained and already quite platform independent, barring a couple of possible small bugs and a hack to avoid the lack of a supported font provider on Android. Vispy also uses numpy heavily, but this was already supported by python-for-android and seems to work fine. The Vispy support is itself quite experimental and there are some performance issues that will need resolving, but it was very simple to set up with the new toolchain. Here’s a screenshot of one of Vispy’s 3D examples running on Android - there are a few small visual artifacts, but I think these are small bugs in Vispy’s OpenGL ES 2 support that the Vispy team are actively addressing:

Image of Vispy running on Android

Support for binary distribution and multiple architectures are both partially implemented but (at the time of writing) not yet working. However, the toolchain is built around them, so there should be no major issues. The initial idea with binary distribution will be to simply make available a number of prebuilt distributions (i.e. Android projects with the Python interpreter) that include common dependencies, so that when the user adds modules as requirements when calling python-for-android they can automatically be checked and an appropriate choice downloaded, with this process being transparent to the user and not requiring any special options. This should not only make many builds faster but also work on Windows, one of our most requested features but something that was not possible when the toolchain required that everything be locally compiled. Likewise, the toolchain has semi-implemented support for multiple architectures internally, but none other than armeabi are yet supported and there will be bugs to work out when more are enabled. Still, these will (fingers crossed) be things to look forward to in the relatively near future.

I should note here that this model of binary distribution is what I initially targeted as a natural extension of python-for-android’s distribution system - although we never made much of it, the first step is already to build a standalone android package which later can be distributed separately and populated with app details by a user, and the above just involves making such prebuilt distributions available to download and use automatically. I found more recently the method of the pybee project/Toga toolkit’s Python-Android-template. This is a similar idea (built by a modified python-for-android) but implemented much better as a standalone project with app details populated by cookiecutter and the packaging of the user’s Python modules taken care of using ant itself, the normal APK build and deployment interface - I didn’t know this was even possible! This means that the user can just write their Python code, drop it in place, and run the APK build, a very neat process. It should be quite easy to modify python-for-android’s dist output to easily create such templates, its dist system is functionally the same thing with a different and less standard-Android interface, and I hope that doing so could make it even more convenient for users to build and distribute different kinds of Python apps.

This leaves the elephant in the room, support for Python 3 on Android, which is perhaps the most requested feature for Kivy itself. The new toolchain makes significant progress on this in a couple of ways. The first is by removing hardcoded use of Python 2, so that now (barring remaining bugs) a Python 3 build should be well behaved once a recipe and appropriate Android patch set are added. The second is that the old Kivy Android bootstrap probably needed significant changes to support Python 3, but this is sidestepped entirely by moving to the new SDL2 backend which should have no issues supporting Python3 more or less the same as on the desktop. However, the missing link here is still actually being able to compile Python 3 for Android, and I don’t yet know how to do this. Some of Kivy’s main Python 2 patches come from this site, but this has Python 3 patches only up to 3.2.2 and it would be ideal to target 3.4 or 3.5 (and to be able to support new versions as they appear). I’ve looked around and seen a few different discussions of this, but I’m not sure what’s the best direction to try. If anyone has any information about places to find more up to date patch sets I would be very interested. I can’t make any other predictions about this as I don’t know much about the Python compilation process or what difficulties might be involved in doing the work we need.

That’s all for now. I’ll note again that this is an initial announcement of the new toolchain; I hope that people may be interested to look and try it, and it should support most of what the old toolchain does when it comes to compiling Pygame APKs, but there are likely to be bugs and missing features particularly surrounding (but not limited to) the new additions. If you’re interested in making this toolchain work with other modules or backends, or just have any questions, comments or criticisms, let us know! If you want to keep informed, watch this space, I’ll make further announcements as things develop. If there is developer interest and people start switching from the old toolchain, I hope development will speed up a lot and quickly approach proper feature parity.

tl;dr (I was told I should have one): Kivy’s python-for-android build tools have been revamped to have a better interface, build apps based on SDL2, build non-Kivy apps (currently Vispy apps), and be more extensible. Further semi-complete features include binary distribution, Windows support, and multiple architecture targets. Python 3 is brought closer but needs direct compilation work.

Python on Android

There are an increasing number of resources about different ways of running Python on Android. Kivy (and its subprojects) are commonly mentioned, as one of the most mature and popular ways to do so, but one thing that gets less attention is the details of what you can do with Python itself once it’s running on the device - what are the limitations of this? Can we use any Python module? What about calling Android APIs, can we perform all of the functions of a Java application? These are all somewhat leading questions, they are things addressed by Kivy or its associated projects, and in this post I’ll summarise some of the most interesting and important details.

python-for-android

Before anything else, let’s look quickly at the tool Kivy actually uses to get Python on Android; the unimaginatively-named python-for-android project. The basic functionality of this tool is to first build a distribution, an Android project directory that includes all the components Kivy needs to run, compiled for Android by its NDK. This includes in particular the Python interpreter itself, plus Kivy and the libraries it depends on - currently Pygame and SDL amongst others, although we are working to modernise this bit. The distribution also includes a Java bootstrap, a normal app structure whose job is to display Kivy’s OpenGL surface and to mediate between Kivy and Android. All these components can then be bundled into an APK with the user’s Python script and different settings (icon, name, orientation etc.) to taste.

This is only the basic procedure, the APK can (and does) include much more than just these essentials. Amongst other things, most of the Python standard library is built in by default, and pure Python modules can be included easily so in general you can perform tasks using just the same libraries you would on the desktop. Libraries with compiled components are more complex, but can be built and included as long as python-for-android has a compilation recipe for them (or you provide your own) - these are often quite simple, just setting some compilation flags and running the normal build procedure, although some modules need additional patching. Python-for-android includes quite a few recipes by default, including very popular modules like numpy, sqlite3, twisted and even django!

The above is the basics of how python-for-android works but is far from the whole story, and you can check the documentation for more information about building your own APKs - in particular, we recommend using Buildozer, which gives python-for-android a more convenient interface and can manage some dependencies (in particular the Android SDK and NDK) automatically. This is also quite focused on Kivy itself, but we’re trying to move to make it easier for other projects to use the same toolchain - the core process of building and including Python should be similar, but there’s no need for the bootstrap app at the end to support only Kivy’s specific needs.

Calling Android APIs with PyJNIus

In normal Android application development, interaction with the Android API is an important part of how your app behaves - getting sensor data, creating notifications, vibrating, pausing and restarting, or just about anything else. Kivy takes care of the essentials for you, but many of these are things you’ll still want to manage yourself from Python. For this reason we have the PyJNIus project, also developed under the Kivy organisation, which automatically wraps Java code in a Python interface.

As a simple example, here’s the Python code to have an Android device vibrate for 10s:

from jnius import autoclass

# We need a reference to the Java activity running the current
# application, this reference is stored automatically by
# Kivy's PythonActivity bootstrap:
PythonActivity = autoclass('org.renpy.android.PythonActivity')
activity = PythonActivity.mActivity

Context = autoclass('android.content.Context')
vibrator = activity.getSystemService(Context.VIBRATOR_SERVICE)

vibrator.vibrate(10000)  # the argument is in milliseconds

If you’re familiar with the Android API, you’ll notice that this is very similar to the Java code you’d use for the same task; PyJNIus just lets us call the same API directly from Python. Most of the Android API can be called from Python in the same way, letting you achieve the same things as a normal Java application.

The main disadvantages of using PyJNIus directly are that it requires some understanding of how the Android API is structured, and that it is quite verbose - though the latter just reflects the nature of the equivalent Java code. For this reason, the Kivy project set includes Plyer.

Plyer: A platform-independent API for platform-specific features

The Plyer project takes a step back from the specific implementation details of individual platforms in order to try to create a simple, pythonic interface for a subset of (mostly) shared functionality. For instance, the vibration example above would become

from plyer.vibrator import vibrate
vibrate(10)  # in Plyer, the argument is in seconds

Further, Plyer is not just for Android but would try to do something appropriate on any of its supported platforms - currently Android, iOS, Linux, Windows and OS X (on iOS, PyOBJus fulfils a similar role to PyJNIus on Android). The vibrator is actually a bad example as only Android is currently implemented, but other APIs such as checking the battery (from plyer import battery; print(battery.status)) or text-to-speech (from plyer import tts; tts.speak('hello world')) would already work on both desktop and mobile devices, and others such as the compass or gyroscope sensors or sending SMS messages would work on both Android and iOS.

Plyer is very much under development, with new API wrapper contributions very welcome, and is the subject of a (second) GSoC project this year. We hope that it will become increasingly feature-complete.

Not just for Kivy

All of these tools have been shaped in their current form by the needs of Kivy, but are really more generic Python tools; Plyer specifically avoids any Kivy dependency, and PyJNIus only makes an assumption about how to access the JNI environment on Android. We hope that these tools can be more generally useful to anyone running Python on Android; for instance, you can already experiment with PyJNIus using the QPython Android app. Python-for-android is more tied to Kivy’s current toolchain but this is a detail under review, and we’re happy to discuss the details of Android compilation with anyone interested.

Overall, a lot is possible with Python on Android, despite how different the Python environment is to the Java development that is directly targeted. But there’s much more that could be done - if you’re interested, now is a great time to dive in!

Python on Android

There are an increasing number of resources about different ways of running Python on Android. Kivy (and its subprojects) are commonly mentioned, as one of the most mature and popular ways to do so, but one thing that gets less attention is the details of what you can do with Python itself once it’s running on the device - what are the limitations of this? Can we use any Python module? What about calling Android APIs, can we perform all of the functions of a Java application? These are all somewhat leading questions, they are things addressed by Kivy or its associated projects, and in this post I’ll summarise some of the most interesting and important details.

python-for-android

Before anything else, let’s look quickly at the tool Kivy actually uses to get Python on Android; the unimaginatively-named python-for-android project. The basic functionality of this tool is to first build a distribution, an Android project directory that includes all the components Kivy needs to run, compiled for Android by its NDK. This includes in particular the Python interpreter itself, plus Kivy and the libraries it depends on - currently Pygame and SDL amongst others, although we are working to modernise this bit. The distribution also includes a Java bootstrap, a normal app structure whose job is to display Kivy’s OpenGL surface and to mediate between Kivy and Android. All these components can then be bundled into an APK with the user’s Python script and different settings (icon, name, orientation etc.) to taste.

This is only the basic procedure, the APK can (and does) include much more than just these essentials. Amongst other things, most of the Python standard library is built in by default, and pure Python modules can be included easily so in general you can perform tasks using just the same libraries you would on the desktop. Libraries with compiled components are more complex, but can be built and included as long as python-for-android has a compilation recipe for them (or you provide your own) - these are often quite simple, just setting some compilation flags and running the normal build procedure, although some modules need additional patching. Python-for-android includes quite a few recipes by default, including very popular modules like numpy, sqlite3, twisted and even django!

The above is the basics of how python-for-android works but is far from the whole story, and you can check the documentation for more information about building your own APKs - in particular, we recommend using Buildozer, which gives python-for-android a more convenient interface and can manage some dependencies (in particular the Android SDK and NDK) automatically. This is also quite focused on Kivy itself, but we’re trying to move to make it easier for other projects to use the same toolchain - the core process of building and including Python should be similar, but there’s no need for the bootstrap app at the end to support only Kivy’s specific needs.

Calling Android APIs with PyJNIus

In normal Android application development, interaction with the Android API is an important part of how your app behaves - getting sensor data, creating notifications, vibrating, pausing and restarting, or just about anything else. Kivy takes care of the essentials for you, but many of these are things you’ll still want to manage yourself from Python. For this reason we have the PyJNIus project, also developed under the Kivy organisation, which automatically wraps Java code in a Python interface.

As a simple example, here’s the Python code to have an Android device vibrate for 10s:

from jnius import autoclass

# We need a reference to the Java activity running the current
# application, this reference is stored automatically by
# Kivy's PythonActivity bootstrap:
PythonActivity = autoclass('org.renpy.android.PythonActivity')
activity = PythonActivity.mActivity

Context = autoclass('android.content.Context')
vibrator = activity.getSystemService(Context.VIBRATOR_SERVICE)

vibrator.vibrate(10000)  # the argument is in milliseconds

If you’re familiar with the Android API, you’ll notice that this is very similar to the Java code you’d use for the same task; PyJNIus just lets us call the same API directly from Python. Most of the Android API can be called from Python in the same way, letting you achieve the same things as a normal Java application.

The main disadvantages of using PyJNIus directly are that it requires some understanding of how the Android API is structured, and that it is quite verbose - though the latter just reflects the nature of the equivalent Java code. For this reason, the Kivy project set includes Plyer.

Plyer: A platform-independent API for platform-specific features

The Plyer project takes a step back from the specific implementation details of individual platforms in order to try to create a simple, pythonic interface for a subset of (mostly) shared functionality. For instance, the vibration example above would become

from plyer.vibrator import vibrate
vibrate(10)  # in Plyer, the argument is in seconds

Further, Plyer is not just for Android but would try to do something appropriate on any of its supported platforms - currently Android, iOS, Linux, Windows and OS X (on iOS, PyOBJus fulfils a similar role to PyJNIus on Android). The vibrator is actually a bad example as only Android is currently implemented, but other APIs such as checking the battery (from plyer import battery; print(battery.status)) or text-to-speech (from plyer import tts; tts.speak('hello world')) would already work on both desktop and mobile devices, and others such as the compass or gyroscope sensors or sending SMS messages would work on both Android and iOS.

Plyer is very much under development, with new API wrapper contributions very welcome, and is the subject of a (second) GSoC project this year. We hope that it will become increasingly feature-complete.

Not just for Kivy

All of these tools have been shaped in their current form by the needs of Kivy, but are really more generic Python tools; Plyer specifically avoids any Kivy dependency, and PyJNIus only makes an assumption about how to access the JNI environment on Android. We hope that these tools can be more generally useful to anyone running Python on Android; for instance, you can already experiment with PyJNIus using the QPython Android app. Python-for-android is more tied to Kivy’s current toolchain but this is a detail under review, and we’re happy to discuss the details of Android compilation with anyone interested.

Overall, a lot is possible with Python on Android, despite how different the Python environment is to the Java development that is directly targeted. But there’s much more that could be done - if you’re interested, now is a great time to dive in!

Python on Android

There are an increasing number of resources about different ways of running Python on Android. Kivy (and its subprojects) are commonly mentioned, as one of the most mature and popular ways to do so, but one thing that gets less attention is the details of what you can do with Python itself once it’s running on the device - what are the limitations of this? Can we use any Python module? What about calling Android APIs, can we perform all of the functions of a Java application? These are all somewhat leading questions, they are things addressed by Kivy or its associated projects, and in this post I’ll summarise some of the most interesting and important details.

python-for-android

Before anything else, let’s look quickly at the tool Kivy actually uses to get Python on Android; the unimaginatively-named python-for-android project. The basic functionality of this tool is to first build a distribution, an Android project directory that includes all the components Kivy needs to run, compiled for Android by its NDK. This includes in particular the Python interpreter itself, plus Kivy and the libraries it depends on - currently Pygame and SDL amongst others, although we are working to modernise this bit. The distribution also includes a Java bootstrap, a normal app structure whose job is to display Kivy’s OpenGL surface and to mediate between Kivy and Android. All these components can then be bundled into an APK with the user’s Python script and different settings (icon, name, orientation etc.) to taste.

This is only the basic procedure, the APK can (and does) include much more than just these essentials. Amongst other things, most of the Python standard library is built in by default, and pure Python modules can be included easily so in general you can perform tasks using just the same libraries you would on the desktop. Libraries with compiled components are more complex, but can be built and included as long as python-for-android has a compilation recipe for them (or you provide your own) - these are often quite simple, just setting some compilation flags and running the normal build procedure, although some modules need additional patching. Python-for-android includes quite a few recipes by default, including very popular modules like numpy, sqlite3, twisted and even django!

The above is the basics of how python-for-android works but is far from the whole story, and you can check the documentation for more information about building your own APKs - in particular, we recommend using Buildozer, which gives python-for-android a more convenient interface and can manage some dependencies (in particular the Android SDK and NDK) automatically. This is also quite focused on Kivy itself, but we’re trying to move to make it easier for other projects to use the same toolchain - the core process of building and including Python should be similar, but there’s no need for the bootstrap app at the end to support only Kivy’s specific needs.

Calling Android APIs with PyJNIus

In normal Android application development, interaction with the Android API is an important part of how your app behaves - getting sensor data, creating notifications, vibrating, pausing and restarting, or just about anything else. Kivy takes care of the essentials for you, but many of these are things you’ll still want to manage yourself from Python. For this reason we have the PyJNIus project, also developed under the Kivy organisation, which automatically wraps Java code in a Python interface.

As a simple example, here’s the Python code to have an Android device vibrate for 10s:

from jnius import autoclass

# We need a reference to the Java activity running the current
# application, this reference is stored automatically by
# Kivy's PythonActivity bootstrap:
PythonActivity = autoclass('org.renpy.android.PythonActivity')
activity = PythonActivity.mActivity

Context = autoclass('android.content.Context')
vibrator = activity.getSystemService(Context.VIBRATOR_SERVICE)

vibrator.vibrate(10000)  # the argument is in milliseconds

If you’re familiar with the Android API, you’ll notice that this is very similar to the Java code you’d use for the same task; PyJNIus just lets us call the same API directly from Python. Most of the Android API can be called from Python in the same way, letting you achieve the same things as a normal Java application.

The main disadvantages of using PyJNIus directly are that it requires some understanding of how the Android API is structured, and that it is quite verbose - though the latter just reflects the nature of the equivalent Java code. For this reason, the Kivy project set includes Plyer.

Plyer: A platform-independent API for platform-specific features

The Plyer project takes a step back from the specific implementation details of individual platforms in order to try to create a simple, pythonic interface for a subset of (mostly) shared functionality. For instance, the vibration example above would become

from plyer.vibrator import vibrate
vibrate(10)  # in Plyer, the argument is in seconds

Further, Plyer is not just for Android but would try to do something appropriate on any of its supported platforms - currently Android, iOS, Linux, Windows and OS X (on iOS, PyOBJus fulfils a similar role to PyJNIus on Android). The vibrator is actually a bad example as only Android is currently implemented, but other APIs such as checking the battery (from plyer import battery; print(battery.status)) or text-to-speech (from plyer import tts; tts.speak('hello world')) would already work on both desktop and mobile devices, and others such as the compass or gyroscope sensors or sending SMS messages would work on both Android and iOS.

Plyer is very much under development, with new API wrapper contributions very welcome, and is the subject of a (second) GSoC project this year. We hope that it will become increasingly feature-complete.

Not just for Kivy

All of these tools have been shaped in their current form by the needs of Kivy, but are really more generic Python tools; Plyer specifically avoids any Kivy dependency, and PyJNIus only makes an assumption about how to access the JNI environment on Android. We hope that these tools can be more generally useful to anyone running Python on Android; for instance, you can already experiment with PyJNIus using the QPython Android app. Python-for-android is more tied to Kivy’s current toolchain but this is a detail under review, and we’re happy to discuss the details of Android compilation with anyone interested.

Overall, a lot is possible with Python on Android, despite how different the Python environment is to the Java development that is directly targeted. But there’s much more that could be done - if you’re interested, now is a great time to dive in!

Hy (lisp) and Kivy

I was recently reminded of the super cool Hy project. Hy is a lisp that compiles to python’s own abstract syntax tree, so it works perfectly with existing Python code (including with Cython etc.) but also exposes all the power of lisp.

For instance, here’s a simple Kivy application that simply displays a Label with the obligatory Hy pun, but written in Hy. I’ve included the normal Python code as comments so you can see exactly what the code is doing. If you’re new to Kivy and want to understand what the code actually does, check out my Kivy crash course:

(import [kivy.app [App]]
        [kivy.uix.label [Label]])
;; from kivy.app import App
;; from kivy.uix.label import Label


(defclass HyApp [App]
  [[build
    (fn [self]
      (apply Label [] {"text" "Hy world!"
                       "font_size" 100
                       "color" (, 0 1 0 1)}))]])

;; class HyApp(App):
;;     def build(self):
;;         return Label(text="Hy world!",
;;                      font_size=100,
;;                      color=(0, 1, 0, 1))

(.run (HyApp))

;; HyApp().run()

This works great, though only with python3 due to a small bug in Kivy - the kwargs of Label are eventually read in cython with a variable typed as str, which in python2 excludes the unicode Hy passes. Still, that’s not surprising even if it’s cool - part of the point of Hy is to interoperate perfectly with Python.

A tougher problem is how to use Kivy’s kv language with Hy. kv is a simple domain-specific language for declaring widget trees, making it easy to define event-driven interactions between the different properties of widgets. It’s really useful and we tend to recommend using it as much as possible, so it’d be great to have it work with Hy. I won’t explain the language here (you can see the Kivy doc or my own tutorials), but the key point is that much of it consists of interpreting normal python code, which I’d like to replace with Hy code.

It turns out making this work is actually really easy. Here’s the relevant part of lang.py in Kivy’s source, the file containing the code for the kv parser:

self.co_value = compile(value,
                        self.ctx.filename or '<string>',
                        mode)

value is the string of Python code whose output will set a property of a widget or be run when an event is registered. For instance, a line of kv code might be color: (1, 0, 0, some_function_of(self.alpha)), in which case value would be "(1, 0, 0, some_function_of(self.alpha))".

To make a line of Hy code work instead of Python, we can do an awful hack, replacing the above line with:

if value[-3:] == '#hy':
    from hy.importer import (ast_compile,
                             import_buffer_to_ast)
    from hy.compiler import hy_compile
    import ast
    ast_part = import_buffer_to_ast(value[:-3], '<stdin>')
    if mode == 'eval':
        ast_part = ast.Expression(ast_part.body[0].value)
    self.co_value = ast_compile(ast_part,
                                self.ctx.filename or '<string>',
                                mode)
else:
    self.co_value = compile(value,
                            self.ctx.filename or '<string>',
                            mode)

This new code checks if the line of Python ends with #hy, and if so runs the code through Hy’s own equivalent of compile (effectively parsing the Hy code to ast before doing the same thing as the normal Python code). I also have the extra muckiness of taking apart this ast if the compilation is in eval mode, because I couldn’t get Hy to return an ast.Expression in the first place. This is probably very easily and neatly fixed, but I’ve left it like this because a silly hack is good enough for a proof of concept. All credit for this part goes to the friendly Hy people on their irc channel, #hy on Freenode.

With this in place, we can write a new Python program, but this time use our Hy+kv language to define the widget tree. Here’s the new code on the Python (now Hy) side:

(import [kivy.app [App]]
        [kivy.lang [Builder]])

;; from kivy.app import App
;; from kivy.lang import Builder

(setv root (Builder.load_file "hy.kv"))

;; root = Builder.load_file("hy.kv")

(defclass HyApp [App]
  [[build
    (fn [self]
      root)]])

;; class HyApp(App):
;;     def build(self):
;;         return root

(.run (HyApp))

;; HyApp().run()

This obviously depends on our new kv file, "hy.kv", whose contents are as below. Kivy users will notice this file would normally be loaded automatically because the app name starts with Hy, but something about Hy seems to have broken this so I manually loaded it with the Builder.

BoxLayout:
    orientation: "vertical"
    Label:
        id: label
        text: "What is your name?"
    TextInput:
        id: ti
        text: ""
    Button:
        text: (.format "Greet me as {}" ti.text) #hy
        on_press: (setv label.text (.format "Hy there {}" ti.text)) #hy

# as normal kv, except the final 2 rules would normally be:
# text: "Greet me as {}".format(ti.text)
# on_press: label.text = "Hy there {}".format(ti.text)

Running the code…it works perfectly! Here’s a picture after typing my name and clicking the button:

Image of Kivy program after running Hy code

For those not familiar with kv, one of its features is that it automatically detects property changes and updates dependent properties - in this case, the text of the button should change every time ti.text changes (i.e. every time a letter is typed in the TextInput). This works too with the new Hy interface, because the parser detects the dependency by searching the string for substrings like ti.text, and these have been unmodified by the move to Hy. Hy does support syntax that would break this relationship, but it’s quite convenient as it is.

So…there we go, Hy support in Kivy! The hack to make kv language work is pretty terrible, but it looks like a proper solution with this basis would work fine - we could subclass the kv parsing Builder to support a Hy loading option, removing the need for the #hy at the end of each Hy line.