python-for-android 0.4 released, now available on PyPI

We’ve just officially released python-for-android 0.4, and pushed it to PyPI for the first time!

python-for-android is a packaging tool for turning Python scripts and apps into Android APKs. It was originally created for use with the Kivy graphical framework, but now supports multiple kinds of Python app including Kivy, PySDL2, a webview interface with Flask or other webserver backend, plain Python scripts without a GUI, or other possibilities such as Python builds for use in other applications.

This release is the culmination of all the work over the last year to replace Kivy’s old Android toolchain with something more flexible and useful for other projects. Major features added in this time include the fully Python toolchain itself, support for SDL2 and other bootstraps, (experimental) python3 support via the CrystaX NDK, multiple architecture support, and many general improvements to the backend. Many thanks to all the contributors who have made this possible!

From now on we intend to move to regular versioned releases rather than the previous rolling master branch. Short term targets for the next release include bringing the python3 build up to full functionality and stability, and some argument restructuring to make command line usage simpler and clearer.

As of this release, you can now install python-for-android with simply:

pip install python-for-android

For full instructions and further information, see the python-for-android documentation.

Android apps with Python, Flask and a WebView

python-for-android has just gained support for a new webview app interface, an alternative to the existing SDL2 or Pygame backends. Under this mode of operation the app gui consists entirely of a browser window directed to open a webpage on localhost, and the Python backend can then run any web framework (I tested with Flask, but others like Bottle or even Django should work), serving this website and managing the app backend.

Example Flask app running on Android

This idea is not itself new; I think SL4A has supported a kind of webview interface for some time and certainly does so now, and we’ve previously seen users running web servers alongside Kivy. The difference to other projects is that apps can take advantage of python-for-android’s relatively extensive toolchain including python3.5 support, the ability to build popular libraries like numpy, support for multiple architectures, and access to the Android API via PyJNIus or Plyer rather than SL4A.

In the image of my testing app above, each of the vibration and orientation buttons sends a request to a Flask url that calls the Android API with PyJNIus to achieve the desired result.

Building a webview app

You can use the webview backend by adding --bootstrap=webview to your python-for-android command line (see the documentation for more details), or including webviewjni in your --requirements argument list. Note that this is incompatible with using SDL or Kivy because the webview bootstrap does not start or manage an OpenGL context. If for any reason you want to run a web server alongside a Kivy app, this is possible but you’ll need to use a different bootstrap and manage the webview yourself via PyJNIus from your Kivy code.

You should also add your chosen web framework to the --requirements argument, or include it your app directory so that it will be imported locally. If there isn’t a recipe for it and it’s a pure Python module, make sure you also add its Python dependencies as these aren’t automatically included right now (letting pip resolve dependencies causes issues when they include compiled modules that must be built separately). python-for-android now includes a recipe for Flask that automatically installs its dependencies (jinja2, werkzeug, markupsafe, itsdangerous and click), so you only need to add flask to the requirements in that case.

Technical details

It turns out that very little hackery is necessary to make a webview type app work. The APK seems to need the INTERNET permission to use a WebView, but Android is very happy for the Python code to run a web server with no further problems.

Making PyJNIus work required a little extra work, as it previously relied on the now-absent SDL to access a pointer to the current JNIEnv. This was fairly simple to fix by using only the relevant code from SDL2 - the important parts are only a small fraction of what SDL provides, as SDL has to worry about all the app input and output going via JNI. For now, python-for-android just patches PyJNIus before building it, but now that there are three different ways to get the JNIEnv on Android this will need addressing somehow in PyJNIus itself.

Android apps with Python, Flask and a WebView

python-for-android has just gained support for a new webview app interface, an alternative to the existing SDL2 or Pygame backends. Under this mode of operation the app gui consists entirely of a browser window directed to open a webpage on localhost, and the Python backend can then run any web framework (I tested with Flask, but others like Bottle or even Django should work), serving this website and managing the app backend.

Example Flask app running on Android

This idea is not itself new; I think SL4A has supported a kind of webview interface for some time and certainly does so now, and we’ve previously seen users running web servers alongside Kivy. The difference to other projects is that apps can take advantage of python-for-android’s relatively extensive toolchain including python3.5 support, the ability to build popular libraries like numpy, support for multiple architectures, and access to the Android API via PyJNIus or Plyer rather than SL4A.

In the image of my testing app above, each of the vibration and orientation buttons sends a request to a Flask url that calls the Android API with PyJNIus to achieve the desired result.

Building a webview app

You can use the webview backend by adding --bootstrap=webview to your python-for-android command line (see the documentation for more details), or including webviewjni in your --requirements argument list. Note that this is incompatible with using SDL or Kivy because the webview bootstrap does not start or manage an OpenGL context. If for any reason you want to run a web server alongside a Kivy app, this is possible but you’ll need to use a different bootstrap and manage the webview yourself via PyJNIus from your Kivy code.

You should also add your chosen web framework to the --requirements argument, or include it your app directory so that it will be imported locally. If there isn’t a recipe for it and it’s a pure Python module, make sure you also add its Python dependencies as these aren’t automatically included right now (letting pip resolve dependencies causes issues when they include compiled modules that must be built separately). python-for-android now includes a recipe for Flask that automatically installs its dependencies (jinja2, werkzeug, markupsafe, itsdangerous and click), so you only need to add flask to the requirements in that case.

Technical details

It turns out that very little hackery is necessary to make a webview type app work. The APK seems to need the INTERNET permission to use a WebView, but Android is very happy for the Python code to run a web server with no further problems.

Making PyJNIus work required a little extra work, as it previously relied on the now-absent SDL to access a pointer to the current JNIEnv. This was fairly simple to fix by using only the relevant code from SDL2 - the important parts are only a small fraction of what SDL provides, as SDL has to worry about all the app input and output going via JNI. For now, python-for-android just patches PyJNIus before building it, but now that there are three different ways to get the JNIEnv on Android this will need addressing somehow in PyJNIus itself.

Kivy Android app showcase

A natural question when people hear about Kivy as a way to create Android apps in Python is…what can you do with it? Is it performant enough for games, can you call the Android APIs, do all apps look the same? One of the best resources for these kinds of question are existing apps, and in this post I’ll give a quick impression of three of my favourites. This is obviously highly subjective, but I’m focusing in particular on features of technical interest, apps that push Kivy beyond what’s normal to show what it capable of.

If you’re interested in other examples, there’s a fairly extensive (but far from exhaustive) list on the Kivy wiki, including winners of our programming contests and many contributions from users. If you’d like to make your own apps in Python, check out Kivy (which also runs on Windows, Linux, OS X and iOS) and python-for-android (which can also package non-Kivy Python apps).

Boardz

You can download Boardz here.

I’ve put Boardz first because it’s my single favourite Kivy app. It’s actually a work in progress (and in fact hasn’t been updated for a while), but is already a fun game showcasing some of Kivy’s more impressive performance potential.

Boardz homescreen and gameplay

Boardz homescreen (left) and gameplay (right). The black ring on the right is the input circle controlling rider posture.

Boardz is a snowboarding physics game; you control your snowboarder by touching the screen, then moving your finger with respect to its initial position to control your posture; quick movements throw your weight around and can cause you to jump, spin, or fall over, while just positioning the rider differently helps you to pick up speed or navigate barriers. The objective of the game is to get to the end of each stage, with different obstacles including slopes and jumps, collapsing structures, falling rocks, or even multidirectional gravity and rocket boosters. You can fail if your head collides with another object with too much force, or if you simply get stuck and can no longer reach the finish.

What’s immediately impressive is that all this runs well as a Python powered game running on a smartphone. It achieves this by being built using the KivEnt game engine developed by Jacob Kovac, one of Kivy’s core developers. This entity based system lets you write game code in Python but internally is highly optimised in Cython, using Kivy’s OpenGL API extremely efficently as well as interfacing with the popular Chipmunk Physics engine.

Boardz wipeout failure and ad example

Boardz wipeout failure by fatal collision (left), and an ad (right).

Boardz betrays its in-progress nature in other ways; you can see in the above screenshots that its UI isn’t very polished, and in this sense it’s the worst of the apps I’m showing here. However, it makes up for this with its surprisingly engaging gameplay, and a breadth of entertaining features not showcased here, including leader boards, racing your ghost, and different riders with different physics attributes.

A final technical feature interesting to Kivy app developers is that Boardz includes ad integration. Regardless of your feelings about ads themselves, the ability to use them is a major feature enquiry from new Kivy users. The problem here is that integrating with a normal ad provider normally requires adding to the Java components of your app, which it may not be immediately obvious how to do from Python. There are actually a number of resources for this nowadays, with a key point being that python-for-android tries to make it easy to include extra Java code, with which you can interact from Python using Pyjnius. KivEnt’s implementation, pictured above, is a nice demonstration.

Kognitivo

You can download Kognitivo here.

Kognitivo is perhaps the single most polished Kivy app on the Play store, being relatively complex, extensively customised, and exhibiting a number of nice Android API interactions. It is also (deservedly) possibly the most popular Kivy app on Google Play.

Kognitivo introduction and main screens

Kognitivo tutorial (left), game instructions (centre) and homescreen (right).

Kognitivo is a brain training and performance monitoring app. The basic structure is to perform a series of simple exercises intended to test different aspects of cognitive performance, being rated on accuracy and speed, and with the results compiled over time in order to detect and act on trends.

As I’ve said already, the nice thing about Kognitivo is its huge amount of polish. It is extensively themed such that no trace of the Kivy defaults remains, runs extremly smoothly, and includes many nice animation tweaks (unfortunately not captured in screenshots) to feel responsive and active.

Kognitivo gameplay, Android notification and in-app purchase

Kognitivo training game (left), Android notification (centre) and in-app purchase option (right).

On the technical side, Kognitivo exhibits a number of features not normally included in Kivy applications but possible through interaction with the Android API via some Java code and/or the aforementioned Pyjnius. These include notifications, interaction with your calendar, and in-app purchases.

The author of Kognitivo, Sergey Cheparev, has his own writeup of Kognitivo’s development on his blog, including discussion of the advantages and disadvantages of Kivy development, and of the experience of putting together all these features. This is a great resource on its own; I don’t agree with some of the author’s criticisms and some of Kivy’s features have been improved since then, but it’s an excellent overview of the experience. Most of all, he enthusiastically captures some of my own reasons for finding Python on Android interesting:

I think the most beautiful thing in it is to use the almightiness of [Python]’s frameworks. I used sqlalchemy and sqlite as a backend, and it worked like a charm! Python is the most powerfull language because of it’s frameworks, you can even start Django on your smartphone! It’s amazing! Or twisted for asynchro communication with server. Or nltk for in-app natural language processing. Or maybe you want make a mobile equations solver with scipy and numpy. This makes all the dreams come true.

Barly

You can download Barly here, or visit its own website.

Barly is the most recent of these apps to appear on Google Play. I’ve chosen to include it as a nice example of pulling off its concept quite well while making good use of Kivy; like Kognitivo the app is themed very differently to Kivy’s defaults (though it doesn’t look like a normal Android app either), and is generally well put together.

Barly homescreen, palate options, and beer example

Barly homescreen (left), palate options (centre) and a beer search result (right).

To quote its Google Play blurb, Barly is ‘your personal beer expert’. It provides a convenient interface to browse beers via data from standard popular websites, and according to your description of your own palate. Barly’s most interesting feature is the ability to take a picture of a beer menu and have it automatically detect what beers are listed, followed by downloading information about them to help you choose. That kind of image analysis has to be tricky, but actually didn’t perform badly when I tested it.

The interface to this functionality is quite nice, switching to the Android camera app to get the image before uploading it to a server for processing (during which you can input your preferences). This functionality is possible with Pyjnius as mentioned previously, but actually in this case is an API also exposed in pure Python by Plyer (another Kivy sister project, wrapping platform-specific APIs in a Python frontend). Not all APIs can be conveniently exposed this way, and actually Barly may not even be using this particular method, but it’s a good example of functionality that can be achieved with Kivy in a particularly cross-platform way.

Beyond this, Barly does not make such wide use of the Android API or unusual Kivy features, but nor does it try to; it is a nice example of a complete and self-contained Kivy app using the power of Python for an unusual and interesting goal.

Kivy Android app showcase

A natural question when people hear about Kivy as a way to create Android apps in Python is…what can you do with it? Is it performant enough for games, can you call the Android APIs, do all apps look the same? One of the best resources for these kinds of question are existing apps, and in this post I’ll give a quick impression of three of my favourites. This is obviously highly subjective, but I’m focusing in particular on features of technical interest, apps that push Kivy beyond what’s normal to show what it is capable of.

If you’re interested in other examples, there’s a fairly extensive (but far from exhaustive) list on the Kivy wiki, including winners of our programming contests and many contributions from users. If you’d like to make your own apps in Python, check out Kivy (which also runs on Windows, Linux, OS X and iOS) and python-for-android (which can also package non-Kivy Python apps).

Boardz

You can download Boardz here.

I’ve put Boardz first because it’s my single favourite Kivy app. It’s actually a work in progress (and in fact hasn’t been updated for a while), but is already a fun game showcasing some of Kivy’s more impressive performance potential.

Boardz homescreen and gameplay

Boardz homescreen (left) and gameplay (right). The black ring on the right is the input circle controlling rider posture.

Boardz is a snowboarding physics game; you control your snowboarder by touching the screen, then moving your finger with respect to its initial position to control your posture; quick movements throw your weight around and can cause you to jump, spin, or fall over, while just positioning the rider differently helps you to pick up speed or navigate barriers. The objective of the game is to get to the end of each stage, with different obstacles including slopes and jumps, collapsing structures, falling rocks, or even multidirectional gravity and rocket boosters. You can fail if your head collides with another object with too much force, or if you simply get stuck and can no longer reach the finish.

What’s immediately impressive is that all this runs well as a Python powered game running on a smartphone. It achieves this by being built using the KivEnt game engine developed by Jacob Kovac, one of Kivy’s core developers. This entity based system lets you write game code in Python but internally is highly optimised in Cython, using Kivy’s OpenGL API extremely efficently as well as interfacing with the popular Chipmunk Physics engine.

Boardz wipeout failure and ad example

Boardz wipeout failure by fatal collision (left), and an ad (right).

Boardz betrays its in-progress nature in other ways; you can see in the above screenshots that its UI isn’t very polished, and in this sense it’s the worst of the apps I’m showing here. However, it makes up for this with its surprisingly engaging gameplay, and a breadth of entertaining features not showcased here, including leader boards, racing your ghost, and different riders with different physics attributes.

A final technical feature interesting to Kivy app developers is that Boardz includes ad integration. Regardless of your feelings about ads themselves, the ability to use them is a major feature enquiry from new Kivy users. The problem here is that integrating with a normal ad provider normally requires adding to the Java components of your app, which it may not be immediately obvious how to do from Python. There are actually a number of resources for this nowadays, with a key point being that python-for-android tries to make it easy to include extra Java code, with which you can interact from Python using Pyjnius. KivEnt’s implementation, pictured above, is a nice demonstration.

Kognitivo

You can download Kognitivo here.

Kognitivo is perhaps the single most polished Kivy app on the Play store, being relatively complex, extensively customised, and exhibiting a number of nice Android API interactions. It is also (deservedly) possibly the most popular Kivy app on Google Play.

Kognitivo introduction and main screens

Kognitivo tutorial (left), game instructions (centre) and homescreen (right).

Kognitivo is a brain training and performance monitoring app. The basic structure is to perform a series of simple exercises intended to test different aspects of cognitive performance, being rated on accuracy and speed, and with the results compiled over time in order to detect and act on trends.

As I’ve said already, the nice thing about Kognitivo is its huge amount of polish. It is extensively themed such that no trace of the Kivy defaults remains, runs extremly smoothly, and includes many nice animation tweaks (unfortunately not captured in screenshots) to feel responsive and active.

Kognitivo gameplay, Android notification and in-app purchase

Kognitivo training game (left), Android notification (centre) and in-app purchase option (right).

On the technical side, Kognitivo exhibits a number of features not normally included in Kivy applications but possible through interaction with the Android API via some Java code and/or the aforementioned Pyjnius. These include notifications, interaction with your calendar, and in-app purchases.

The author of Kognitivo, Sergey Cheparev, has his own writeup of Kognitivo’s development on his blog, including discussion of the advantages and disadvantages of Kivy development, and of the experience of putting together all these features. This is a great resource on its own; I don’t agree with some of the author’s criticisms and some of Kivy’s features have been improved since then, but it’s an excellent overview of the experience. Most of all, he enthusiastically captures some of my own reasons for finding Python on Android interesting:

I think the most beautiful thing in it is to use the almightiness of [Python]’s frameworks. I used sqlalchemy and sqlite as a backend, and it worked like a charm! Python is the most powerfull language because of it’s frameworks, you can even start Django on your smartphone! It’s amazing! Or twisted for asynchro communication with server. Or nltk for in-app natural language processing. Or maybe you want make a mobile equations solver with scipy and numpy. This makes all the dreams come true.

Barly

You can download Barly here, or visit its own website.

Barly is the most recent of these apps to appear on Google Play. I’ve chosen to include it as a nice example of pulling off its concept quite well while making good use of Kivy; like Kognitivo the app is themed very differently to Kivy’s defaults (though it doesn’t look like a normal Android app either), and is generally well put together.

Barly homescreen, palate options, and beer example

Barly homescreen (left), palate options (centre) and a beer search result (right).

To quote its Google Play blurb, Barly is ‘your personal beer expert’. It provides a convenient interface to browse beers via data from standard popular websites, and according to your description of your own palate. Barly’s most interesting feature is the ability to take a picture of a beer menu and have it automatically detect what beers are listed, followed by downloading information about them to help you choose. That kind of image analysis has to be tricky, but actually didn’t perform badly when I tested it.

The interface to this functionality is quite nice, switching to the Android camera app to get the image before uploading it to a server for processing (during which you can input your preferences). This functionality is possible with Pyjnius as mentioned previously, but actually in this case is an API also exposed in pure Python by Plyer (another Kivy sister project, wrapping platform-specific APIs in a Python frontend). Not all APIs can be conveniently exposed this way, and actually Barly may not even be using this particular method, but it’s a good example of functionality that can be achieved with Kivy in a particularly cross-platform way.

Beyond this, Barly does not make such wide use of the Android API or unusual Kivy features, but nor does it try to; it is a nice example of a complete and self-contained Kivy app using the power of Python for an unusual and interesting goal.

python-for-android now supports Python 3 APKs

It’s been a long time coming, but we can finally make the announcement… python-for-android now supports Python 3 Android apps! This naturally includes Kivy, but also should work for anything else you can package with python-for-android, such as apps made with PySDL2. Using Python 3 remains experimental for now, it works but doesn’t yet perform all possible optimisations and hasn’t been as widely tested as Python 2. However, there should be no extra application requirements (beyond actually being written for Python 3), and the remaining issues and optimisations are being worked on.

Overview

python-for-android is a packaging tool for turning Python applications into Android APKs. It was originally created to make apps with the very cross-platform Kivy graphical framework (though it didn’t arise in a vacuum, I think it built in particular on previous work by the Ren’Py project). However, the original python-for-android had flaws including being quite inconvenient to modify for non-Kivy apps (several other projects seem to have used modified versions, but each was performing similar changes), hard to extend for multiple architecture support, hard to extend internally (both in general and from the perspective of new contributors, as much of the toolchain was a big shell script), and only supported building apps with Python 2.

We recently completed and released a fully revamped version of python-for-android aimed at fixing all of these problems, as discussed in several previous posts on this blog (originally here). Almost all of the original goals are now complete, with Python 3 support the major missing one until now, though not for lack of trying. Some technical details and basic instructions for testing the new support are given below, and you can also see the online documentation for further information.

Our Python 3 support depends on the prebuilt Python distributions provided with the CrystaX NDK, a drop-in replacement for Google’s own Android NDK with many fixes and improvements. The technical details of this choice are given below, and we’ll try to further support a locally-built Python 3 option in the future. Thanks to the CrystaX team for making it so (relatively) easy!

Technical details

python-for-android works by bundling a Python interpreter, compiled for Android devices and architectures (usually arm, though other choices are supported), into an Android APK. The APK includes a simple Java bootstrap application, which mostly starts a Python script via JNI. The script then runs essentially as normal, almost all of the Python standard library is present and works fine, and python-for-android supports including other modules or non-Python dependencies. Pure Python modules will mostly work without special treatment, though things requiring compilation need a special recipe. Many common modules such as numpy and sqlalchemy are supported this way. Following its revamp python-for-android is now designed to support multiple kinds of java bootstrap, but the current main support is for GUI apps via Pygame (for Kivy’s old Android support) or SDL2 (both for Kivy and for anything else that can use it); SDL2 also now does much of the heavy lifting of handling events etc. itself, via its own Android support.

The main problems with compiling and including Python are first that it must be patched to compile (as Android’s libc doesn’t support some things very well or at all), and second that it must be unpacked and started on the device via its C API. The second point is fiddly but ultimately not that different to working this way on the desktop. The first is (in my opinion) harder because it needs some understanding of Python’s internals, of Android’s limitations, of appropriate fixes, and of how to test and debug these problems.

Such patches have been made by a number of different people for different Android versions, and I believe there has been activity on Python itself to fix some issues (including this current issue to make Python build natively on Android). For Python 2, I think python-for-android’s original patches came from here, though extended with further modifications. However, the main thing holding up my efforts to get Python 3 working was the inability to find a similar working patchset; I tried a few sources, achieving a working compilation using SL4A’s python3 patches, but I couldn’t get Python working on the device. I’m sure this was my own technical mistakes, since other projects do have it working, but it’s what was holding up the feature.

I eventually resolved this by using the new Python on Android support from the CrystaX NDK. As mentioned above, this is a drop-in replacement for Google’s own NDK (the Native Development Kit providing compilers etc for targeting Android with non-Java code), including many improvements to the build environment. As of version 10.3.0, they provide prebuilt Python packages for Android on all architectures - and all of their NDK improvements mean that Python no longer even needs patching for this compilation to work. Python is provided as a zipped standard library (Python can automatically load modules from zip files), and a folder of the compiled components like ctypes (as it’s hard to dynamically load from zips). From the perspective of python-for-android, supporting Python 3 means modifying the build to load CrystaX’s prebuilt components (both in the Android project structure and in python-for-android’s support for compiling other modules), and modifying the C initialisation code for Python 3. This takes some work, but all told wasn’t very hard and the Python bundles worked with no issues, so we owe a lot to CrystaX; thanks again.

I’d still like to come back to the issue of local python3 compilation; CrystaX’s versions are fine, but I’ve learned a lot from making them work, and have a much better idea of what I may have been doing wrong. However, the focus for now will be on resolving the remaining issues with what’s already working.

Building apps with Python 3

Building Python 3 APKs is only supported in the revamped python-for-android toolchain which was merged to the master branch a while ago. It can be installed and used as described in its documentation. If you use Buildozer, it currently does not support this new toolchain, though tito has been working on this. There is also the new restriction that you must (for obvious reasons) use the CrystaX NDK, which can be obtained here; simply refer to its filepath when setting the NDK directory, and everything else should work automatically.

To build for Python 3, add the python3crystax recipe to the requirements option, e.g. --requirements=python3crystax,kivy. It should mostly work automatically with existing recipes, though at this stage there may be bugs or problems with a few, and some will need modification. The exact syntax above may also change in the future as the Python 3 support becomes better integrated, but not significantly.

There’s also one big change whose importance I’m not sure about; the Python 3 mechanism doesn’t currently build a local Python 3 to use as a hostpython, instead using the system python. This means that you must have python3.5 (3.4 may also work) installed locally in order for python-for-android to build Python 3 APKs. This will be fixed soon, adding a hostpython3 recipe to avoid weird bugs with system-specific differences, but you need to bear it in mind for now.

Future work

For now, this Python 3 support remains experimental. I anticipate no major issues, but it’s internally a quite different method to the Python 2 support and needs further work to duplicate some of the old optimisations, and undoubtedly to fix bugs in the toolchain that will appear as it stabilises. Amongst other things, Python 3 shared libraries are not currently collected and merged (with Python 2 we did this originally to get around an Android limit but also for optimisation), python files are not precompiled to bytecode (it can make a big loading speed difference), and some features of the old pygame bootstrap have not yet been implemented in SDL2. All this and more will come in the future, but shouldn’t be hard to add now that the toolchain all works.

The SDL2 bootstrap is also missing a few features that users of the old toolchain will be used to, like the splash screen image and at least one Kivy-specific function. These too are being actively worked on, especially as more people start to move their apps to SDL2.

I’ve also phrased this as Python 2 built locally vs Python 3 from CrystaX, but actually CrystaX also supports Python 2.7 and I hope to add this option in the near future. As discussed in the technical details, it also should absolutely be possible to have a local Python 3 build, which I’d like to eventually come back to.

python-for-android now supports Python 3 APKs

It’s been a long time coming, but we can finally make the announcement… python-for-android now supports Python 3 Android apps! This naturally includes Kivy, but also should work for anything else you can package with python-for-android, such as apps made with PySDL2. Using Python 3 remains experimental for now, it works but doesn’t yet perform all possible optimisations and hasn’t been as widely tested as Python 2. However, there should be no extra application requirements (beyond actually being written for Python 3), and the remaining issues and optimisations are being worked on.

Overview

python-for-android is a packaging tool for turning Python applications into Android APKs. It was originally created to make apps with the very cross-platform Kivy graphical framework (though it didn’t arise in a vacuum, I think it built in particular on previous work by the Ren’Py project). However, the original python-for-android had flaws including being quite inconvenient to modify for non-Kivy apps (several other projects seem to have used modified versions, but each was performing similar changes), hard to extend for multiple architecture support, hard to extend internally (both in general and from the perspective of new contributors, as much of the toolchain was a big shell script), and only supported building apps with Python 2.

We recently completed and released a fully revamped version of python-for-android aimed at fixing all of these problems, as discussed in several previous posts on this blog (originally here). Almost all of the original goals are now complete, with Python 3 support the major missing one until now, though not for lack of trying. Some technical details and basic instructions for testing the new support are given below, and you can also see the online documentation for further information.

Our Python 3 support depends on the prebuilt Python distributions provided with the CrystaX NDK, a drop-in replacement for Google’s own Android NDK with many fixes and improvements. The technical details of this choice are given below, and we’ll try to further support a locally-built Python 3 option in the future. Thanks to the CrystaX team for making it so (relatively) easy!

Technical details

python-for-android works by bundling a Python interpreter, compiled for Android devices and architectures (usually arm, though other choices are supported), into an Android APK. The APK includes a simple Java bootstrap application, which mostly starts a Python script via JNI. The script then runs essentially as normal, almost all of the Python standard library is present and works fine, and python-for-android supports including other modules or non-Python dependencies. Pure Python modules will mostly work without special treatment, though things requiring compilation need a special recipe. Many common modules such as numpy and sqlalchemy are supported this way. Following its revamp python-for-android is now designed to support multiple kinds of java bootstrap, but the current main support is for GUI apps via Pygame (for Kivy’s old Android support) or SDL2 (both for Kivy and for anything else that can use it); SDL2 also now does much of the heavy lifting of handling events etc. itself, via its own Android support.

The main problems with compiling and including Python are first that it must be patched to compile (as Android’s libc doesn’t support some things very well or at all), and second that it must be unpacked and started on the device via its C API. The second point is fiddly but ultimately not that different to working this way on the desktop. The first is (in my opinion) harder because it needs some understanding of Python’s internals, of Android’s limitations, of appropriate fixes, and of how to test and debug these problems.

Such patches have been made by a number of different people for different Android versions, and I believe there has been activity on Python itself to fix some issues (including this current issue to make Python build natively on Android). For Python 2, I think python-for-android’s original patches came from here, though extended with further modifications. However, the main thing holding up my efforts to get Python 3 working was the inability to find a similar working patchset; I tried a few sources, achieving a working compilation using SL4A’s python3 patches, but I couldn’t get Python working on the device. I’m sure this was my own technical mistakes, since other projects do have it working, but it’s what was holding up the feature.

I eventually resolved this by using the new Python on Android support from the CrystaX NDK. As mentioned above, this is a drop-in replacement for Google’s own NDK (the Native Development Kit providing compilers etc for targeting Android with non-Java code), including many improvements to the build environment. As of version 10.3.0, they provide prebuilt Python packages for Android on all architectures - and all of their NDK improvements mean that Python no longer even needs patching for this compilation to work. Python is provided as a zipped standard library (Python can automatically load modules from zip files), and a folder of the compiled components like ctypes (as it’s hard to dynamically load from zips). From the perspective of python-for-android, supporting Python 3 means modifying the build to load CrystaX’s prebuilt components (both in the Android project structure and in python-for-android’s support for compiling other modules), and modifying the C initialisation code for Python 3. This takes some work, but all told wasn’t very hard and the Python bundles worked with no issues, so we owe a lot to CrystaX; thanks again.

I’d still like to come back to the issue of local python3 compilation; CrystaX’s versions are fine, but I’ve learned a lot from making them work, and have a much better idea of what I may have been doing wrong. However, the focus for now will be on resolving the remaining issues with what’s already working.

Building apps with Python 3

Building Python 3 APKs is only supported in the revamped python-for-android toolchain which was merged to the master branch a while ago. It can be installed and used as described in its documentation. If you use Buildozer, it currently does not support this new toolchain, though tito has been working on this. There is also the new restriction that you must (for obvious reasons) use the CrystaX NDK, which can be obtained here; simply refer to its filepath when setting the NDK directory, and everything else should work automatically.

To build for Python 3, add the python3crystax recipe to the requirements option, e.g. --requirements=python3crystax,kivy. It should mostly work automatically with existing recipes, though at this stage there may be bugs or problems with a few, and some will need modification. The exact syntax above may also change in the future as the Python 3 support becomes better integrated, but not significantly.

There’s also one big change whose importance I’m not sure about; the Python 3 mechanism doesn’t currently build a local Python 3 to use as a hostpython, instead using the system python. This means that you must have python3.5 (3.4 may also work) installed locally in order for python-for-android to build Python 3 APKs. This will be fixed soon, adding a hostpython3 recipe to avoid weird bugs with system-specific differences, but you need to bear it in mind for now.

Future work

For now, this Python 3 support remains experimental. I anticipate no major issues, but it’s internally a quite different method to the Python 2 support and needs further work to duplicate some of the old optimisations, and undoubtedly to fix bugs in the toolchain that will appear as it stabilises. Amongst other things, Python 3 shared libraries are not currently collected and merged (with Python 2 we did this originally to get around an Android limit but also for optimisation), python files are not precompiled to bytecode (it can make a big loading speed difference), and some features of the old pygame bootstrap have not yet been implemented in SDL2. All this and more will come in the future, but shouldn’t be hard to add now that the toolchain all works.

The SDL2 bootstrap is also missing a few features that users of the old toolchain will be used to, like the splash screen image and at least one Kivy-specific function. These too are being actively worked on, especially as more people start to move their apps to SDL2.

I’ve also phrased this as Python 2 built locally vs Python 3 from CrystaX, but actually CrystaX also supports Python 2.7 and I hope to add this option in the near future. As discussed in the technical details, it also should absolutely be possible to have a local Python 3 build, which I’d like to eventually come back to.

python-for-android status update

It’s been a while since Kivy’s python-for-android project was revamped, so here’s a quick status update.

There have since been well over 200 commits from 15 different contributors, cleaning up the missing pieces of the new toolchain and adding new features that weren’t previously possible. Thanks to everyone who has contributed.

These fixes include progress on the remaining major goals of the python-for-android revamp. In particular, compilation is now supported for multiple target architectures - in principle anything targeted by the Android SDK (i.e. ARM, ARMv7a, x86, x86_64 and MIPS options), though I’ve tested only with the ARM and x86 ones. This means that Kivy applications, or other Python projects built with these tools, can be built for devices with e.g. intel atom processors. Even without this compilation it was often possible to run Kivy apps as many devices include libhoudini, but directly targeting them means such apps should now always work. A further advantage is that Kivy apps can be built for and tested on the Android emulator, which was not previously possible.

The architecture target support does need some more work to create fully multiarch APKs (i.e. including .so files for each target, so a single APK can work on different types of device). The problem here is that we need to duplicate as little as possible, as Kivy APKs are already made large by including the python interpreter, and it is undesirable to include two or more copies of everything. Using a single python installation and loading the .so dependencies as appropriate should be possible but needs more work. However, this is not a problem if uploading an APK to a store like Google Play; in this case you can include multiple APKs, one for each arch target, and the user will receive one that is appropriate.

Another important feature that I’ve worked on, but unfortunately unsuccessfully so far, is support for python3 APKs. The problem to be solved is to patch the interpreter to compile for Android (it cannot do so by default, due to problems with the Android platform like poor locale support), to modify the python-for-android bootstrap to load it correctly (it builds things a little differently to python2), and to modify the initialisation code to have it start successfully. I’ve only partially succeeded with the first two of these; using patches from the SL4A python-for-android tools (plus extras for our own modifications to python loading) allows the interpreter to be built, but it fails during Py_Initialize when run on the device, apparently raising an exception when calculating the python install path. Work on this will continue, but it’s hard to know how long it might take to resolve this error. If you know of any other projects patching python3.4+ for Android, I’d love to heard about it to compare their methods.

python-for-android status update

It’s been a while since Kivy’s python-for-android project was revamped, so here’s a quick status update.

There have since been well over 200 commits from 15 different contributors, cleaning up the missing pieces of the new toolchain and adding new features that weren’t previously possible. Thanks to everyone who has contributed.

These fixes include progress on the remaining major goals of the python-for-android revamp. In particular, compilation is now supported for multiple target architectures - in principle anything targeted by the Android SDK (i.e. ARM, ARMv7a, x86, x86_64 and MIPS options), though I’ve tested only with the ARM and x86 ones. This means that Kivy applications, or other Python projects built with these tools, can be built for devices with e.g. intel atom processors. Even without this compilation it was often possible to run Kivy apps as many devices include libhoudini, but directly targeting them means such apps should now always work. A further advantage is that Kivy apps can be built for and tested on the Android emulator, which was not previously possible.

The architecture target support does need some more work to create fully multiarch APKs (i.e. including .so files for each target, so a single APK can work on different types of device). The problem here is that we need to duplicate as little as possible, as Kivy APKs are already made large by including the python interpreter, and it is undesirable to include two or more copies of everything. Using a single python installation and loading the .so dependencies as appropriate should be possible but needs more work. However, this is not a problem if uploading an APK to a store like Google Play; in this case you can include multiple APKs, one for each arch target, and the user will receive one that is appropriate.

Another important feature that I’ve worked on, but unfortunately unsuccessfully so far, is support for python3 APKs. The problem to be solved is to patch the interpreter to compile for Android (it cannot do so by default, due to problems with the Android platform like poor locale support), to modify the python-for-android bootstrap to load it correctly (it builds things a little differently to python2), and to modify the initialisation code to have it start successfully. I’ve only partially succeeded with the first two of these; using patches from the SL4A python-for-android tools (plus extras for our own modifications to python loading) allows the interpreter to be built, but it fails during Py_Initialize when run on the device, apparently raising an exception when calculating the python install path. Work on this will continue, but it’s hard to know how long it might take to resolve this error. If you know of any other projects patching python3.4+ for Android, I’d love to heard about it to compare their methods.

python-for-android status update

It’s been a while since Kivy’s python-for-android project was revamped, so here’s a quick status update.

There have since been well over 200 commits from 15 different contributors, cleaning up the missing pieces of the new toolchain and adding new features that weren’t previously possible. Thanks to everyone who has contributed.

These fixes include progress on the remaining major goals of the python-for-android revamp. In particular, compilation is now supported for multiple target architectures - in principle anything targeted by the Android SDK (i.e. ARM, ARMv7a, x86, x86_64 and MIPS options), though I’ve tested only with the ARM and x86 ones. This means that Kivy applications, or other Python projects built with these tools, can be built for devices with e.g. intel atom processors. Even without this compilation it was often possible to run Kivy apps as many devices include libhoudini, but directly targeting them means such apps should now always work. A further advantage is that Kivy apps can be built for and tested on the Android emulator, which was not previously possible.

The architecture target support does need some more work to create fully multiarch APKs (i.e. including .so files for each target, so a single APK can work on different types of device). The problem here is that we need to duplicate as little as possible, as Kivy APKs are already made large by including the python interpreter, and it is undesirable to include two or more copies of everything. Using a single python installation and loading the .so dependencies as appropriate should be possible but needs more work. However, this is not a problem if uploading an APK to a store like Google Play; in this case you can include multiple APKs, one for each arch target, and the user will receive one that is appropriate.

Another important feature that I’ve worked on, but unfortunately unsuccessfully so far, is support for python3 APKs. The problem to be solved is to patch the interpreter to compile for Android (it cannot do so by default, due to problems with the Android platform like poor locale support), to modify the python-for-android bootstrap to load it correctly (it builds things a little differently to python2), and to modify the initialisation code to have it start successfully. I’ve only partially succeeded with the first two of these; using patches from the SL4A python-for-android tools (plus extras for our own modifications to python loading) allows the interpreter to be built, but it fails during Py_Initialize when run on the device, apparently raising an exception when calculating the python install path. Work on this will continue, but it’s hard to know how long it might take to resolve this error. If you know of any other projects patching python3.4+ for Android, I’d love to heard about it to compare their methods.