python-for-android 2019.08.09 released: running under Python 2 no longer supported

python-for-android is a packaging tool for Python apps on Android. You can create your own Python distribution including the modules and dependencies you want, and bundle it in an APK along with your own code.


python-for-android 2019.08.09 has just been released! I haven’t been making a blog post for every new version now that we’ve switched to a monthly releases, but this one includes an especially major change: python-for-android no longer runs under Python 2. If this affects you, simply run python-for-android (or buildozer) under Python 3.6 or newer, they should run exactly the same as ever.

Note that you can still target Python 2 on Android, simply add python2 to your requirements as you would normally. However, this won’t last forever, support for Python 2 will probably be totally removed early in 2020.

Other changes in this release include further testing improvements, significantly improved documentation on developing with python-for-android, and some preparations for further big changes in the future. See the release note for full details.

An update on python-for-android: v2019.06.06 released and future plans

python-for-android is a packaging tool for Python apps on Android. You can create your own Python distribution including the modules and dependencies you want, and bundle it in an APK along with your own code.


python-for-android 2019.06.06 has just been released! This release contains 198 commits from 31 different contributors. Many thanks to everyone involved.

Major changes in this release include:

  • Added support for running your setup.py when packaging your app, enabling your code to be installed as a module in the on-device Python environment. This also makes it easy to build Cython or other compiled components.
  • Added some tooling for requesting and checking for app permissions at runtime.
  • Added initial support for matplotlib.
  • Updated many recipes, and especially the SDL2 backend components, for improved performance and stability on Android.
  • Ongoing improvements to our test coverage and infrastructure.
  • Removed a significant amount of dead code relating to long-deprecated build configurations, and unified other parts of the build to reduce duplication of effort between bootstraps.
  • Updated the release model to target regular, smaller releases.

Of course there have also been a wide range of bugfixes and improvements. See the release notes for a full changelog.

Release model

In this release we’ve transitioned to a calendar-based version system. Future releases will continue to be of the form YYYY.MM.DD. We’re initially targeting one release every four weeks. This scheme represents how python-for-android is normally best used: many changes are driven by updates in the Android ecosystem and build toolchain, and in practice it’s usually best to be working from the most recent possible python-for-android release.

This has been made possible by the hard work of various contributors, setting up and continuing to improve python-for-android’s suite of tests and continuous integration services. In the past we haven’t done a great job of keeping up releases alongside major improvements, but this should now be much more straightforward.

If you’re using buildozer then this doesn’t directly affect you, as buildozer was already configured to use a more recent python-for-android version. Buildozer will now automatically transition to use the latest release, represented by the master branch in the python-for-android git repository.

Future work

We’ve had some questions about Google’s plan to require 64-bit support for apps on Google Play, starting in August 2019. In fact python-for-android already supports this, just pass the required architecture as an argument to the build:

p4a apk --arch=arm64-v8a

That said, we’re working to improve our testing and documentation around 64-bit builds, to make sure everything works as expected.

We also don’t currently support multi-architecture builds in a single output APK. This should be technically possible, but hasn’t ever been a focus because it would significantly increase the APK size. We may revisit this, but in the meantime you can upload one APK of each type to Google Play and python-for-android will automatically handle versioning different architectures correctly (i.e. arm64-v8a is an ‘upgrade’ for devices that support it, so that APK will be preferred).

The other current focus is on improving our test infrastructure, especially increasing test coverage and automation. This should further increase the ease of making regular releases, and our confidence that everything continues to work correctly!

These points are just general goals for python-for-android. There is always other maintenance work to do, and contributions of all types are always welcome.

Running buildozer in a virtual machine

This guide describes how to turn your Kivy/Python app into an APK, by running the buildozer build tool in a virtual machine. This is not the only way to run buildozer, it can work natively on Linux or MacOS or be run from the Windows Subsystem for Linux. See the Kivy documentation for more general instructions.

Creating a Virtual Machine

I’ll be using VirtualBox. Other virtualisation software should also work, but you’ll need to adapt the specific instructions.

We also need a target OS. I recommend Lubuntu 18.04, available here (or direct download link). Lubuntu is a light weight Ubuntu variant. You can also use a different distro if you like, but may need to adapt the later instructions.

Once you have downloaded the Lubuntu iso file, start VirtualBox and press New to create a new virtual machine. You’ll see a dialog like the following:

New VM dialog

Fill in the other options as shown in the image. It’s fine to set a larger memory size if you have enough available, or less may also work fine. Then press Create to continue.

Disk image creation dialog

You now need to select a file size for your virtual hard disk. 15 GB should be sufficient, but it’s safest to double that. Leave the other options unchanged and choose Create to continue.

Now, select your new VM and click Start in the main VirtualBox GUI. You should be prompted to select a virtual optical disk to boot from:

Choose iso to boot

Navigate to your Lubuntu 18.04 iso downloaded earlier, as shown, and press Start to continue. The first screen you see should look something like the following:

Select language

Select your language to see the boot menu:

Choose boot option

Choose the second option, "Install Lubuntu". It doesn’t matter if you accidentally press enter to "Try Lubuntu without installing", in this case there should be an Install Lubuntu application on the desktop that you can click to continue the install process.

You’ll be shown a series of dialogs to help prepare the install process. Clicking through with the defaults is fine, or select other options if you prefer.

The fourth screen will ask what kind of install to use, as shown:

Choose install type

The options shown above should be the defaults, and are what you want to use, so go ahead and continue.

Choose partitioning options

Next, select "Erase disk and install Lubuntu". Note that this is only erasing the (emtpy) virtual disk image created earlier, it won’t affect your host operating system.

Click through again, and you’ll eventually reach the user creation screen. It doesn’t matter what your username is, I used kivyuser:

User creation dialog

Click "Continue" to finally start the install. You’ll be asked a few more questions, but eventually will just have to wait for the installation to complete. This shouldn’t take too long. You’ll be prompted to "Restart Now", which you should go ahead and do.

Restart Now screen

If you have any issues with the virtual machine failing to reboot, go ahead and select Machine > Reset in the VirtualBox menu, it doesn’t matter how you do it as long as the machine is reset. If all goes well, Lubuntu should now automatically boot to a login screen - congratulations, your virtual machine is ready to use! Enter your username and password, and proceed to the next section of instructions.

Login screen

Setting up buildozer

We can now go ahead and set up buildozer ready to build your app. Open an LXTerminal as below:

LXTerminal location in menu

We now have to run a few commands to install everything buildozer needs to run. Run the following command to do so, and enter your user’s password if prompted:

sudo apt-get install python3-pip openjdk-8-jdk autoconf libtool python3-venv

That should give us everything we need for a basic app, so we can go ahead and install buildozer:

.. code-block:: sh
python3 -m venv buildozer-env source buildozer-env/bin/activate pip install buildozer cython

Note that we installed cython as well, this is also required for building the APK.

You only have to create the virtual environment once, but if you reboot the virtual machine you’ll need to run source buildozer-env/bin/activate again. See the Python documentation for more details.

The final step before running buildozer is to have your app ready in the virtual machine. You can access a folder in your host machine using VirtualBox shared folders (in the Devices > Shared Folders menu), but I won’t cover the details here. Note though that if you do this you must copy the folder contents to a different folder within the virtual machine, the buidozer process will not work if run within a shared folder.

In the following instructions I’ll assume you’ve created a folder named app_dir and placed a main.py file inside it containing your application code. Navigate to this folder in the terminal (cd app_dir) and run:

.. code-block:: sh
buildozer init

This will create a buildozer.spec file alongside your main.py:

Creating buildozer.spec

Edit the buildozer.spec to set any options you like. In this example I’ve changed only the title and pacakge.name options:

Editing buildozer.spec

I recommend changing very little for this first build, to make sure everything works. It won’t cause any problems if you edit the buildozer.spec again later.

Running buildozer

We’re now ready to actually build the app into an APK file. Start the process with:

.. code-block:: sh
buildozer -v android debug

The -v option asks for verbose output. This is recommended so that you can keep an eye on what’s happening - the details aren’t too important, but you should be able to see that the process never stops in one place for too long.

Buildozer will now download the Android tools it needs. This may take a while.

At some point you’ll be asked to accept the Android SDK license agreement, which is printed for you as in the following image:

SDK license agreement

At this point, press "y" and then enter to accept the agreement (or abort the process if you don’t agree). This is necessary even if you don’t see any text asking you to do so, due to a bug in buildozer (fixed in the next release).

After downloading everything it needs, buildozer will work through the build process compiling and packaging each of the components for your app. This may take a while, but as long as it doesn’t crash then everything is fine. Future builds will be much faster unless you change the build options, as only the contents of your app itself will need re-packaging.

Eventually the build will complete, you’ll see a screen like the following:

Build complete

That’s it, you’re done! You can find the finished APK in the bin directory, as noted in the final message buildozer prints.

Kivy Hackathon and FOSDEM 2019

For the last 3 days several of the Kivy Core Developers gathered in Brussels, Belgium for the first ever core developer hackathon. Not only is this the first time we’ve gathered to work on Kivy framework issues, but for most of us the first time we’ve ever met - even after years of collaboration!

From left to right: Peter Badida (KeyWeeUsr), Gabriel Pettier (tshirtman), Mathieu Virbel (tito), Alexander Taylor (inclement) and Andre Miras (AndreMiras)

From left to right the attendees are Peter Badida (KeyWeeUsr), Gabriel Pettier (tshirtman), Mathieu Virbel (tito), Alexander Taylor (inclement) and Andre Miras (AndreMiras).

We used this opportunity to make amazing progress on a range of Kivy issues. Most especially:

  • 55 closed issues and 36 merged pull requests across all Kivy projects, specifically targeting long standing bugs and the most important current issues.
  • A full update of buildozer to work well with the latest SDK, NDK and other dependencies for Android development. New release to follow!
  • Rapid progress on ncis, a new remote inspector/debugger for all types of Python application.
  • Many improvements to the python-for-android continuous integration, and some major bugfixes for stability with Python 3 and the latest toolchain. Preliminary 0.7.0 release already available via PyPI and buildozer, more to follow.
  • Extensive cleanup and improvements for PyJNIus.
  • A further ~170 closed python-for-android issues and ~110 buildozer issues, as a general clean up of stale and outdated reports to focus on the ones that really matter.

For the next two days, the same team will be attending FOSDEM 2019. Let us know by discord or other communication if you’d like to say hello!

Cached and templated files in python-for-android builds

I’ve more than once seen people confused by how python-for-android constructs an Android project that can be compiled into an APK. Since p4a uses various cached and templated files, it’s easy to get confused trying to edit things only to find your changes are overwritten when you run p4a again.

Here’s a quick summary of what’s what.

Bootstraps

The core project files are all in the bootstraps directory. A bootstrap is basically an Android project, containing java sources, JNI stuff to be compiled, and project management files like the AndroidManifest.xml and build.gradle.

Note: Some of these files are built from templates, e.g. in the main SDL2 bootstrap you can find build.tmpl.gradle, AndroidManifest.tmpl.xml etc. in the templates subdirectory.

If you edit the code in these bootstrap directories, the changes will always take effect if you rebuild your whole project, but might not affect anything if you repeat a previous build. This is due to caching of previous build components. You can always guarantee that everything is cleared to be rebuilt using p4a clean dists builds (or there are other tricks if you look into it).

Build directories

Most of the individual python-for-android recipes are built in separate build directories. The location of these depends on the OS: on Linux the default is ~/.local/share/python-for-android, especially ~/.local/share/python-for-android/other_builds. When you build a recipe, it is copied here for the build to be run. These folders also cache builds, so if you e.g. build two different projects using python3 then the same build directory is used both times, reducing time spent compiling.

You can edit code in ~/.local/share/python-for-android/other_builds, but it isn’t recommended unless you have a good idea what python-for-android will overwrite or cache on each run. It is, however, sometimes useful during development.

Dists

Dists are p4a’s fully compiled Android projects, including the bundled output of all the different requirements specified by the user. On Linux, their default location is ~/.local/share/python-for-android/dists.

Every time you build an APK using p4a apk ..., this essentially runs gradle from an appropriate dist directory (i.e. the one with the recipes you wanted). At this point, nothing new is built or copied from the build directories, so changes you make to p4a’s recipes or build directories have no effect on the output. However, templated files such as AndroidManifest.tmpl.xml are rebuilt every time. That means that you must edit the templates themselves if you want your changes to make it to the APK.

You can always run p4a clean dists to delete the existing Android projects. Next time you run p4a apk ..., a new dist will be created.

Buildozer

When using buildozer, everything works basically the same except that the build and dist directories can be found in $PROJECT_DIR/.buildozer/android/platform/build instead of ~/.local/share/python-for-android. The python-for-android source code is stored in $PROJECT_DIR/.buildozer/android/platform/python-for-android.

python-for-android 0.6 released

We’ve just officially released python-for-android 0.6. The new version can be downloaded via pip, or will be used by buildozer automatically in new installations. This release contains about 130 new commits from 14 different contributors. Thanks to everyone involved!

python-for-android is a packaging tool for turning Python scripts and apps into Android APKs. It was originally created for use with the Kivy graphical framework, but now supports multiple kinds of Python app including Kivy, PySDL2, a webview interface with Flask or other webserver backend, plain Python scripts without a GUI, or other possibilities such as Python builds for use in other applications.

As planned following the release of python-for-android 0.5, the new version includes some relatively major changes and improvements. In particular, python-for-android should now work with all recent versions of the Android SDK and NDK. On the SDK side this means python-for-android now uses gradle if available, although this doesn’t require any changes to the configuration on the user side.

For full instructions and further information, see the python-for-android documentation.

python-for-android 0.5 released

We’ve just officially released python-for-android 0.5. The new version can be downloaded via pip, or will be used by buildozer automatically in new installations. This release contains about 300 commits from almost 40 different contributors. Thanks to everyone involved!

python-for-android is a packaging tool for turning Python scripts and apps into Android APKs. It was originally created for use with the Kivy graphical framework, but now supports multiple kinds of Python app including Kivy, PySDL2, a webview interface with Flask or other webserver backend, plain Python scripts without a GUI, or other possibilities such as Python builds for use in other applications.

This release contains many fixes and improvements to all parts of the toolchain. Now that these have been released as stable, we intend to move quickly to make some larger improvements including supporting gradle builds, and better support for Python 3 in some recipes.

For full instructions and further information, see the python-for-android documentation.

Pyonic interpreter 1.3 released: Adds support for loading and executing files

I’ve just released Pyonic interpreter 1.3. As usual you can download it from Google Play, for Python 2.7 or Python 3.6. The APKs can also be downloaded directly from Github (where the source code is also available).

The new filebrowser interface in Pyonic interpreter.

The main addition in this release is a file browser interface, which gives Pyonic the ability to load and execute files in the interpreter. This is mildly useful on its own, and I’ve had comments that people would like to be able to do it, but it’s also groundwork for full support for file editing support. I hope to add these features in a future version.

Pyonic interpreter 1.2 released: Now supports Python 3.6 and input functions

I’ve just released Pyonic interpreter 1.2. As usual, you can get it from Google Play, now for Python 2.7 or Python 3.6. The APKs can also de bownloaded directly from Github (where the source code is also available).

Pyonic interpreter showing docstring and autocompletion options

This is the first release to target Python 3.6 on Android (not just Python 3.5), which is made possible by recent additions to python-for-android. I expect to do a separate python-for-android release to announce this shortly.

The main change to the app this release is support for the input and (in Python 2) raw_input functions. These would previously crash as the interpreter isn’t really being run in a shell, so the way they try to take input doesn’t work. They are now overridden with new replacements, which should hopefully behave roughly the same way as the originals are supposed to, but via a more convenient popup gui for the text to be entered.

I’m still working on file editing and other Python management functions, but there didn’t seem to be any reason to delay a release since according to the Google Play reviews people are trying and failing to use the input functions.

Pyonic interpreter 1.1 released: Python 2/3 interpereter for Android, now with autocompletion

I’ve just released Pyonic interpreter 1.1. As usual, you can get it from Google Play for Python 2.7 or Python 3.5, or download the APKs directly from Github (where the source code is also available).

Pyonic interpreter showing docstring and autocompletion options

The major feature in this release is autocompletion support via the excellent jedi library, as is used by many editors and IDEs. Pyonic now automatically gives a list of autocompletion options as you write Python code, any of which can be selected by tapping it. There’s also a new help button, which when pressed shows the call signature and docstring of the Python object reference currently under the cursor.

As a further bonus, I’ve reduced the size of the Python 3 APK by a further ~25%, it’s now around 11MB. This is probably still a little larger than it needs to be, but is much better than the massive 19MB version that I first published! I’ll continue to try to improve this with tweaks in python-for-android’s Python 3 build process.

For the next release, I intend to go back to improving Pyonic’s process handling, and from there to add support for pip installation of new modules and file editing. The latter of these will also benefit from the new autocompletion integration.