Pyonic interpreter 1.1 released: Python 2/3 interpereter for Android, now with autocompletion

I’ve just released Pyonic interpreter 1.1. As usual, you can get it from Google Play for Python 2.7 or Python 3.5, or download the APKs directly from Github (where the source code is also available).

Pyonic interpreter showing docstring and autocompletion options

The major feature in this release is autocompletion support via the excellent jedi library, as is used by many editors and IDEs. Pyonic now automatically gives a list of autocompletion options as you write Python code, any of which can be selected by tapping it. There’s also a new help button, which when pressed shows the call signature and docstring of the Python object reference currently under the cursor.

As a further bonus, I’ve reduced the size of the Python 3 APK by a further ~25%, it’s now around 11MB. This is probably still a little larger than it needs to be, but is much better than the massive 19MB version that I first published! I’ll continue to try to improve this with tweaks in python-for-android’s Python 3 build process.

For the next release, I intend to go back to improving Pyonic’s process handling, and from there to add support for pip installation of new modules and file editing. The latter of these will also benefit from the new autocompletion integration.

Posted in android, Kivy, pyonic, Python | Comments Off on Pyonic interpreter 1.1 released: Python 2/3 interpereter for Android, now with autocompletion

Pyonic interpreter 1.0 released

I’ve just released Pyonic interpreter 1.0. You can get it from Google Play for Python 2.7 or Python 3.5, or download the APKs directly from Github (update: changed to v1.0.1 following a small bugfix).

The primary change in this release is that both APKs are about 25% smaller than before, thanks to optimisations in the Python distributions that I’ve added to python-for-android - in particular, making sure Python files are shipped as .pyo files (which may also speed things up a bit) and stripping unneeded symbols from object files with Python 3. Both of these were things python-for-android has been missing for a while, so it’s nice to get them working and immediately see the benefits.

I’ve also been working on some backend improvements in Pyonic and python-for-android in order to support multiple interpreter processes. This will be convenient for using pip and running Python code from files, but isn’t ready yet and so hasn’t made it into this release.

Posted in android, Kivy, pyonic, Python | Comments Off on Pyonic interpreter 1.0 released

Pyonic interpreter 0.7 released: Python for Android, now supports Python 3

Pyonic interpreter 0.7 has just been released. There are now two versions on Google Play, one for Python 2.7 and one for Python 3.5. The APKs are also available directly from Github. Other features in this release include a new settings screen and improved gui arrangement.

Three screenshots of Pyonic interpreter.

The app is written in Python using Kivy, and uses exactly the same code under both Python versions. This code is open source and available online on Github.

This release includes most of the short term improvements I had planned, since supporting Python 3 didn’t raise any major issues. I expect that development will now focus on adding a few more usability tweaks, then working with the Python packaging to add features like pip installs for new modules, code editing (rather than just the interpreter interface), and support for GUI creation via Kivy.

Posted in android, Kivy, pyonic, Python | Comments Off on Pyonic interpreter 0.7 released: Python for Android, now supports Python 3

Pyonic interpreter: a Python interpreter GUI for Android, written in Python

I’ve just released a new app, Pyonic Python 2 interpreter. Pyonic interpreter is a Python interpreter app for Android, providing a convenient interface adapted to mobile devices. The app itself is written entirely in Python using Kivy.

Screenshot of the interpreter app.

I put this together because I’ve always thought it would be nice to have a Python interpreter app that is itself written in Python, and in principle Kivy and python-for-android provide all the necessary components. In practice this worked even better than I expected, Kivy handled almost everything perfectly - I actually underestimated its maturity here! As part of the project, I’ve tried to round a number of corners that Kivy apps sometimes can to have, so that the interpreter (hopefully) behaves nicely in all situations. Within the interpreter, all of the standard library is available, and it’s possible to interrupt execution (equivalent to the normal ctrl+c behaviour) or to restart the interpreter process. No external modules are included yet except those necessary for the app to run, but I’ll probably include some major ones like numpy in a future release, and in the long term the aim is to support pip installs of new modules.

This has also been a great stimulus for working on python-for-android; I’ve fixed a number of bugs, added several new features, and improved documentation in several places, just thanks to needing these things in a real app.

On a technical level, Pyonic interpreter runs under Python 2, consisting of the app itself and a background Service running a second instance of the interpreter. I’ll be working on Python 3 support in the near future, in fact I originally wrote the app using Python 3 but switched to Python 2 partly due to incompatibilities in Kivy’s osc library (which should be easily fixed or avoided by just using a better communication library) and partly the more well-tested nature of python-for-android’s Python 2 build.

The interpreter works by passing submitted Python code to the background Python process where it is parsed as ast and compiled in ‘exec’ or ‘single’ mode as appropriate to replicate the output printing behaviour of the normal Python interpreter. Doing things this way is a little awkward and feels like reinventing the wheel, although I’m not sure how to better achieve the same thing. An alternative might be to just call the python binary in a subprocess and manipulate its stdin/stdout - I’ll be looking into this option, but it doesn’t eliminate the need for message passing and may need some small changes in python-for-android, assuming also that android doesn’t impose any important limits on subprocessing.

In the short term future, I expect to work first on releasing an improved version that adds a number of useful settings options (sneak peek in the image below), followed by working on a Python 3 version, and then to investigate some of these technical questions. I’d like to look into iOS support, as everything should work almost the same way there, but I don’t have the hardware or developer mempership for iOS development; if anyone would like to try it, let me know. Longer term, Pyonic interpreter is an experimental step towards creating a larger suite of mobile Python tools, in tandem with using this experience to improve python-for-android. There are many features to be added directly to the interpreter, but I’d also like to add surrounding tools including a full code editor, the ability to use pip to install other modules locally, and GUI support via additional Kivy activities.

Screenshot of the settings screen in the development version of Pyonic interpreter.

Settings screen in development to appear in the next release.

Posted in android, Kivy, pyonic, Python | Comments Off on Pyonic interpreter: a Python interpreter GUI for Android, written in Python

python-for-android 0.4 released, now available on PyPI

We’ve just officially released python-for-android 0.4, and pushed it to PyPI for the first time!

python-for-android is a packaging tool for turning Python scripts and apps into Android APKs. It was originally created for use with the Kivy graphical framework, but now supports multiple kinds of Python app including Kivy, PySDL2, a webview interface with Flask or other webserver backend, plain Python scripts without a GUI, or other possibilities such as Python builds for use in other applications.

This release is the culmination of all the work over the last year to replace Kivy’s old Android toolchain with something more flexible and useful for other projects. Major features added in this time include the fully Python toolchain itself, support for SDL2 and other bootstraps, (experimental) python3 support via the CrystaX NDK, multiple architecture support, and many general improvements to the backend. Many thanks to all the contributors who have made this possible!

From now on we intend to move to regular versioned releases rather than the previous rolling master branch. Short term targets for the next release include bringing the python3 build up to full functionality and stability, and some argument restructuring to make command line usage simpler and clearer.

As of this release, you can now install python-for-android with simply:

pip install python-for-android

For full instructions and further information, see the python-for-android documentation.

Posted in android, Kivy, Python | Comments Off on python-for-android 0.4 released, now available on PyPI

python-for-android 0.4 released, now available on PyPI

We’ve just officially released python-for-android 0.4, and pushed it to PyPI for the first time!

python-for-android is a packaging tool for turning Python scripts and apps into Android APKs. It was originally created for use with the Kivy graphical framework, but now supports multiple kinds of Python app including Kivy, PySDL2, a webview interface with Flask or other webserver backend, plain Python scripts without a GUI, or other possibilities such as Python builds for use in other applications.

This release is the culmination of all the work over the last year to replace Kivy’s old Android toolchain with something more flexible and useful for other projects. Major features added in this time include the fully Python toolchain itself, support for SDL2 and other bootstraps, (experimental) python3 support via the CrystaX NDK, multiple architecture support, and many general improvements to the backend. Many thanks to all the contributors who have made this possible!

From now on we intend to move to regular versioned releases rather than the previous rolling master branch. Short term targets for the next release include bringing the python3 build up to full functionality and stability, and some argument restructuring to make command line usage simpler and clearer.

As of this release, you can now install python-for-android with simply:

pip install python-for-android

For full instructions and further information, see the python-for-android documentation.

Posted in android, Kivy, Python | Comments Off on python-for-android 0.4 released, now available on PyPI

Kivy at PyCon

Jacob Kovac, Kivy core developer and creator of the KivEnt game engine, is at PyCon 2016.

Click here or see below to watch his talk, Revitalizing Python Game Development: Packaging, Performance, and Platforms.

Posted in kivent, Kivy, Python | Comments Off on Kivy at PyCon

Android apps with Python, Flask and a WebView

python-for-android has just gained support for a new webview app interface, an alternative to the existing SDL2 or Pygame backends. Under this mode of operation the app gui consists entirely of a browser window directed to open a webpage on localhost, and the Python backend can then run any web framework (I tested with Flask, but others like Bottle or even Django should work), serving this website and managing the app backend.

Example Flask app running on Android

This idea is not itself new; I think SL4A has supported a kind of webview interface for some time and certainly does so now, and we’ve previously seen users running web servers alongside Kivy. The difference to other projects is that apps can take advantage of python-for-android’s relatively extensive toolchain including python3.5 support, the ability to build popular libraries like numpy, support for multiple architectures, and access to the Android API via PyJNIus or Plyer rather than SL4A.

In the image of my testing app above, each of the vibration and orientation buttons sends a request to a Flask url that calls the Android API with PyJNIus to achieve the desired result.

Building a webview app

You can use the webview backend by adding --bootstrap=webview to your python-for-android command line (see the documentation for more details), or including webviewjni in your --requirements argument list. Note that this is incompatible with using SDL or Kivy because the webview bootstrap does not start or manage an OpenGL context. If for any reason you want to run a web server alongside a Kivy app, this is possible but you’ll need to use a different bootstrap and manage the webview yourself via PyJNIus from your Kivy code.

You should also add your chosen web framework to the --requirements argument, or include it your app directory so that it will be imported locally. If there isn’t a recipe for it and it’s a pure Python module, make sure you also add its Python dependencies as these aren’t automatically included right now (letting pip resolve dependencies causes issues when they include compiled modules that must be built separately). python-for-android now includes a recipe for Flask that automatically installs its dependencies (jinja2, werkzeug, markupsafe, itsdangerous and click), so you only need to add flask to the requirements in that case.

Technical details

It turns out that very little hackery is necessary to make a webview type app work. The APK seems to need the INTERNET permission to use a WebView, but Android is very happy for the Python code to run a web server with no further problems.

Making PyJNIus work required a little extra work, as it previously relied on the now-absent SDL to access a pointer to the current JNIEnv. This was fairly simple to fix by using only the relevant code from SDL2 - the important parts are only a small fraction of what SDL provides, as SDL has to worry about all the app input and output going via JNI. For now, python-for-android just patches PyJNIus before building it, but now that there are three different ways to get the JNIEnv on Android this will need addressing somehow in PyJNIus itself.

Posted in android, Kivy, Python | Comments Off on Android apps with Python, Flask and a WebView

Kivy Android app showcase

A natural question when people hear about Kivy as a way to create Android apps in Python is…what can you do with it? Is it performant enough for games, can you call the Android APIs, do all apps look the same? One of the best resources for these kinds of question are existing apps, and in this post I’ll give a quick impression of three of my favourites. This is obviously highly subjective, but I’m focusing in particular on features of technical interest, apps that push Kivy beyond what’s normal to show what it capable of.

If you’re interested in other examples, there’s a fairly extensive (but far from exhaustive) list on the Kivy wiki, including winners of our programming contests and many contributions from users. If you’d like to make your own apps in Python, check out Kivy (which also runs on Windows, Linux, OS X and iOS) and python-for-android (which can also package non-Kivy Python apps).

Boardz

You can download Boardz here.

I’ve put Boardz first because it’s my single favourite Kivy app. It’s actually a work in progress (and in fact hasn’t been updated for a while), but is already a fun game showcasing some of Kivy’s more impressive performance potential.

Boardz homescreen and gameplay

Boardz homescreen (left) and gameplay (right). The black ring on the right is the input circle controlling rider posture.

Boardz is a snowboarding physics game; you control your snowboarder by touching the screen, then moving your finger with respect to its initial position to control your posture; quick movements throw your weight around and can cause you to jump, spin, or fall over, while just positioning the rider differently helps you to pick up speed or navigate barriers. The objective of the game is to get to the end of each stage, with different obstacles including slopes and jumps, collapsing structures, falling rocks, or even multidirectional gravity and rocket boosters. You can fail if your head collides with another object with too much force, or if you simply get stuck and can no longer reach the finish.

What’s immediately impressive is that all this runs well as a Python powered game running on a smartphone. It achieves this by being built using the KivEnt game engine developed by Jacob Kovac, one of Kivy’s core developers. This entity based system lets you write game code in Python but internally is highly optimised in Cython, using Kivy’s OpenGL API extremely efficently as well as interfacing with the popular Chipmunk Physics engine.

Boardz wipeout failure and ad example

Boardz wipeout failure by fatal collision (left), and an ad (right).

Boardz betrays its in-progress nature in other ways; you can see in the above screenshots that its UI isn’t very polished, and in this sense it’s the worst of the apps I’m showing here. However, it makes up for this with its surprisingly engaging gameplay, and a breadth of entertaining features not showcased here, including leader boards, racing your ghost, and different riders with different physics attributes.

A final technical feature interesting to Kivy app developers is that Boardz includes ad integration. Regardless of your feelings about ads themselves, the ability to use them is a major feature enquiry from new Kivy users. The problem here is that integrating with a normal ad provider normally requires adding to the Java components of your app, which it may not be immediately obvious how to do from Python. There are actually a number of resources for this nowadays, with a key point being that python-for-android tries to make it easy to include extra Java code, with which you can interact from Python using Pyjnius. KivEnt’s implementation, pictured above, is a nice demonstration.

Kognitivo

You can download Kognitivo here.

Kognitivo is perhaps the single most polished Kivy app on the Play store, being relatively complex, extensively customised, and exhibiting a number of nice Android API interactions. It is also (deservedly) possibly the most popular Kivy app on Google Play.

Kognitivo introduction and main screens

Kognitivo tutorial (left), game instructions (centre) and homescreen (right).

Kognitivo is a brain training and performance monitoring app. The basic structure is to perform a series of simple exercises intended to test different aspects of cognitive performance, being rated on accuracy and speed, and with the results compiled over time in order to detect and act on trends.

As I’ve said already, the nice thing about Kognitivo is its huge amount of polish. It is extensively themed such that no trace of the Kivy defaults remains, runs extremly smoothly, and includes many nice animation tweaks (unfortunately not captured in screenshots) to feel responsive and active.

Kognitivo gameplay, Android notification and in-app purchase

Kognitivo training game (left), Android notification (centre) and in-app purchase option (right).

On the technical side, Kognitivo exhibits a number of features not normally included in Kivy applications but possible through interaction with the Android API via some Java code and/or the aforementioned Pyjnius. These include notifications, interaction with your calendar, and in-app purchases.

The author of Kognitivo, Sergey Cheparev, has his own writeup of Kognitivo’s development on his blog, including discussion of the advantages and disadvantages of Kivy development, and of the experience of putting together all these features. This is a great resource on its own; I don’t agree with some of the author’s criticisms and some of Kivy’s features have been improved since then, but it’s an excellent overview of the experience. Most of all, he enthusiastically captures some of my own reasons for finding Python on Android interesting:

I think the most beautiful thing in it is to use the almightiness of [Python]’s frameworks. I used sqlalchemy and sqlite as a backend, and it worked like a charm! Python is the most powerfull language because of it’s frameworks, you can even start Django on your smartphone! It’s amazing! Or twisted for asynchro communication with server. Or nltk for in-app natural language processing. Or maybe you want make a mobile equations solver with scipy and numpy. This makes all the dreams come true.

Barly

You can download Barly here, or visit its own website.

Barly is the most recent of these apps to appear on Google Play. I’ve chosen to include it as a nice example of pulling off its concept quite well while making good use of Kivy; like Kognitivo the app is themed very differently to Kivy’s defaults (though it doesn’t look like a normal Android app either), and is generally well put together.

Barly homescreen, palate options, and beer example

Barly homescreen (left), palate options (centre) and a beer search result (right).

To quote its Google Play blurb, Barly is ‘your personal beer expert’. It provides a convenient interface to browse beers via data from standard popular websites, and according to your description of your own palate. Barly’s most interesting feature is the ability to take a picture of a beer menu and have it automatically detect what beers are listed, followed by downloading information about them to help you choose. That kind of image analysis has to be tricky, but actually didn’t perform badly when I tested it.

The interface to this functionality is quite nice, switching to the Android camera app to get the image before uploading it to a server for processing (during which you can input your preferences). This functionality is possible with Pyjnius as mentioned previously, but actually in this case is an API also exposed in pure Python by Plyer (another Kivy sister project, wrapping platform-specific APIs in a Python frontend). Not all APIs can be conveniently exposed this way, and actually Barly may not even be using this particular method, but it’s a good example of functionality that can be achieved with Kivy in a particularly cross-platform way.

Beyond this, Barly does not make such wide use of the Android API or unusual Kivy features, but nor does it try to; it is a nice example of a complete and self-contained Kivy app using the power of Python for an unusual and interesting goal.

Posted in android, Kivy, Python | Comments Off on Kivy Android app showcase

python-for-android now supports Python 3 APKs

It’s been a long time coming, but we can finally make the announcement… python-for-android now supports Python 3 Android apps! This naturally includes Kivy, but also should work for anything else you can package with python-for-android, such as apps made with PySDL2. Using Python 3 remains experimental for now, it works but doesn’t yet perform all possible optimisations and hasn’t been as widely tested as Python 2. However, there should be no extra application requirements (beyond actually being written for Python 3), and the remaining issues and optimisations are being worked on.

Overview

python-for-android is a packaging tool for turning Python applications into Android APKs. It was originally created to make apps with the very cross-platform Kivy graphical framework (though it didn’t arise in a vacuum, I think it built in particular on previous work by the Ren’Py project). However, the original python-for-android had flaws including being quite inconvenient to modify for non-Kivy apps (several other projects seem to have used modified versions, but each was performing similar changes), hard to extend for multiple architecture support, hard to extend internally (both in general and from the perspective of new contributors, as much of the toolchain was a big shell script), and only supported building apps with Python 2.

We recently completed and released a fully revamped version of python-for-android aimed at fixing all of these problems, as discussed in several previous posts on this blog (originally here). Almost all of the original goals are now complete, with Python 3 support the major missing one until now, though not for lack of trying. Some technical details and basic instructions for testing the new support are given below, and you can also see the online documentation for further information.

Our Python 3 support depends on the prebuilt Python distributions provided with the CrystaX NDK, a drop-in replacement for Google’s own Android NDK with many fixes and improvements. The technical details of this choice are given below, and we’ll try to further support a locally-built Python 3 option in the future. Thanks to the CrystaX team for making it so (relatively) easy!

Technical details

python-for-android works by bundling a Python interpreter, compiled for Android devices and architectures (usually arm, though other choices are supported), into an Android APK. The APK includes a simple Java bootstrap application, which mostly starts a Python script via JNI. The script then runs essentially as normal, almost all of the Python standard library is present and works fine, and python-for-android supports including other modules or non-Python dependencies. Pure Python modules will mostly work without special treatment, though things requiring compilation need a special recipe. Many common modules such as numpy and sqlalchemy are supported this way. Following its revamp python-for-android is now designed to support multiple kinds of java bootstrap, but the current main support is for GUI apps via Pygame (for Kivy’s old Android support) or SDL2 (both for Kivy and for anything else that can use it); SDL2 also now does much of the heavy lifting of handling events etc. itself, via its own Android support.

The main problems with compiling and including Python are first that it must be patched to compile (as Android’s libc doesn’t support some things very well or at all), and second that it must be unpacked and started on the device via its C API. The second point is fiddly but ultimately not that different to working this way on the desktop. The first is (in my opinion) harder because it needs some understanding of Python’s internals, of Android’s limitations, of appropriate fixes, and of how to test and debug these problems.

Such patches have been made by a number of different people for different Android versions, and I believe there has been activity on Python itself to fix some issues (including this current issue to make Python build natively on Android). For Python 2, I think python-for-android’s original patches came from here, though extended with further modifications. However, the main thing holding up my efforts to get Python 3 working was the inability to find a similar working patchset; I tried a few sources, achieving a working compilation using SL4A’s python3 patches, but I couldn’t get Python working on the device. I’m sure this was my own technical mistakes, since other projects do have it working, but it’s what was holding up the feature.

I eventually resolved this by using the new Python on Android support from the CrystaX NDK. As mentioned above, this is a drop-in replacement for Google’s own NDK (the Native Development Kit providing compilers etc for targeting Android with non-Java code), including many improvements to the build environment. As of version 10.3.0, they provide prebuilt Python packages for Android on all architectures - and all of their NDK improvements mean that Python no longer even needs patching for this compilation to work. Python is provided as a zipped standard library (Python can automatically load modules from zip files), and a folder of the compiled components like ctypes (as it’s hard to dynamically load from zips). From the perspective of python-for-android, supporting Python 3 means modifying the build to load CrystaX’s prebuilt components (both in the Android project structure and in python-for-android’s support for compiling other modules), and modifying the C initialisation code for Python 3. This takes some work, but all told wasn’t very hard and the Python bundles worked with no issues, so we owe a lot to CrystaX; thanks again.

I’d still like to come back to the issue of local python3 compilation; CrystaX’s versions are fine, but I’ve learned a lot from making them work, and have a much better idea of what I may have been doing wrong. However, the focus for now will be on resolving the remaining issues with what’s already working.

Building apps with Python 3

Building Python 3 APKs is only supported in the revamped python-for-android toolchain which was merged to the master branch a while ago. It can be installed and used as described in its documentation. If you use Buildozer, it currently does not support this new toolchain, though tito has been working on this. There is also the new restriction that you must (for obvious reasons) use the CrystaX NDK, which can be obtained here; simply refer to its filepath when setting the NDK directory, and everything else should work automatically.

To build for Python 3, add the python3crystax recipe to the requirements option, e.g. --requirements=python3crystax,kivy. It should mostly work automatically with existing recipes, though at this stage there may be bugs or problems with a few, and some will need modification. The exact syntax above may also change in the future as the Python 3 support becomes better integrated, but not significantly.

There’s also one big change whose importance I’m not sure about; the Python 3 mechanism doesn’t currently build a local Python 3 to use as a hostpython, instead using the system python. This means that you must have python3.5 (3.4 may also work) installed locally in order for python-for-android to build Python 3 APKs. This will be fixed soon, adding a hostpython3 recipe to avoid weird bugs with system-specific differences, but you need to bear it in mind for now.

Future work

For now, this Python 3 support remains experimental. I anticipate no major issues, but it’s internally a quite different method to the Python 2 support and needs further work to duplicate some of the old optimisations, and undoubtedly to fix bugs in the toolchain that will appear as it stabilises. Amongst other things, Python 3 shared libraries are not currently collected and merged (with Python 2 we did this originally to get around an Android limit but also for optimisation), python files are not precompiled to bytecode (it can make a big loading speed difference), and some features of the old pygame bootstrap have not yet been implemented in SDL2. All this and more will come in the future, but shouldn’t be hard to add now that the toolchain all works.

The SDL2 bootstrap is also missing a few features that users of the old toolchain will be used to, like the splash screen image and at least one Kivy-specific function. These too are being actively worked on, especially as more people start to move their apps to SDL2.

I’ve also phrased this as Python 2 built locally vs Python 3 from CrystaX, but actually CrystaX also supports Python 2.7 and I hope to add this option in the near future. As discussed in the technical details, it also should absolutely be possible to have a local Python 3 build, which I’d like to eventually come back to.

Posted in android, Kivy, Python | Comments Off on python-for-android now supports Python 3 APKs