cx-freeze, runpy and multiprocessing – multiple paths to failure

Each Answer to this Q is separated by one/two green lines.

This is a bit of a complex one, and may take some of your time.

The basic problem is, that on linux (Ubuntu in my test case) a cx-freeze’d version of my program (Omnitool) is not able to create subprocesses. It works on Windows 7, however. Or when running directly from source code. Unfortunately it’s not as simple as forgetting freeze_support.

The Problem

Default behaviour of starting a subprocess, is that the X Server crashes. Specifically like this:

XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"
      after 23 requests (23 known processed) with 0 events remaining.
[xcb]Unknown sequence number while processing queue 
[xcb] Most likely this is a multi-threaded client and XInitThreads has not been called 
[xcb] Aborting, sorry about that. 
Omnitool: ../../src/xcb_io.c:274: poll_for_event: Assertion `!xcb_xlib_threads_sequence_lost' failed. 
Aborted (core dumped)

There is no python traceback. I tried to manually call XInitThreads with ctypes, it returns 0 for successfully set, but crashes regardless. Shorty before the crash, the pygame UI bugs out, so I expect that something there faults.

Now, setting multiprocessing.set_start_method() changes problems around:
“forkserver” gives me this lovely traceback, that tells me nothing:

Pastebin Example

Setting it to spawn, instead, just makes it not do anything. The process starts, and goes through __main__, as I can prove with prints, but never enters the target function for the subprocess.

Trying yourself

Make sure you have Ubuntu or comparable Linux with python 3.4.
Then to get all dependencies:

Download omnitool as zip or clone from git:
requirements.txt in the following code is from Omnitool.

sudo apt-get update -qq
sudo apt-get install --fix-missing mercurial subversion python3-dev python3-numpy libav-tools libsdl-image1.2-dev libsdl-mixer1.2-dev libsdl-ttf2.0-dev libsmpeg-dev libsdl1.2-dev libportmidi-dev libswscale-dev libavformat-dev libavcodec-dev python3-pip
pip3 install -r requirements.txt
pip3 install cx_Freeze

To freeze, run with the buildargument.

I’ve also tried freezing with pyinstallers python 3 fork, which fails very similar. The spec file isn’t git tracked though.

I used cx_freeze for a project at work. I’m not sure if this is your problem… but I was using the Anaconda distribution, and cx_freeze was not properly gathering the .dll’s that I needed for my project.

The solution was to:

  1. Install a plane version of Python
  2. make an environment with the packages that I needed for that project
  3. Run cx_freeze.

Magically, all of the problems went away and everything compiled like it was supposed too.

I know that probably this is not what you want to hear, but the reality is that you will never be able to succeed with this approach and here is why:

  • Linux does not have portably Python package binaries, meaning that a compiled numpy on one distro will probably not work on another. It can even happen with the same distribution and version, just because one system library received an update.
  • manylinux1 is supposed to solve this but is really hard to build packages for it and only few started using it.
  • cx-freeze project had latest release in December 2014, which on Python timescale puts it as an abandoned project.

Well the problem you are facing may be due to the lack of dependencies.
cx_Freeze have some dependencies.
In order to distribute your work in python many lightweight packages are avalible.
You can use:-

  1. Pyinstaller: it can be installed by

    pip install pyinstaller
    pyinstaller is really good for making executables with less size

  2. Py2exe

The answers/resolutions are collected from stackoverflow, are licensed under cc by-sa 2.5 , cc by-sa 3.0 and cc by-sa 4.0 .