I used python to write a small utility that i wanted to share with a coworker, ended up having to rewrite it in Go because i spent 3 hours trying to compile the python version to an exe so that it was easier to run on their machine.
I wonder why isn't there an easy way to just package the interpreter inside an EXE for people that don't care about binary size just so that it would make it seamless to package up utilities written in python.
It's kind of frustrating to deal with python stuff as an user. I wish it didn't have to be like this, because there's a lot of interesting stuff written in python that's nearly impossible to get to run.
There are too many environemnt systems (conda, venv, I think I'm forgetting one more), not all of them build or ship on all systems, then you need to manage them, and sometimes to have the exact correct python version installed because god help you if you're on 3.12 and not 3.11; and set as PATH for pip to find the correct dependencies, but you need to do that before you set up your venv and run pip, otherwise you need to tear that down and star tover. Sometimes the dependencies build, and break, because some package is missing from your system.
I also end up avoiding python not because it's a bad language, but because it's so much more convenient to work with compiled artifacts that worry about shipping around a bunch of source or object files and a full compiler. Not to mention the speed of using Go and the ability to write concurrent code without a lot of effort.
The only time I really use python is when I need a quick and dirty script.
As of a few months ago, pex supports including an interpreter, either directly inline or lazily downloaded (and cached): https://docs.pex-tool.org/scie.html
I've come to realize that putting everything into a container is the only viable way to share a python program.
I'll certainly checkout PEX, I think that distribution of a binary is likely the largest one for Python right now. Java solved it with a JAR file, static-compiled binaries for most compiled languages.
At Google PAR files have been quite a useful way to handle this for at least 18 years now, hope this can get reasonably solved everywhere.
It's probably on the list that you've tried, but I've had most luck with Nuitka (in standalone mode, not onefile). Unlike pyinstaller, it really is the executable it claims to be, not just an elaborate zip file containing python.exe and your source files.
I've always been curious why Nim never took off as the solution for use cases like this. It combines the simplicity and clarity of Python-like syntax with the performance of a natively compiled language, which looks like the best of both worlds.
Seems like i have it in my browsing history, i remember not being able to run the executable it produced, not being able to find a library it was supposed to have. This was just using the default "pyinstaller your_program.py", and i was frustrated enough to not go deeper into why that was. Will definitely give it a try again in the future
PyInstaller-made executables also used to have a habit of getting flagged by security software as malicious (maybe that's why you couldn't run it?) -- apparently, so many malware writers used it that it ruined the party for everyone.
Fortunately, that was only the 32-bit version of Python 2.7. Using 64-bit versions or Python 3 was enough to not get flagged as malicious. I figured that out when I decided I didn't want to teach myself Go just then to deploy something that had worked the day before.
Using the spec files for persistent readable configuration also goes a long way, if you treat pyinstaller as a python module you can automate it whole with just python, including the spec files as it executes them as python scripts
Yep I used it at my last job and it worked great! Startup times were horrible, but that didn't matter so much to us and it solved tons of problems we had with people messing up their python environments. Takes some tweaking to get certain modules (like scikit-rf) to work, but never found an issue that couldn't be solved.
I'm always on the lookout for ways of packaging Python programs/script-piles into single-file executables, so thank you for posting this!
The GitHub README says it builds on the Python .ZIP App format (PEP441), which in turns says you can put your app in a .ZIP file and Python will run it as an app.
I think I went this route with PyInstaller's one-file output option. But I found it to be too slow because (1) my app was a CLI app and I needed it to start, run, and end quickly, (2) I imported something from Django because it was useful (which ballooned the size), and (3) the single-file .ZIP needed to be extracted into a temporary directory, run, and then deleted every time (!!!).
Does anyone know how Pex / PEP441 deals with this? Is the extract-run-delete thing normal/standard? Is there a way to cache the extracted app?
My guess is that without operating system support or equivalent, there is not an easy way to avoid extracting the zip file. (You could patch syscalls in the Python interpreter to read from the zip file for certain paths, but that might be hacky.)
I did find https://github.com/google/mount-zip which might be useful, but you would likely have to still mount the zip manually. However, you don’t have to worry as much about cleaning up after yourself.
I think the easiest way to do this now is with uv and their support for inline metadata. At this point you can just have the user install uv with a single command and then have the user run the script with `uv run example.py`.
I know it's not very nice to post that here, but given it's hard to get good info about python because the ecosystem is so huge, I'd rather give it straight:
Pex doesn't work on windows, is slower than shiv and shiv solves the resource files problem that you will encounter with dependencies like django. To achieve this, shiv unzips the whole zipapp in a cache on disk and create a transparent proxy to it.
Still, I wish zipapps had better tooling honestly. Using either pex or shiv is more complex than it needs to be.
I hope uv will eventually give you something like "uv zipapp" and make it good and simple.
Right now, to bundle all deps for shiv you have to download first all wheels, then figure out the proper incantation to include them. And if you use WSGI, you probably want some kind of dedicated entry point.
The tech is cool, but it deserves maturing to be a true ".war for python".
FWIW, pex now also has options to unzip the archive to a cache directory on startup (I believe this happens by default now, but am not at a computer to confirm), to side step the zipapp limitations that you reference.
In the early 2000’s open source hubris, next year was always the year of the Linux desktop. Since then consumer windows matured into a totally ok OS and one with the best support for graphics and C++ development at that.
Non-windows support nowadays is a fairly strong signal of a non-serious software offering if there is no obvious reason for it. And that’s totally fine, hobby tools developed by enthusiasts rock - but they are not industrial in scope as such.
A lot of serious software offerings are only concerned with the server use case, modern servers run Linux unless there's a good reason not to, and modern windows has more than one acceptable way to run Linux binaries if you absolutely have to.
While we're posting not-nice things, I'll throw my current favorite out: pipx ( https://github.com/pypa/pipx ). I use Python's Poetry to build a wheel and then pipx to install it. Super-easy, barely an inconvenience :)
I'm not a pipx expert (I'm a "I got it to work for my use case and then stopped" person :) ) but I was able to get my project working after installing a local wheel file (so i don't think you need access to the Internet to install things).
I thought the wheel file has all the dependencies since (my understanding of) pipx uses a venv for each individual wheel file/app. Certainly I'm using a bunch of packages that I needed to pip install into my development environment, but it's entirely possible that my pipx-managed wheels are using those installs, too.
I think the last two points are accurate, and I've never tried to use APIs like wsgi with it.
It would today, because macos requires signing, and windows has "the mark of the web".
They don't apply to scripts, so Python is fine, but they do apply to executables.
Having to sign your zip every time you produce it would nullify the benefit of the concept. And because the checksum of the executable file change according to the content of the zip, you can't just sign the cosmopolitan part.
- The underlying python code cannot use open() to access its own resources, but must use pkg_resources, and most packages on pypi don't do that because most dev don't even know it exists or what problem is soves.
- _ _ file _ _ is not available, breaking a lot of things. zip_safe is a build option, but actual compliance to it is hard work and rare.
I was thinking more along the lines of using namespaces and FUSE. I'm aware that Python kinda treats them like folders transparently, but not completely for reasons (and more) that you described.
I can see how .pex or shiv's zippaps live at the Pareto threshold where converting a Python app to an 80% (or less...) executable takes only 20% (or less!) of the effort that would be spent re-writing the app in eg C, C++, Rust, or Golang. Still, it breaks my heart to see all this work being done to keep developers employed as "Python developers", instead of leveling them up to a broader programming skillset. It seems as if the industry is investing rather heavily in a local maximum.
But, of course, the market is has information that I don't.
It is often the case that the nifty Python thing you want to pass around uses one or more nifty Python libraries that have no C/C++/Rust/Golang equivalent (or no obvious equivalent), and so rewriting it becomes a herculean task.
Eh I wouldn't call PEX proper executables that'd replace the languages above as they still need python, but I still prefer python for things like simple Qt apps even if it means using e.g. PyInstaller
We've been using pex for a long while to package PySpark jobs with dependencies into a single file. Saves a ton of time vs. the old way of building/deploying docker images.
I thought this was going to be a post about plumbing and then I read further and was even more excited to say the phrase "python executable". Very interesting. Thanks!
I used python to write a small utility that i wanted to share with a coworker, ended up having to rewrite it in Go because i spent 3 hours trying to compile the python version to an exe so that it was easier to run on their machine.
I wonder why isn't there an easy way to just package the interpreter inside an EXE for people that don't care about binary size just so that it would make it seamless to package up utilities written in python.
It's kind of frustrating to deal with python stuff as an user. I wish it didn't have to be like this, because there's a lot of interesting stuff written in python that's nearly impossible to get to run.
There are too many environemnt systems (conda, venv, I think I'm forgetting one more), not all of them build or ship on all systems, then you need to manage them, and sometimes to have the exact correct python version installed because god help you if you're on 3.12 and not 3.11; and set as PATH for pip to find the correct dependencies, but you need to do that before you set up your venv and run pip, otherwise you need to tear that down and star tover. Sometimes the dependencies build, and break, because some package is missing from your system.
It's kind of a miserable experience.
I also end up avoiding python not because it's a bad language, but because it's so much more convenient to work with compiled artifacts that worry about shipping around a bunch of source or object files and a full compiler. Not to mention the speed of using Go and the ability to write concurrent code without a lot of effort.
The only time I really use python is when I need a quick and dirty script.
As of a few months ago, pex supports including an interpreter, either directly inline or lazily downloaded (and cached): https://docs.pex-tool.org/scie.html
I've come to realize that putting everything into a container is the only viable way to share a python program.
I'll certainly checkout PEX, I think that distribution of a binary is likely the largest one for Python right now. Java solved it with a JAR file, static-compiled binaries for most compiled languages.
At Google PAR files have been quite a useful way to handle this for at least 18 years now, hope this can get reasonably solved everywhere.
It's probably on the list that you've tried, but I've had most luck with Nuitka (in standalone mode, not onefile). Unlike pyinstaller, it really is the executable it claims to be, not just an elaborate zip file containing python.exe and your source files.
I've always been curious why Nim never took off as the solution for use cases like this. It combines the simplicity and clarity of Python-like syntax with the performance of a natively compiled language, which looks like the best of both worlds.
There is: PyInstaller
Seems like i have it in my browsing history, i remember not being able to run the executable it produced, not being able to find a library it was supposed to have. This was just using the default "pyinstaller your_program.py", and i was frustrated enough to not go deeper into why that was. Will definitely give it a try again in the future
PyInstaller-made executables also used to have a habit of getting flagged by security software as malicious (maybe that's why you couldn't run it?) -- apparently, so many malware writers used it that it ruined the party for everyone.
Fortunately, that was only the 32-bit version of Python 2.7. Using 64-bit versions or Python 3 was enough to not get flagged as malicious. I figured that out when I decided I didn't want to teach myself Go just then to deploy something that had worked the day before.
Pyinstaller can be a bit fiddly to get right initially if you have a lot of package dependencies, but other than that it works really well.
Using the spec files for persistent readable configuration also goes a long way, if you treat pyinstaller as a python module you can automate it whole with just python, including the spec files as it executes them as python scripts
I've had automated builds running mostly untouched for years here https://github.com/Numerlor/Auto_Neutron/blob/master/pyinsta..., though it doesn't have any particularly troublesome packages other than PySide6
Yep I used it at my last job and it worked great! Startup times were horrible, but that didn't matter so much to us and it solved tons of problems we had with people messing up their python environments. Takes some tweaking to get certain modules (like scikit-rf) to work, but never found an issue that couldn't be solved.
Would recommend!
I'm always on the lookout for ways of packaging Python programs/script-piles into single-file executables, so thank you for posting this!
The GitHub README says it builds on the Python .ZIP App format (PEP441), which in turns says you can put your app in a .ZIP file and Python will run it as an app.
I think I went this route with PyInstaller's one-file output option. But I found it to be too slow because (1) my app was a CLI app and I needed it to start, run, and end quickly, (2) I imported something from Django because it was useful (which ballooned the size), and (3) the single-file .ZIP needed to be extracted into a temporary directory, run, and then deleted every time (!!!).
Does anyone know how Pex / PEP441 deals with this? Is the extract-run-delete thing normal/standard? Is there a way to cache the extracted app?
My guess is that without operating system support or equivalent, there is not an easy way to avoid extracting the zip file. (You could patch syscalls in the Python interpreter to read from the zip file for certain paths, but that might be hacky.)
I did find https://github.com/google/mount-zip which might be useful, but you would likely have to still mount the zip manually. However, you don’t have to worry as much about cleaning up after yourself.
Pex and Shiv both require the end user to have Python installed, so I think PyOxidizer is the best tool in this category: https://pyoxidizer.readthedocs.io/en/stable/pyoxidizer_compa...
How does Pyoxidizer compare to pyinstaller and nuitka?
Great question! That was actually covered in the link I posted.
Compared to PyInstaller, it doesn't dump any temporary files on disk.
Compared to Nuitka, it uses the real CPython, not a different implementation.
I think the easiest way to do this now is with uv and their support for inline metadata. At this point you can just have the user install uv with a single command and then have the user run the script with `uv run example.py`.
—script appears to be missing if you meant https://peps.python.org/pep-0723/
I know it's not very nice to post that here, but given it's hard to get good info about python because the ecosystem is so huge, I'd rather give it straight:
Don't use pex, use shiv: https://shiv.readthedocs.io/
Pex doesn't work on windows, is slower than shiv and shiv solves the resource files problem that you will encounter with dependencies like django. To achieve this, shiv unzips the whole zipapp in a cache on disk and create a transparent proxy to it.
Still, I wish zipapps had better tooling honestly. Using either pex or shiv is more complex than it needs to be.
I hope uv will eventually give you something like "uv zipapp" and make it good and simple.
Right now, to bundle all deps for shiv you have to download first all wheels, then figure out the proper incantation to include them. And if you use WSGI, you probably want some kind of dedicated entry point.
The tech is cool, but it deserves maturing to be a true ".war for python".
FWIW, pex now also has options to unzip the archive to a cache directory on startup (I believe this happens by default now, but am not at a computer to confirm), to side step the zipapp limitations that you reference.
I just checked, and there is indeed a `--pex-root` option with and even "-c" to specify a custom entry point.
Thanks for pointing it out.
”Pex doesn't work on windows”
Oh that’s such a weird limitation.
In the early 2000’s open source hubris, next year was always the year of the Linux desktop. Since then consumer windows matured into a totally ok OS and one with the best support for graphics and C++ development at that.
Non-windows support nowadays is a fairly strong signal of a non-serious software offering if there is no obvious reason for it. And that’s totally fine, hobby tools developed by enthusiasts rock - but they are not industrial in scope as such.
A lot of serious software offerings are only concerned with the server use case, modern servers run Linux unless there's a good reason not to, and modern windows has more than one acceptable way to run Linux binaries if you absolutely have to.
While we're posting not-nice things, I'll throw my current favorite out: pipx ( https://github.com/pypa/pipx ). I use Python's Poetry to build a wheel and then pipx to install it. Super-easy, barely an inconvenience :)
pipx is orthogonal to the problem pex and shiv are solving since:
- it doesn't provide a single file for a full project with all dependencies.
- it requires internet access on the target machine to install things.
- it can't give you access to underlying API for things like wsgi.
- it requires pipx on the target machine.
- it requires a wheels made from a build, which many projects don't have (such as a lot of web projects).
Hello!
I'm not a pipx expert (I'm a "I got it to work for my use case and then stopped" person :) ) but I was able to get my project working after installing a local wheel file (so i don't think you need access to the Internet to install things).
I thought the wheel file has all the dependencies since (my understanding of) pipx uses a venv for each individual wheel file/app. Certainly I'm using a bunch of packages that I needed to pip install into my development environment, but it's entirely possible that my pipx-managed wheels are using those installs, too.
I think the last two points are accurate, and I've never tried to use APIs like wsgi with it.
> .war for python
It shouldn't be that hard to get a general 'jar/war', not just for python.
Perhaps jart's cosmolitan libc with a zip file appendage which it auto extracts and optionally caches on first use.
A heavyweight solution, but still lighter than Docker.
It would today, because macos requires signing, and windows has "the mark of the web".
They don't apply to scripts, so Python is fine, but they do apply to executables.
Having to sign your zip every time you produce it would nullify the benefit of the concept. And because the checksum of the executable file change according to the content of the zip, you can't just sign the cosmopolitan part.
Pex works with older versions than shiv.
I feel like, in general, you should be able to treat an archive as a filesystem. Zipapps could be so much easier if it did that transparently.
It can, but:
- It's slow.
- The underlying python code cannot use open() to access its own resources, but must use pkg_resources, and most packages on pypi don't do that because most dev don't even know it exists or what problem is soves.
- _ _ file _ _ is not available, breaking a lot of things. zip_safe is a build option, but actual compliance to it is hard work and rare.
I was thinking more along the lines of using namespaces and FUSE. I'm aware that Python kinda treats them like folders transparently, but not completely for reasons (and more) that you described.
That's a cool idea.
Wonder what the FUSE equivalent for windows is.
Probably Windows Projected FileSystem: https://learn.microsoft.com/en-us/windows/win32/projfs/proje...
I can see how .pex or shiv's zippaps live at the Pareto threshold where converting a Python app to an 80% (or less...) executable takes only 20% (or less!) of the effort that would be spent re-writing the app in eg C, C++, Rust, or Golang. Still, it breaks my heart to see all this work being done to keep developers employed as "Python developers", instead of leveling them up to a broader programming skillset. It seems as if the industry is investing rather heavily in a local maximum.
But, of course, the market is has information that I don't.
It is often the case that the nifty Python thing you want to pass around uses one or more nifty Python libraries that have no C/C++/Rust/Golang equivalent (or no obvious equivalent), and so rewriting it becomes a herculean task.
I wonder if this isn't reflective of some universal law of niftiness.
Eh I wouldn't call PEX proper executables that'd replace the languages above as they still need python, but I still prefer python for things like simple Qt apps even if it means using e.g. PyInstaller
We've been using pex for a long while to package PySpark jobs with dependencies into a single file. Saves a ton of time vs. the old way of building/deploying docker images.
https://spark.apache.org/docs/latest/api/python/user_guide/p...
We shiv all the dependencies into a single file, bake it into a Docker layer, and refer to it in each scripts shebang.
I thought this was going to be a post about plumbing and then I read further and was even more excited to say the phrase "python executable". Very interesting. Thanks!