Hashicorp has go-plugin: https://github.com/hashicorp/go-plugin. It does similar with support for net/rpc and grpc. With grpc, you could have the external process in Python. Unix domain sockets and TCP are supported. The framework handles spawning the process and managing it.
It is used extensively within hashicorp's products - nomad, packer etc.
I wish python had proper sandbox configuration. This would be great for user defined scripts but only if I could disable filesystem and network syscalls for the user defined code.
This uses UDS (Unix Domain Sockets) to communicate, which are file-like and can be mounted from the host filesystem inside a container.
As long as the socket is writable, the rest of the filesystem(s) don't have to be. Same goes for the networking, which can be very isolated and restricted.
Hashicorp has go-plugin: https://github.com/hashicorp/go-plugin. It does similar with support for net/rpc and grpc. With grpc, you could have the external process in Python. Unix domain sockets and TCP are supported. The framework handles spawning the process and managing it. It is used extensively within hashicorp's products - nomad, packer etc.
One of the main reasons I switched to Rust, awesome work!
But now that I mostly code in Rust I don’t think I’ll be returning to Go, it’s a fun language but very limited
I wish python had proper sandbox configuration. This would be great for user defined scripts but only if I could disable filesystem and network syscalls for the user defined code.
You could use starlark but I don't have context for what you're using Python for specifically. Starlark has stronger hermetic guarantee.
This uses UDS (Unix Domain Sockets) to communicate, which are file-like and can be mounted from the host filesystem inside a container.
As long as the socket is writable, the rest of the filesystem(s) don't have to be. Same goes for the networking, which can be very isolated and restricted.
It should be possible to do what you want.
chroot ? :)
Pretty sure chroot is not an isolation method, it's way too easy to bypass when used as such
Hopefully we can write an Airflow replacement using this.
How is "running a pool of Python worker processes" different from having a microservice?