CLI is first class in proxmox, I use the qm command for managing vms all the time. The networking is also just a file in `/etc/network/interfaces` that I modify with vim as needed.
CLI is dandy but Terraform is even nicer. I run something like ten Proxmox clusters now around the world, and an AI agent can deploy a fully terraformed VM for me, tied into monitoring, asset mgmt, EDR, etc in a few minutes...
I'm a big LXD and now Incus fan. But I went with NixOS rather than IncusOS for my latest build because I prefer the LTS linux kernel over the mainline kernel.
Another proxmox-esque project I've been watching is Sylve, a control plane for FreeBSD that provides a web interface to jails, bhyve VMs, and containers [1]. It's new-ish, but it looks like a possible sweet spot replacement for both proxmox and TrueNAS (which was originally also built on FreeBSD before they switched to Linux IIRC), at least for my homelab-ing use case. Potentially eventually for environments at greater scale also. The company behind it is a software consultancy and Sylve is built with their actual business needs in mind, and is BSD-licensed like the OS it runs atop.
One of the developers building Sylve gave a talk last year [2].
I don't if you know this, but there was a collection for LXD. Last time I checked, it didn't seem very popular so not maintained very well but it did work. Maybe that could be used for inspiration.
I remember Stéphane worked on adding support for incus containers to opentofu which seemed more popular than using ansible to describe the resources.
NixOS has transformed my use of Proxmox. I configure, build, and deploy everything from my nix workstation. I don't need to use the PVE gui at all. Proxmox is just a target, and I've abstracted things enough to where I can deploy the same machines to libvirt on a local machine too. Why would I need to let my agent into my PVE box? I haven't looked at incus, but if I wanted to run the full stack declaratively, nixos and LLMs are so powerful now that I would probably just say to run libvirt and ZFS on nixos natively.
> I configure, build, and deploy everything from my nix workstation. I don't need to use the PVE gui at all.
I would love to know more about how you do this, particularly the deploy part. I'm considering moving away from Ansible, but haven't had the time to dedicate to exploring a similar Nix experience.
This seems very cool and I will probably try it, but I think I’m missing something. I run Proxmox so that I can have multiple VMs running on my NUC. This doesn’t really solve that right? I cant spin up a windows 11 vm one weekend for a random experiment.
Incus is roughly a frontend for qemu, so you can launch an empty VM and use the ISO to install the OS. You don't have to use a preconfigured base image.
VM support has been there for quite a while now and works nicely. I think that's what they're referring to. It started out with only system containers and then gained support for VMs and now there's work to support launching OCI images directly.
> But fundamentally, Proxmox is built around clicking buttons. It is a GUI-first paradigm.
Uhh, whut? It provides a button-y interface, but you can do everything via config files and `pct` on the command line if you prefer. I know that’s not full nix-style declarative, but you don’t have to mislead to sell me on the advantages of declarative infra.
I haven't abandoned Proxmox yet, but the take here resonates with me. I do not like configuring appliances. I prefer defining infra as code, having that diffable, assertable, etc. I have had pretty good luck managing Proxmox clusters with the Proxmox API (https://pve.proxmox.com/pve-docs/api-viewer/index.html) or just letting the agent shell in as root (lol). I built a very simple provisioning tool called vmfactory that takes some really somple config on disk, bakes a fresh qcow image, pushes it to proxmox and then configures networking and boots it. It's extremely rudimentary but has been working well for me.
I did abandon TrueNAS, however. It really is a locked-down appliance. Good luck installing custom software on the base OS. I have a domain-joined Ubuntu/ZFS box that inherits a lot of policy from FreeIPA and/or Ansible config that is all backed by files on disk. It's been really easy to orchestrate what many would consider overkill in my homelab because literally everything is represented in a single Github repo.
I run Ansible against Proxmox hosts to ensure the config on them is what I want, and then Terraform all VMs into place on them. It's not too far off from having your own mini-EC2, minus some of the nice trappings like load balancers.
I get the advantage of fully declarative systems such as Nix, but everything you do in proxmox via GUI can be done via CLI: https://pve.proxmox.com/pve-docs/#_command_line_interface
CLI is first class in proxmox, I use the qm command for managing vms all the time. The networking is also just a file in `/etc/network/interfaces` that I modify with vim as needed.
CLI is dandy but Terraform is even nicer. I run something like ten Proxmox clusters now around the world, and an AI agent can deploy a fully terraformed VM for me, tied into monitoring, asset mgmt, EDR, etc in a few minutes...
I'm also considering migrating from Proxmox to Incus, but I'd look into IncusOS rather than having to manage the host OS myself.
I'm a big LXD and now Incus fan. But I went with NixOS rather than IncusOS for my latest build because I prefer the LTS linux kernel over the mainline kernel.
Another proxmox-esque project I've been watching is Sylve, a control plane for FreeBSD that provides a web interface to jails, bhyve VMs, and containers [1]. It's new-ish, but it looks like a possible sweet spot replacement for both proxmox and TrueNAS (which was originally also built on FreeBSD before they switched to Linux IIRC), at least for my homelab-ing use case. Potentially eventually for environments at greater scale also. The company behind it is a software consultancy and Sylve is built with their actual business needs in mind, and is BSD-licensed like the OS it runs atop.
One of the developers building Sylve gave a talk last year [2].
[1]: https://sylve.io/
[2]: https://youtu.be/wo4oD5UON30
I've been using incus for a while now and actually run it on a side project in production for the better part of a year. Rock solid performance.
Incus is great. I've been trying to revive an unmaintained ansible collection to manage incus resources https://github.com/sbstp/ansible-collection-incus
I don't if you know this, but there was a collection for LXD. Last time I checked, it didn't seem very popular so not maintained very well but it did work. Maybe that could be used for inspiration.
I remember Stéphane worked on adding support for incus containers to opentofu which seemed more popular than using ansible to describe the resources.
NixOS has transformed my use of Proxmox. I configure, build, and deploy everything from my nix workstation. I don't need to use the PVE gui at all. Proxmox is just a target, and I've abstracted things enough to where I can deploy the same machines to libvirt on a local machine too. Why would I need to let my agent into my PVE box? I haven't looked at incus, but if I wanted to run the full stack declaratively, nixos and LLMs are so powerful now that I would probably just say to run libvirt and ZFS on nixos natively.
> I configure, build, and deploy everything from my nix workstation. I don't need to use the PVE gui at all.
I would love to know more about how you do this, particularly the deploy part. I'm considering moving away from Ansible, but haven't had the time to dedicate to exploring a similar Nix experience.
Just use Terraform against Proxmox and don't leave mysterious things nobody else knows how to work with.
What do you mean exactly? You have a nix workstation and a physically separate PVE server, and configure the second from the first?
This seems very cool and I will probably try it, but I think I’m missing something. I run Proxmox so that I can have multiple VMs running on my NUC. This doesn’t really solve that right? I cant spin up a windows 11 vm one weekend for a random experiment.
It sounds like all his containers are Linux, so that's why Incus is such a good fit for him. For your use case, yeah, proxmox is likely a better fit.
This small project makes running Windows on Incus a breeze: https://github.com/antifob/incus-windows
Incus is roughly a frontend for qemu, so you can launch an empty VM and use the ISO to install the OS. You don't have to use a preconfigured base image.
I think incus can be a frontend for qemu, but it's primary mode of operating is to run containers. It's a fork of LXD.
VM support has been there for quite a while now and works nicely. I think that's what they're referring to. It started out with only system containers and then gained support for VMs and now there's work to support launching OCI images directly.
> But fundamentally, Proxmox is built around clicking buttons. It is a GUI-first paradigm.
Uhh, whut? It provides a button-y interface, but you can do everything via config files and `pct` on the command line if you prefer. I know that’s not full nix-style declarative, but you don’t have to mislead to sell me on the advantages of declarative infra.
I haven't abandoned Proxmox yet, but the take here resonates with me. I do not like configuring appliances. I prefer defining infra as code, having that diffable, assertable, etc. I have had pretty good luck managing Proxmox clusters with the Proxmox API (https://pve.proxmox.com/pve-docs/api-viewer/index.html) or just letting the agent shell in as root (lol). I built a very simple provisioning tool called vmfactory that takes some really somple config on disk, bakes a fresh qcow image, pushes it to proxmox and then configures networking and boots it. It's extremely rudimentary but has been working well for me.
I did abandon TrueNAS, however. It really is a locked-down appliance. Good luck installing custom software on the base OS. I have a domain-joined Ubuntu/ZFS box that inherits a lot of policy from FreeIPA and/or Ansible config that is all backed by files on disk. It's been really easy to orchestrate what many would consider overkill in my homelab because literally everything is represented in a single Github repo.
I yanked vmfactory out and into a standalone repo if anyone is interested: https://github.com/whalesalad/vmfactory
https://github.com/Telmate/terraform-provider-proxmox ...
I run Ansible against Proxmox hosts to ensure the config on them is what I want, and then Terraform all VMs into place on them. It's not too far off from having your own mini-EC2, minus some of the nice trappings like load balancers.