For anyone mystified about what a NIF is that doesn't want to go read the docs.
The BEAM VM (which is the thing that runs erlang / elixir / gleam / etc) has 3 flavors of functions.
- BIFs - Built-in functions, these are written in C and ship with the VM
- NIFs - Natively implemented functions, these are written in any language that can speak the NIF ABI that BEAM exposes and allows you to provide a function that looks like a built-in function but that you build yourself.
- User - User functions are written in the language that's running on BEAM, so if you write a function in erlang or elixir, that's a user function.
NIFs allow you to drop down into a lower level language and extend the VM. Originally most NIFs were written in C, but now a lot more languages have built out nice facilities for writing NIFs. Rust has Rustler and Zig now has Zigler, although people have been writing zig nifs for a while without zigler and I'm sure people wrote rust nifs without rustler.
It’s important to note that while Erlang has protections against user code crashing an Erlang process and recovering, a faulty NIF can take down the entire virtual machine.
There's a series of things that a NIF must do to be a good citizen. Not crashing is a big one, but also not starving the VM by never yielding (in case the NIF is long-running) is important, plus a few secondary things like using the BEAM allocator so that tooling that monitors memory consumption can see resources consumed by the NIF.
The creator of Zigler has a talk from ElixirConf 2021 on how he made Zig NIFs behave nicely:
Is any of this code open source? As an outsider, I'm kind of at a loss for why anyone wants this or what you kids are doing over there and how offended I should be by it.
It's also important to point out ports, because as you mention, NIFs are a way to integrate external code. But as someone else points out, NIFs can crash the entire BEAM VM. Ports are a safer way to integrate external code because they are just another BEAM process that talks to an external program. If that program crashes, then the port process crashes just like any other BEAM process but it won't crash the entire BEAM VM.
Zig is also used in an excellent way by burrito[0]. I've also used zig for compiling NIFs written in C/C++/Objective-C, since `zig cc` makes cross-compiling much nicer.
I wish zig got more use and attention in the Erlang ecosystem, but rustler seems more popular.
Does anyone actually enjoy using these systems that encourage you to embed programming-language X code in programming-language Y heredocs?
I always find actually doing that — and then maintaining the results over time — to be quite painful: you don't get syntax highlighting inside the string; you can no longer search your worktree reliably using extension-based filtering; etc.
I personally find the workflow much more sane if/when you just have a separate file (e.g. `foo.zig`) for the guest-language code, and then your host-language code references it.
I've done some assembly in C, and for big functions, yeah, I want it in its own file, but smaller things often make sense to embed. I'm not sure if I'd like my nif code embedded into my erl files (assuming this works for Erlang as well), but it could conceivably make the nasty bit of boilerplate around ERL_NIF_INIT in the NIF (which I have to do in C anyway) and exit(nif_library_not_loaded) in the erl go away, which would be nice.
It's certainly possible to get syntax highlighting on the embedded code, but you'll need to work with your syntax highlighter; it certainly helps if you're not the only person using it.
But then again, I worked without syntax highlighting for years, so I'm happy when it works, but when it doesn't, I'm ok with that too.
But, if all you do is write elixir wrappers around the zig function, to completely hide the foreign language functions, keeping both the wrapper and implementation in the same file, even if two different languages doesn't seem horrible, but again, keeping them in two file doesn't seem like a huge difference too
I think its really a matter of taste, both options viable
I use zig a lot in elixir nif, for things like audio and video processing, it works great. But I do not use zigler as I prefer the code to live in their own codebases. But zigler is really nice and it provides an easy way to do computational heavy tasks in elixir.
For anyone mystified about what a NIF is that doesn't want to go read the docs.
The BEAM VM (which is the thing that runs erlang / elixir / gleam / etc) has 3 flavors of functions.
- BIFs - Built-in functions, these are written in C and ship with the VM
- NIFs - Natively implemented functions, these are written in any language that can speak the NIF ABI that BEAM exposes and allows you to provide a function that looks like a built-in function but that you build yourself.
- User - User functions are written in the language that's running on BEAM, so if you write a function in erlang or elixir, that's a user function.
NIFs allow you to drop down into a lower level language and extend the VM. Originally most NIFs were written in C, but now a lot more languages have built out nice facilities for writing NIFs. Rust has Rustler and Zig now has Zigler, although people have been writing zig nifs for a while without zigler and I'm sure people wrote rust nifs without rustler.
It’s important to note that while Erlang has protections against user code crashing an Erlang process and recovering, a faulty NIF can take down the entire virtual machine.
There's a series of things that a NIF must do to be a good citizen. Not crashing is a big one, but also not starving the VM by never yielding (in case the NIF is long-running) is important, plus a few secondary things like using the BEAM allocator so that tooling that monitors memory consumption can see resources consumed by the NIF.
The creator of Zigler has a talk from ElixirConf 2021 on how he made Zig NIFs behave nicely:
https://www.youtube.com/watch?v=lDfjdGva3NE
Hence why Rustler is of so much interest since it provides more protections against this happening.
Discord is a big Erlang + Rustler user.
Are they really? Their projects don't look so active
Is any of this code open source? As an outsider, I'm kind of at a loss for why anyone wants this or what you kids are doing over there and how offended I should be by it.
https://github.com/discord/sorted_set_nif
Do you mean Rustler?
Yes, it's Apache 2.0
https://github.com/rusterlium/rustler
Don't be like me and do a 20ms page fault in a NIF
It's also important to point out ports, because as you mention, NIFs are a way to integrate external code. But as someone else points out, NIFs can crash the entire BEAM VM. Ports are a safer way to integrate external code because they are just another BEAM process that talks to an external program. If that program crashes, then the port process crashes just like any other BEAM process but it won't crash the entire BEAM VM.
Zig is also used in an excellent way by burrito[0]. I've also used zig for compiling NIFs written in C/C++/Objective-C, since `zig cc` makes cross-compiling much nicer.
I wish zig got more use and attention in the Erlang ecosystem, but rustler seems more popular.
Does anyone actually enjoy using these systems that encourage you to embed programming-language X code in programming-language Y heredocs?
I always find actually doing that — and then maintaining the results over time — to be quite painful: you don't get syntax highlighting inside the string; you can no longer search your worktree reliably using extension-based filtering; etc.
I personally find the workflow much more sane if/when you just have a separate file (e.g. `foo.zig`) for the guest-language code, and then your host-language code references it.
Syntax highlighting here can work correctly, actually.
Also, I'm not sure why it's not better documented in Zigler, but you can also write the code in a separate file just fine.
I've done some assembly in C, and for big functions, yeah, I want it in its own file, but smaller things often make sense to embed. I'm not sure if I'd like my nif code embedded into my erl files (assuming this works for Erlang as well), but it could conceivably make the nasty bit of boilerplate around ERL_NIF_INIT in the NIF (which I have to do in C anyway) and exit(nif_library_not_loaded) in the erl go away, which would be nice.
It's certainly possible to get syntax highlighting on the embedded code, but you'll need to work with your syntax highlighter; it certainly helps if you're not the only person using it.
But then again, I worked without syntax highlighting for years, so I'm happy when it works, but when it doesn't, I'm ok with that too.
I initially agree
But, if all you do is write elixir wrappers around the zig function, to completely hide the foreign language functions, keeping both the wrapper and implementation in the same file, even if two different languages doesn't seem horrible, but again, keeping them in two file doesn't seem like a huge difference too
I think its really a matter of taste, both options viable
Completely lacking a description that made it clear, but basically, from what I can tell, this lets you embed Zig code inside Elixir code
I use zig a lot in elixir nif, for things like audio and video processing, it works great. But I do not use zigler as I prefer the code to live in their own codebases. But zigler is really nice and it provides an easy way to do computational heavy tasks in elixir.
Great! But, what is a nifs, please? :'D
Natively Implemented Functions
https://www.erlang.org/doc/system/nif.html
Are sigils (~) restricted to one char? To me seems ~Zig would be more clear and short enough.
Erlang sigils are not, they can be any length, limited to characters allowed in atoms.
Elixir sigils also allow multiple characters in the name, but chars after the first must be upper case, according to the docs.
So for Elixir, it would have to be something like ~zIG
According to the docs, must be all upper case:
> Custom sigils may be either a single lowercase character, or an uppercase character followed by more uppercase characters and digits.
https://hexdocs.pm/elixir/sigils.html
Ah yeah, you're right.