MQTT turns 25

(andypiper.co.uk)

206 points | by andypiper 8 hours ago ago

51 comments

  • gorbypark 4 hours ago

    I think my first project that's still "in production" and used daily was taking an existing SVG map of a water distribution system (snowmaking and fire suppression pipelines/pumps/valves/etc at a major ski resort). I made a MQTT topic for each pump, valve and section of pipe along with some state for each (water direction, pump or valve on/off, pressures, etc). It's a website, so I used mqtt.js and jQuery to update the colours and fills of the SVG to represent the various states. It's statically hosted and there's a mqtt broker that's been running in a container, untouched for nearly 10 years now. Mqtt.js is running over a web socket, so if the state is updated, it just magically updates for everyone else.

  • kitd 6 hours ago

    About 15 years ago, Andy Stanford Clark's house became something of a news item, in the days before tweeting IoT devices were a thing:

    https://www.bbc.co.uk/blogs/technology/2009/06/things_that_t...

    The core protocol, devised in the days when 1 byte over a satellite link cost $1, is incredibly efficient and simple to implement.

  • this_user 5 hours ago

    Having recently used MQTT for a project, I can't say that I'm a huge fan. There are a lot of options in the protocol where it's not immediately clear what they do and why they are important, or in what combination they need to be used to make sure things work as intended, and the documentation is often not great at explaining. Part of this may also be on the Eclipse Mosquitto Python client that I had been using. I took me several days to figure out that the client was running into a race condition on a slower system that resulted in subscriptions to topics silently being ignored which caused the associated callbacks to malfunction despite following their documentation 100%. Overall, this has been one the messiest experienced I have ever had with a protocol, certainly one that isn't that old.

    • DannyBee 5 hours ago

      So your client just sucks here.

      My experience with the eclipse clients (paho, etc) is similar to yours in both python and C++, - it makes it overly complicated and seems very low level. It also is somewhat buggy because of the architecture.

      I believe (perhaps wrongly), it's all basically maintained by one person or close to it (only one person has contributed to the C++ client in the past 6 months, for example, and nobody has contributed in the past 3), so i kind of understand how it's gotten this way over time.

      I filed a simple PR (1 word change) to fix an obvious bug and it took 2 years to review it and accept it with no changes. Again not a complaint, just trying to portray how the state feels - a lot of libraries, bugs, and work, and few overworked people helping get it all done.

      I moved to other clients and the experience is much much better in python, rust, C#, and C++.

      Most of them have a good combination of high and low level API's, so if you just want to send a message on a topic, you don't have to worry about acks, retries, etc.

      But if you need control, you can get it.

      Honestly, i worry at this point that keeping paho/etc alive in this state is doing more harm than good - if they were officially dead it would at least force the issue. Right now you end up with users who have an experience like yours, and then either give up on MQTT or assume they are doing it wrong :)

      • vv_ 4 hours ago

        After reading your post, I went looking for good C/C++ implementations suitable for embedded development, and honestly, I couldn’t find many! MQTT is typically straightforward to implement, so most companies develop their own version to better adapt to the specific systems they use for data transmission. That said, maybe this is an opportunity to create a well-designed MQTT client implementation tailored specifically for embedded devices! :)

      • ta988 4 hours ago

        Could you list the clients you are using? Because like many I fell into paho and things are not great.

    • jorl17 5 hours ago

      My experience using MQTT has been through the paho python MQTT library, and, while we definitely have managed to get a lot done, it has been a terrible experience.

      Everything about it rubs me the wrong way. The API design, the poor documentation, even the way they seem to not adhere to typical python conventions.

      It looks like it's easy to start using it, but then the breadth of protocols and implementations start getting to you. At a point in time, I remember the only reliable way we had of knowing if we had successfully connected to a server was to try to subscribe to a topic twice and catch a specific error code on the on_connect message (which actually was documented as a success code at the time). I know how crazy it sounds and maybe there was a better way to do it, but, if there was, I don't think it was that easy to find.

      It's easy to complain, though. I'm very grateful so many people have worked to create this library. If they hadn't, I wouldn't have built what we've built with it. I really admire people who take on these huge projects.

    • HankB99 4 hours ago

      I've used the Paho APIs in Python, C and C++ and moved on to not using any of these APIs. Where possible I use mosquitto_sub and mosquitto_pub to support the protocol and then just read/write standard input/output. It's not a matter of bugs (though I have encountered at least one) but rather it's just easier to use a program that's been written and tested to manage the connection to the broker.

      The only place where I was unable to do this effectively was with the last will and testament message (and I might not have tried hard enough.) It also doesn't work with microcontrollers that don't run something like Linux.

    • qudat an hour ago

      While not directly competing with mqtt, https://pipe.pico.sh is a great pubsub tool that leverages ssh for communication, effectively creating an authenticated, networked *nix pipe system.

    • clumsysmurf 3 hours ago

      The eclipse paho MQTT Java client is also terrible, lots of open bugs, terrible API, and no release in over 4 years. For my case, I was able to use NATS instead.

      https://github.com/eclipse/paho.mqtt.java/issues

  • miljen 6 hours ago

    Fun fact: at the same time, the most famous C++ library, Boost, is reviewing the `async-mqtt5` implementation (https://github.com/mireo/async-mqtt5) to be included in Boost as Boost.MQTT: https://lists.boost.org/Archives/boost/2024/10/index.php

    • formerly_proven 5 hours ago

      Is Boost still something people reach for in newer projects? Anecdotally most adoption I've seen happened in the 00s and very early 2010s (pre everyone mandating C++0x/C++11), I only rarely see it around these days.

      edit: boost.org is a blast from the past, still looks exactly like I remember it from like 2008. Down to the "Get Boost" shopped on an emergency off button!

      • happosai 22 minutes ago

        Boost appears to have become some kind of staging area for stdlib. Stuff that becomes popular in boost get eventually included in stdlib.

      • spacechild1 5 hours ago

        AFAICT Boost.Asio is still the go-to networking library. (Though Asio is also available as a standalone outside of Boost, I didn't know that until recently.)

        Apart from that, Boost has quite a few goodies such as Boost.Json, Boost.Program_options, Boost.Interprocess, Boost.Lockfree, Boost.Unordered, Boost.Dynamic_Bitset, etc.

        Some libraries, like Boost.Atomic, Boost.Thread, Boost.Chrono and Boost.Filesystem, simply became obsolete with modern C++ versions. In fact, they served as the blueprint for the corresponding C++ standard libraries.

        Personally, I have been wary of using Boost in the past because it's such huge library, but CMake integration is actually quite good these days and I found it pretty easy to use. Documentation is also quite good IMO.

      • miljen 5 hours ago

        That is an interesting question. Here you can see the actual popularity of each library within Boost: https://grafikrobot.github.io/boost_lib_stats/. Also, see the comments at https://www.reddit.com/r/cpp/comments/130bzj8/has_boost_lost.... My company has been heavily utilizing Boost.Asio and Boost.Beast, so we've definitely reached for Boost in newer projects as well.

  • ronameels 5 hours ago

    MQTT is being used a lot more in recent years inside of factories for sharing data between machines. Historically it's been used in Oil & Gas for SCADA (getting data from remote well sites).

    10+ years ago we added it to Kepware (OPC server) and streamed tag values to "the cloud". I was at a conference giving a presentation on it when Arlen Nipper, one of the creators of MQTT, came up after the presentation and said I did a "decent job". It was humbling :). Fast forward to today, and we have a new company (HighByte) modeling factory data at the edge and sending it via MQTT, SparkplugB (protocol over MQTT), direct to S3, Azure Blob, etc, etc.

    All that to say, MQTT is a big driver in Industry 4.0, and it's cool to see it so heavily used all these years later.

    • doctoboggan an hour ago

      We are currently using the Kepware IoT plugin to stream around 800k tags per second via MQTT ultimately into a Victoria metrics DB. It's a little clunky and requires more processing steps than I would like. I am currently moving away from this solution (since Kepware charges a reoccurring license fee every year for the IoT plugin), and toward using telegraf to read the OPC-UA data directly from kepware without the IoT plugin.

      Do/Did you work at Kepware?

    • eska 2 hours ago

      I also work in that industry and am quite puzzled by the technological inferiority, scope creep and NIH syndrome of the OPC foundation’s standards. I wish they just used sparkplug b and implemented their specs on top of that for semantics.

      Then again the async stuff they’re doing now is so over-engineered and terrible as well.. I took part in their teams meetings some time, only to find they had never read the mqtt spec (less than 50 easy to read pages) and didn’t understand what the headers are for for example (they wanted to put stuff in there that actually belongs in the payload). Some microsoft guy also took offense at my suggestion to first look at what competitors are doing with mqtt, because he wanted to create something new rather than copy. My company will just treat opc ua at the very edge and isolate it from our tech as much as possible on my suggestion.

    • fartfeatures 5 hours ago

      Is MQTT being used in places where previously Modbus would have been used?

      • InDubioProRubio 3 hours ago

        There is also MQTT integrating bus-hardware- you basically plug in as bus-hardware replacement which allows to send data directly from the bus to the broker. The PLC never know what hit it.

        https://www.ifm.com/de/de/product/AL1332

        Pretty wild times.

        PS: You can web GET/POST set and get ios with these. Robot experience that tragic irony POST {NOOO} and he just dropped it like its hot.

      • ronameels 4 hours ago

        In some cases yes, controllers are adding in MQTT support directly or SparkplugB.

        It works in simple cases, but the lack of transactions & ability to control what and how often you're reading from the client side, can create challenges.

  • 8fingerlouie 5 hours ago

    It's such a great little protocol, and besides being "small enough" for hobby projects, it also scales enough to be used for something like Facebook Messenger [1]

    [1]: https://engineering.fb.com/2011/08/12/android/building-faceb...

  • bouvin 7 hours ago

    I have been using MQTT for years in my IoT courses – it has proven to be a very versatile tool, and it is convenient that it is also supported over web sockets.

  • Evidlo an hour ago

    I remember one of the annoying things I found about actually using MQTT was that many clients didn't implement any sort of connect/disconnect callback and expected you to handle reconnection logic (e.g. exponential backoff) yourself. Also it didn't support RPC style requests for some reason.

    Also since MQTT is TCP-based you can't natively speak the protocol from a webbrowser without the broker also listening on a websocket.

    Ended up switching to WAMP-proto [0], which checked all my boxes and just did everything over websockets by default.

    0: https://wamp-proto.org/

  • NortySpock 5 hours ago

    My most recent foray into MQTT was using the MQTT 3.1.1 compatibility mode in NATS, which was pretty easy to set up once I got the permissions sorted out.

    Works great for my smart home needs while also letting me explore latency and throughput load testing if I want to have a little fun.

    • artooro 4 hours ago

      I'm using NATS as well, natively when possible but bridged to MQTT for devices that need it.

  • joezydeco 6 hours ago

    I had a lot of fun on an embedded systems project recently where I used MQTT as an interprocess messaging system. Broker and clients on the same machine.

    If I needed to sniff or debug something it was as easy as putting the device on the network and using MQTT Explorer to record or inject messages. I could even let coworkers that were working remotely work with the system by opening the port outside the LAN.

    • klysm 5 hours ago

      Haven’t seen this done much but it does seem to have some desirable properties. My biggest concern when using it as a system component has been durability guarantees. I don’t have a lot of trust on broker implementations not to lose data.

      • logifail 5 hours ago

        > My biggest concern when using it as a system component has been durability guarantees. I don’t have a lot of trust on broker implementations not to lose data

        (If we ignore retained messages) MQTT brokers are not designed to store data, so there's no need to be concerned about durability because there simply isn't any...(!)

        It is (just) a messaging protocol.

      • joezydeco 4 hours ago

        MQTT, by design, isn't going to have the confirmation mechanisms and safeguards that you could get from other methods or even sockets. But once you release yourself from that paranoia, it's pretty liberating.

        The nature of my project was that a missed message wouldn't be fatal, and message retention was good enough protection for a process that crashed and restarted.

    • thebruce87m 4 hours ago

      We use ZeroMQ for this purpose. No broker needed.

      • omani 2 hours ago

        this statement is a bit confusing. because with zeromq, one side plays the broker (I dont mean the broker pattern). I mean it in the sense that someone, somewhere, needs to listen on a port. which in the end is the same as having a central mqtt broker sitting somewhere listening on a port. hence the statement "no broker needed" is a bit confusing.

        • thebruce87m 2 hours ago

          > one side plays the broker

          Hopefully most people understand that I meant that you don’t need a stand-alone broker, which on embedded systems means one more thing to deploy and worry about the license for.

          Edit: Also:

          > The philosophy of ZeroMQ starts with the zero. The zero is for zero broker (ZeroMQ is brokerless)

          https://zeromq.org/get-started/

    • roland35 4 hours ago

      Would zeromq be an alternative here?

      • joezydeco 4 hours ago

        It's an alternative, but there are some aspects to having the broker that I liked. If there was a bug I could halt the system, attach/subscribe an external dev machine, and grab all retained messages from the broker. I don't believe ZMQ allows that.

  • ErneX 5 hours ago

    Most of my home automations depend on it, it’s been great honestly. Cheers to mqtt!

  • vv_ 5 hours ago

    There aren’t many telemetry transfer protocols suitable for IoT besides MQTT (especially for resource constrained systems). It is incredibly simple to implement, has numerous libraries available in various programming languages, is supported by major cloud providers and is designed to operate over TCP (meaning you avoid DTLS).

    We have shipped many products that use MQTT for telemetry transfer to various different vendor servers and it always just worked. It was never a pain-point in stark contrast to CoAP or some proprietary protocols.

  • baruchthescribe 6 hours ago

    Reposting my comment from five years ago:

    I once had a client where the only port available for me to use on their firewalls was for MQTT (1883) because that's how we were getting sensor data from them. They would not open anything else for us no matter how we implored them so I wrote a live TCP wrapper over MQTT to get around it. It was a local multithreaded TCP daemon that listened for outbound requests on a certain port, wrapped them in MQTT and then published them using a unique topic. The server daemon would detect these topics and unwrap them before forwarding to our server processes. So the client machine thought it was making a live TCP connection to our server but in the middle was a funky invisible MQTT wrapper. It was really elegant once it worked but my goodness was it a pain to debug - a couple of months before I got the whole thing right because of all the blind alleys I went down.

    • ericvsmith 6 hours ago

      I’ve worked at places where subverting firewall rules like this would get you fired. I now refer to such situations where I’m forced to twiddle my thumbs while no work gets done as “letting the process work”.

      • exe34 6 hours ago

        I always kick this kind of decision upwards: we can do it this way and break this rule, so I need director level approval, or we can wait until the customer blinks.

        • n_plus_1_acc 6 hours ago

          Yes. Ask your boss for approval and tell them otherwise nothing will get done.

  • InDubioProRubio 3 hours ago

    A heap of dead "replacements" at the feets - what more can a communication standard want.

  • Uptrenda 7 hours ago

    The cool thing about MQTT is there's so many servers for it that are completely public -- and intentionally so. You can use it for real time messages but I think it also has quality modes that allow for some persistence.

    Some of these servers have stat pages that show how much data they transfer. Usually phenomenal amounts of data. It's pleasing to me as someone doing a lot of stuff with open source networking to have such an establish protocol (instead of say -- 10 crappy ones.) MQTT works and it works well.

  • sunshine-o 2 hours ago

    I never understood why we don't use MQTT for simple systems monitoring, with a push model.

    I mean it was almost designed for this use case.

    Also if using NATS things become very simple to setup.

  • dloranc 5 hours ago

    What do you think about LwM2M in comparison with MQTT?

    • vv_ 5 hours ago

      It is not directly comparable because LwM2M was originally developed for device management, while MQTT is primarily used for transferring telemetry or other opaque data.

      My personal gripe with LwM2M is that it uses CoAP (Constrained Application Protocol) as its primary transport protocol. Since CoAP is based on UDP, you’re forced to rely on DTLS for authentication and encryption. Managing sessions with DTLS can be quite frustrating, and unlike TCP, there’s no reliable indication of when a session is closed by your peer.

      Another issue is that it doesn't strictly follow the client-initiated communication pattern and can receive unsolicited messages from the server. I have found this to be very unreliable in real-world conditions for Device Management use cases.