A broker that works is not the same as a broker you can read

Up to here the series has been about making things happen: installing EMQX, connecting a client, writing an authentication rule, sending data on to somewhere else. Everything in this part is about reading what already happened. That is a different skill, and it is the one that saves your evening.

Three ordinary situations, and where each of them is answered.

None of those needs a terminal, and none of them needs guesswork. The routine below is the spine of this whole part: each decision lands on a page covered in one of the sections that follow.

flowchart TD
  A["A device is set up.
Nothing arrives"] --> B{"Is it listed in
Monitoring, Clients?"}
  B -->|"not at all"| B1["It never connected.
Check the listener,
the credential, the ACL"]
  B -->|"listed, offline"| B2["The session outlived
the connection. Read
Session Information"]
  B -->|"listed, connected"| C{"Is the topic listed
under Subscriptions?"}
  C -->|"no"| C1["It subscribed to a
different topic filter.
Compare them character
by character"]
  C -->|"yes"| D{"Does a test publish
from WebSocket Client
reach it?"}
  D -->|"it arrives"| D1["The broker is fine.
The publisher is
the problem"]
  D -->|"nothing"| D2["Check QoS and No Local
on that subscription row"]

Who is connected, and who only looks connected

The Clients page shows the clients connected right now — and the sessions that have not expired yet. Those are two different things, and mixing them up is the reason the list sometimes disagrees with what you can see with your own eyes.

A connection is one live MQTT channel between a client and EMQX. A session is the state EMQX keeps on that client's behalf: its subscriptions and its queued messages. When a connection drops, the session does not necessarily go with it. If the client asked for a persistent session and gave a session expiry interval, the session lives on for that long, waiting for the device to come back.

stateDiagram-v2
  direction LR
  state "Connected" as C
  state "Disconnected, session kept" as D
  state "Session expired and gone" as G
  [*] --> C: the client connects
  C --> D: link drops, or Kick Out
  D --> C: reconnects in time
  D --> G: expiry passes
  C --> G: no persistent session
  G --> [*]

Opening the list and finding one device

1. Step 1

   Open the Clients list

   In the sidebar, pick Monitoring → Clients. It shows the currently connected clients by default, with columns for client ID, username, connection status, IP, heartbeat, session information and the time the connection completed.

2. Step 2

   Find one device with a filter

   The search bar at the top does a fuzzy search on client ID or username. Expand the arrow on its right for the proper filter fields: connection status, a time range and a target IP. Select Column at the top of the list chooses which columns are shown; Refresh resets every filter and reloads from scratch.

3. Step 3

   Look at one client in detail

   Click a client ID to open its connection detail page. The top right lets you refresh by hand and clear the session by hand. Further down is the list of topics this connection currently subscribes to.

4. Step 4

   Kick a client out, if you have to

   Back in the list, tick a client and press Kick Out to break the connection. Read the next callout before you rely on this.

The IP address column puts the client's source IP and the port it came in on together in one field, which is handy when two devices sit behind the same router. The detail page then adds three things the list does not carry: the protocol version the connection uses — MQTT 3.1.1 or 5.0, for example — whether the session is to be cleaned up once the client goes offline, and, if it is offline already, the time it last went offline.

The top of the detail page splits into two panels: Connection Information on the left and Session Information on the right. The session panel is the one worth learning, because it holds six fields that explain most “it is connected but slow” complaints.

Below those two panels come the traffic, message and packet statistics, which make it easy to read the in and out volume of a single client rather than the whole broker. The bottom of the page lists the topics it subscribes to right now, with Add Subscription to add a simple subscription by hand and Unsubscribe in the list to cancel one.

What the device actually asked for

The Monitoring → Subscriptions page lists every subscription on the broker as one row per client ID and topic pair. This is where you find out that the device subscribed to home/sensor/temp while you have been publishing to home/sensors/temp all afternoon. There is no clever way to catch that; you read the two strings side by side.

Each row also carries the QoS and the new MQTT 5.0 subscription options. Three of them come up often enough to be worth knowing by name. Read them as three separate settings rather than three dials on the same thing: one is about your own messages being sent back to you, and the other two are about the RETAIN flag — one on the way out, one on subscribe.

The search bar on this page carries three filter fields by default — Node, Client ID and Topic — and expanding the arrow adds two more: QoS and the shared subscription name, Shared Name.

On a Topics row, Create Monitor in the Actions column takes you to Diagnose → Topic Metrics to set monitoring up for that topic. That is an Enterprise page — more on it later in this part.

When it is online but slow (Enterprise)

Being connected and being served promptly are different problems. When a device is clearly online but slow to receive messages, EMQX's Slow Subscriptions measures the latency from the moment a message reaches EMQX to the moment it finishes being sent. Enable it under Diagnose → Slow Subscriptions, and four settings become yours to set.

The resulting list is sorted from the longest latency down, with five columns: Client ID, Topic, Duration, Node and Updated. Clicking a Client ID opens that client's detail page, which puts you back in the previous section with a specific device to look at.

Two kinds of number, and four ways to read them

A broker that works is not the same as a broker that is healthy under load. To keep the message backbone of a house running you want to know how many connections and subscriptions there are right now, how much memory the node has left, and whether the rate of messages in and out has blown up. The Monitoring module gathers all of that in one place.

Before the pages themselves, the distinction that everything else rests on. EMQX splits its monitoring data into two kinds:

Cluster Overview and the Nodes tab

1. Step 1

   Open the Monitoring module

   In the sidebar, choose Monitoring → Cluster Overview. The first thing on the page is the cluster-level connection, subscription and topic counts.

2. Step 2

   Switch the group to Nodes

   In the top half, switch to Nodes to expand every node in the cluster. A home add-on install is usually a single node, so the first entry here is your add-on itself.

3. Step 3

   Read the resources and the version

   The node row shows the connection count, the version, uptime, the number of Erlang processes and memory and CPU usage. The Version column is the most direct place to confirm that the EMQX you are running is 5.8.9. Click the node name to open the node details, where the system paths and the log path are written out.

4. Step 4

   Use the Metrics tab for the cumulative counters

   Switch to the Metrics tab and work through the four groups of counters one at a time. Always read them together with the time range you have selected — a counter without a window around it is not information.

A node drawn in gray has stopped. That is worth committing to memory, because the instinct on seeing flat numbers is to assume the chart is broken. Check the node status column first. The time series chart below the list can be set from the last 1 hour or 6 hours up to 7 days, which is where you watch connection counts and message volume trend rather than twitch. If you want the Dashboard to start counting again from zero, Reset Monitoring Data clears what has accumulated so far.

On the Metrics tab, EMQX's counters come in four dimensions:

The Statistics side has its own names, and these are the ones you will see again on the REST API and in Prometheus, so they are worth reading once slowly. Most of them are shown as a current value alongside an all-time maximum, which is what makes them useful for a long-term view.

Four ways out of the Dashboard

The Dashboard is only one of the doors onto this data. If you would rather not open it — because you want a chart on the wall, or an alert on your phone — there are three other routes.

flowchart LR
  E["EMQX 5.8.9"] --> D["Dashboard
Monitoring pages"]
  E --> R["REST API
the same numbers, as JSON"]
  E --> S["System topics
starting with $SYS/"]
  E --> P["Prometheus"]
  P --> G["Grafana charts
Alertmanager alerts"]

Prometheus is the one most people end up using, because it is what puts EMQX on the same chart as everything else you measure on the host, and what lets you draw a Grafana dashboard at home and have Alertmanager tell you when something is off. On the Dashboard's Management → Monitoring page, open the Integration tab and choose the Prometheus settings. EMQX supports two modes.

Pull is what most people use, and the official EMQX documentation recommends Pull too — precisely because the Pushgateway mode currently covers only the basic metrics and is not as complete as auth or data_integration.

Retained messages, and the two things that are not one

MQTT works on publish and subscribe, and the broker does not file messages away and keep them forever. A message goes out to whoever is subscribed at that instant, and then it is gone. The one exception is the retained message, and understanding it answers most of the “where is that value now” questions in a smart home.

When a client publishes a message with the RETAIN flag set, EMQX stores it. From then on, any client that newly subscribes to that topic receives that message immediately, without waiting for the next publish. By default it never expires, unless you delete it by hand.

sequenceDiagram
  participant P as A thermostat
  participant B as EMQX
  participant S as A dashboard, later
  P->>B: PUBLISH with RETAIN set
  B->>B: keep it as this topic's retained message
  S->>B: SUBSCRIBE, hours afterwards
  B->>S: the stored message arrives at once
  P->>B: PUBLISH an empty body with RETAIN
  B->>B: the stored message is cleared

The Retained Messages page

1. Step 1

   Open the list

   In the sidebar, choose Monitoring → Retained Messages to see every retained message in the system right now — topic, QoS, publisher and publish time.

2. Step 2

   Read one message's payload

   In that row's Actions column, press Show Payload and the content opens below. You can pick a format such as JSON or Hex, and Copy at the bottom right copies it out.

3. Step 3

   Delete one, or all of them

   Press Delete on the row to remove one. Clear All clears the retained messages across the whole cluster — read the warning below before you use it.

4. Step 4

   Try it yourself with the WebSocket Client

   In the sidebar, choose Diagnose → WebSocket Client, add a connection, and use subscribe and publish to check a topic and its retained behavior in a few seconds. This is also the fastest way to prove the broker is publishing at all when a device says it is not receiving.

At the top of that list, Refresh reloads it and Settings takes you to Management → MQTT Settings → Retainer, where the feature can be turned on or off and five parameters set.

Topic Metrics: one topic, counted (Enterprise)

Diagnose → Topic Metrics counts message volume for one specific topic and nothing else. Add a monitor there, or use Create Monitor on a Topics row, and give it a topic name. In the list's Actions column, View opens the details broken down by QoS, Reset starts the count over and Delete removes the entry.

Delayed Publish: the other thing that is not retain (Enterprise)

Delayed Publish is an EMQX extension to MQTT. When the topic a client publishes to starts with $delayed/, EMQX holds the message back for a while and then publishes it on the real topic. The format is:

{DelayInterval} can go up to 4294967 seconds. If it cannot be parsed as an integer, EMQX drops the message, so a typo in that segment costs you the message rather than delaying it. To manage the feature, open Management → Delayed Publish in the sidebar, where you can enable or disable it and cap the number of delayed messages.

Your change did nothing, and here is the chain that explains it

EMQX parameters are scattered across several places: the ones you adjust dynamically in the Dashboard, the ones tucked away in a config file, and the ones you override with an environment variable. In a home setup it is easy to write the same setting into two layers by accident and then find your change does nothing, with no obvious way to tell which layer won.

Start from the two kinds of directory. Static configuration lives in etc and is usually read-only — the main config file, emqx.conf, is here, and you mostly change it at deployment or upgrade time. Dynamic configuration lives in data/configs and is writable: changes you make from the Dashboard, the REST API or the CLI are saved into cluster.hocon.

Then the chain. Precedence runs lowest to highest as base.hocon — which only exists from 5.8.4 on — then cluster.hocon, then emqx.conf, then environment variables. The further along that chain, the higher it wins.

flowchart LR
  A["base.hocon
weakest"] --> B["cluster.hocon
Dashboard writes here"]
  B --> C["emqx.conf
the static file"]
  C --> D["EMQX_ variables
strongest"]

The half of this you can do without a restart

Everything so far reads like configuration is a file you edit and a service you restart. Half of it is not. The Dashboard's Management module is the way in to dynamic changes, and four areas live there: Cluster Settings, MQTT Settings, Logging and Monitoring. Those can all be applied while EMQX is running, with no restart.

Another way to hold the split: a Management page is a switch you flip on a machine that is already running, while emqx.conf and env_vars are edits to the instructions the machine reads when it starts. The first shows up the moment you save. The second waits for the next start.

To see them, open the sidebar and choose Management → Cluster Settings. That is where you adjust MQTT, Listener and Logging settings, and they take effect as soon as you save. There is no file to find and nothing to restart. What you save is still a dynamic change, so it lands in cluster.hocon — which is why a higher layer can overrule it later, exactly as the chain above says.

HOCON, and the two forms of the same line

From EMQX 5.0 on, configuration is written in HOCON, a superset of JSON. That means you can use nested objects or flat paths, and they mean the same thing. The nested form opens a node { } block and sets name and cookie inside it; the flat form puts the whole path on one line:

Environment variables and the add-on's env_vars

Three rules convert a config-file path into an environment variable, and all three have to hold or nothing happens.

In the Woow EMQX add-on this is exposed through the env_vars option in config.yaml. Each entry is a name and a value pair:

1. Step 1

   Open the add-on's configuration

   Home Assistant → the EMQX add-on in the sidebar → Configuration. Fill in name and value under env_vars.

2. Step 2

   Use a name the add-on will accept

   The add-on's config.yaml validates environment variable names against the regular expression ^EMQX_([A-Z0-9_])+$. Only names starting with EMQX_, in capitals, get through. A misspelled name shows up as a warning in the startup log rather than as a rejected form.

3. Step 3

   Save and restart the add-on

   Nothing takes effect until you save and restart. This is the single most common reason an env_vars entry “does not work”.

4. Step 4

   Read the value back in the Dashboard

   Change one thing, restart, and confirm the new value in the Dashboard before you change the next. If it did not take, you now know it is that one entry and not a combination.

Secrets, and the comment trap that eats your password

EMQX configuration has a Secret type for sensitive values such as passwords and tokens. Two habits go with it. First, whenever you write down what goes into an add-on field or a Dashboard field — in a note to yourself, in a document, in a message to somebody helping you — use a placeholder such as <your-password> in place of the real value, and never the actual secret.

Second, and this one bites people who did nothing wrong: in HOCON, # starts a comment. If a password contains a # — say MQtt#123 — and it is not quoted, the parser reads it as MQtt and throws #123 away as a comment. Your password is now four characters long and you have no idea why login fails. Wrap it in HOCON-level double quotes so the whole literal survives:

Note the shape of that: single quotes on the outside for the shell, double quotes on the inside for HOCON. Do the same for any value containing : or =. And do not reach for URL encoding to dodge the problem — %23 does not help here, because EMQX does not decode URL encoding in environment variables.

The symptom you see, and the page that explains it

These are the situations that come up most across the four areas in this part, matched to what to do about them. Read the symptom column first; most of these look like faults and are not.

The ones that come up again and again

Where to go from here