The Dashboard shows you that something is wrong, not why

Everything so far in this series has been about making the broker do things: connect clients, check passwords, decide who may publish where, move messages into other systems. Part 7 added the pages that show you the state of it all. Those pages are excellent at the general question and useless at the specific one.

Four questions, three sets of tools. The first two are answered by the log and by Log Trace, with the Diagnose module and the REST API alongside them. The third is answered by the ngrok tunnel built into the add-on, and by a hard prerequisite that comes before it. The fourth is answered by knowing what a backup of /data/emqx actually contains — and, just as importantly, what it leaves out.

Everything below is written against EMQX 5.8.9 (Open Source) as shipped in the Woow EMQX add-on. Where a feature belongs to EMQX Enterprise instead, it says so — several of the Diagnose tools do, and so does the Dashboard's own backup page.

Six levels, and the one you should be sitting on

EMQX writes its log to two output streams: Console Log, out to the console, and File Log, into a file. Both are adjusted on the same Dashboard page, at Management → Logging, which has a tab for each. The default level for both is warning, and a change takes effect immediately — there is no restart to plan around.

EMQX supports 6 levels, which are 6 of the 8 defined in RFC 5424. From weakest to strongest:

The source guide describes five of the six in words. They are worth reading slowly, because picking a level is really a question of which of these you want to see:

• debug — fine-grained debugging data such as variables and functions.
• info — minor anomalies such as an authorization denial, and the result of a configuration change that succeeded.
• warning — things that may need a look: a dropped connection, a connection timeout, an authentication failure.
• error — cannot reach an external database, a subscription that does not exist, and similar errors.
• critical — a configuration error that stops a component from starting, for example.

notice sits between info and warning in that ordering and the source does not describe it separately, so this article does not invent a description for it. What matters in practice is the shape: everything at or above the level you set gets written, everything below it does not. Sitting at warning means you keep dropped connections and authentication failures, and you throw away the per-variable detail that would fill the disk.

What else the file handler lets you set

The File Log tab carries more than a level. The defaults are sensible and worth knowing before you change them:

Log Throttling, and why your log looks incomplete

EMQX also has Log Throttling. Within a time window it records only the first occurrence of a repeated event and counts the rest, which is what keeps one misbehaving device from producing a hundred thousand identical lines overnight. It is on by default, with a window of 1 minute.

Debug on one client, instead of on the whole broker

Turning the whole node down to debug to investigate one device is the blunt instrument. Log Trace is the precise one: live debug-level logging aimed at a single named target and nothing else. That is what makes it usable in production, where the broker is still carrying everybody else's traffic while you investigate. You will find it under Diagnose → Log Trace.

1. Step 1

   Diagnose → Log Trace → Create

   The list page holds whatever traces are currently running. Create opens the form.

2. Step 2

   Choose the Type, and give it an exact target

   Under Type you choose Client ID, Topic, IP Address or Rule ID. Client ID and IP Address have to be entered in full — a partial client ID matches nothing. Topic is the only type that takes wildcards, and the wildcard rules are the ordinary MQTT ones: + stands for exactly one level, and # is allowed only at the end of the filter.

3. Step 3

   Set the start and end times, then Create

   Collection starts as soon as the trace is created. Bounding it with an end time is the difference between a trace that stops on its own and one you have to remember to go back and clear up.

4. Step 4

   View or download the log from the list

   Each trace in the list can be viewed in the browser or downloaded. The files also live on the server itself, in the /data/trace directory, which is where to look if the Dashboard is not cooperating.

Two limits are worth committing to memory before you start creating traces liberally. The system runs at most 30 traces at a time. And each node caps its trace logs at 512MB — once that is full it stops appending and raises a warning in the main log, so a trace that seems to have gone quiet may simply have hit the ceiling.

flowchart TD
  A["Warning level is not telling you
enough about this problem"] --> B{"Can you name one client ID,
IP address, topic filter or rule ID?"}
  B -->|"yes"| C["Log Trace, under Diagnose.
Debug detail for that target only,
while the rest of the broker
stays at its normal level"]
  B -->|"no, it is broker-wide"| D["Management then Logging.
Drop the level for the
length of the investigation"]
  C --> E["The file lands in /data/trace.
Capped at 512MB per node,
at most 30 traces at once"]
  D --> F["At debug, Log Throttling is off,
so repeated events are no
longer folded into a count"]

What else is in the Diagnose module

Diagnose is the folder where the Dashboard keeps its debugging tools. Five things live there, and three of them belong to EMQX Enterprise:

The one you will use constantly is WebSocket Client. When a device is not receiving what you think it should, opening a connection here and subscribing to the same topic filter settles in ten seconds whether the message is reaching the broker at all — which splits the problem cleanly into “the publisher is not publishing” and “the subscriber is not subscribing.”

Two ways to drive the same engine

The Dashboard is one interface onto EMQX. The REST API is the other, and everything you can do by clicking has an HTTP equivalent. It follows the OpenAPI 3.0 specification, and every path starts with /api/v5. There is a browsable copy on the broker itself: open http://<host>:18083/api-docs/ and you can try calls straight from the Swagger UI.

Authentication is the part that catches people. The REST API authenticates with an API Key, which you create in the Dashboard under System → API Key. It arrives as a pair. Over HTTP Basic, the API Key is the username and the Secret Key is the password. And the rule that produces most of the confusion: a Dashboard user account cannot call the API directly. That has been true from EMQX 5.0 onward.

flowchart LR
  subgraph U["A Dashboard user account"]
    direction LR
    U1["Sign in on port 18083"] --> U2["Full use of the
Dashboard in a browser"]
    U1 --> U3["A call to any path
under /api/v5:
401 Unauthorized"]
  end
  subgraph K["An API Key from System then API Key"]
    direction LR
    K1["HTTP Basic:
API Key as the username,
Secret Key as the password"] --> K2["Any path under /api/v5"]
    K2 --> K3["JSON comes back"]
  end

The call to prove it with is the node list. This is one command spread over three lines, with the trailing backslashes joining them:

The two angle-bracketed pieces are placeholders. Substitute the Key and the Secret you were shown when you created the API Key — and nothing else in the command changes.

The health check endpoint

If you are putting a load balancer in front of EMQX, there is a purpose-built endpoint for it:

A healthy node answers 200, meaning it can take connections. A node that has been evacuated — or has already left the cluster — answers 503. That is precisely the shape a health check in HAProxy or nginx wants, and it means a node being drained stops receiving new connections without anyone editing the balancer's configuration.

One switch that puts 1883 on the public internet

The EMQX on your home network accepts connections from the local network only. Outside devices cannot reach it at all, which is a security posture rather than a limitation — until the afternoon you want to check the living-room sensor after you have gone out, or turn off the hallway light from somewhere else. Then you need a path between the local network and the outside world.

From add-on version 5.9.0 onward, Woow EMQX has ngrok built in, and one setting turns raw MQTT on port 1883 into a public TCP tunnel. ngrok is a tunneling service: an agent runs on your machine, opens a connection out to the ngrok cloud, and traffic arriving from outside is forwarded back down that connection to a local port.

Turning it on

1. Step 1

   Check your authentication first

   In the EMQX Dashboard, go to Access Control → Authentication and create a user, then set least privilege under Authorization. At a minimum, make sure that going public will not leave anonymous connections from the internet open.

2. Step 2

   Turn on the add-on's ngrok settings

   In Home Assistant, go to Settings → Add-ons → Woow EMQX and open the Configuration tab. Find the three ngrok options and set ngrok_enabled to true.

3. Step 3

   Fill in the authtoken, which is required

   Paste your ngrok authtoken into ngrok_authtoken — shown throughout this guide as <your-ngrok-authtoken>. If you have already reserved a TCP address on your ngrok account, put it in ngrok_tcp_addr; otherwise leave that one empty.

4. Step 4

   Restart, and read the log

   Go back to the Overview tab and click Restart. Once the add-on is up, open the Log and look for a line of this form:

   >>> MQTT ngrok: <public_url>

   That public address, port included, is what an outside MQTT client connects to. It is not your Home Assistant address on the local network, and configuring the remote client with the local address is the most common reason a connection from outside fails.

The three options, and exactly what each one does

Three options control the whole feature. Their types and defaults come straight from the add-on's config.yaml, where ngrok_enabled is declared bool with a default of false, ngrok_authtoken is password? and ngrok_tcp_addr is str?.

The authtoken is a secret in the full sense — it carries the permissions on your ngrok account. Do not write your own token into a document, and do not paste it into a log.

flowchart TD
  A["The add-on starts"] --> B{"ngrok_enabled"}
  B -->|"false"| C["The ngrok service idles.
Nothing is exposed"]
  B -->|"true"| D{"Is ngrok_authtoken filled in?"}
  D -->|"empty"| E["The service logs an error
and stays idle.
EMQX itself keeps running"]
  D -->|"filled in"| F{"Is ngrok_tcp_addr set?"}
  F -->|"empty"| G["Runs ngrok tcp 1883.
ngrok assigns a temporary address,
which can change on every restart"]
  F -->|"a reserved address"| H["Runs ngrok tcp 1883 with the
remote-addr option set to it.
The endpoint survives a restart"]
  G --> I["ngrok-announce prints the
public URL into the add-on log"]
  H --> I

The runtime behavior in words, because it is worth being able to predict: with ngrok_enabled set to false, the ngrok service simply idles. Enabled but with an empty authtoken, the service logs an error and stays idle — and EMQX itself keeps running as usual. Only once the authtoken has a value does it run ngrok tcp 1883; and if you filled in ngrok_tcp_addr, it runs ngrok tcp 1883 --remote-addr=<address> instead.

If you only want outside access now and then, the simplest setup is an authtoken with everything else left empty, and let ngrok assign a temporary address.

Where the public address comes from

When ngrok is enabled the add-on runs a second service called ngrok-announce. It polls the ngrok agent's own local API at http://127.0.0.1:4040/api/tunnels every two seconds, for up to about 120 seconds, and once it has the public URL of the first tunnel it prints it into the add-on log.

So the place to read the public URL is Home Assistant → Settings → Add-ons → Woow EMQX → Log. If it is not there immediately after a start, the service is most likely still waiting for ngrok to come up, which takes up to about two minutes. If it still has not appeared after that, look at the ngrok service's own log — the usual cause is an authtoken that was not filled in correctly.

The risks, stated plainly

Exposing raw MQTT is not a matter of pushing a port out and forgetting about it. Read these four before you decide:

• Anyone can attempt a connection. The ngrok address is public by definition, so it hands the broker's 1883 entrance to the whole internet. With Authentication not set up, EMQX lets anonymous clients through.
• TLS is a separate job. The ngrok TCP tunnel is raw TCP and it does not encrypt the connection for you. If you need encryption, switch to or add an MQTTS (8883) listener.
• A temporary address does not last. With no reserved address filled in, the endpoint ngrok assigns can change on every restart.
• Keep exposure minimal. If you need it only occasionally, open it for a set window and close it again when you are done. For long-term exposure, finish authentication, the ACL and whatever TLS you need first.

When it does not come up

• Enabled, but no URL in the log. First confirm that ngrok_enabled is true and the authtoken is filled in, then check whether the log holds an “enabled but authtoken is empty” error. If both are right and there is still nothing, restart the add-on or look at the ngrok service's own log.
• All you see is “ngrok not enabled”. The setting has not taken effect. Go back to the Configuration tab, check ngrok_enabled, restart, and read the log again.
• An outside client cannot connect. Check that it is using the full URL and port that ngrok printed rather than a local-network address, that you have created the account in EMQX, and that the ACL allows that topic. If the address keeps changing on you, switch to a reserved address in ngrok_tcp_addr.
• Any client on the internet can get into 1883. Turn ngrok off first. Go back to EMQX, set up Authentication and Authorization, and only then decide whether to expose it again. A public broker that accepts anonymous connections is an obvious warning sign, not an edge case.

What /data/emqx holds, and what it quietly leaves out

Over time EMQX builds up two kinds of thing you do not want to lose. The configuration is the structure: Authentication, Authorization, rules, connectors and sinks. The data is what sits in the built-in database: accounts, API keys, the blacklist and retained messages. Lose either and you are rebuilding by hand from memory.

EMQX keeps its runtime data in one place, the data directory, and the add-on puts it at /data/emqx. The add-on README states the principle in one line: a backup contains everything under /data/emqx. Here is the exact scope, including the one path that is not in it:

flowchart LR
  A["A Home Assistant backup
takes the add-on's /data"] --> B["/data/emqx/data
EMQX data and the persisted
built-in database"]
  A --> C["/data/emqx/etc
config files"]
  A --> D["/data/emqx/plugins
plugin files"]
  E["/config/log, and any certificate
or ACL file stored outside
the data directory"] -.-> F["Not in the archive.
Copy these yourself,
before you need them"]

Two ways to take one

A Home Assistant backup covers the add-on's /data, and therefore /data/emqx with it. That makes it the least-effort way to get everything back in one go, and it is the habit worth forming first.

1. Step 1

   Create a backup in Home Assistant

   Go to Settings → System → Backups and click Back up now. A full backup takes in the add-on's /data. Then sync the backup file somewhere safe — a backup that only exists on the machine you are protecting is not a backup.

2. Step 2

   Optionally, export a config bundle from the EMQX CLI

   Run emqx ctl data export in the add-on terminal or inside the container. It packs the configuration and the built-in database into a file named on this pattern:

   emqx-export-YYYY-MM-DD-HH-mm-ss.sss.tar.gz

   and writes it into the backup subfolder of the data directory. This is the file-level route, and it suits you when you want the configuration on its own, or want to move it to another EMQX of the same version. The matching command to bring it back is emqx ctl data import.

3. Step 3

   Know the order you would restore in

   To restore the whole host, roll straight back to that backup. If you are replacing EMQX only — after a reinstall, say — stop the add-on first, put the contents of /data/emqx back where they belong, then start EMQX. Finally, check in the Dashboard that the accounts and the ACL rules are all there. Checking is part of the procedure, not an optional flourish.

4. Step 4

   Update the add-on, backup in hand

   On the add-on page, click Check for updates at the bottom. Only click update once a new version actually shows up — 5.9.0, for example — and restart as prompted when it finishes. Take a backup before you update. If a release only adds ngrok and leaves the EMQX core alone, the regression risk is relatively low, but low is not zero.

What an export archive actually covers

Read against the EMQX documentation, an export tar usually covers:

• Authentication and authorization settings.
• Data Integration — Rules, Connectors, Sinks and Sources.
• Listener and Gateway settings.
• The built-in database: Dashboard users and REST API keys, client password-based authentication and enhanced authentication, PSK, Authorization rules, and the Blacklist.
• Retained messages.

TLS certificates and ACL files that sit inside the data directory travel with it. Ones that sit outside it do not, and you have to move those back by hand before you restore.

The conditions on a CLI import

When you import from the CLI you can name the file by absolute path, by relative path, or — once it is in the backup subfolder of the data directory — by file name alone. The EMQX 5.8 documentation attaches four hard conditions:

• The EMQX node has to be running when you import.
• In a core plus replica cluster, import on a core node only.
• Data exported from the Enterprise edition cannot be imported into Open Source.
• The file you import must not be renamed.

When the restore does not go to plan

• /data/emqx is not in the backup. Check that the Home Assistant backup includes add-on data — in some cases you back up the system only, or only certain add-ons. If you are replacing EMQX alone and want the existing configuration back at the file level, a CLI export gives you more control.
• The Dashboard will not open after a restore. Check whether another service is holding 1883 or 18083, and stop whatever conflicts first. Then read the Log and confirm the data directory went back to the right place.
• The CLI import reports “file not found”. The import file must not be renamed. If it sits in the backup subdirectory, the basename alone is enough. A relative path is resolved against the EMQX root directory.
• The version looks unchanged after an update. Read the CHANGELOG first. Add-on 5.9.0 only adds the ngrok TCP tunnel; the EMQX core is still 5.8.9. A newer release in the add-on store does not necessarily mean the broker version has moved — the CHANGELOG is what decides. If what you were expecting was a new core feature, you may have to wait for another version.

Only one add-on can hold 1883

Plenty of people arrive at EMQX from the Mosquitto add-on, and there is one fact that shapes the whole migration: Mosquitto and Woow EMQX cannot run at the same time, because both of them bind port 1883.

flowchart LR
  M["The Mosquitto add-on"] --> P["Port 1883"]
  E["The Woow EMQX add-on"] --> P
  P --> R["They cannot run
at the same time"]

1. Step 1

   Stop Mosquitto

   Stop — or remove — the Mosquitto add-on to free up 1883.

2. Step 2

   Move the configuration to EMQX

   Recreate the same users in EMQX; you can keep the credentials you set up before. Then rebuild the ACL rules there, which EMQX manages under Access Control.

3. Step 3

   Point everything at EMQX

   Change the broker address in Home Assistant's MQTT integration, in Zigbee2MQTT and on each external device to EMQX — homeassistant, port 1883 — and sign in with the EMQX users.

4. Step 4

   Verify, and finish up

   Confirm that Home Assistant, Zigbee2MQTT and every device connect, and that you can see them on the Clients page. If the old Mosquitto had ACL rules built up over a long time, reviewing them one by one as you convert them to EMQX Authorization is what keeps you from letting something through by mistake.

The two things most often missed are the ACL and the integrations. Mosquitto has no graphical interface and its ACL lives in a config file, so there is nothing on screen to remind you it exists; on EMQX you rebuild it under Access Control → Authorization. And the broker address has to change in Home Assistant's MQTT integration, in Zigbee2MQTT and on your external devices at the same time — a device still pointed at the old broker fails quietly rather than loudly.

The ones that come up every time

Where to go from here