MQTT Gateway - disconnects after a day

Hello,

We have an MQTT gateway and two wireless transmitters in our setup:


By changing the stage of each input on either transmitter we receive an MQTT message properly. However, after a day of letting the system run, we no longer get any MQTT messages from either transmitter. The LED status on the gateway is still green but it needs a restart to work properly again. All of these devices are powered up with power supplies.

How can we make the connection more stable?

Thanks

What MQTT Broker are you connecting The MQTT Gateway to?

It’s connected to a Mosquitto MQTT installed on a Raspberry Pi in the local area network.

I have another customer who has reported this same issue but he was connecting The MQTT Gateway to Ubidots. I am going to look into this over the next couple of days and see if I can replicate the problem. It’s extremely difficult to troubleshoot something when it takes a long time for the problem to occur in use so it may take me a few days. When I do resolve the problem it will be possible for you to update the firmware on your MQTT Gateway without returning it, I’ll provide instructions on that procedure when I have the issue resolved.

Thank you,
Travis Elliott

I had another user update his gateway with the latest firmware and it corrected his problem.

You could try updating your MQTT Gateway module to the latest firmware though. Updating the firmware on the module requires this script:

The guide for it can be found here:

Great, thanks for this. I’ll try updating the firmware to see if that solves the problem.

Hello,

We were trying to upgrade the firmware but looks like there is a SSL certificate issue. Here’s the terminal log:

Scanning for Serial Ports
Please wait for the scan to complete
Serial Port Options:
[1]: /dev/cu.Bluetooth-Incoming-Port
[2]: /dev/cu.SLAB_USBtoUART

Please enter the number of the desired Serial Port above: 2
Firmware Choices:
[1]: WiFi AWS Gateway
[2]: WiFi Azure Gateway
[3]: WiFi MQTT Gateway
[4]: WiFi Google IoT Gateway
[5]: Mega Modem
[6]: Cellular MQTT Gateway

Please enter the number of the desired firmware: 3
https://ncd-esp32.s3.amazonaws.com/WiFi_MQTT/firmware.bin
Traceback (most recent call last):
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py”, line 1317, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py”, line 1230, in request
self._send_request(method, url, body, headers, encode_chunked)
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py”, line 1276, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py”, line 1225, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py”, line 1004, in _send_output
self.send(msg)
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py”, line 944, in send
self.connect()
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py”, line 1399, in connect
self.sock = self._context.wrap_socket(self.sock,
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py”, line 500, in wrap_socket
return self.sslsocket_class._create(
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py”, line 1040, in _create
self.do_handshake()
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py”, line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “ncd_flasher.py”, line 179, in
firmware_file = urllib.request.urlretrieve(str(firmware.get(‘firmware’)), ‘./firmware.bin’)
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py”, line 247, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py”, line 222, in urlopen
return opener.open(url, data, timeout)
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py”, line 525, in open
response = self._open(req, data)
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py”, line 542, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py”, line 502, in _call_chain
result = func(*args)
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py”, line 1360, in https_open
return self.do_open(http.client.HTTPSConnection, req,
File “/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/urllib/request.py”, line 1320, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)>

@jacob can you take a look at this?

I double checked permissions on the server files and they look correct.

I’d chalk this up to a local computer certificate issue. Either not sufficient permissions to access the appropriate certs via Python or some other arcane issue with CA files.

Try the tips on this stack overflow and try and manually pull in certs: https://stackoverflow.com/questions/50236117/scraping-ssl-certificate-verify-failed-error-for-http-en-wikipedia-org

Let me know if that doesn’t solve the issue.

Ok, so we could fix the certificate issue and upgrade the firmware with that stack overflow solution - thanks.

However, the problem still exists and the gateway doesn’t publish the incoming messages to the MQTT broker (after ~20 hours) even though the LED is still green.

This has made major problems for our system as we are using these components (transmitter and gateway) to detect the limit switch triggers and if they are not probably detected, the whole system would get out of the rail, which has happened to us a few times and has costs us several thousand dollars. We really need to get this fixed.

I’m guessing there could be some sort of lifetime/keepalive time for the connection where if no messages are published to the broker for a certain period of time the connection is not alive anymore. If so, is there a way to disable that lifetime or implement a heartbeat message publishing mechanism for the gateway so that it will publish a message every once a while to make sure the connection stays alive? Let me know your thoughts.

Hi,

The MQTT protocol actually implements a keep alive natively so the gateway sends a keep alive message to the broker every 30 seconds or so. MQTT protocol mandates that if a client does not submit a keep alive message after 1.5 times the keep alive interval determined at time of initial connection that the client must be disconnected and that socket should be closed. That being said are you able to check with your broker to see if it still detects connection from the client? It should be able to tell you if the client is connected or not.

Hello,

Yes, the connection with the client was very stable. Looks like we are having a lot less issues with the gateway after that firmware update. No disconnections so far after that first accident.

Thanks!