AWS Micro Gateway

I have a few questions about the AWS Micro Gateway.

  • Can the gateway be used to configure the various sensors that connect to it? I tried putting the sensors in config mode and entering the setup of the gateway but the sensors did not seem to pop up and be configurable there. The sensors and gateway to communicate with AWS IoT fine though.

  • Can the gateway be configured to send ALL the sensors data at once as opposed to 1 request to AWS for sensors transmission? and if it can could it send the data by “group” ie… send all the temp/humidity sensor nodes at once then all of each of the other sensors?

  • Ive found the setup guide for the AWS gateway but is there a user manual for it similar to the Mega Modem?

Cheers

Hi James,

Configuring sensors through the gateways is still in Beta. It works fairly well but I wouldn’t call it 100%. Here are some steps for configuring sensors through the gateway:

  1. Put the Gateway into setup mode by pressing and holding the CFG button inside the gateway until the LED flashes Blue.
  2. Open a web interface to the Gateway and click on the Devices Tab.
  3. Power up a sensor or press the reset button on one of the sensors you’d like to configure. It will send a transmission, the gateway will receive it, then it will be listed on the devices page. You can repeat this process to get all sensors listed on the Devices page.
  4. Click on a sensor to configure it individually. Enter settings as desired.
  5. Click the Configuration mode button at the bottom of the Gateway’s Device page, this puts the Gateway into a mode where it listens for sensors in configuration mode.
  6. Put one of the sensors into configuration mode by pressing and holding the C button, then press and release the R button, count 8 seconds then release the C button. The Sensor should send a configuration mode packet. I only recommend configuring one sensor at a time so do not put another sensor into configuration mode until the gateway is done with the current sensor.
  7. Once the gateway receives the configuration mode packet it will send your configuration settings to the sensor. It will also display the update status of the sensor on the devices page.
  8. Once the sensors are configured press the button on the Devices page to put the Gateway back into Run mode(Exit Config Mode).
  9. Now reboot all sensors by either pressing the reset button on them or power cycle.

As I said this process is not 100% yet so it’s possible you might have to try a couple of times. Make sure antennas are attached to the sensors and the gateway and that the gateway is at least 5 ft away from the sensors.

It is not possible for the Gateway to report groups of sensors. That is something we will keep in mind for future versions though. We will discuss the viability of that option. If I might ask though why would you want this?

Currently the setup guide is the only documentation for the AWS gateway. I plan on revising it and adding additional information though.

Thank you,
Travis Elliott

Awesome thx for the quick response! I will give those beta config steps a try tonight.

There are a couple of reasons I think I want to send groups of nodes at once to AWS or at least all of the nodes associated with a gateway (once I get you guys to set that up on future orders). First would be creating of a report from various sensor data without having to necessarily have to query my DB to get the data or have some measure of knowing when all sensors have reported so I could generate such a report. Another example would be to reduce to requests to AWS and off load some logic to AWS Lambda before storing the data. Perhaps there are several temp/humidity sensors in a large room and I am only concerned with storing the avg of the sensors each time they report. With the sensors passing up their data to AWS in the “nodes” object I could parse that data in Lambda, maybe preform some calculations and then store only that data. This would also greatly reduce the number of requests to IoT (it would increase the message size though), it would also reduce the number of rules triggered, Lambdas executed and additional AWS things that are charged by use.

ok, sounds good. I look forward to the new docs. Can the AWS gateway be used with NodeRed or similar or would I need the little USB dongle?

Cheers,
James

1 other thing Im having a hard time with is the devices Thing Shadow does not appear to get updated with the request to ‘things/[thingID]/shadow/update’. I can see the data coming through the test page in AWS IoT but after setting up a shadow for the device and triggering a packet the shadow is not updated and `things/[thingID]/shadow/update/accepted’ is never trigger. Is this a bug with the sensor? Perhaps there are more steps needed in the guide to connect the IAM user with the right permissions (I set up IoT:*)?

Cheers,

James

Hi James,

The “Thing” should provision itself using your IAM credentials. Is it listed under Manage on IoT Central? Is the LED on the Gateway Green? If so then it is provisioned and will report data when it receives transmissions from sensors. If you are not seeing any updates to the Thing Reported Shadow then the Gateway is not receiving transmissions from sensors. Make sure you have the sensor powered on as we do not ship them powered up, you’ll need to open it and either flip the switch on or move the on board jumper.

Sorry for the confusion… it provisioned itself properly and I do see the sensor data in AWS IoT Test when I subscribe to the topic ‘/things/[thingiD/shadow/update’ BUT when I actually inspect the shadow of the device there is only the “Welcome to AWS IoT” message both in the reported and desired fields. My understanding is the “reported” data should be written to the shadow… override any fields matching and leave the rest of the shadow as is.

I can manually update the information and fields in the shadow document but it is not updated when the gateway sends its info. Is it possible the JSON request being sent from the gateway to AWS IoT is malformed and not quite to the AWS IoT API specs to preform the update and trigger the ‘/shadow/update/accepted’ topic BUT is in a good enough form to trigger the 'shadow/update/ topic?

Hi James,

It is my understanding that update/accepted is a topic which AWS IoT writes to in order to verify it received an update to the /update topic. I believe it only sends this to the client which sent the message to the /update topic. It would be a method for the Gateway to verify that the update it sent was verified, however other clients could not monitor that topic. This is why you are not able to see messages on /update/accepted (your client ID did not send anything to /update, the sensor did).

I don’t believe AWS IoT has much in the way of a UI for visualizing data coming in other than the MQTT Client. Actually viewing the Shadow there does not seem to do much.

Here is the doc where I got my information:
https://docs.aws.amazon.com/iot/latest/developerguide/device-shadow-mqtt.html

This is just my interpretation of how this functions from my reading. I could of course be wrong. If you find out additional information please let me know. I am fairly certain however that the topic and payloads coming from the Gateway are not malformed an seem to be going to the proper topic.

Im not sure I agree at all about not being able to see the messages because the sensors updated. The gateway is sending its data like so…

things/{BLUR}/shadow/update

Jun 10, 2019 3:47:38 PM -0300ExportHide

{
  "state": {
    "reported": {
      "Gateway_Config": {
        "mac": "{BLUR}",
        "ip": "{BLUR}",
        "xbee_address": "{BLUR}",
        "network_id": "{BLUR}",
        "preamble": "0",
        "tx_power": "4",
        "xbee_ready": "1"
      }
    }
  }
}

Which “should” be adding a field “Gateway_Config” to the device shadow with its data. It does not.

Last update: Jun 10, 2019 3:44:42 PM -0300
Shadow state:

  "desired": {
    "welcome": "aws-iot"
  },
  "reported": {
    "welcome": "aws-iot"
  }
}
Metadata:
{
  "metadata": {
    "desired": {
      "welcome": {
        "timestamp": 1560192282
      }
    },
    "reported": {
      "welcome": {
        "timestamp": 1560192282
      }
    }
  },
  "timestamp": 1560192659,
  "version": 12
}

the sensors are sending info through the gateway to the SAME shadow document of the gateway like so…

things/{blur}/shadow/update
Jun 10, 2019 3:48:34 PM -0300
 {
  "state": {
    "reported": {
      "nodes": {
        "{blur}": {
          "transmission_count": 1,
          "battery_level": 3.29406,
          "type": 18,
          "node_id": 0,
          "rssi": 100,
          "frequency": 0,
          "duty cycle": 0
        }
      }
    }
  }
}

and

things/{blur}/shadow/update
Jun 10, 2019 3:48:50 PM -0300
{
  "state": {
    "reported": {
      "nodes": {
        "{blur}": {
          "transmission_count": 1,
          "battery_level": 3.29406,
          "type": 1,
          "node_id": 1,
          "rssi": 100,
          "humidity": 40.48,
          "temperature": 24.35
        }
      }
    }
  }
}

these should be adding new fields to the gateway device shadow under “nodes” and then under their respective XBEE addresses with the sensor data. This is also not happening.

I can subscribe to the ‘update’ and ‘update/accepted’ topic in the IoT console and AWS never fires the ‘update/accepted’… meaning to me that the update was in fact not accepted as is suggested with the shadow not being updated. At the very least from what you are saying the shadow should be getting updated with the packet info the gateway sends, it does not. I dont believe it matters if the gateway is listening for the ‘update/accepted’ topic… AWS still sends it and we can listen for it in AWS IoT Test.

I agree that the messages from the gateway and sensors are received by AWS IoT at ‘things/{id}/shadow/update’ but I dont think AWS does anything with it like update the shadow. I concede I may be misunderstanding the whole process or at least the functionality of the NCD sensors but something isnt quite jiving haha. Happy to continue to track down this issue with you

cheers,

James

Any updates on this @Travis? I was just re-reading your comments and this part “I don’t believe AWS IoT has much in the way of a UI for visualizing data coming in other than the MQTT Client. Actually viewing the Shadow there does not seem to do much.” I feel is a little telling as to the problem. You are right that there is not much in the way of UI visualization but the shadow document listed in AWS MQTT Client does live update the shadow on that page for RaspberryPi’s doing the same communication. AWS has a full tutorial showing that shadow document being updated in realtime and live on the screen when new data is received.

Here is the AWS example for a RaspberryPi updating the shadow document. https://docs.aws.amazon.com/iot/latest/developerguide/iot-plant-step3.html

Has there been any progress made into where the problem might be here? Is the gateway functioning as NCD expects by not updating the shadow document and just sending the data to the ‘shadow/update’ topic?

Id like to keep the ball rolling and the communication open on this issue but until this is resolved it is preventing my company from making further orders and that is beginning to stall project development.

Hi James,

I apologize for the delayed response. I have been pulled in several directions the past couple days.

I understand what you were referencing now and can see the problem. Currently the AWS gateway is publishing data to the topic things/thingName/shadow/update. However it should be publishing data to the topic $aws/things/thingName/shadow/update. The issue with this is the $ preceding the topic. Generally topics preceded with a $ are reserved as system status topics. The MQTT library I am using for this project does not support publishing to or subscribing to system topics as that is not suppose to be used(in some people’s opinion). I’m still sorting out a way of solving this problem but don’t have a solution just yet.

Ouuu… Good catch @Travis. That sounds like exactly the thing to cause the problem. If there is anything I can do to help just let me know. AWS IoT is our chosen solution for a couple of active projects and I love the quality of the NCD hardware and team members Ive spoke with so far. Happy to help sort out any software issues if it makes the hardware better :+1:

Thank you for your patience on this James. These are very new products and new products just have bugs like this. This one is obviously pretty large but it happens sometimes. I developed the product and unfortunately I didn’t get several eyes on it prior to release. Thank you very much for pointing this out. I’ll let you know as soon as I have something. I have the problem pinpointed now, just need to resolve it.

No problem at all, glad it has been identified and can get fixed. We’ll continue with the backend dev and swing back around to the App-sensor live data connection. Ive got the structure of the sensor/gateway data and thats enough to seed some DB tables and keep going. Safe to assume this can be fixed with the daily gateway check for firmware updates?

Fingers crossed on OTA firmware update on the Gateway. That has been “Spotty” at best so far I would say.

Is that something I will be able to update locally?

Hi James,

If OTA does not work it is possible to flash firmware to the board. It’s not the simplest thing in the world but I can provide instructions for doing so. There are several tools that must be installed. If you want to get a jump start on that or you’re interested in this sort of thing these gateways are based on ESP32 modules. You can look into ESPTools which is a python application that allows for uploading firmware and SPIFFS files to the board. Hopefully OTA just works so this isn’t required.

Hi James,

I have good news and bad news. The good news is I found and corrected the problem of updating the shadow. The bad news is OTA on your module will not work. I have OTA working well on this latest version of firmware but we will need to get you a new processor. Would it be acceptable to ship this module to you? I will send you a private message which you can respond to with the address you would like the module shipped to.

Sounds good, thank you @Travis. Ive PM’d you. Will future AWS gateway orders built include this fix and the OTA updates?

Is there somewhere to see what the latest available version of firmware? I’d like to verify that my gateways are up to date. Would also be nice to see release notes. :slight_smile:

Hi,

Firmware version monitoring is on my “todo list” but I don’t have anything for it just yet. That said you can update your firmware at any time with this Python update script. After this update you will however need to re enter your settings.

All of this said no change has been made to gateway functionality only the addition of supporting new sensors as they are released. So if you do not have a new sensor which isn’t supported by the gateway there is no sense in updating the firmware.