Collecting data from a device

• Overview
• Terms and concepts
  • Data sample
  • Time series
• Playbook
  • Enable the time series auto-extraction
  • Send data samples

Time to complete: 10 min.

Overview

Welcome to the second tutorial in the Kaa getting started guide! We assume that you have successfully connected your first device to Kaa, so make sure to review the first tutorial before proceeding here.

From this tutorial you will learn some additional concepts of the Kaa platform and discover how to:

• collect telemetry data from a device
• transform raw telemetry data into well-structured time series
• use the time series auto-extraction feature
• visualize time series data on the Kaa UI

Terms and concepts

Let us start by reviewing some new terms and concepts related to the Kaa data collection feature.

Data sample

Think of data sample as a block of data in JSON format that a client sends to the platform for data collection.

Let’s consider an example.

You have a greenhouse monitoring station that consists of temperature, humidity and light sensors, and a gateway (e.g. Arduino) that all sensors are connected to. Every minute Arduino polls all connected sensors for their current values. Then the Arduino formats sensor values into a JSON object: this is a data sample.

{
  "temperature": 25,
  "humidity": 46,
  "lux": 8600
}

Time series

Time series is a named sequence of data points. Each data point contains a timestamp and one or more named values. A set of value names and their types (numeric, string, boolean) defines a time series.

You may want different time series defined for various things. For example, a fuel level time series may only have one numeric value: level. On the other hand, a geo-location time series may have several numeric values: latitude, longitude, and in some applications altitude.

You can configure Kaa to transform data samples received from endpoints into time series for displaying them on charts, gauges, maps, etc. Kaa can use a timestamp present in a data sample or use a data sample receipt timestamp. For simplicity, in this tutorial we will use data receipt timestamps.

Also, Kaa has an auto-extraction feature that stores each numeric top-level data sample field into a separate time-series. All auto-extracted time series have a name that follows the pattern auto~<field name> and one numeric value with the name value. So, if your endpoint sends data a sample with two fields, e.g.:

  {
    "temperature": 23,
    "humidity": 48
  }

and the auto-extraction feature is enabled, Kaa creates two time-series: auto~temperature and auto~humidity.

Let’s now send some data to Kaa and see the auto-extraction feature in action.

Playbook

We assume that you have already created an application, application version, and endpoint with a token while following the “connecting your first device” tutorial. You can reuse them or create new ones.

Start by logging into your Kaa Cloud account.

Enable the time series auto-extraction

To keep things simple, at this time we will not define any configurable time series, but rather just use the auto-extraction feature. Let us make sure that it is turned on in your application. Go to the Device management -> Applications -> expand your application -> select “epts” (Kaa component that is responsible for time series processing) and enable the “Autoextract” checkbox.

With this function enabled, Kaa will automatically create a time series for each numeric field it encounters at the root of data samples your endpoints in this application will submit. You will then be able to view these time series in Kaa UI, no extra configuration required.

Go to the pre-defined endpoint dashboard and scroll to the “Device telemetry” widget. (The endpoint dashboard is available from Device management -> Devices -> select your endpoint in the list.) Now we are all set to send our first data sample to the platform.

Send data samples

You can submit telemetry data via a variety of MQTT- and HTTP-based interfaces. Let’s take a look at a couple of the simplest ones: plain HTTP and plain MQTT.

• HTTP
• MQTT

curl --location --request POST 'https://connect.cloud.kaaiot.com:443/kp1/<app-version-name>/dcx/<endpoint-token>/json' \
--data-raw '{
  "temperature": 23,
  "humidity": 48
}'

# Simple MQTT-based data collection client for the Kaa IoT platform.
# See https://docs.kaaiot.io/KAA/docs/current/Tutorials/getting-started/collecting-data-from-a-device/.

import json
import random
import signal
import string
import time

import paho.mqtt.client as mqtt

KPC_HOST = "mqtt.cloud.kaaiot.com"  # Kaa Cloud plain MQTT host
KPC_PORT = 1883                     # Kaa Cloud plain MQTT port

ENDPOINT_TOKEN = ""       # Paste your endpoint token
APPLICATION_VERSION = ""  # Paste your application version


class DataCollectionClient:

    def __init__(self, client):
        self.client = client
        self.data_collection_topic = f'kp1/{APPLICATION_VERSION}/dcx/{ENDPOINT_TOKEN}/json'

    def connect_to_server(self):
        print(f'Connecting to Kaa server at {KPC_HOST}:{KPC_PORT} using application version {APPLICATION_VERSION} and endpoint token {ENDPOINT_TOKEN}')
        self.client.connect(KPC_HOST, KPC_PORT, 60)
        print('Successfully connected')

    def disconnect_from_server(self):
        print(f'Disconnecting from Kaa server at {KPC_HOST}:{KPC_PORT}...')
        self.client.loop_stop()
        self.client.disconnect()
        print('Successfully disconnected')

    def compose_data_sample(self):
        return json.dumps({
            'timestamp': int(round(time.time() * 1000)),
            'temperature': random.randint(15, 25),
            'humidity': random.randint(35, 60),
        })


def on_message(client, userdata, message):
    print(f'<-- Received message on topic "{message.topic}":\n{str(message.payload.decode("utf-8"))}')


def main():
    # Initiate server connection
    client = mqtt.Client(client_id=''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(6)))

    data_collection_client = DataCollectionClient(client)
    data_collection_client.connect_to_server()
    client.on_message = on_message

    # Start the loop
    client.loop_start()

    # Send data samples in loop
    listener = SignalListener()
    while listener.keepRunning:

        payload = data_collection_client.compose_data_sample()

        result = data_collection_client.client.publish(topic=data_collection_client.data_collection_topic, payload=payload)
        if result.rc != 0:
            print('Server connection lost, attempting to reconnect')
            data_collection_client.connect_to_server()
        else:
            print(f'--> Sent message on topic "{data_collection_client.data_collection_topic}":\n{payload}')

        time.sleep(3)

    data_collection_client.disconnect_from_server()


class SignalListener:
    keepRunning = True

    def __init__(self):
        signal.signal(signal.SIGINT, self.stop)
        signal.signal(signal.SIGTERM, self.stop)

    def stop(self, signum, frame):
        print('Shutting down...')
        self.keepRunning = False


if __name__ == '__main__':
    main()

Verify that the auto~temperature and auto~humidity time series appeared on the right side of the “Device telemetry” chart and the chart is filled with data points.