Kaa Documentation

ESP32 Over-the-Air (OTA) Updates using Kaa Platform

Written by Andrew Pasika and Viktor Kazanin

Overview

In this tutorial, we will explore what OTA (Over-the-Air) updates are and how to implement them using the Kaa Platform on an ESP32 board.

What is OTA?

OTA stands for Over-the-Air update.
It allows you to update your device wirelessly without requiring a direct USB connection.
This is particularly useful for updating multiple devices simultaneously or when the devices are located in hard-to-reach places.
OTA is a crucial feature for IoT devices with internet access.

How OTA Works with Kaa

The Kaa Platform is responsible for hosting and distributing software updates.
In this tutorial, the device will connect to the Kaa Platform through MQTT to check for available firmware updates.
When a new firmware version is detected, the device will download the binary files and automatically apply the update to itself.

Prerequisites

  1. You have a Kaa Cloud account.
  2. This tutorial is designed for the ESP32, so the code and libraries may not work with other WiFi-enabled boards.
  3. You should already know how to connect your ESP32 to the Kaa Platform.
    If not, refer to Getting Started.
  4. You should have Arduino IDE 1.x or 2.x installed to flash the code onto the ESP32 and export firmware binaries.

Setting up Kaa OTA

Creating Your First OTA Update

  1. Log in to your Kaa Cloud account and create a new application and endpoint. Save the application version and the token, you’ll need them later.
  2. In the side menu, navigate to Device Management -> Software OTA. This is where we will manage and host our software updates.
  3. This update won’t be automatically downloaded by devices. It will serve as the starting point for your chain of firmware versions.
  4. Set the Semantic version number to 1.0.0, leave the other fields as they are, and click Create.

Create initial firmware

Sketch of version 1.0.0 we will install locally on your device, as it’ll be explained later.

Preparing the Arduino IDE

  1. Connect your ESP32 to your desktop.
  2. Open the Arduino IDE.
  3. Before proceeding, configure your IDE to recognize the ESP32 board.
    This involves selecting the correct board under Tools -> Board and the appropriate port under Tools -> Port options.
  4. Install the required libraries: ArduinoJson and PubSubClient.
    These libraries are needed for the code that will be flashed to your ESP32.
  5. Verify the setup by compiling and uploading any example sketches to your ESP32.
    Navigate to File -> Examples and select an example sketch to ensure everything is functioning properly.

Testing arduino ide setup

Flashing and Provisioning the Initial Update on Your ESP32

To begin, configure the code below and flash your ESP32 with it.
This code connects to WiFi using the provided credentials and uses the endpoint token and app version to establish a connection with the Kaa Platform.
It also listens for software update messages about new versions.

// This is a basic Kaa OTA implementation for a generic ESP32 microcontroller
#include <WiFi.h>
#include <PubSubClient.h>
#include <ArduinoJson.h>
#include <esp_err.h>
#include <esp_http_client.h>
#include <esp_https_ota.h>
#include <esp_crt_bundle.h>

// --- IMPORTANT - WiFi credentials and MQTT server configuration
const char* ssid = "<your-wifi-name>";            // WiFi name (case-sensitive)
const char* password = "<your-wifi-password>";    // WiFi password

const String TOKEN = "<your-endpoint-token>";     // Endpoint token - provided during device provisioning
const String APP_VERSION = "<your-app-version>";  // Application version
const String CURRENT_VERSION = "1.0.0";           // Version of the current firmware

const char* mqttServer = "mqtt.cloud.kaaiot.com";   // MQTT server address
const char* client_id = "19665f9f";                 // MQTT client ID

// Initialize the MQTT client and WiFi client
WiFiClient espClient;              
PubSubClient client(espClient);   

// Initial setup function before main `void loop()`
void setup() {
  // Open serial connection
  Serial.begin(9600);

  // Set up MQTT server and callback for handling OTA messages
  client.setServer(mqttServer, 1883);
  client.setCallback(handleOtaUpdate);

  // Establish initial server connection
  initServerConnection();

  // Allocate memory for MQTT payloads
  if (client.setBufferSize(1024)) {
    Serial.println("Successfully reallocated internal buffer size");
  } else {
    Serial.println("Failed to reallocated internal buffer size");
  }

  delay(1000);

  // Report current firmware version and request a new firmware update
  reportCurrentFirmwareVersion();
  requestNewFirmware();
}

// Main loop function
void loop() {
  // Maintain server connection and handle periodic MQTT work
  initServerConnection();
  delay(1000);
}

// Report the current firmware version to the server
void reportCurrentFirmwareVersion() {
  String reportTopic = "kp1/" + APP_VERSION + "/cmx_ota/" + TOKEN + "/applied/json";
  String reportPayload = "{\"configId\":\"" + CURRENT_VERSION + "\"}";
  Serial.println("Reporting current firmware version on topic: " + reportTopic + " and payload: " + reportPayload);
  client.publish(reportTopic.c_str(), reportPayload.c_str());
}

// Request a new firmware update from the server
void requestNewFirmware() {
  int requestID = random(0, 99);  // Generate a random request ID to avoid conflicts
  String firmwareRequestTopic = "kp1/" + APP_VERSION + "/cmx_ota/" + TOKEN + "/config/json/" + requestID;
  Serial.println("Requesting firmware using topic: " + firmwareRequestTopic);
  client.publish(firmwareRequestTopic.c_str(), "{\"observe\":true}"); // Use observe to indicate acceptance of server push
}

// Establish WiFi and MQTT server connections
void initServerConnection() {
  setupWifi();
  if (!client.connected()) {
    reconnect();
  }
  client.loop();
}

// Handle OTA update messages from the MQTT server
void handleOtaUpdate(char* topic, byte* payload, unsigned int length) {
  Serial.printf("\nHandling firmware update message on topic: %s and payload: ", topic);

  // Parse incoming JSON payload
  DynamicJsonDocument doc(1024);
  deserializeJson(doc, payload, length);
  JsonVariant json_var = doc.as<JsonVariant>();
  Serial.println(json_var.as<String>());

  // Check if the payload contains valid OTA information
  if (json_var.isNull()) {
    Serial.println("No new firmware version is available");
    return;
  }

  // Confirm that the payload is valid
  unsigned int statusCode = json_var["statusCode"].as<unsigned int>();
  if (statusCode != 200) {
    String reasonPhrase = json_var["reasonPhrase"].as<String>();
    Serial.printf("Firmware message's status code is not 200, but: %d\n", statusCode);
    Serial.println("Response: " + reasonPhrase);
    return;
  }

  // Extract the firmware download link, which may be located in one of two different places depending on the OTA update configuration.
  String systemFileLink = json_var["config"]["system"]["files"][0].as<String>();
  String downloadLink = json_var["config"]["link"].as<String>();
  String newVersionNumber = json_var["configId"].as<String>();
  // Determine which firmware link to use
  String firmwareLink = systemFileLink && !systemFileLink.equals(String("null")) 
    ? systemFileLink 
    : downloadLink && !downloadLink.equals(String("null"))
    ? downloadLink 
    : "";

  Serial.println("firmwareLink: " + firmwareLink);
  
  // Attempt the firmware update
  Serial.println("Updating...");
  esp_err_t ret = performFirmwareUpgrade(firmwareLink);

  // Handle the result of the OTA update
  if (ret == ESP_OK) {
    Serial.println("Firmware was updated successfuly from '" + CURRENT_VERSION + "' to '" + newVersionNumber + "'. Restarting...");
    esp_restart();
  } else {
    Serial.println("Firmware update failed. Skipping...");
  }
}

// Perform OTA firmware upgrade
esp_err_t performFirmwareUpgrade(String firmwareLink) {
  esp_http_client_config_t config = {
    .url = firmwareLink.c_str(),
    .crt_bundle_attach = esp_crt_bundle_attach, // Default certificate bundle
  };
  esp_https_ota_config_t ota_config = {
    .http_config = &config,
  };

  esp_err_t ret = esp_https_ota(&ota_config);
  return ret;
}

// Setup WiFi connection
void setupWifi() {
  if (WiFi.status() != WL_CONNECTED) {
    delay(200);
    Serial.println();
    Serial.printf("Connecting to [%s]\n", ssid);
    WiFi.begin(ssid, password);
    connectWiFi();
  }
}

// Wait until connected to WiFi
void connectWiFi() {
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.println("Connecting to WiFi...");
  }
  Serial.println();
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

// Attempt to reconnect to the MQTT server if disconnected
void reconnect() {
  while (!client.connected()) {
    Serial.println("Attempting MQTT connection...");
    
    if (client.connect(client_id)) {
      Serial.println("Connected to WiFi");
      subscribeToFirmwareUpdates();
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      delay(5000);
    }
  }
}

// Subscribe to firmware-related topics for OTA updates
void subscribeToFirmwareUpdates() {
  String serverPushOnConnect = "kp1/" + APP_VERSION + "/cmx_ota/" + TOKEN + "/config/json/#";
  client.subscribe(serverPushOnConnect.c_str());
  Serial.println("Subscribed to server firmware push on topic: " + serverPushOnConnect);

  String serverFirmwareResponse = "kp1/" + APP_VERSION + "/cmx_ota/" + TOKEN + "/config/json/status/#";
  client.subscribe(serverFirmwareResponse.c_str());
  Serial.println("Subscribed to server firmware response on topic: " + serverFirmwareResponse);

  String serverFirmwareErrorResponse = "kp1/" + APP_VERSION + "/cmx_ota/" + TOKEN + "/config/json/status/error";
  client.subscribe(serverFirmwareErrorResponse.c_str());
  Serial.println("Subscribed to server firmware response on topic: " + serverFirmwareErrorResponse);
}

  1. Paste the provided code into your Arduino IDE.

  2. Fill in the ssid, password with WiFi credentials. Also, fill in TOKEN, and APP_VERSION using values from the endpoint you created earlier.
    For now, leave CURRENT_VERSION set to 1.0.0, as mentioned earlier. This version will be installed locally on your device to start the chain of firmware updates.

  3. Flash the code onto your device and monitor the output in the serial monitor.
    You should see a response confirming the connection to WiFi and the status of the connection to the Kaa Platform.

Flashing ESP32 with new code

Compile and Host New Firmware Binary on the Kaa Platform

  1. After successfully running the initial code, you can now try updating the firmware via OTA.
  2. In the code above, update the CURRENT_VERSION to 1.1.0 to reflect the new firmware version. Do not upload it to ESP32 after changes, since it will be uploaded via OTA.
  3. To prepare for the update, compile your sketch into a binary by selecting Sketch -> Export Compiled Binary in Arduino IDE.
  4. Once the compilation is finished, go to Sketch -> Show Sketch Folder.
    From there, locate the binary file:
    • For Arduino IDE 1.x, you’ll look for a file [your sketch name].ino.esp32.bin.
    • For Arduino IDE 2.x, look for a file in the build folder and named [your sketch name].ino.bin.

Compiled Binary

The compiled binary will be used to update the firmware of your ESP32. When the device receives this binary via OTA from Kaa, it will attempt to update itself and reboot upon success. After reboot, the device will use the new code from version 1.1.0 to reconnect to WiFi and listen for Kaa MQTT messages until it receives another update (for example 2.0.0), just like it did with version 1.0.0.

Defining a New Firmware Version for Provisioning in Kaa

To provision the new firmware, go to the Kaa platform UI and navigate to Device Management -> Software OTA -> Add software version.

From there:

  1. Set the Semantic version number to 1.1.0 (this should match the CURRENT_VERSION in your sketch).
  2. Upload the newly compiled sketch with the updated CURRENT_VERSION changed from 1.0.0 to 1.1.0.
  3. In the Upgradable From field, select 1.0.0 to specify that devices running version 1.0.0 can upgrade to 1.1.0.
  4. Choose All for the rollout configuration.
    This ensures that all endpoints in your application can update to this firmware version.
  5. Click Create to add the new firmware version.

Create the Second Firmware

Monitoring the update process

  1. After submitting the new firmware version 1.1.0 via Kaa UI, your ESP32, running the 1.0.0 sketch, will immediately detect this change and attempt to download the binary we compiled earlier to update itself.
  2. In Arduino IDE open Serial Monitor. Monitor the serial output on your ESP32 to confirm that it automatically fetched and applied the update from version 1.0.0 to 1.1.0.
  3. You can now disconnect the ESP32 from your computer and connect it to a remote power supply.

Monitoring Firmware Update

Summary

Now, it’s just a process that you can repeat to enjoy the OTA updates:

  1. Add more custom logic in your sketch in the loop() method and update CURRENT_VERSION.
  2. Compile the binary.
  3. In Kaa, create a firmware version with the same value as CURRENT_VERSION, upload the binary, and set Upgradable From to the previous/initial version.

Issues and solutions

Infinite updates

If you don’t set Semantic version number to the same value as CURRENT_VERSION, and Upgradable From to any previously existing version, it might create problems, such as the code infinitely updating itself or failing to update at all.

MQTT client issues

To troubleshoot any issues related to communication between the client and the Kaa platform, you can use the following Python script.

import itertools
import json
import queue
import random
import string
import sys
import time

import paho.mqtt.client as mqtt

KPC_HOST = "mqtt.cloud.kaaiot.com"  # Kaa Cloud plain MQTT host
KPC_PORT = 1883                     # Kaa Cloud plain MQTT port

CURRENT_SOFTWARE_VERSION = ""   # Specify software that device currently uses (e.g., 0.0.1)

APPLICATION_VERSION = ""     # Paste your application version
ENDPOINT_TOKEN = ""          # Paste your endpoint token


class SoftwareClient:

    def __init__(self, client):
        self.client = client
        self.software_by_request_id = {}
        self.global_request_id = itertools.count()
        get_software_topic = f'kp1/{APPLICATION_VERSION}/cmx_ota/{ENDPOINT_TOKEN}/config/json/#'
        self.client.message_callback_add(get_software_topic, self.handle_software)

    def handle_software(self, client, userdata, message):
        if message.topic.split('/')[-1] == 'status':
            topic_part = message.topic.split('/')[-2]
            if topic_part.isnumeric():
                request_id = int(topic_part)
                print(f'<--- Received software response on topic {message.topic}')
                software_queue = self.software_by_request_id[request_id]
                software_queue.put_nowait(message.payload)
            else:
                print(f'<--- Received software push on topic {message.topic}:\n{str(message.payload.decode("utf-8"))}')
        else:
            print(f'<--- Received bad software response on topic {message.topic}:\n{str(message.payload.decode("utf-8"))}')

    def get_software(self):
        request_id = next(self.global_request_id)
        get_software_topic = f'kp1/{APPLICATION_VERSION}/cmx_ota/{ENDPOINT_TOKEN}/config/json/{request_id}'

        software_queue = queue.Queue()
        self.software_by_request_id[request_id] = software_queue

        print(f'---> Requesting software by topic {get_software_topic}')
        payload = {
            "configId": CURRENT_SOFTWARE_VERSION
        }
        self.client.publish(topic=get_software_topic, payload=json.dumps(payload))

        try:
            software = software_queue.get(True, 5)
            del self.software_by_request_id[request_id]
            return str(software.decode("utf-8"))
        except queue.Empty:
            print('Timed out waiting for software response from server')
            sys.exit()

def main():
    # Initiate server connection
    print(f'Connecting to Kaa server at {KPC_HOST}:{KPC_PORT} using application version {APPLICATION_VERSION} and endpoint token {ENDPOINT_TOKEN}')

    client_id = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(6))
    client = mqtt.Client(client_id=client_id)
    client.connect(KPC_HOST, KPC_PORT, 60)
    client.loop_start()

    software_client = SoftwareClient(client)

    # Fetch available software
    retrieved_software = software_client.get_software()
    print(f'Retrieved software from server: {retrieved_software}')

    time.sleep(5)
    client.disconnect()


if __name__ == '__main__':
    main()