Modernizing Home Telemetry: Why I Switched from Direct Writes to MQTT

I'm finally taking a step forward with my home automation setup, and I'm genuinely excited about it. For a while, I'd been running a handful of ESP32s around the house, each directly pushing temperature and humidity data into InfluxDB. It worked... kind of. But it was never quite stable. Any time InfluxDB was down, the data was lost. If network latency got in the way, my ESPs would hang or drop readings. It was a fragile system that felt more like a proof-of-concept than something I could rely on long-term.

This week, I rethought the architecture. Instead of writing directly to InfluxDB, I'm using MQTT to decouple my ESP32s from the rest of the telemetry stack. This simple change opened the door to a more durable and flexible pipeline.

The New Stack

Here's the updated flow:

ESP32 (sensors) --> MQTT (Mosquitto) --> Telegraf --> InfluxDB

Each ESP32 publishes sensor readings as JSON to an MQTT topic. Telegraf subscribes to those topics and forwards the data to InfluxDB, where I can visualize and analyze it in Grafana.

Services Overview

A brief look at the services I deployed in my home lab to support this pipeline:

Mosquitto (MQTT Broker)

Simple message broker where ESP32s publish sensor data:

services:
  mosquitto:
    image: eclipse-mosquitto:2
    container_name: mosquitto
    ports:
      - "1883:1883"    # MQTT
      - "9002:9001"    # WebSocket (optional)
    volumes:
      - ./mosquitto.conf:/mosquitto/config/mosquitto.conf
      - ./data:/mosquitto/data
      - ./log:/mosquitto/log
    restart: always
    logging:
      driver: "gelf"
      options:
        tag: "docker.mosquitto"
        gelf-address: "udp://${LOGSTASH_HOST}:12201"

Telegraf

Subscribes to MQTT and writes to InfluxDB:

services:
  telegraf:
    image: telegraf:1.34.4
    container_name: telegraf
    restart: always
    environment:
      - HOST_PROC=/host/proc
      - HOST_SYS=/host/sys
      - HOST_ETC=/host/etc
    volumes:
      - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
    logging:
      driver: "gelf"
      options:
        tag: "docker.telegraf"
        gelf-address: "udp://${LOGSTASH_HOST}:12201"

InfluxDB

Time-series database storing all ESP32 metrics:
https://github.com/18visions-ci/homelab-influxdb

---
- name: Install and configure InfluxDB 3.0.3
  hosts: influxdb
  become: yes
  become_method: sudo
  become_user: root
  vars:
    influxdb_admin_user: "admin"
    influxdb_admin_password: "{{ lookup('env', 'influxdb_admin_pw') }}"
    influxdb_org: "homelab"
    influxdb_bucket: "esp32_data"
    influxdb_token: "{{ lookup('env', 'influxdb_tkn') }}"
    influxdb_retention: "17520h"
    influxdb_version: "3.0.3"

  roles:
    - influxdb_install
    - influxdb_configure

Each of these services runs in my Proxmox-based homelab using lightweight VMs or containers, managed via Ansible and GitHub Actions.

Why I Switched

When I was posting directly to InfluxDB:

• If InfluxDB was down, the ESP32s lost data
• The HTTP client on the ESPs was fragile
• Writing from embedded code meant I needed to hardcode authentication, bucket, org, and token logic

With MQTT + Telegraf:

• ESP32s are now fire-and-forget. They just publish JSON to a broker.
• Telegraf handles batching, retries, and formatting for InfluxDB.
• If Influx or Telegraf go down, Mosquitto still holds messages in memory (or disk, if configured).

Pros and Cons

Direct to InfluxDB (old setup):

• ✅ Simple architecture (1 device talks to 1 endpoint)
• ❌ Prone to failure when the network or InfluxDB is unavailable
• ❌ Requires HTTPS, token handling, and manual JSON formatting
• ❌ ESP32s do all the heavy lifting

MQTT + Telegraf (new setup):

• ✅ Decouples producer from consumer
• ✅ More resilient (in case of failure)
• ✅ Easier to expand (add a second database, push to cloud, etc.)
• ✅ Lightweight JSON from ESPs
• ❌ Adds more services to maintain (Telegraf, Mosquitto)
• ❌ Slightly steeper initial setup

A Bonus: OTA Updates

Another big win this time around was setting up OTA (Over-The-Air) firmware updates for my ESP32s. Once I had stable Wi-Fi and MQTT connections, I added ArduinoOTA support to each sketch and was able to push firmware updates from my laptop without ever plugging in a USB cable. It took some trial and error with partition schemes and port binding, but now it's working flawlessly.

Wrapping It Up

The new setup already feels so much more stable. My ESP32s don’t care if InfluxDB is momentarily unavailable. OTA lets me make improvements quickly, and MQTT + Telegraf smooth out the edges between my devices and my database.

I'll add code snippets and configurations in follow-up posts, but for now, I'm thrilled to have a telemetry pipeline that actually feels production-ready — even if it’s just for a home lab.

Stay tuned for more home automation posts coming soon!