iotDashboard/.github/copilot-instructions.md

The following concise instructions help AI coding agents become productive in this repository.

Purpose

• This repo is a small Django-based IoT dashboard that ingests sensor data via MQTT, stores metadata in Django models, temporarily queues messages in Redis (streams/hashes), and persistently stores timeseries in Postgres/Timescale via background tasks (Huey).

Big Picture

• Components:
  • iotDashboard/ — Django app (models, views, templates, tasks)
  • manage.py — Django CLI
  • mqtt_service.py — standalone MQTT client that subscribes to device topics and writes to Redis
  • tasks.py — Huey periodic tasks that read Redis and write to Postgres
  • Redis — used for device metadata (mqtt_devices), per-sensor streams and latest-value hashes
  • Postgres/Timescale — final storage for sensor_readings table (raw SQL used in places)

Key Files To Read First

• iotDashboard/settings.py — central settings; environment variables expected: SECRET_KEY, CONNECTION_STRING, MQTT_BROKER, MQTT_USER, MQTT_PASS, REDIS_HOST.
• iotDashboard/models.py — Device, Sensor, SensorType; these shape how devices and sensors are represented.
• mqtt_service.py — where MQTT messages are received and written to Redis. Important for stream naming and payload format.
• iotDashboard/tasks.py — Huey tasks that consume Redis and insert into the DB. Shows ingestion logic and timescale interactions.
• iotDashboard/views.py and templates/chart.html — how the UI reads mqtt_latest/Timescale data and what format it expects.

Important Conventions & Patterns

• Redis usage: repo stores device metadata under mqtt_devices (JSON), and the code uses Redis streams and hashes inconsistently. When changing stream behavior, update both mqtt_service.py and tasks.py to remain compatible.
• Topic/Stream canonicalization: adopt a single convention: MQTT topic devices/{device_id}/{sensor} and Redis stream mqtt_stream:{device_id}:{sensor}. Latest-value hash pattern: mqtt_latest:{device_id}.
• No requirements.txt in repo; use python-dotenv + redis, paho-mqtt, huey, psycopg2-binary, requests, Django (4.2) — add a requirements.txt before running.
• Avoid import-time side-effects: tasks.py currently opens Redis and calls devices_to_redis() at import time — refactor to lazy init or a management command.

Developer Workflows (commands & notes)

• Run Django dev server (use virtualenv and install deps):
  • pip install -r requirements.txt (create this file if missing)
  • python manage.py migrate
  • python manage.py runserver
• Run MQTT service locally (requires Redis & MQTT broker):
  • python mqtt_service.py
  • Example publish: mosquitto_pub -t "devices/esp32/test_temperature" -m "23.5"
• Huey tasks:
  • The project uses huey.contrib.djhuey; run workers with Django settings: python manage.py run_huey (ensure huey is installed and configured in HUEY setting).
• Inspect Redis during debugging:
  • redis-cli KEYS "mqtt*"
  • redis-cli XREVRANGE mqtt_stream:mydevice:temperature + - COUNT 10
  • redis-cli HGETALL mqtt_latest:mydevice

Integration Points & Gotchas

• Environment variables: many hosts/credentials are taken from .env via python-dotenv. If missing, code sometimes falls back to defaults or will raise at runtime. Add .env or set env vars in the system.
• DB access: tasks.py sometimes uses psycopg2.connect(settings.CONNECTION_STRING) while views use Django connections. If you change DB config, update both patterns or consolidate to Django connections.
• Topic parsing: mqtt_service.py expects at least 3 topic parts (it reads topic_parts[2]) — be defensive when editing.
• Stream payloads: xadd must receive simple string fields (no nested dicts). When changing stream layout, update the reader in tasks.py accordingly.
• Logging: repo uses print widely. Prefer converting prints to Python logging for maintainability.

What AI agents should do first

• Do not change stream/topic names unless you update both mqtt_service.py and tasks.py.
• Remove import-time Redis initializations and exit() calls; move to lazily-created client getters or management commands.
• Centralize config in settings.py and import from django.conf import settings in scripts instead of hardcoded IPs.
• When making API/DB changes, prefer to update views.py and tasks.py together and add short integration tests using pytest and a Redis test double (or local docker-compose).

Examples (copyable snippets)

• XADD to create canonical stream entry:
  • redis_client.xadd(f"mqtt_stream:{device_id}:{sensor}", {"value": str(sensor_value), "time": datetime.utcnow().isoformat()})
• Create/read consumer group (ingest):
  • redis_client.xgroup_create(stream, "ingest", id="0", mkstream=True)
  • entries = redis_client.xreadgroup("ingest", consumer_name, {stream: ">"}, count=10, block=5000)

If you add or change docs

• Update README.md with a simple docker-compose.yml recipe for Redis/Postgres/Mosquitto and document environment variables. Update env.sample with REDIS_HOST, CONNECTION_STRING, MQTT_BROKER, MQTT_USER, MQTT_PASS.

If anything in these instructions looks off or incomplete for your current refactor, tell me what you'd like to focus on and I'll iterate.