How the Smart OLED Learning Kit Was Built

This project began with a simple idea: learning should feel playful, interactive, and adaptive. I wanted to go beyond the usual single-screen projects by creating a kit where questions appear first, followed by answer options — all on the same OLED display — with a timed transition based on difficulty. To make the experience more meaningful, the questions and words are generated live using Google Gemini, ensuring every session feels new. The Digikey My-List has been given below for exact components that are used in the project.

Digikey My-List 

https://www.digikey.in/en/mylists/list/D7ZNFW46CF 

Video

1. Concept & Early Planning

The design started by mapping the learning flow. A child selects the learning mode, then the difficulty level, after which the ESP32 requests AI-generated content from the backend.

Each question appears for a specific duration:

• Easy: 10 seconds
• Medium: 7 seconds
• Hard: 5 seconds

After the timer ends, the same OLED switches to a clean options screen, allowing the child to choose an answer using a joystick.

For alphabet mode, the kit behaves differently: words starting with the selected letter are shown one by one, and the child can select a favorite word. That chosen word is uploaded to Firebase and accessed later through a mobile app.

2. Hardware Selection & Assembly

The ESP32-C6 was chosen as the main controller because of its fast Wi-Fi and reliable performance. A single SSD1306 OLED display handled all visuals, and a joystick module made navigation intuitive. A buzzer and LED pair helped provide immediate feedback for correct and incorrect answers.

The components were assembled on a breadboard with clean wiring. A separate tactile button was added for quick resetting, and the ESP32-C6’s internal RGB LED was disabled during resets to keep the interface distraction-free.

3. Wiring & Initial Testing

Before writing the full logic, every component was tested independently:

• Checking OLED initialization and text rendering
• Reading joystick analog values through the serial monitor
• Testing buzzer tones and LED indicators
• Verifying Wi-Fi connectivity
• Testing the reset button behavior

Early validation ensured the hardware remained consistent and troubleshooting later became much simpler. The connections diagram has been shown above and the detailed explanation has been given in the table below.

4. Programming the Firmware

All firmware was developed in the Arduino IDE.
The main challenge was creating a smooth two-stage interface on a single OLED:

Stage 1 — Question Screen

Displays the generated question along with a countdown timer.

Stage 2 — Options Screen

After the timer ends, only the answer choices (A/B/C/D) are shown, and the joystick becomes active for selection.

The firmware handles:

• Mode and difficulty selection
• Countdown timers
• Switching question → options
• Reading joystick navigation
• Sending JSON requests to the backend server
• Playing feedback sounds
• Uploading selected alphabet-mode words to Firebase

Even with one OLED, the timed transitions give the kit a structured, game-like feel.

5. Building the Backend

A lightweight Flask API hosted on Vercel handles all AI-related tasks. When the ESP32 requests a question or word list, the backend prompts Gemini to generate:

• Child-friendly questions
• Four options and a correct index
• Simple words for alphabet mode

The backend cleans the response, formats valid JSON, and sends it back to the device.
Firebase integration was included as well, allowing the ESP32 to upload user-selected words from alphabet practice.

6. MIT App Inventor Integration

To make alphabet learning more interactive, an MIT App Inventor mobile app was created. Whenever the child selects a favorite word in the alphabet mode, the ESP32 uploads that word to Firebase.

The app then:

• Retrieves the stored words from Firebase
• Displays them inside the app
• Plays their pronunciation using text-to-speech

This gives children a mobile extension of the learning kit — turning the experience from hardware-only to a connected digital learning tool.

7. Code Blocks

The code mentioned below is to be implemented in the Arduino IDE and uploaded to the ESP32-C6 after making the connections as mentioned in the table given above.

#include <Wire.h>
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>
#include <WiFi.h>
#include <HTTPClient.h>
#include <ArduinoJson.h>

// ==================== I2C and OLED Setup ======================
TwoWire WireQuestion = TwoWire(0); // For Question OLED
TwoWire WireAnswer = TwoWire(1);   // For Answer OLED

#define SCREEN_WIDTH 128
#define SCREEN_HEIGHT 64

#define OLEDQ_SDA 6
#define OLEDQ_SCL 7
#define OLDA_SDA 6
#define OLDA_SCL 7

// ==================== Input / Output Pins ======================
#define VRX 0
#define VRY 1
#define SW 10

#define GREEN_LED 8
#define RED_LED 9
#define BUZZER 3

// ==================== OLED Objects ======================
Adafruit_SSD1306 oledQ(SCREEN_WIDTH, SCREEN_HEIGHT, &Wire, -1);
Adafruit_SSD1306 oledA(SCREEN_WIDTH, SCREEN_HEIGHT, &Wire, -1);

// ==================== WiFi and Server ======================
const char* ssid = "Asw_S";
const char* password = "There is no password";
String serverUrl = "https://edu-kit-rose.vercel.app/generate";
String firebaseUrl = "https://test-7de2c-default-rtdb.asia-southeast1.firebasedatabase.app/word.json";  // <-- Replace this

DynamicJsonDocument doc(8192);
int questionIndex = 0, totalQuestions = 5, score = 0;

String questionText;
String options[4];
int correctIndex = 0;

String mode = "";
String difficulty = "";
String grade = "";

int cursor = 0;
bool pressed = false;

// ================================================================
// Helper Functions
// ================================================================
void drawList(String title, String arr[], int n, int highlight) {
  oledQ.clearDisplay();
  oledQ.setTextSize(1);
  oledQ.setTextColor(SSD1306_WHITE);
  oledQ.setCursor((SCREEN_WIDTH - title.length() * 6) / 2, 0);
  oledQ.println(title);

  for (int i = 0; i < n; i++) {
    if (i == highlight) oledQ.fillRect(0, 14 + i * 12, SCREEN_WIDTH, 12, SSD1306_WHITE);
    oledQ.setTextColor(i == highlight ? SSD1306_BLACK : SSD1306_WHITE);
    oledQ.setCursor(5, 15 + i * 12);
    oledQ.println(arr[i]);
  }
  oledQ.display();
}

void drawCenter(String msg, int textSize = 1) {
  oledQ.clearDisplay();
  oledQ.setTextSize(textSize);
  oledQ.setTextColor(SSD1306_WHITE);
  int x = (SCREEN_WIDTH - msg.length() * 6 * textSize) / 2;
  oledQ.setCursor(x, (SCREEN_HEIGHT - textSize * 8) / 2);
  oledQ.println(msg);
  oledQ.display();
}

void showCountdown(int seconds, String qNo) {
  for (int i = seconds; i > 0; i--) {
    oledQ.clearDisplay();
    oledQ.setTextSize(1);
    oledQ.setTextColor(SSD1306_WHITE);
    oledQ.setCursor(10, 0);
    oledQ.print("Q");
    oledQ.print(qNo);
    oledQ.print(" - ");
    oledQ.print(i);
    oledQ.print("s");
    oledQ.setCursor(0, 16);
    oledQ.println(questionText);
    oledQ.display();
    delay(1000);
  }
}

void selectWithJoystick(String prompt, String arr[], int n, int &selected) {
  cursor = 0;
  bool chosen = false;
  while (!chosen) {
    drawList(prompt, arr, n, cursor);
    int xVal = analogRead(VRX);
    int yVal = analogRead(VRY);
    int swVal = digitalRead(SW);
    if (yVal > 3000) cursor = min(cursor + 1, n - 1);
    if (yVal < 1000) cursor = max(cursor - 1, 0);
    if (swVal == LOW && !pressed) {
      pressed = true;
      selected = cursor;
      chosen = true;
    } else if (swVal == HIGH) pressed = false;
    delay(150);
  }
}

// ================================================================
// Firebase Upload
// ================================================================
void uploadToFirebase(String word) {
  HTTPClient http;
  // Upload directly to /word.json (overwrite the same key each time)
  http.begin(firebaseUrl);
  http.addHeader("Content-Type", "application/json");

  // Send the word directly as a JSON string (not as {"word": "value"})
  String data = "\"" + word + "\"";

  int httpResponseCode = http.PUT(data);  // use PUT to overwrite instead of POST

  oledQ.clearDisplay();
  oledQ.setTextSize(1);
  oledQ.setTextColor(SSD1306_WHITE);
  if (httpResponseCode > 0) {
    oledQ.setCursor(0, 20);
    oledQ.println("Uploaded: " + word);
  } else {
    oledQ.setCursor(0, 20);
    oledQ.println("Upload Failed!");
  }
  oledQ.display();
  delay(1500);
  http.end();
}


// ================================================================
// Words Mode (Alphabet Navigation + Word List)
// ================================================================
void startWordsMode() {
  char alphabet = 'A';
  bool inWordList = false;

  while (true) {
    if (!inWordList) {
      // Show alphabet selection
      oledQ.clearDisplay();
      oledQ.setTextSize(5);
      oledQ.setTextColor(SSD1306_WHITE);
      oledQ.setCursor((SCREEN_WIDTH - 30) / 2, 10);
      oledQ.println(alphabet);
      oledQ.display();

      int xVal = analogRead(VRX);
      int swVal = digitalRead(SW);

      if (xVal > 3000) {  // right → next letter
        alphabet = (alphabet == 'Z') ? 'A' : alphabet + 1;
        delay(250);
      } else if (xVal < 1000) {  // left → previous letter
        alphabet = (alphabet == 'A') ? 'Z' : alphabet - 1;
        delay(250);
      } else if (swVal == LOW && !pressed) {
        pressed = true;
        inWordList = true;
        oledQ.clearDisplay();
        oledQ.display();
      } else if (swVal == HIGH) {
        pressed = false;
      }
    } 
    else {
      // Fetch actual word list from Gemini backend
      HTTPClient http;
      http.begin(serverUrl);
      http.addHeader("Content-Type", "application/json");
      String jsonData = "{\"mode\":\"words\", \"difficulty\":\"easy\", \"letter\":\"" + String(alphabet) + "\"}";
      int httpResponseCode = http.POST(jsonData);

      String words[5];
      if (httpResponseCode > 0) {
        String payload = http.getString();
        DynamicJsonDocument resp(2048);
        deserializeJson(resp, payload);
        JsonArray arr = resp["words"].as<JsonArray>();
        for (int i = 0; i < 5 && i < arr.size(); i++) {
          words[i] = arr[i].as<String>();
        }
      } else {
        for (int i = 0; i < 5; i++) words[i] = "Error";
      }
      http.end();

      int selected = 0;
      bool exitWordList = false;

      while (!exitWordList) {
        drawList(String(alphabet) + " Words", words, 5, selected);

        int yVal = analogRead(VRY);
        int xVal = analogRead(VRX);
        int swVal = digitalRead(SW);

        if (yVal > 3000) selected = min(selected + 1, 4);
        if (yVal < 1000) selected = max(selected - 1, 0);
        if (xVal > 3000) { // Move joystick down to go back
          inWordList = false;
          exitWordList = true;
          delay(250);
          break;
        }

        if (swVal == LOW && !pressed) {
          pressed = true;
          uploadToFirebase(words[selected]);
          delay(500);
        } else if (swVal == HIGH) {
          pressed = false;
        }
        delay(150);
      }
    }
  }
}



// ================================================================
// Quiz Logic
// ================================================================
void showQuestion() {
  int seconds = (difficulty == "Easy") ? 7 : (difficulty == "Medium" ? 5 : 3);

  while (questionIndex < totalQuestions) {
    drawCenter("Q" + String(questionIndex + 1), 3);
    delay(1000);

    JsonObject q = doc["questions"][questionIndex];
    questionText = q["question"].as<String>();
    for (int i = 0; i < 4; i++) options[i] = q["options"][i].as<String>();
    correctIndex = q["correct"].as<int>();

    showCountdown(seconds, String(questionIndex + 1));

    int sel = 0;
    selectWithJoystick("Pick option", options, 4, sel);

    drawCenter(sel == correctIndex ? "Correct!" : "Wrong!", 2);
    if (sel == correctIndex) {
      digitalWrite(GREEN_LED, HIGH);
      tone(BUZZER, 1000, 200);
      delay(800);
      digitalWrite(GREEN_LED, LOW);
      score++;
    } else {
      digitalWrite(RED_LED, HIGH);
      tone(BUZZER, 300, 400);
      delay(800);
      digitalWrite(RED_LED, LOW);
    }

    questionIndex++;
  }

  oledQ.clearDisplay();
  oledQ.setTextSize(2);
  oledQ.setTextColor(SSD1306_WHITE);
  oledQ.setCursor(20, 20);
  oledQ.print("Score:");
  oledQ.setCursor(40, 45);
  oledQ.print(String(score) + "/" + String(totalQuestions));
  oledQ.display();
  delay(5000);

  drawCenter("Test Complete!", 1);
  oledQ.setCursor(10, 50);
  oledQ.setTextSize(1);
  oledQ.println("Press to Restart");
  oledQ.display();

  while (true) {
    if (digitalRead(SW) == LOW) {
      delay(300);
      questionIndex = 0;
      score = 0;
      showQuestion();
    }
    delay(100);
  }
}

// ================================================================
// Setup
// ================================================================
void setup() {
  Serial.begin(115200);
  pinMode(VRX, INPUT);
  pinMode(VRY, INPUT);
  pinMode(SW, INPUT_PULLUP);
  pinMode(GREEN_LED, OUTPUT);
  pinMode(RED_LED, OUTPUT);
  pinMode(BUZZER, OUTPUT);

  Wire.begin(OLEDQ_SDA, OLEDQ_SCL);

  if (!oledQ.begin(SSD1306_SWITCHCAPVCC, 0x3C)) {
    Serial.println("Question OLED not found!");
    while (1);
  }

  Wire.begin(OLDA_SDA, OLDA_SCL);

  if (!oledA.begin(SSD1306_SWITCHCAPVCC, 0x3C)) {
    Serial.println("Answer OLED not found!");
    while (1);
  }

  drawCenter("Connecting WiFi...");
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  drawCenter("WiFi Connected!");
  delay(1000);

  String modes[5] = {"Alphabet", "Numbers", "Math", "Quiz"};
  int modeSel;
  selectWithJoystick("Select Mode", modes, 4, modeSel);
  mode = modes[modeSel];

  if (mode == "Alphabet") {
    startWordsMode(); // Jump directly to words mode loop
    return;
  }

  String diffs[3] = {"Easy", "Medium", "Hard"};
  int diffSel;
  selectWithJoystick("Select Difficulty", diffs, 3, diffSel);
  difficulty = diffs[diffSel];

  HTTPClient http;
  http.begin(serverUrl);
  http.addHeader("Content-Type", "application/json");
  String body = "{\"mode\":\"" + mode + "\",\"difficulty\":\"" + difficulty + "\"}";
  int code = http.POST(body);
  if (code > 0) {
    String payload = http.getString();
    deserializeJson(doc, payload);
    JsonArray questions = doc["questions"].as<JsonArray>();
    totalQuestions = min((int)questions.size(), 5);
  } else {
    drawCenter("HTTP Err", 2);
    while (true);
  }
  http.end();

  showQuestion();
}

void loop() {
  // Nothing here, handled in functions
}

The python code that needs to be deployed in the Vercel which is responsible for the prompting in the Gemini using Gemini API is given below and this needs to be uploaded to the Github and then cnnect it with the Vercel.

from flask import Flask, request, jsonify
import requests
import os
import json
import re
import random
import datetime

app = Flask(__name__)

# --- Load Gemini API Key ---
GEMINI_API_KEY = os.getenv("GEMINI_API_KEY")
if not GEMINI_API_KEY:
    raise ValueError("GEMINI_API_KEY not found. Please set it as environment variable.")

GEMINI_URL = "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent"

# --- Gemini Request Function ---
def fetch_questions(mode, difficulty, letter=None):
    random_seed = random.randint(1000, 9999)
    time_stamp = datetime.datetime.now().strftime("%H:%M:%S")

    if mode.lower() == "words" and letter:
        # Word learning mode
        prompt = (
            f"Generate 5 simple words that start with the letter '{letter.upper()}'. "
            f"Words should be suitable for children (easy to pronounce, common). "
            f"Return only a JSON array of words, like ['Apple', 'Ant', 'Axe', 'Air', 'Arm']. "
            f"Make them different each time. "
            f"Session: {random_seed}-{time_stamp}."
        )

        payload = {"contents": [{"parts": [{"text": prompt}]}]}
        headers = {"Content-Type": "application/json", "x-goog-api-key": GEMINI_API_KEY}

        response = requests.post(GEMINI_URL, headers=headers, json=payload)
        if response.status_code != 200:
            raise Exception(f"Gemini API error: {response.text}")

        result = response.json()
        text = result["candidates"][0]["content"]["parts"][0]["text"].strip().strip("`")
        if text.startswith("json"):
            text = text[4:].strip()

        # Normalize Gemini output (replace single quotes and fix)
        text = text.replace("'", '"')
        text = re.sub(r'(\w+):', r'"\1":', text)  # ensure keys have quotes


        try:
            words = json.loads(text)
            if not isinstance(words, list):
                raise ValueError("Invalid word format")
            return {"letter": letter, "words": words}
        except json.JSONDecodeError:
            match = re.search(r"\[.*\]", text, re.DOTALL)
            if match:
                return {"letter": letter, "words": json.loads(match.group(0))}
            else:
                raise Exception("Gemini returned invalid JSON:\n" + text)

    # Other normal quiz modes
    else:
        prompt = (
            f"Generate 5 unique {mode} learning questions for children of age 1 - 7 based on the difficulty. "
            f"Difficulty: {difficulty}. "
            f"Each question must not exceed 7 words and the options should only be a single words and should be distinct. "
            f"Return only a valid JSON array in the format: "
            f"[{{'question': '...', 'options': ['A','B','C','D'], 'correct': 0/1/2/3}}]."
        )

        if mode == "Numbers":
            prompt += "Include questions like handling counts, what comes after what, based on shapes and so on and be clear with each question"
        elif mode == "Math":
            prompt += "Include questions using arithmetic operations (+, -, *, /) based on the difficulty that is being chosen (one digit (+,-,*,/) for easy, two digit (+,-,*,/) for medium and hard)"
        elif mode == "Quiz":
            prompt += "Include general knowledge based on animals, fruits, currency, countries, planets, and etc.."

        payload = {"contents": [{"parts": [{"text": prompt}]}]}
        headers = {"Content-Type": "application/json", "x-goog-api-key": GEMINI_API_KEY}

        response = requests.post(GEMINI_URL, headers=headers, json=payload)
        if response.status_code != 200:
            raise Exception(f"Gemini API error: {response.text}")

        result = response.json()
        text = result["candidates"][0]["content"]["parts"][0]["text"].strip().strip("`")
        if text.startswith("json"):
            text = text[4:].strip()

        try:
            questions = json.loads(text)
            if not isinstance(questions, list):
                raise ValueError("Invalid question format")
            return {"mode": mode, "difficulty": difficulty, "questions": questions}
        except json.JSONDecodeError:
            match = re.search(r"\[.*\]", text, re.DOTALL)
            if match:
                return {"mode": mode, "difficulty": difficulty, "questions": json.loads(match.group(0))}
            else:
                raise Exception("Gemini returned invalid JSON:\n" + text)


# --- Flask Routes ---
@app.route("/", methods=["GET"])
def home():
    return jsonify({"status": "Edu-Kit Gemini API running!"})


@app.route("/generate", methods=["POST"])
def generate_questions():
    data = request.json
    mode = data.get("mode", "math")
    difficulty = data.get("difficulty", "easy")
    letter = data.get("letter", None)

    try:
        result = fetch_questions(mode, difficulty, letter)
        return jsonify(result)
    except Exception as e:
        import traceback
        print(traceback.format_exc())
        return jsonify({"error": str(e)}), 500


if __name__ == "__main__":
    app.run(debug=True, port=5000)

8. Final Touches

With both firmware and backend stable, the kit works seamlessly:

1. Device boots → connects to Wi-Fi
2. Child selects mode and difficulty
3. Question appears on OLED with countdown
4. Screen switches to options
5. Joystick selects answer
6. LED feedback
7. For alphabet mode: selected word goes to Firebase → accessed later in the MIT App

The experience is clean, fast, and surprisingly fun for learners.