|
| 1 | +# Node.js Kafka Demo |
| 2 | + |
| 3 | +Real-time dashboard application that demonstrates Apache Kafka message streaming with Node.js using KafkaJS, Express, and Server-Sent Events (SSE). |
| 4 | + |
| 5 | +## Architecture |
| 6 | + |
| 7 | +``` |
| 8 | +┌─────────────┐ ┌───────────┐ ┌─────────────┐ ┌───────────────┐ |
| 9 | +│ Generators │───>│ Producer │───>│ Kafka │───>│ Consumer │ |
| 10 | +│ (fake data) │ │ │ │ Broker │ │ │ |
| 11 | +└─────────────┘ └───────────┘ └───────────┘ └──────┬────────┘ |
| 12 | + │ |
| 13 | + │ SSE |
| 14 | + v |
| 15 | + ┌───────────────┐ |
| 16 | + │ Dashboard │ |
| 17 | + │ (browser) │ |
| 18 | + └───────────────┘ |
| 19 | +``` |
| 20 | + |
| 21 | +1. **Generators** create random simulated data (orders, notifications, logs) |
| 22 | +2. **Producer** publishes messages to three Kafka topics every 3 seconds |
| 23 | +3. **Consumer** reads messages from Kafka and forwards them to the Express server |
| 24 | +4. **Express server** broadcasts messages to connected browsers via SSE |
| 25 | +5. **Dashboard** displays messages in real-time across three columns |
| 26 | + |
| 27 | +## About the displayed data |
| 28 | + |
| 29 | +All data shown in the dashboard is **simulated and fictitious**. The generators produce random data to demonstrate the Kafka messaging flow: |
| 30 | + |
| 31 | +- **Orders** — Random products (GPU, Monitor, Mouse, etc.) with random prices, quantities, and statuses |
| 32 | +- **Notifications** — Random types (Info, Warning, Error, Success) with messages like "SSL certificate expiring soon" or "User login detected" |
| 33 | +- **Logs** — Random levels (DEBUG, INFO, WARN, ERROR) from fictitious services (auth-service, api-gateway, payment-service, etc.) |
| 34 | + |
| 35 | +**None of the errors, warnings, or log messages represent real issues.** They are generated purely for demonstration purposes. |
| 36 | + |
| 37 | +## Prerequisites |
| 38 | + |
| 39 | +- Node.js >= 24 |
| 40 | +- A Kafka broker (local or remote) |
| 41 | + |
| 42 | +## Setup |
| 43 | + |
| 44 | +1. Install dependencies: |
| 45 | + |
| 46 | +```bash |
| 47 | +npm install |
| 48 | +``` |
| 49 | + |
| 50 | +2. Create a `.env` file from the example: |
| 51 | + |
| 52 | +```bash |
| 53 | +cp .env.example .env |
| 54 | +``` |
| 55 | + |
| 56 | +3. Edit `.env` with your Kafka broker settings: |
| 57 | + |
| 58 | +```env |
| 59 | +PORT=3000 |
| 60 | +KAFKA_BROKER=localhost:9092 |
| 61 | +PRODUCE_INTERVAL_MS=3000 |
| 62 | +KAFKA_USERNAME= |
| 63 | +KAFKA_PASSWORD= |
| 64 | +KAFKA_MECHANISM=scram-sha-256 |
| 65 | +``` |
| 66 | + |
| 67 | +When `KAFKA_USERNAME` and `KAFKA_PASSWORD` are set, SASL authentication and SSL are enabled automatically. Leave them empty for unauthenticated local connections. |
| 68 | + |
| 69 | +## Running |
| 70 | + |
| 71 | +```bash |
| 72 | +# Production |
| 73 | +npm start |
| 74 | + |
| 75 | +# Development (auto-reload) |
| 76 | +npm run dev |
| 77 | +``` |
| 78 | + |
| 79 | +Open http://localhost:3000 in your browser to view the dashboard. |
| 80 | + |
| 81 | +## Running with Docker Compose |
| 82 | + |
| 83 | +This starts both Kafka (KRaft mode, no Zookeeper) and the application: |
| 84 | + |
| 85 | +```bash |
| 86 | +docker compose up --build |
| 87 | +``` |
| 88 | + |
| 89 | +## Kafka Topics |
| 90 | + |
| 91 | +| Topic | Description | |
| 92 | +|-----------------|----------------------------------| |
| 93 | +| `orders` | Simulated e-commerce orders | |
| 94 | +| `notifications` | Simulated system notifications | |
| 95 | +| `logs` | Simulated application logs | |
| 96 | + |
| 97 | +## Environment Variables |
| 98 | + |
| 99 | +| Variable | Default | Description | |
| 100 | +|----------------------|------------------|--------------------------------------| |
| 101 | +| `PORT` | `3000` | Express server port | |
| 102 | +| `KAFKA_BROKER` | `localhost:9092` | Kafka broker address | |
| 103 | +| `PRODUCE_INTERVAL_MS`| `3000` | Interval between produced messages | |
| 104 | +| `KAFKA_USERNAME` | — | SASL username (optional) | |
| 105 | +| `KAFKA_PASSWORD` | — | SASL password (optional) | |
| 106 | +| `KAFKA_MECHANISM` | `scram-sha-256` | SASL mechanism (`scram-sha-256`, `scram-sha-512`, `plain`) | |
0 commit comments