Writing

The Hackathon Demo That Needed No Setup

Jun 3, 2025 · 3 min read

Most hackathon submissions are scripts or notebooks. Fragile prototypes that break once the lights are off. We entered the Airbyte + MindsDB Hackathon and built a deployable system instead: data ingestion, training, and inference running in real Kubernetes.

ProfiteerIO is an automated inventory forecasting engine. Simulates e-commerce traffic, pipes data through an ELT stack, and predicts purchasing trends in real time.

Here is how we built it, why we chose this stack, and how we solved memory constraints on a localized cluster.

The Architecture

We designed the system to be modular. You can rip out the synthetic data generator and plug in a Shopify API, or swap the inference engine without breaking the ingestion pipeline.

The data flows through five distinct stages:

  1. Simulation (FastAPI): Generates high-fidelity synthetic sales events.
  2. Ingestion (Airbyte): Handles ELT, moving data from REST endpoints to the warehouse.
  3. Storage (PostgreSQL): Stores raw logs, metadata, and ML predictions.
  4. Inference (MindsDB): Connects to Postgres to train models and write forecasts back to the DB.
  5. Visualization (React): Queries predictions for the user dashboard.

1. Generating Realistic Data (The Source)

We didn't have access to proprietary sales logs. We built a FastAPI service to simulate e-commerce traffic instead. Random data is useless for ML. The model needs patterns to learn from.

We programmed specific heuristic behaviors into the generator to test if the model could actually "learn" them later:

  • Time-based Seasonality: Traffic ramps up in the evening (18:00–22:00) and flatlines overnight (03:00–06:00).
  • Price Elasticity: We introduced a logic where higher prices increase "View" counts (perceived premium quality) but drastically lower "Cart Conversion" rates.
  • Rating Variance: Ratings fluctuate hourly to mimic user sentiment shifts.

This gave us a controlled dataset to validate our models. If the model didn't predict a dip in sales at 4:00 AM, we knew the training configuration was wrong.

2. The ML Engineering (The Brain)

We integrated MindsDB directly with PostgreSQL. Machine learning models became database tables. Query forecasts using standard SQL.

Model Selection

We initially tested K-Nearest Neighbors (KNN), but it struggled with noise in our synthetic data. We shifted to XGBoost. It handled feature interactions better, particularly the relationship between time_of_day_bucket and category_popularity.

Training Configuration:

  • Target: purchases
  • Lookback Window: 24 hours (to predict the next 1 hour).
  • Key Features: views, cart_adds, price_ratio, rating.

Handling Resource Constraints

Resource management was the biggest challenge. During retraining, MindsDB processes use heavy memory. Our initial runs hit 8GB RAM, causing Kubernetes pods to OOM (Out of Memory) crash.

To stabilize the cluster on standard hardware:

  1. Reduce batch sizes for ingestion and training.
  2. Enforce strict windowing. Discard logs older than 48 hours to keep memory footprint manageable.

3. Infrastructure as Code (The Delivery)

A major factor in our win was reproducibility. We didn't want the judges to spend hours setting up Python virtual environments.

We bundled the entire stack (FastAPI, Postgres, Airbyte, MindsDB, React, and NGINX) into a single Helm chart.

Deployment takes 6 minutes from helm install to live dashboard. Requires 2 CPUs and 7GB RAM. The Helm chart is self-contained. We iterated fast, and judges got a working environment immediately.

Retrospective

Building ProfiteerIO highlighted a trade-off: ease of use vs. resource cost.

MindsDB abstracted away complex MLOps pipelines. No airflow DAGs to write. But it shifted complexity to infrastructure. Tuning memory limits to prevent database starvation by the training process was hard.

The trade-off worked. A single engineer can deploy a full-stack ML pipeline that simulates, ingests, learns, and predicts. One command.

Resources