Spark data pipeline and diagnostics app at Lyten

An event triggered Spark pipeline on Databricks and a FastAPI and React app with an MCP server that replaced a manual diagnostics workflow for process engineers.

Problem

Process engineers at Lyten ran diagnostics by hand in SharePoint and Excel. Each run meant 421 serial historian queries per lot and 60 to 90 minutes of waiting.

What I built

An event triggered Python and Spark pipeline on Databricks that ingests 7M+ historian time series rows into Delta Live Tables. It replaces the serial queries with 5 batch queries per lot and stamps cycle membership at write time. A multi signal state machine classifies production cycles automatically. A FastAPI and React app serves the results to 6 process engineers with bounded SQL warehouse concurrency and tiered LRU/TTL caching. Additionally, for convenience, an MCP server was set up for quick analysis, served to all the process engineers.

Stack

Python · PySpark · Databricks · Delta Live Tables · FastAPI · MCP · React · SQL

Result

The daily run dropped from 60 to 90 minutes to 12 (5 to 7x). Cycle classification reached 96% validated accuracy with exact row level parity with the manual analyst process across 10+ lots. The app responds in under a second, and caching cut warehouse load by 80%.