A Unified AI System For Data Quality Control and DataOps Management in Regulated Environments
Devender Saini, Bhavika Jain, Nitish Ujjwal, Philip Sommer, Dan Romuald Mbanga, Dhagash Mehta

TL;DR
This paper introduces a unified AI-driven framework for data quality control and DataOps management in regulated environments, integrating multiple QC methods into a continuous, governed system to enhance data integrity, auditability, and compliance.
Contribution
It presents a novel integrated architecture combining rule-based, statistical, and AI methods for data QC within a continuous DataOps framework, specifically designed for regulated sectors.
Findings
Improved anomaly detection recall in financial data pipelines.
Reduced manual effort in data remediation processes.
Enhanced auditability and traceability in high-throughput workflows.
Abstract
In regulated domains such as finance, the integrity and governance of data pipelines are critical - yet existing systems treat data quality control (QC) as an isolated preprocessing step rather than a first-class system component. We present a unified AI-driven Data QC and DataOps Management framework that embeds rule-based, statistical, and AI-based QC methods into a continuous, governed layer spanning ingestion, model pipelines, and downstream applications. Our architecture integrates open-source tools with custom modules for profiling, audit logging, breach handling, configuration-driven policies, and dynamic remediation. We demonstrate deployment in a production-grade financial setup: handling streaming and tabular data across multiple asset classes and transaction streams, with configurable thresholds, cloud-native storage interfaces, and automated alerts. We show empirical gains…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsData Quality and Management · Scientific Computing and Data Management · Advanced Database Systems and Queries
