Data Cleaning and Normalization Tool
A data cleaning tool for detecting and fixing data quality issues like duplicates, missing values, formatting inconsistencies, and outliers. Provides rule-based cleaning, preview, and audit trail w...
Target user
Data analysts who prepare messy datasets for analysis
Problem
Analysts spend significant time cleaning messy data with duplicates, missing values, and formatting issues without a tool that automates detection and suggests fixes.
Requirements
- Detect duplicates and missing values
- Fix formatting inconsistencies
- Identify statistical outliers
- Apply rule-based cleaning operations
- Preview changes with audit trail
Core technologies
Python, pandas, FastAPI
Completion criteria
Users can detect, fix, and audit data quality issues with rule-based cleaning and statistical analysis.
Optional extensions
- Add ML-based anomaly detection
- Implement custom cleaning rules
- Add data profiling
- Support batch processing
Project phases
Phase 1: Phase 1: Issue Detection
Tasks: Set up Python project with pandas, Implement duplicate detection, Add missing value identification, Build outlier detection
Phase 2: Phase 2: Cleaning Operations
Tasks: Build formatting normalization, Implement rule-based cleaning, Add missing value imputation, Create preview interface
Phase 3: Phase 3: Audit and Export
Tasks: Implement audit trail logging, Add statistical summaries, Create export functionality, Build cleaning report