Enterprise Data Management Platform
Built a microservices platform for dynamic form schemas, data validation, and audit logging for a large enterprise client.
A large organization requiring secure, compliant software systems
Project Overview
Developed a microservices platform for a large enterprise client that manages dynamic form schemas, validates data submissions, tracks changes through audit journals, and provides enumeration management for lookup values. The system enables mission-critical workflows while maintaining strict security and compliance requirements.
The Challenge
Large organizations require flexible data collection systems that can adapt to changing requirements while maintaining rigorous validation, audit trails, and security controls. The platform needed to support dynamic form definitions, validate submissions against complex rule sets, and provide comprehensive logging for compliance purposes.
Technical Architecture
The platform consists of four interconnected microservices, each handling a specific domain:
Core API Service
The primary backend service manages form schemas, handles data submissions, and coordinates with other services. Built with:
- Framework: Node.js with Koa for high-performance HTTP handling
- Database: PostgreSQL with Sequelize ORM for data persistence
- API Documentation: OpenAPI/Swagger specifications with automated validation
- Admin Interface: AdminJS for internal data management
Validation Service
A dedicated service for validating data against JSON Schemas and external rule engines:
- JSON Schema validation using AJV (Another JSON Validator)
- Integration with the Rules and Policy Engine (RPE) for complex business rules
- Machine feedback logging via Apache Pulsar for validation analytics
Enumeration Service
Manages dynamic lookup values and reference data:
- CRUD operations for enumeration types and values
- Search functionality across enumeration datasets
- Version control for enumeration changes
Journal Service
Provides comprehensive audit logging for compliance:
- Activity tracking across all platform operations
- Immutable audit trail for regulatory requirements
- Event streaming via Apache Pulsar
Key Contributions
Schema Management System
Built the core functionality for managing form schemas through their lifecycle:
- Schema versioning with status transitions (draft, review, published)
- Prevention of duplicate fields and invalid configurations
- UI schema generation for frontend form rendering
Validation Pipeline
Developed the validation service that ensures data integrity:
- JSON Schema validation for structural correctness
- Integration with external rules engines for business logic
- Comprehensive error reporting with actionable feedback
Security Remediation
Led efforts to address critical security vulnerabilities across all services:
- Dependency updates to resolve CVEs
- Security scanning integration with ThreadFix
- Secure configuration management with HashiCorp Vault
Enumeration Management
Built the enumeration service from the ground up:
- RESTful API for managing lookup values
- Search and filtering capabilities
- Bulk import/export functionality
Infrastructure & DevOps
The platform runs on a secure cloud infrastructure:
- Container Orchestration: Kubernetes with Helm charts for deployment
- CI/CD: GitLab CI pipelines with security scanning
- Messaging: Apache Pulsar for event-driven communication between services
- Secrets Management: HashiCorp Vault for secure credential storage
- Monitoring: Centralized logging and metrics collection
Security & Compliance
Working with sensitive data required adherence to strict security protocols:
- Secure development practices with regular security scans
- Audit logging for all data access and modifications
- Network isolation and access controls
- Compliance with enterprise security frameworks
Technologies Deep Dive
The technology choices balanced modern development practices with enterprise security requirements:
- Koa provided a lightweight, middleware-based framework well-suited for API development
- Sequelize offered type-safe database access with migration support for schema evolution
- Apache Pulsar enabled reliable, scalable messaging between services with built-in persistence
- Helm simplified Kubernetes deployments across multiple environments (dev, staging, production)
Technologies
Key Highlights
- Developed microservices architecture with four interconnected backend services
- Built schema validation service with JSON Schema and custom rules engine integration
- Implemented enumeration management system for dynamic lookup values
- Created audit journal service for compliance and activity tracking
- Addressed critical security vulnerabilities across all services