Vermont department of Public safety

The Vermont Department of Public Safety (DPS), in collaboration with the Vermont Agency of Digital Services (ADS), sought to modernize its data infrastructure.

The Challenge The department relied on a Computer Aided Dispatch Records Management System (CAD RMS) running on MySQL, alongside supplemental data scattered across SQL server instances and flat files within SharePoint and internal file stores. To harness this data effectively, they needed to build a Public Services Lakehouse environment that could unify these disparate sources.

The Solution Cogent Infotech facilitated the implementation of AWS Lake Formation to centrally govern, secure, and share data for analytics and machine learning.

• Centralized Governance: Leveraged Lake Formation to break down data silos by centralizing structured and unstructured data from Amazon S3 and on-premises databases.
• Hybrid Access Mode: Implemented a hybrid access model allowing data administrators to secure cataloged data using both Lake Formation permissions and IAM policies. This enabled selective and incremental onboarding of data use cases.
• Data Ingestion & Management: Utilized AWS Glue crawlers to extract schemas and store them in the AWS Glue Data Catalog, while using JDBC to import data from external sources.

Key Technologies The solution integrated a robust stack of AWS services:

• Storage & Ingestion: Amazon S3, AWS Glue, Amazon Kinesis, and AWS Transfer Family.
• Processing & Analytics: Amazon EMR, AWS Lambda, Amazon Redshift, and Amazon Athena.
• Security & Governance: AWS IAM, AWS KMS, AWS Lake Formation, and AWS Config.
• AI & Monitoring: Amazon SageMaker, Amazon CloudWatch, and AWS CloudTrail.

Best Practices Implemented To ensure long-term success, the project adhered to rigorous best practices including:

• Data Organization: Structured partitioning and metadata management.
• Security: Comprehensive compliance checks, regular audits, and disaster recovery planning.
• Automation: Implementation of CI/CD pipelines and continuous improvement workflows.
• Risk Management: Detailed analysis of risk probability and impact on project schedules.

‍