Data Expert

• Developed and implemented a standardized big data architecture for group-wide platform services in the Transport & Logistics sector on Azure
• Automated solutions using Infrastructure as Code (Terraform, Ansible)
• Presented and discussed sub-project architectures on Azure
• Implemented real-time data streaming with Apache Kafka and monitoring solutions
• Advised on Azure platform strategy and reference architectures
• Developed mechanisms for proactive elimination of vulnerabilities in Azure and Kubernetes clusters
• Conceptualized container orchestration platforms with Kubernetes CI/CD
• Created user and authorization concepts according to group specifications
• Managed operational services within an agile team

Services: Azure Purview, Azure Synapse Analytics, Azure Data Factory, Azure Databricks, Terraform, GitLab Runner, Azure DevOps

• Guided internal e-commerce product teams in developing, implementing, and maintaining high-performance data processing and integration systems
• Migrated existing data services, pipelines, and assets to a new event-based serverless architecture
• Developed and executed Lambda functions and PySpark jobs
• Designed architecture and integration with Kafka for real-time processing and analysis of event data
• Implemented PySpark transformations, filtering, and aggregations
• Ensured efficient and reliable connection with Kafka, configured security settings, and integrated with other components
• Established extensive testing and monitoring mechanisms
• Delivered a high-performance, scalable event system enabling data-driven decision-making

Services: AWS Glue, Apache Spark, Data Catalog, S3, Athena, Redshift, Lambda, ECS, Step Functions

• Spearheaded development of a robust data strategy and governance framework to streamline and enhance data handling capabilities
• Constructed a sophisticated data management platform on Databricks
• Designed and implemented an efficient data hub ingestion platform
• Led the design and establishment of an organization-wide data strategy aligned with business goals
• Developed a comprehensive data governance framework ensuring data accuracy, privacy, and compliance
• Oversaw deployment and customization of the data management platform on Databricks
• Enhanced data processing, analysis, and reporting capabilities with Power BI
• Engineered a robust data hub with advanced ingestion pipelines based on AWS EventBridge
• Optimized data flow from diverse sources to centralized storage systems (Data Lake House on Azure)
• Collaborated with cross-functional teams to integrate the data management platform with existing IT infrastructure
• Conducted training sessions and workshops to foster a data-driven culture and enhance data literacy

Services: Azure Databricks, Databricks Data Catalog, AWS EventBridge, Kinesis, Event Hub, Structured Streaming, Apache Spark