Daniel Pape
Professional Development
Erfahrungen
Jan. 2022 - Nov. 2023
11 MonatenProfessional Development
- In a proactive phase focused on enhancing my skills and knowledge, I dedicated my time effectively during a break from client projects:
- Certified Cloud Practitioner: Attained certification as an AWS Certified Cloud Practitioner, validating foundational cloud knowledge and proficiency in AWS services.
- New Programming Language: Mastered the foundations of Rust as new programming language through intensive self-study, including reading books, completing online courses, and contributing to open-source projects.
- Serverless Web Application (In Development): Development of a serverless web application using AWS and TypeScript/React in combination with D3 as part of a personal project this year. Although the application is still undergoing continuous development, I am prepared to provide a comprehensive demo or walkthrough to showcase its current functionalities and future potential. AWS services used include RDS, Lambda, Polly and Amplify. CDK is used for infrastructure management. This ongoing project has provided valuable hands-on experience in architecting serverless applications on AWS cloud infrastructure. Throughout the development process, I have gained proficiency in integrating AWS services such as RDS (Relational Database Service) for efficient database management, Lambda for serverless compute, Polly for text-to-speech functionality, and Amplify for seamless deployment and hosting.
- Continuous Learning: I consistently stay updated with industry trends and best practices through self-education, webinars, and online workshops. Recently, I delved into the Data Mesh approach and explored FastAPI.
Juni 2022 - Dez. 2022
7 MonatenData Engineer
Leading German multinational automotive manufacturer
- As a data engineer, I developed the replacement of existing clustering batch jobs for a software company belonging to a leading German multinational automotive manufacturer. This clustering uses location and additional metadata from worldwide distributed electric charging stations.
- As part of the company's team responsible for POI data management (POI=point of interest), I collaborated closely with the developers who previously worked on this task and utilised my expertise with Spark and Airflow to design and implement the new solution using Databricks on Azure.
- In collaboration with another team member, I worked on refactoring the existing Airflow pipeline for the batch jobs and the grown and complex business logic. We also refactored the library code used for the clustering and preprocessing steps. While doing so, I added missing documentation and improved code quality by using techniques from defensive programming.
- Based on the acquired domain knowledge and business requirements for the clustering, I built a new solution in Azure Cloud using Databricks. Tasks involved integrating existing data in AWS S3 and Azure blob storage, development of library code, and Spark job for geospatial clustering of charging station data.
- While the team mainly works with Python, the new solution also uses the official open-source Scala library for graph clustering. For this, I used my experience with Scala and JVM-based development to make the library callable from Python using a suitable wrapper class.
- At the beginning of the project, I also worked with the developers and testing team to eliminate bugs and increase the test coverage for an event-driven service. This Azure Functions-based service detects and removes certain privacy-related information within streams of vehicle signals. Redis is used to cache intermediary results detected in the event streams.
- Other tasks: Contribution to code reviews, PI planning, testing, and documentation.
- Skills and Technologies:
- Programming Languages: Python, Scala, SQL
- Build and Dependency Management: Poetry, SBT
- Data Processing and Analysis: PySpark, Spark, GraphX
- Cloud Services: Azure (Databricks, Blob Storage, Functions) and AWS (S3)
- Workflow Orchestration: Airflow
- In-memory Data Store: Redis
- Version Control and CI/CD: Git, Jenkins, Bitbucket
Mai 2021 - März 2022
11 MonatenData Engineer
Leading German media company
- As a Data Engineer, I managed data and machine learning models for automated customer relationship management. Our cross-functional team, including data scientists and engineers, operated under SaFe. The team's endeavors were instrumental in delivering valuable data insights and predictive models crucial for executing multi-message campaigns and CRM initiatives, supporting a user base of around 7 million.
- Designing and constructing ETL pipelines for contract and usage data, and generating features for machine learning models.
- Implementation of machine learning pipelines in the cloud supporting artificial intelligence use cases for churn scoring, next best actions, and customer behaviour prediction.
- Collaborating on the transition of Data Scientist-developed models from experimentation to production.
- Automating data exports and score transfers into the central event bus for consumption by other services.
- Established data quality monitoring using Great Expectations to ensure the accuracy, completeness, and consistency of incoming data.
- Collaborating with a fellow engineer during a spike to provide guidance for later steps in facilitating MLOps, including assessing workflows, repositories, and researching feature store integration.
- Skills and Technologies:
- Programming Languages: Python, SQL
- Data Processing and Analysis: Pandas, PySpark, NumPy
- Machine Learning Libraries: Scikit-Learn, CatBoost
- Data Quality Monitoring: Great Expectations
- Cloud Services: AWS (S3, Kinesis, Athena, EMR, Glue)
- Workflow Orchestration: Airflow
- Containerization and Orchestration: Docker, Kubernetes
- Infrastructure as Code: Terraform
- Version Control and CI/CD: Git, GitLab
Dez. 2020 - Dez. 2020
1 MonateData Engineer
Multinational telecommunications company
- Supported the client's data engineering team with the conceptual and architectural preparation of the extraction of data from an external API to their AWS-based Big Data Lake and the Redshift data warehouse.
- The API belongs to a SaaS platform used for campaign management and customer analytics, involving topics from Natural Language Processing (NLP) including for example sentiment analysis and phrase as well as keyword detection within customer text comments. Activities included:
- Gathering requirements from project stakeholders and requirement analysis.
- Clarification of issues related to Natural Language Processing and the API design with a contact person from the SaaS platform and stakeholders.
- Extension of documentation in Confluence.
- Preparation of logical data model and conceptual design of the ETL processing pipeline.
- Preparation of proof of concept for streaming data from Kafka cluster to S3 layer in AWS cloud using Databrick’s Delta Lake framework.
- Skills and Technologies:
- Programming Languages: Scala, SQL
- Big Data Processing and Analytics: Spark, Kafka, Delta Lake, Redshift, Natural Language Processing (NLP)
- Cloud Services: AWS (S3)
Jan. 2019 - Bis heute
6 Jahren 6 MonatenFreelance Software Developer
- As a freelancer, I build individual solutions with a focus on Data and Machine Learning Engineering.
Jan. 2019 - Jan. 2020
1 Jahr 1 MonateSoftware Developer
European fashion and lifestyle company
- Responsible as a software developer for the development of an AWS-based cloud data lake divided into several zones, which is also used for a downstream data mart in Redshift used by end-users for analytics and queries via Tableau. (Periods: 01/2019 - 05/2019 and 08/2019 - 01/2020). Activities included:
- Used Apache Nifi during the initial proof of concept phase for the extraction from source systems to S3 and Kinesis.
- Developed several software services with Scala and Spark for automatic extraction and transformation (ETL) of different source systems/databases to and within the AWS cloud, respectively.
- Implemented transformation logic for the creation of fact and dimension tables (data modelling following Kimball).
- Implementation of ETL pipelines using Spark and Scala, based on specifications and blueprints from the data analytics department and already existing Tableau Prep Flows.
- Development, adjustment, and deployment of programmatic workflows for scheduling Spark jobs on EMR clusters via Apache Airflow.
- Transfer of knowledge and mentoring regarding Scala and Spark to coworkers during team programming.
- Development of a custom mini-framework using Scala for working in a type-safe fashion with Spark data frames (worked together via team programming for this). The objective of this framework is to facilitate the development and testing of transformation components used in Spark jobs within ETL pipelines.
- Integration and replacement of new and old data sources, respectively.
- Development of integration and unit tests, debugging, and execution tests.
- Skills and Technologies:
- Programming Languages: Python, Scala, SQL
- Build and Dependency Management: Poetry, SBT
- Data Processing and Analysis: PySpark, Spark, GraphX
- Cloud Services: AWS (EMR, S3, SSM, Kinesis, Redshift)
- Data Integration and Workflow Orchestration: Airflow, NiFi
- Databases: MS SQL Server, SAP HANA
- Containerization and Orchestration: Docker, Kubernetes
- Version Control and CI/CD: GitHub, Jenkins, Bitbucket
- Artifact Management: JFrog Artifactory
Aug. 2015 - März 2018
2 Jahren 8 Monaten Hamburg, Deutschland
Data Engineer
codecentric AG
- codecentric AG is a German IT and services provider with 15 locations in Germany and other European countries headquartered in Solingen. The company develops custom software solutions for its clients and employs around 450+ experts in the area of agile software development.
- At codecentric, I worked as an engineer in a business unit focused on Data Science and Machine Learning and contributed to related customer and in-house projects.
Nov. 2014 - Mai 2015
7 Monaten Hamburg, Deutschland
Scala Developer
DIGITEC GmbH
- DIGITEC GmbH is a German software provider based in Hamburg. It develops, sells, and supports computer software for clients in the finance and banking industry used for trading in the money and foreign exchange markets.
- As a developer in a Scrum team, I implemented features for the upcoming version of the company's flagship desktop software suite.
Zusammenfassung
Software Engineer with 5+ years experience in building data-driven education software solutions. Enabled companies to take advantage of their data by building Machine Learning services to optimize their ad revenue up to building a data lake integrating various data sources to create a company-wide 360-degree view of their most important data in the cloud allowing actionable insights.
Sprachen
Deutsch
Muttersprache
Englisch
Verhandlungssicher
Französisch
Grundkenntnisse
Ausbildung
Georg August University of Göttingen
PhD · Mathematics · Göttingen, Deutschland
University of Wuppertal
Diploma · Mathematics and Computer Science (Minor) · Wuppertal, Deutschland
Zertifikate & Bescheinigungen
AWS Certified Cloud Practitioner
AWS
Confluent Developer Training: Building Kafka Solutions
Confluent
Amazon Detective Deep Dive
A Cloud Guru
Applied AI with Deep Learning
Coursera
BerkeleyX – CS190.1x: Scalable Machine Learning
edX
Building Machine Learning Pipelines on AWS
A Cloud Guru
Cluster Analysis in Data Mining
Coursera
Data Streaming Nanodegree
Udacity
Databricks Lakehouse Fundamentals
Databricks Academy
Deploying Applications with AWS CDK
A Cloud Guru
Developing Data Products
Coursera
Exploratory Data Analysis
Coursera
Functional Programming Principles in Scala
Coursera
Getting Started with Google Kubernetes Engine
Coursera
Introduction to Big Data with Apache Spark
edX
Kotlin for Java Developers
Coursera
Pattern Discovery in Data Mining
Coursera
Practical Event-Driven Security with AWS
A Cloud Guru
R Programming
Coursera
Statistical Inference
Coursera
Terraform Basics
Coursera
Text Mining and Analytics
Coursera
Ähnliche Freelancer
Entdecken Sie andere Experten mit ähnlichen Qualifikationen und Erfahrungen.