Data Engineer

Support and consulting services for an SCM application.

Technologies: Google Cloud, GKE, Kubernetes, SQL, Kafka, Kafka Connect, Kotlin, dbt, GitHub, Pekko.

• Supported the creation of a master data set for a supply chain management application.
• Designed and implemented ETL workflows.
• Analyzed data to improve data quality.
• Built a dashboard to monitor data quality.
• Set up CI/CD pipelines, monitoring, and alerting for operational ETL processes.

Support with migrating a data warehouse from an on-premise environment to the AWS Cloud.

Technologies: AWS Cloud, Airflow, Terraform, Python, Java, SQL, Cloudera, Hadoop, Glue, EMR, LakeFormation, GitLab.

• Designed and implemented a data warehouse using LakeFormation and Glue Catalog.
• Migrated existing Spark jobs to AWS Glue and AWS EMR.
• Implemented workflow management with Airflow.
• Designed and implemented multiple GitLab CI/CD pipelines.
• Trained and transferred knowledge to internal staff.

Design and implementation of infrastructure for sensor data processing, extension of an existing data science environment.

Technologies: AWS Cloud, Kubernetes, CloudFormation, Kafka, Kafka Connect, PySpark, Bamboo, Java, Python, Docker, Helm Charts, InfluxDB.

• Provided advice on design and tools for building a Kubernetes-based infrastructure for sensor data processing.
• Designed and built infrastructure on Kubernetes (Kafka cluster, Spark framework, Zookeeper, ZK manager).
• Designed and built an ETL pipeline for data ingestion.
• Designed and built a CI/CD pipeline with Bamboo and Kubernetes.

Design and implementation of a cloud-based data warehouse / data science environment.

Technologies: AWS Cloud, Kubernetes, Spark, R, NiFi, CloudFormation, Docker, Python, Jupyter Notebook.

• Designed a dynamically scalable data warehouse.
• Implemented infrastructure in CloudFormation (Infrastructure as Code).
• Implemented infrastructure components in Kubernetes.
• Modeled data warehouse and data storage.
• Built ETL processes for data ingestion.

Supported the evaluation of vendors in the Big Data field.

Technologies: Hortonworks, Cloudera, SAP Cloud, Apache NiFi, AWS Cloud, MS Azure.

• Recorded and documented technical and functional requirements for building and running an Apache Hadoop-based data warehouse.
• Collected quotes from various vendors and prepared the information for decision-making.
• Implemented a prototype for data ingestion.

Big Data Technologies Workshop – Introduction and how it works.

Technologies: Hadoop, Spark, AWS Cloud, MapReduce, Hive, Pig, R.

• Conducted a 3-day workshop.
• Introduced the Big Data/Hadoop ecosystem.
• Hands-on exercise on using Big Data in AWS Cloud.

Supported building ETL pipelines for a Hadoop-based data warehouse.

Technologies: Hadoop, Hive, Spark, Redis, Kafka, Avro, Scala, HCatalog, Schedoscope.

• Planned and implemented a Hive export module.
• Implemented Kafka & Redis export module as part of an open source project.
• Implemented an analysis algorithm to evaluate click streams.

Design and implementation of a data warehouse based on big data technologies – OLAP workload.

Technologies: Hadoop, Impala, Hive, ETL, AWS Cloud.

• Planned and implemented the cluster infrastructure.
• Evaluated different input formats for performance.
• Prepared and conducted load tests.

Design and implementation of a Hadoop-based data warehouse for game analytics.

Technologies: Hadoop, Map/Reduce, Kafka, Hive, ETL, Java, Linux.

• Planning and setup of a data warehouse.
• Evaluating different approaches to data collection.
• Selecting suitable technologies.
• Technical lead and coordination of a distributed team (GER, CN, CAN).
• Implementing a distributed, fault-tolerant high-throughput messaging system.

Design and implementation of a Big Data architecture for analyzing telecommunications data.

Technologies: Cloudera, Hadoop, Hive, Flume, Java, Spring, Puppet, Ubuntu Linux, AWS.

• Planning and setup of the network (VPC).
• Planning and setup of a Hadoop cluster (100TB capacity).
• Setting up deployment processes, including monitoring.
• Implementing a data ingestion framework to store about 300GB of data per day.

Reimplementation of an analysis tool as a Map/Reduce application.

Technologies: Cloudera, Hadoop/HBase, Java, Maven, Ganglia, Chef, PHP, Debian Linux.

• Analysis and integration of an existing implementation into the Map/Reduce framework with the Hadoop Streaming API.
• Installation and configuration of a Hadoop cluster including monitoring.
• Setup of a deployment process.

Integration of a payment provider into an existing backend.

Technologies: Ruby/Rails, OAuth, MySQL, Git, Debian Linux.

• Data mapping and text matching with existing data.
• Data preparation, conversion and import into the database.
• Integration of a payment provider.

Implementation and refactoring of an analysis tool in C++.

Technologies: C++, MySQL C/C++ API, Doxygen, Hudson, Ubuntu/Debian Linux.

• Setup of a build environment for C++ projects.
• Refactoring of the prototype.
• Adaptation and extension of the software to the production environment (logging, error handling, unit testing).
• Setup of a deployment process.
• Setup of a build server (continuous integration).

Design and development of a web crawler.

Technologies: C++, Fedora/RedHat Linux, Cassandra.

• Design of the application as a high-performance multithreaded server.
• Implementation as a distributed application using asynchronous sockets (non-blocking I/O).

Design, setup and deployment of a hosting environment for a large website.

Technologies: Apache, Nginx, HAProxy, Mongrel, MySQL, MySQLProxy, BIND, DHCP, Cobbler, Puppet, RedHat Linux.

• Design of the hosting environment.
• Setup of the infrastructure for server provisioning/configuration.
• Configuration of MySQL master-master replication + MySQLProxy.
• Installation of server software, monitoring and logging.
• Migration of the website from the hosting provider to colocation.
• Analysis and optimization to improve overall system performance.

Design and development of an infrastructure to integrate data from external providers.

Technologies: Ruby/Rails, MySQL, Bash, RedHat Linux.

• Building a music database with data from providers like Allmusic, Muze (Macromedia), Rhapsody, MediaNet.
• Data preparation, conversion, and import into the database.
• Mapping data from different providers.

• Responsible for the design, implementation, and deployment of the hosting environment
• Design and implementation of several software projects in C++

Implementation/porting of a method for extracting the main melody line.

Technologies: C++, Matlab, Mandrake Linux.

• Porting an existing Matlab algorithm to C++.
• Optimization of the implementation for performance.

Scientific investigation of data structures to determine the nearest neighbor for a Music Information Retrieval System.

Technologies: Matlab, C++, Perl, Apache, CGI, Mandrake Linux.

• Evaluated several nearest neighbor algorithms for suitability based on the given specifications
• Designed and implemented an algorithm in C++ and integrated it into the existing system (as a dynamic C++ library)
• Conducted and analyzed a test series for evaluation

• Collaborated on software for identity theft detection
• Data preparation and data analysis
• Contributed to the development and implementation of algorithms

• Development of test environments to evaluate algorithms
• Design and implementation of GUIs to present algorithms at trade fairs

• Provision of IT infrastructure