Siddharth N.
Principal Consultant
Experience
Jul 2021 - Present
4 years 6 monthsPrincipal Consultant
StatusNeo Consulting
- Consulted with CXO-level stakeholders on data strategy, architecture reviews, and modernization roadmaps across Hospitality, Finance, Retail, Sports, and Telecom.
- Led an 8-member Data, Analytics & AI team delivering large-scale data engineering and warehousing solutions, mentoring juniors, and collaborating with cross-functional teams for seamless delivery.
- Designed and delivered an end-to-end Medallion Architecture on Microsoft Fabric integrating Opera APIs, ReviewPro, SFTP, and on-prem sources.
- Implemented Direct Lake models in Power BI, reducing dashboard load times from 1+ min to <10 sec.
- Launched an AI-powered Data Agent enabling conversational queries for CXOs.
- Enforced RBAC, governance, and lineage to ensure compliance and security.
- Migrated pipelines from Azure Data Factory to Databricks, orchestrating transformations with dbt and Delta Lake.
- Integrated Salesforce and RDBMS sources using reusable ETL frameworks with SCD1 and SCD2 handling.
- Applied DataOps practices including CI/CD, monitoring, and quality checks for reliability.
- Successfully decommissioned the legacy data warehouse, cutting daily costs from 250 to 50 USD through cost optimization.
- Automated P&L reporting with Alteryx workflows and Power BI dashboards, cutting manual effort by 80%.
- Designed advanced transformations with Crew Macros, improving reporting accuracy.
- Established DataOps pipelines for automated validation and CI/CD.
- Built a real-time Cricket Data Analytics Platform ingesting high-frequency JS events into Cassandra.
- Developed Flask/OpenAPI microservices deployed on AKS for web and mobile APIs.
- Delivered real-time streaming and batch processing pipelines balancing low latency and scalability.
- Enabled instant match and event updates via FCM push notifications with SSL pinning and parameterized queries for security.
- Built preprocessing pipelines for catalog data, establishing Elasticsearch indexes and a Redis serving layer.
- Developed FastAPI suggest APIs for high-performance search and deployed them as microservices on AKS with CI/CD.
Nov 2020 - Jul 2021
9 monthsConsultant
Deloitte USI
- Designed and deployed scalable ETL pipelines using Azure Data Factory and Databricks, reducing ETL runtime by 50%.
- Built reusable dimension and fact frameworks with SCD1 and SCD2 handling on Delta Lake, improving maintainability and historical accuracy.
- Implemented dynamic data masking in Azure Synapse to protect PII and ensure compliance with regulatory requirements.
- Partnered with cross-functional business teams to translate requirements into scalable technical solutions, improving reporting efficiency and data trust.
Jan 2020 - Nov 2020
11 monthsBig Data Engineer
Reddoorz
- Built AWS ETL pipelines with Apache NiFi, Step Functions, and Redshift, optimizing SQL queries and reducing load times by 40%.
- Developed Python-based validation and SCD frameworks for Redshift, with monitoring and orchestration automated via CloudWatch and Lambda.
- Replaced legacy reporting tool “Holistic” by implementing automated pipelines, reducing CXO report generation from hours to minutes.
- Improved data quality and consistency across pipeline outputs, ensuring reliable executive-level reporting.
Jul 2017 - Dec 2019
2 years 6 monthsTechnical Associate
Genpact India Pvt. Ltd.
- Developed reusable ETL and validation frameworks in Python and Spark with SCD1 and SCD2 logic, improving data consistency across ingestion jobs.
- Automated large-scale batch processing pipelines (>100 TB) using AWS Data Pipeline, Step Functions, Lambda, CloudWatch, and EMR, reducing manual oversight and improving reliability.
- Optimized Amazon Redshift schema design and queries, improving performance and reducing storage and compute costs.
- Implemented monitoring and alerting using CloudWatch and integrated workflows with CloudFormation for better orchestration.
- Partnered with onshore and offshore teams to deliver data migration and modernization projects under strict SLAs.
Summary
Accomplished Data Architect with 8+ years of experience helping organizations transform their data into a strategic asset. Proven ability to bridge business and technology by consulting with stakeholders, shaping enterprise architectures, and delivering modern cloud-native platforms. Skilled at leading high-performing teams and guiding clients across industries including hospitality, finance, telecom, sports, power & energy and retail.
Hands-on with AWS, Azure, Microsoft Fabric, and GCP, bringing a holistic approach to data engineering, governance, and analytics that drives innovation and competitive advantage.
Languages
Hindi
NativeEnglish
AdvancedGerman
ElementaryEducation
Oct 2013 - Jun 2017
AKTU
B.Tech, Computer Science Engineering · Computer Science Engineering · Lucknow, India
Need a freelancer? Find your match in seconds.
Try FRATCH GPT More actions
Similar Freelancers
Discover other experts with similar qualifications and experience
- English
- Deutsch
2025 © FRATCH.IO GmbH. All rights reserved.