This session is designed for data professionals, engineers, and enthusiasts who are eager to stay ahead of the curve in the rapidly evolving field of data engineering. We will delve into the key trends, technologies, and strategies that will shape the data engineering landscape over the next few years. <ul><li>Brief Overview of the Data Engineering Landscape</li><li>Current Trends in Data Engineering</li><li>Highlighting the importance of data pipelines, real-time processing, and data governance.</li><li>Tools and Technologies to Embrace (technologies like Apache Spark, Kafka, Delta Lake, and cloud-native solutions)</li><li>Best Practices for Data Pipeline Design</li><li>Discussing approaches to handle the increasing volume and variety of data.</li><li>Exploring options for auto-scaling and distributed computing.</li><li>Q&amp;A Session</li></ul>

• Implemented a data warehouse for approx. 50GB of datasets for Agricultural equipment data using Microsoft Cloud Technologies (Azure Synapse, Azure Data Factory, Azure SQL Server, Azure Storage, Azure Databricks).
• Guided and mentored over four recent college graduates, aiding them in seamlessly integrating into the office environment.
• Recognition - Awarded Best Team (Quarter)



Senior Software Engineer (Big Data)

• Crafted and Migrated Field Reporting Solution for ~500 Pharma Sales Representatives from AWS Cloud Services + MicroStrategy and implemented smooth-scalable-optimized system using Microsoft Cloud Services and saving vendor costs by 10%.
• Conducted the Root Cause Analysis of bad data and queried approx. 1M records loaded into the Big Data Environment and implemented a metadata driven Data Quality Management tool with the help of Azure SQL Server, Data Factory, Data Bricks helping client reduce the data issue resolution time by 75% thereby leading to 70% of client time savings on a weekly basis.
• Helped Client in information decision marketing to combat global pandemic challenge via COVID-19 Impact Analysis Web Dashboards.
• Authored quality content to train 100+ associates in crucial branches of pharma business space.
• Oversaw stakeholder management, led onshore & offshore teams to launch multiple drugs in specialty and oncology space, analyzed its outreach, and created pre-sales promotion (critical for the client) resulting in client saving $400K yearly in incentives to field reps.
• Mentored 10 new members of my team over 2 years.
• Created training curriculum from scratch and led a 60-member team in cross functional trainings in different parts of Pharma domain such as Incentive compensation, online/offline reporting, Call Plan, & Data governance, resulting in turnaround time reduced by 94 working hours, thereby increasing company billables.
• Worked on cloud data pipeline using technologies Athena , Glue, Lambda.
• Recognition – Awarded Best performer for 2 quarters for helping in increasing business understanding of various stakeholders and showcasing excellent technical skills resulting in cost savings for the organization.