Portland Azure User Group
Website
Description
The Portland Azure User Group is a community of people excited about the endless possibilities of cloud computing, data analytics, and security on Microsoft Azure. We believe the Azure platform offers rich possibilities. We keep pace by sharing our learning and doing deep-dive explorations of topics as a community of learners. We run the gamut from new to experienced users and welcome anyone who shares an interest in cloud scalable infrastructure, applications, tools, analytics, identity management, and security.
Some great perks to joining our monthly meetups are networking, knowledge transfer, and community building. We aim to foster a vibrant community that shares success in implementing best practices for Microsoft Azure.
Session Details: Metadata Driven Pipelines for Dynamic Full and Incremental Processing in Azure SQL
Developing ETLs/ELTs can be a complex process when you add in business logic, large amounts of data, and the high volume of table data that needs to be moved from source to target. This is especially true in analytical workloads involving Azure SQL when there is a need to either fully reload a table or incrementally update a table. In order to handle the logic to incrementally update a table or fully reload a table in Azure SQL (or Azure Synapse), we will need to create the following assets: • Metadata table in Azure SQL ◦ This will contain the configurations needed to load each table end to end • Metadata driven pipelines ◦ Parent and child pipeline templates that will orchestrate and execute the ETL/ELT end to end • Custom SQL logic for incremental processing ◦ Dynamic SQL to perform the delete and insert based on criteria the user provides in the metadata table
This session uses Azure SQL DB as the source and sink databases. However, Azure SQL MI, On-Prem SQL, and Synapse Dedicated Pools (along with Synapse Pipelines) will also be compatible for this solution. As a source, you can use databases like MySQL, Oracle, and others. You will just need to adjust the query syntax/connections to match the desired source.
Guest Speaker: Marc Bushong Marc Bushong, Senior Cloud Solution Architect at Microsoft, has extensive experience in healthcare and higher education. He excels in Azure architecture, end-to-end solution design, and full-stack Azure development. His career began as a Radiologic Technologist, providing him with a unique perspective on the healthcare industry, from direct patient care to backend data engineering processes.
linux