Gold Microsoft Partner
Microsoft Azure

Building robust data pipelines

Darren Hutton & Jonathan Abbott

Surreal portrait of a figure holding a tablet against a desert landscape Hands holding a tablet showing charts surrounded by analytics icons Person viewing a technical slide with diagrams on a large tablet Person browsing a dark dashboard of content cards on a tablet Hands holding a tablet with a dark UI in front of a laptop Tablet showing a checklist beside a data table on a wooden desk

Azure resources

  • Communication Services
  • SignalR Service
  • Stream Analytics
  • Data Catalogue / Schema Registry
  • Functions (incl Logic Apps)
  • Storage (Blob, Queues, Tables)
  • App Insights
  • DataFactory
  • Service Bus
End-to-end architecture from EUDs through Landing Zone and Data Lake to reporting

Data processing

Adhoc

Works with

  • Small/Med datasets
  • Semi-conformed data

Considerations

  • Often used as experiments (notebooks)
  • Security / Permissions
Batch

Works with

  • Large datasets
  • Scheduled processing

Considerations

  • How to deal with interruptions
  • Impact on transactional systems
Streaming

Works with

  • Times-series data
  • Event data

Considerations

  • Hard to conform reference data
  • Eventual consistency model

What’s generating
the data?

Front-end client – WebRTC

WebRTC signaling, media stream, STUN and TURN diagram
SignalR Service
Communication Services
Network Traversal

Landing Zone

  • Used as a security gateway
  • Usually a public share
    • Fronted by an API gateway
    • Azure data shares
  • Key place for virus checking
  • Flagging and Quarantine (Human review)
Landing Zone diagram with API Services, Blob Storage, and landing file processing

Medallion what?

Medallion Pattern

Represents an increasing quality of data in the stores

  • Simple data model
  • Allows for multiple processing (replay, time-travel)
  • Supports ACID transactions (atomicity, consistency, isolation, and durability)
  • Supports incremental ETL
Data Lake medallion layers Bronze, Silver, Gold

Bronze layer

The bronze layer contains unvalidated data. Data ingested in the bronze layer:

  • Maintains the raw state of the data source.
  • Is appended incrementally and grows over time.
  • Combination of streaming and batch transactions.

Silver layer

Bronze layer contains the entire data history. Silver layer represents a version of the data that is:

  • Validated
  • Enriched
  • Trusted by downstream analytics.

Gold layer

Highly refined and aggregated, containing data for power analytics, machine learning, and production applications.

Gold represents data that has been transformed into knowledge, rather than just information.

Analysts rely on gold for their core responsibilities.

Aggregations, joins, and filtering are handled before data is written to the gold layer — users should see low latency query performance on data in gold.

Democratising data

Reporting API

  • HAL – Hypertext Application Language
HAL resource structure with resource state, links, and embedded resources Reporting API connecting Data Lake Gold to Power BI, SPSS, and R

Data as a product

  • Create, maintain and validated datasets accessible by all
  • Mindset focused on curating data-as-products, rather than tooling

Thank you