Slide 1: 10 Best Practices for Data Engineers
10 Best Practices for Data Engineers Clarke Patterson | Head of Product Marketing © 2020 Snowflake Inc. All Rights Reserved
Slide 2: A poll...
A poll... I’ve been a data engineer for: 1. Just starting 2. Less than a year 3. More than a year 4. I’m an expert! © 2020 Snowflake Inc. All Rights Reserved
Slide 3: What does a data engineer do, exactly?
What does a data engineer do, exactly? ● Build data pipelines to collect data and move it into storage; ● Prepare the data as part of an ETL or ELT process; ● Stitch the data together with scripting languages; ● Work with the DBA to construct data stores; ● Ensure the data is ready for use; ● Use fr
Slide 4: #1: Enable your pipeline to handle concurrent workloads
#1: Enable your pipeline to handle concurrent workloads © 2020 Snowflake Inc. All Rights Reserved
Slide 5: Operate isolated workloads, concurrently
OPERATE ISOLATED WORKLOADS, CONCURRENTLY Compute Data Resources Science (2XL) TestDev (XL) w/ZeroCopy Clone Batched data Pipeline Executive Dashboards (Med) (Large) BI Users (Med) Streaming data pipeline 3AM 6AM 9AM Noon 3PM 6PM 9PM Midnight Customize for business needs No contention – No data copyi
Slide 6: Real-world use case
REAL-WORLD USE CASE 50% < 1s Interactive 85% < 2s Continuous S3 Dashboard 95% < 5s Loading (4TB/day) <5min SLA Virtual Warehouse Virtual Auto Scale – X-Large x Warehouse Reporting 5 Medium ETL & (Segmented) Maintenance Prod DB Virtual Virtual Warehouse Warehouse 2X-Large Large 4 trillion rows 3+ pet
Slide 7: #2: Tap into existing skills to get the job done
#2: Tap into existing skills to get the job done © 2020 Snowflake Inc. All Rights Reserved
Slide 8: You have a choice... SQL
You have a choice... SQL © 2020 Snowflake Inc. All Rights Reserved
Slide 9: #3: Use data streaming instead of batch ingestion
#3: Use data streaming instead of batch ingestion** © 2020 Snowflake Inc. All Rights Reserved
Slide 10: Snowpipe
Automate ingest when and where possible Snowpipe © 2020 Snowflake Inc. All Rights Reserved
Slide 11: #4: Streamline & operationalize pipeline development
#4: Streamline & operationalize pipeline development © 2020 Snowflake Inc. All Rights Reserved
Slide 12: Time Travel for Data
“TIME TRAVEL” FOR DATA > SELECT * FROM mytable AS OF T0 Previous versions of data automatically retained > SELECT * FROM mytable AS OF T1 Retention period selected by customer Accessed via SQL extensions T0 T1 T2 AS OF for selection CLONE to recreate New data Modified data UNDROP recovers from accid
Slide 13: Zero-Copy Data Cloning
ZERO-COPY DATA CLONING Instant data cloning operations Databases, schema, tables, etc Metadata-only operation Modified data stored as new blocks Unmodified data stored only once No data copying required, no cost! Instant test/dev environments Test code on your entire production dataset Swap tables i
Slide 14: #5: Invest in tools with built-in connectivity
#5: Invest in tools with built- in connectivity © 2020 Snowflake Inc. All Rights Reserved
Slide 15: Relational Database Extended to Semi-Structured Data
RELATIONAL DATABASE EXTENDED TO SEMI-STRUCTURED DATA Semi-structured data > SELECT … FROM (JSON, Avro, XML, Parquet, ORC) … Structured data (e.g., CSV, TSV, …) Storage optimization Query optimization Transparent discovery and storage optimization + Full database optimization for queries of repeated
Slide 16: Ever Expanding Ecosystem
Platform BI/Analytics ETL EVER EXPANDING ECOSYSTEM Data Science Services
Slide 17: A poll...
A poll... What data do you primarily work with today? 1. Structured data 2. JSON 3. XML 4. AVRO 5. All of the above! © 2020 Snowflake Inc. All Rights Reserved
Slide 18: #6: Incorporate extensibility
#6: Incorporate extensibility © 2020 Snowflake Inc. All Rights Reserved
Slide 19: Embrace each others differences!
Embrace each others differences! © 2020 Snowflake Inc. All Rights Reserved
Slide 20: #7: Enable data sharing in your pipelines
#7: Enable data sharing in your pipelines © 2020 Snowflake Inc. All Rights Reserved
Slide 21: Traditional Methods for Sharing Data
TRADITIONAL METHODS FOR SHARING DATA 1 Data 2 API Transfer ● Development ● Maintenance ● Support ● Data extraction, transformation (for providers and consumers) ● Loading into FTP, Cloud Bucket ● Repeat for every update ● Time Intensive ● Sub-par User Experience ● Costly ● Raises Security Concerns ●
Slide 22: A Better Way to Share Data
A BETTER WAY TO SHARE DATA Data Providers Data Consumers No data movement Ready to use Share with unlimited number Consumers can immediately of consumers Live access start querying Data consumers immediately see all updates © 2020 Snowflake Inc. All Rights Reserved 22
Slide 23: #8: Choose the right tool for data wrangling
#8: Choose the right tool for data wrangling © 2020 Snowflake Inc. All Rights Reserved
Slide 24: Continuous Data Pipelines
CONTINUOUS DATA PIPELINES How do I know what changed in the staging table? Where do I keep the Snowflake business logic for my transformations? How do I run this on a schedule? Target Table Table 1 Stream Staging Table Target How do IFiles make this run Task Task Snowpipe Table Stream Table 3 reliab
Slide 25: #9: Build data cataloging into your engineering strategy
#9: Build data cataloging into your engineering strategy © 2020 Snowflake Inc. All Rights Reserved
Slide 26: Data Exchange: Discovery
DATA EXCHANGE Discovery Listings Metrics Control SECURE DATA SHARING © 2020 Snowflake Inc. All Rights Reserved
Slide 27: #10: Rely on data owners to set security policy
#10: Rely on data owners to set security policy © 2020 Snowflake Inc. All Rights Reserved
Slide 28: Security Is a Team Sport
SECURITY IS A TEAM SPORT Access Authentication Application Data Infrastructure © 2020 Snowflake Inc. All Rights Reserved 28
Slide 29: How does Snowflake help?
How does Snowflake help? © 2020 Snowflake Inc. All Rights Reserved
Slide 30: Modern Data Architecture with Snowflake
MODERN DATA ARCHITECTURE WITH SNOWFLAKE Data Sources Data Consumers OLTP Data Data Data Data Data Data Data Databases ETL, Streaming Warehouse Lake Engineering Exchange Applications Science Monetization Enterprise Applications Operational Reporting Third-Party Ad Hoc Web/Log Analysis Data Real-time
Slide 31: Modern Data Architecture with Snowflake
MODERN DATA ARCHITECTURE WITH SNOWFLAKE Data Sources Data Consumers OLTP Data Data Data Data Data Data Data Databases ETL, Streaming Warehouse Lake Engineering Exchange Applications Science Monetization Enterprise Applications Operational Reporting Third-Party Ad Hoc Web/Log Analysis Data Real-time
Slide 32: Snowflake Architecture
SNOWFLAKE ARCHITECTURE Scale Out Services Multi-Cluster Compute Centralized Storage Cloud Agnostic Layer © 2020 Snowflake Inc. All Rights Reserved 32
Slide 33: Snowflake Data Exchange
SNOWFLAKE DATA EXCHANGE © 2020 Snowflake Inc. All Rights Reserved
Slide 34: Snowflake Delivers Performance at Scale
SNOWFLAKE DELIVERS PERFORMANCE AT SCALE 650k users Over 145PB >56PB 355M jobs across 3300+ under scanned run per customers management daily day © 2020 Snowflake Inc. All Rights Reserved * with 8x compression ratio
Slide 35: Proven by Over 3400 Customers
PROVEN BY OVER 3400 CUSTOMERS © 2020 Snowflake Inc. All Rights Reserved 35
Slide 36: Questions?
Questions? © 2020 Snowflake Inc. All Rights Reserved
Slide 37: Thank You
THANK YOU © 2020 Snowflake Inc. All Rights Reserved