Slide 1: Spotify Distributed Systems
Spotify Distributed Systems Jon tind @jooon
Slide 2: Who am I?
Who am I?
Slide 3: Jon Aslund
Jon Aslund
Slide 4: Slide 4
Slide 5: Computer Science
Computer Science
Slide 6: gos,
E> gos, KTH & VETENSKAP S89 OCH KONST && ) 9 Tyce
Slide 7: Shakespeare Programming Language
Shakespeare Programming Language
Slide 8: Joined Spotify Autumn 2006
Joined Spotify Autumn 2006
Slide 9: April 2006: the idea
April 2006 the idea
Slide 10: Slide 10
Slide 11: Slide 11
Slide 12: Music itself is going to become like running water or electricity
“music itself is going to become like running water or electricity” David Bowie, 2002
Slide 13: Slide 13
Slide 14: Slide 14
Slide 15: Slide 15
Slide 16: Slide 16
Slide 17: February 2012
February 2012
Slide 18: 10+ million users
10+ million users (active within previous 30 days)
Slide 19: 15+ million tracks
15+ million tracks
Slide 20: 500+ million playlists
500+ million playlists
Slide 21: 1000+ servers
1000+ servers in 3 countries
Slide 22: Spotify is a distributed system
Spotify is a distributed system
Slide 23: Fast <200ms
Fast <200ms
Slide 24: Always working
Always working
Slide 25: How do you accomplish that?
How do you accomplish that?
Slide 26: Cheat as much as possible
Cheat as much as possible
Slide 27: Fast vs Feels fast
Fast VS Feels fast
Slide 28: Always working vs Pretend there is nothing wrong
Always working vs Pretend there is nothing wrong
Slide 29: Build to fail
Build to fail
Slide 30: Redundancy
Redundancy (talk about waffles)
Slide 31: Error recovery
Error recovery
Slide 32: Don't forget to test your recovery code
Don't forget to test your recovery code
Slide 33: Fire drill: Cut the atlantic cable
Fire drill: Cut the atlantic cable
Slide 34: Netflix Chaos Monkey
Netflix Chaos Monkey
Slide 35: On January 15, 1990, AT&T's long-distance system crashed
“On January [5, 1990, AT&T’s long-distance telephone switching system crashed.”
Slide 36: Isolation
Isolation
Slide 37: Getting the right tool for the job
Getting the right tool for the job
Slide 38: iler
— iler SS DELLIStH WE R, Reiki
Slide 39: Metadata, Master Storage
APe deen enenenenenareeeneneeeeeeresereereeeereeees Metadata Master Storage Spotity Protecot
Slide 40: Consistency, Availability, Partition tolerance
Consistency Availability Partition tolerance http://ksat.me/a-plain-english-introduction-to-cap-theorem/
Slide 41: Availability, Partition Tolerance
Availability Partition Tolerance
Slide 42: user
user
Slide 43: PostgreSQL
PostgreSQL
Slide 44: Slide 44
Slide 45: search
search
Slide 46: search: Lucene index in memory
search Lucene index in memory
Slide 47: search: replaced daily
search replaced daily
Slide 48: read only
read only
Slide 49: scales horizontally
scales horizontally
Slide 50: search: sort of
search sort of
Slide 51: How do you replace an index in memory?
search How do you replace an index in memory when you are using all of it?
Slide 52: storage
storage
Slide 53: storage: number of plays
storage number of plays 5, David Guerra Long +ail , Fest hos Mange dRraumwohnung
Slide 54: storage: lru cache in nginx
storage lru cache in nginx
Slide 55: storage: better use of memory with sharding
storage better use of memory with sharding
Slide 56: Storage
Slide 57: storage: consistent hashing
storage consistent hashing
Slide 58: playlist
playlist
Slide 59: playlist: cassandra
playlist cassandra
Slide 60: playlist: structured key-value store
playlist structured key-value store
Slide 61: playlist: high throughput
playlist high throughput
Slide 62: playlist: choose redundancy and consistency level
playlist choose redundancy and consistency level
Slide 63: playlist: datacentre aware
playlist datacentre aware
Slide 64: p2p
p2p
Slide 65: Data source - ratio - by week
Data source - ratio - by week Weekend Night
Slide 66: thank you
thank you
Slide 67: Master Thesis?
Master Thesis?
Slide 68: Yes, please!
Yes, please!
Slide 69: or come and work for us
or come and work for us
Slide 70: Oh, and we have cake as well!
Oh, and we have cake as well! :D spotify.com/jobs