DCOS Presentation
-
Upload
jan-repnak -
Category
Internet
-
view
761 -
download
0
Transcript of DCOS Presentation
![Page 2: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/2.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Agenda
- Containers- Apache Mesos - Marathon- Datacenter Operating System
(DCOS)- Hands-On
2
![Page 3: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/3.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved. 3
LINUX CONTAINERS
![Page 4: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/4.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Linux Containers
The why and the what
● Containers are not VMs …
● … they are process groups for app-level dependency management
● … they are lightweight (startup time, footprint, average runtime)
● security considerations
● pets vs cattle
4
![Page 5: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/5.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Linux Containers
● namespaces
● Isolate PIDs between processes
● Isolate network resources (stacks, devices, etc.)
● Isolate hostname/NIS (UTS)
● Isolate filesystem mount (chroot)
● Isolate inter process communication (IPC)
● Isolate users/groups
● cgroups
● https //sysadmincasts.com/episodes/14-introduction-to-linux-control-groups-cgroups
5
![Page 6: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/6.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Docker
6
![Page 7: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/7.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
What Is This All About?Pets vs Cattle
● Pets:treat machines as individuals that you give names and when they get ill you nurse them back to health.
● Cattle:anonymous, identical to each other you assign numbers and when they get ill you get rid of it.
http://www.theregister.co.uk/2013/03/18/servers_pets_or_cattle_cern/
7
![Page 8: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/8.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Consequences of going all-in with Cattle approachPets vs Cattle
● scale out on commodity hardware
● elasticity
● 'cheap' & 'simple'
● R U on pager duty? Just sleep through!
● social ≫ technology challenge
● new technical challenges such as service discovery
http://www.theregister.co.uk/2013/03/18/servers_pets_or_cattle_cern/
8
![Page 9: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/9.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Container Life Cycle
9
![Page 10: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/10.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved. 10
Apache Mesos
![Page 11: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/11.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Mesosphere is well funded$50M by Tier 1 investors Andreessen Horowitz and Khosla Ventures
2009 2013 2014 2015
First Mesosphere DCOS lighthouse customersTens of thousands of containers launched
Expanded OperationsSan Francisco (2013)Hamburg (2014)New York (2015)
Mesosphere FoundedKey engineering leaders from Twitter, Airbnb - companies behind open-source tech
Apache Mesos built at UC Berkeley● Core technology from AMPlab
● Corporate large-scale production deployments begin (e.g., Apple, Twitter, Salesforce)
Growing Ecosystem & DCOS General AvailabilityNow a breeze to install modern app services (e.g., Hadoop, Spark, Cassandra)
MESOSPHERE HISTORY
11
![Page 12: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/12.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
WORKLOADS*
12
![Page 13: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/13.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
MESOS KERNEL APPLIES LESSONS FROM EARLY INNOVATORS
Tupperware/BistroBorg/Omega Apache Mesos
ProprietaryProprietary Open Source (Apache license)
~2007~2001 2010+
Production-proven Web Scale Cluster Managers
● Built at UC Berkeley AMPLab by Ben Hindman (Mesosphere Co-founder) ● Built in collaboration with Google to overcome some Borg Challenges● Production proven at scale +80K hosts @ Twitter
13
![Page 14: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/14.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
MESOS IS THE IDEAL KERNEL FOR POWERING A DCOS
Designed to be flexible
● Aggregates all resources in the datacenter for modern apps
● Intentionally simple to enable massive scalability
● Handles different types of tasks - long running, batch, and real-time
● Two-level scheduler architecture enables multiple scheduling logics (a key challenge at Google)
● Extensible to work with new technologies
Dow
nloa
ds
Mesos daily downloads, July 2014 - November 2015
Gaining massive adoption
14
![Page 15: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/15.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
What Is This All About?Pets vs Cattle
● A top-level ASF project
● A cluster resource negotiator
● Scalable to 10,000s of nodes but also useful for a handful of nodes
● Fault-tolerant, battle-tested
● An SDK for distributed apps
● Native Docker support
15
![Page 16: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/16.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
What does a scheduler do?
16
![Page 17: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/17.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
ResourcesApache Mesos
● resource == anything a task/executor consumes in order to do their work
● standard resources: cpu, mem, disk, ports
● at its core: DRF algorithm, for fair sharing across-resource types
17
![Page 18: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/18.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved. 18
Marathon
![Page 19: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/19.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Marathon
● An init System for datacenters
● starts instances of a long-running service somewhere in the cluster, for example, as Docker containers
● restarts the instances if they crash
● provides composition primitives
● supports health checks
● supports rolling upgrades
19
![Page 20: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/20.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Marathon
● Basics
● apps and groups
● health checks
● HTTP API
● curl, http
● DCOS CLI
● Team Player
● Integrates nicely into the DCOS ecosystem
● Doesn’t try to solve everything itself
20
![Page 21: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/21.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Marathon
21
![Page 22: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/22.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Marathon{ "id": "webserver", "cmd": "python3 -m http.server 8080", "cpus": 0.5, "mem": 32.0, "container": { "type": "DOCKER", "docker": { "image": "python:3", "network": "BRIDGE", "portMappings": [ { "containerPort": 8080, "hostPort": 0 } ] } }, "acceptedResourceRoles": [ "slave_public" ], "constraints": [ [ "hostname", "UNIQUE" ] ]} 22
![Page 23: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/23.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Service Discovery
23
![Page 24: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/24.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved. 24
DatacenterOperating System(DCOS)
![Page 25: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/25.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved. 25
THE DATACENTER IS THE NEW SERVER
PHYSICAL (x86) VIRTUAL HYPERSCALEMAINFRAME
SERVER VIRTUAL MACHINEPARTITION (LPAR)UNIT OF INTERACTION
● ERP, CRM, PRODUCTIVITY, MAIL & WEB SERVER
● LINUX, WINDOWS
● DATA / TRANSACTION PROCESSING
● UNIX, IBM OS/360
DEFINITIVE APPS AND OS
● ERP, CRM, PRODUCTIVITY, MAIL & WEB SERVER
● HYPERVISOR + GUEST OS
● BIG DATA, INTERNET OF THINGS, MOBILE APPS
● ???
???DATACENTER
NEW FORM FACTOR FOR DEVELOPING AND RUNNING APPS
● BIG DATA, INTERNET OF THINGS, MOBILE APPS
● DATACENTER OPERATING SYSTEM (DCOS)
![Page 26: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/26.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
TRADITIONAL DATACENTER
Docker
App 1
Hadoop Cluster Spark Cluster
Cloud Foundry
App 2
VMware Cluster OpenStack
• Many “snowflakes”
• Management nightmare
• Lengthy cycles to deploy code
• Low utilization
26
![Page 27: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/27.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
MODERN DATACENTER
• High performance and resource isolation
• Easy scalability and multi-tenancy
• Fault-tolerant and highly available
• Highly efficient with highest utilization
• Complete workload portability
Mesos
Docker
Big Data Analytics (Hadoop, Spark, etc.)
Cloud Foundry
Stateful Service(All)
Deploys on-premise in cloud or both
27
![Page 28: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/28.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Meet the Datacenter Operating System
28
![Page 29: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/29.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
DCOS is a Distributed Operating System
● kernel == Apache Mesos, scaling to 10,000 of nodes
● fault-tolerant in all components, rolling upgrades throughout
● containers first class citizens (LXC, Docker)
● local OS per node (container enabled)
● scheduling (long-lived, batch)
● service discovery, monitoring, logging, debugging
29
![Page 30: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/30.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Local OS vs. Distributed OS
http://bitly.com/os-vs-dcos 30
![Page 31: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/31.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Benefits
● Run stateless services such as Web servers, app servers (via Marathon) and stateful services like Crate, Kafka, HDFS, Cassandra, ArangoDB etc. together on one cluster
● Dynamic partitioning of your cluster, depending on your needs (business requirements)
● Increased utilization (10% → 80% and more)
31
![Page 32: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/32.jpg)
32
![Page 33: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/33.jpg)
33
![Page 34: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/34.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
DCOS Architecture
34
![Page 35: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/35.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved. 35
MICROSOFT AZURE CONTAINER SERVICE (ACS)
Challenges
● Needed a production grade native container service that would work on premises and on Azure, at massive scale
● Must easily integrate with Azure CI/CD, app management and auto scaling infrastructure
● Microsoft- and Linux-friendly technology
Mesosphere Solution
● After independent evaluation, MS team determined Mesos/Mesosphere was the right fit
● Currently integrating Mesosphere DCOS as the core technology for Azure Container Service
![Page 36: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/36.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
WEBINAR
DCOS ON AZURE WEBINARTomorrow (Jan 26), 10 00 AM
http://bit.ly/dcosonazure
36
![Page 37: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/37.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved.
Resources
● MESOSPHEREhttps //mesosphere.com
● DCOS COMMUNITY EDITIONhttps //mesosphere.com/product/https //docs.mesosphere.com/getting-started/tutorials/
● MICROSOFT AZURE CONTAINER SERVICEhttps //mesosphere.com/blog/2015/09/29/mesosphere-and-mesos-power-the-microsoft-azure-container-service/
● APACHE MESOS FOR WINDOWS SERVERhttps //mesosphere.com/blog/2015/08/20/mesos-everywhere-apache-mesos-for-windows-server/
37
![Page 38: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/38.jpg)
© 2015 Mesosphere, Inc. All Rights Reserved. 38
Demo: Oinker
![Page 39: DCOS Presentation](https://reader031.fdocuments.mx/reader031/viewer/2022021920/586e8c521a28aba0038b840d/html5/thumbnails/39.jpg)