Friday, May 27, 2016

Beyond ad-hoc automation: to structured platforms | QCon San Francisco 2015

Beyond ad-hoc automation: to structured platforms | QCon San Francisco 2015



Beyond Ad-hoc Automation: To Structured Platforms















  • Advice on navigating DevOps Tooling and Platforms
  • Learn to work with the right level of abstraction with DevOps Tooling
  • Hear stories of successful and not so successful solutions
  • Grok how different DevOps pieces fit together into a cohesive solution

How Docker Helped Yelp Leave the Monolith Behind - The New Stack

How Docker Helped Yelp Leave the Monolith Behind - The New Stack

Yelp's Journey Into Containers

Inside Yelp's SOA Infrastructure



Kyle Anderson discusses Yelp's internal platform as a service (PaaSTA),
which takes a holistic view for taking services from an idea and git
repo, to monitored in production. Kyle includes lots of details on how
to tie Mesos, Docker, SmartStack, Haproxy, Git, and Sensu all together
into a coherent system that developers can use to ship their code in a
self-serve way. 

What's Next for C# and VB.NET | DEVintersection 2016 | Channel 9

What's Next for C# and VB.NET | DEVintersection 2016 | Channel 9



ASP.NET Core 1.0 Deep Dive | DEVintersection 2016 | Channel 9





From Zero to Front-end Hero (Part 2) — Free Code Camp

From Zero to Front-end Hero (Part 2) — Free Code Camp



first read about MVC at Chrome Developers. After that, read Understanding MVC And MVP (For JavaScript And Backbone Developers). In that article, don’t worry about learning Backbone, just go through the parts with explanations of MVC and MVP.



Addy Osman also wrote about MVVM in Understanding MVVM — A Guide For JavaScript Developers. To learn about the origins of MVC and why it came about, read Martin Fowler’s essay on GUI Architectures. Finally, read the section, JavaScript MV* Patterns, in Learning JavaScript Design Patterns. Learning JavaScript Design Patterns is a fantastic free online book.

Creating a new Rembrandt with big data - Microsoft UK Developers

Creating a new Rembrandt with big data - Microsoft UK Developers:



Thursday, May 26, 2016

Advanced Neo4j at FiftyThree: Reading, Writing & Scaling – Oh My!

Advanced Neo4j at FiftyThree: Reading, Writing & Scaling – Oh My!





When you really dive into the subtleties of both reading and
writing, there is a lot that we may be taking for granted. For example,
with reading, consistency is a very important aspect to your data, while
with writing, atomicity is an important fundamental principle.

Wednesday, May 25, 2016

Exciting Features of SQL Server 2016

Exciting Features of SQL Server 2016



Temporal tables
in the database engine is another feature that provides some great
benefits. Tracing history on tables is something that I run into pretty
regularly as a requirement. I like that fact that SQL Server handles the
bulk of the complexity of managing historical copies of the data.


Support for JSON in the database engine is also big.

Container Days 2016 - Boston Notes: Cruise control for your distributed infrastructure - Lightweight Container Virtualization

Logging and monitoring stacks
-. Prometheus
-. .Grafana
-. .InfluxDB

Hosted logs
-. Loggly
-. Sumologic

Docker compose with...
-. Logstash stack
-. Logspout
-. Kibana
-. ElasticSearch

Resiliency, orchestration & SDN
-. Clustering
-. Networking
-. Service discovery
-. Scheduling
-. Health checks
-. Load balancing
-. Upgrades

Metadata services
-. Configuration injection
--- Persistent storage
-. Access control
--. Audit
-. Secret management

Gluster process: building a Gluster pod

PaaS sits on top of Scheduler
- DEIS
- clodFoundry

Scale-out vs Scale-up

PaaSTa
- deploy small services
- crosscutting concerns are abstracted
- service registry, discovery, health checks


Jenkins pipeline
- tests
- Security verification
- performance tests


SmartStack: Service Discovery in the Cloud - Airbnb Engineering

As traffic (and the number of engineers working on the product) grows, this approach doesn’t scale. The code base becomes too complicated, concerns are not cleanly separated, changes from many engineers touching many different parts of the codebase go out together, and performance is determined by the worst-performing sections in the application.
The solution to this problem is services: individual, smaller code bases, running on separate machines with separate deployment cycles, that more cleanly address more targeted problem domains. This is called a services-oriented architecture: SOA.


Container Builds are not Reproducible



Tom De Wolf and Tom Holvoet, Emergence Versus Self-Organisation: Different Concepts but Promising When Combined, http://www.uvm.edu/~pdodds/teaching/courses/2009-08UVM-300/docs/others/2005/dewolf2005a.pdf, 2005



The Autopilot Pattern and the rise of app-centric micro-orchestration - Blog - Joyent

To make sense of how it’s possible to make a complex, even stateful application so portable, we need to start with the relationship between the scheduler and the application (a single Docker container). Consider the minimum requirements for each:
Scheduler Application
  • Start containers
  • Stop containers
  • You’re looking for more, including discovery, configuration, orchestration, load balancing, monitoring and support for rolling upgrades? Read on...
  • Register itself in a service catalog for use by other apps
  • Look to the service catalog to find the apps it depends on
  • Configure itself when the container starts, and reconfigure itself over time

By making our applications effectively stateless to the scheduler, we can make even the most complex applications simple enough to manage just by setting the scale of their components.

The Autopilot Pattern, on the other hand, works across all schedulers because it takes advantage of the common interface they provide to the application: the scheduler starts the container, the application takes it from there.

The features of containers that have made them so developer-friendly also make deployments easier. The dream of immutable infrastructure that was so hard to achieve without containers is within reach with them, now that the challenges of managing dependencies for application components and delivering them to the server with each update can be wrapped up and isolated in each container. Rather than upgrading servers and dealing with the configuration management of those servers and all the libraries the applications depend on, we're simply starting new containers running the newest image.

Kubernetes handles cluster management, container scheduling, and scheduler-centric orchestration as a tightly-coupled system. Like Marathon, it was designed for stateless services. There are efforts to overcome that limitation within Kubernetes and with PersistentVolumeClaims,

By moving the orchestration details into the application container, we simplify both the scheduler and application.
Consider these additional benefits of app-centric micro-orchestration:
  1. It's visible and self-documenting to all developers and operators
  2. It's portable so that it works on (almost) any infrastructure, including developer and operator laptops
  3. It's testable and repeatable everywhere the application goes, making it easy to bring up isolated staging environments for every change
This simplification allows us to focus our attention where it belongs: on our apps.



Running and debugging CoreCLR ASP.NET web applications on Triton - Blog - Joyent



Self-Organizing Microservices - Evaluating ContainerPilot on Mantl - Container Solutions

The core concept of ContainerPilot is to have an agent
inside each container that deals with changes in infrastructure – for
instance, if a microservice A depends on another one (B), the agent
inside container A can update the configuration of microservice A
whenever B changes location.

For this simple idea to work, the agent must be embedded inside each
container, and start as the first process in the container (PID 1).
Effectively this means that it is behaves like an init system, but
unlike initd or systemd, it is meant to supervise only one process – the
microservice. When the process fails beyond recovery, the agent exits
as well, which effectively ends the lifecycle of the container.



How to build reliable software systems out of many moving parts (microservices)?
If we look at the market for solutions that try to answer the same
question, I can currently see two approaches. First, deployments tools
(like Puppet, Chef, Ansible etc) are now being adapted to the world of
microservices (see Puppet’s Blueshift
project). The tools already know the structure of systems they deploy,
so it’s tempting to use them for the coordination of microservices at
run-time. The problem is that deployment tools were never designed to
operate in real-time, and we need something that can react very fast
(microservices can change their state at any moment).


The second approach are schedulers – AWS Auto-scaling groups,
Mesos(w/Marathon), Kubernetes, Swarm, etc.. Their knowledge of system
structure is typically minimal – they were simply designed to find
resources for a new task – but in contrary to deployment tools, they
were designed to operate in real-time. That’s why for simple systems
(e.g. only stateless containers) a scheduler can be sufficient to manage
even large clusters.


The two products have one thing in common – they both manage
microservices from outside i.e. orchestrate. There is another option: to
let them self-organize – i.e. manage from inside.




Introducing PaaSTA: An Open, Distributed, Platform as a Service
Yelp/dumb-init: A minimal init system for Linux containers

PaaSTA works by integrating existing open-source components that do their jobs very well:
  • Docker for code delivery and containment
  • Apache Mesos for code execution and scheduling (runs Docker containers)
  • Mesosphere’s Marathon for managing long-running services
  • Chronos for running things on a timer (nightly batches)
  • SmartStack for service registration and discovery
  • Sensu for monitoring/alerting
  • Jenkins (optionally) for continuous deployment


Unikernels - Rethinking Cloud Infrastructure
emc-advanced-dev/unik: The Unikernel Compilation and Deployment Platform
Unikernels are unfit for production - Blog - Joyent

MirageOS
Since Xen powers most public cloud computing infrastructure such as Amazon EC2 or Rackspace, this lets your servers run more cheaply, securely and with finer control than with a full software stack.



Overview of the Service Fabric Reliable Service programming model | Microsoft Azure

Reliable Collections | Microsoft Azure : Recommendations
Overview of Service Fabric | Microsoft Azure
Microservices: An application revolution powered by the cloud | Blog | Microsoft Azure


Azure-Samples/service-fabric-dotnet-getting-started: Get started with Service Fabric with these simple introductory sample projects.

  • Deploy different versions of the same application side by side, each independently upgradable.
  • Manage the lifecycle of your stateful applications without any downtime, including breaking and nonbreaking upgrades.
  • Manage applications using .NET APIs, PowerShell, or REST interfaces.
  • Upgrade and patch microservices within applications independently.
  • Monitor and diagnose the health of your applications and set policies for performing automatic repairs.
  • Scale up or scale down your Service Fabric cluster easily, knowing that the applications scale according to available resources.

  • Develop applications composed of microservices, using the Service Fabric programmming model, or simply host guest executables, and other application frameworks of your choice, such as ASP.NET Core 1, node.js, etc.
  • Develop stateless and stateful microservices and make these highly reliable.
  • Simplify the design of your application by using stateful microservices in place of caches and queues.

Stateful Services

Stateless Applications


How does storage in Containers work

Stateful Services in Containers yes or no
One container per service. 
Tip: Much faster to have one common base image. Issue: builds are'nt reproducible

Scale-out vs Scale-up : Stateful services in Containers




Cinder: Block Storage service for OpenStack. present storage resources to end users consumed by OpenStack Compute

Docker Toolbox with Kubernetes and Persistent Volumes

Challenges Adopting Containers: Part 1

Run your stateful apps with Mesos and Docker


https://github.com/emccode/vagrant
https://github.com/emccode/rexray
https://github.com/emccode/mesos-module-dvdi
https://github.com/jonasrosland/simplehttp

EMC Rex_Ray: ties in with Schedulers
Nomad: Shared state; Kubernetes, Docker Swarm, Mesos: plugging in your own Scheduler for containers



Continuous Integration Platform Using Docker Containers: Jenkins, SonarQube, Nexus, GitLab - codecentric Blog : codecentric Blog
Using Jenkins Job DSL for Job Lifecycle Management - codecentric Blog : codecentric Blog





tools and frameworks:
  • UI tests: Selenium, Robot Framework, Protractor
  • Service tests: JBehave, FitNesse, SoapUI, JMeter Performance Tests
  • Unit tests: JUnit, TestNG, Mockito (Mock Framework)



Agenda

A New Era: Containers in the Enterprise

A History of DevOps, and How You Can Too

Applications on Autopilot

Containers & PaaS: Complementary or Mutually Exclusive?

Next steps!


Framework Architecture Resource granularity Multi-scheduler Pluggable logic Priority preemption Re-scheduling Oversubscription Resource estimation Avoid interference
O
P
E
N
Kubernetes monolithic multi-dimensional N[v1.2, DD, Issue] Y[DD] N[Issue] N[Issue] Y[DD] N N
Swarm monolithic multi-dimensional N N N[Issue] N N N N
YARN monolithic/
two-level
RAM/CPU slots Y N[app-lvl. only] N[JIRA] N N[JIRA] N N
Mesos two-level multi-dimensional Y Y[framework-lvl.] N[JIRA] N Y[v0.23, Doc] N N
Nomad shared-state multi-dimensional Y Y N[Issue] N[Issue] N[Issue] N N
Sparrow fully-distributed fixed slots Y N N N N N N
C
L
O
S
E
D
Borg monolithic[7] multi-dimensional N[7] N[7] Y Y Y Y N
Omega shared-state multi-dimensional Y Y Y Y Y Y N
Apollo shared-state multi-dimensional Y Y Y Y N N N
Figure 2: Architectural classifaction and feature matrix of widely-used orchestration frameworks, compared to closed-source systems.

 

Introducing Commissaire — Project Atomic
Firmament Blog: The evolution of cluster scheduler architectures


 

Onward…

Project Atomic Getting Started Guide — Project Atomic
Project Atomic hosts ship with Docker, Flannel, and Kubernetes to build clusters for container based services. Docker provides the container runtime, Flannel provides overlay networking, and Kubernetes provides scheduling and coordination of containers on hosts.

Docker for Developers

 

detailed approaches to:
  • Containerize build environments
  • Package code in containers
  • Build and deploy test environments
  • Implement CI with Jenkins
  • Build an application stack with Rancher
  • Push new code to production
  • Upgrade containerized applications

   Todo

Best practices for writing Dockerfiles
Docker Toolbox | Docker
Overview of CoreOS and 7 reasons why you should be using CoreOS (with Docker)
The Netflix Tech Blog: Global Continuous Delivery with Spinnaker
DevOps questions: patterns for multi-data center and multi-cloud applications - Blog - Joyent
Modern Architectures | Apcera The Trusted Cloud Platform
3 Reasons I Should Build My Containerized Applications on RHEL and OpenShift – Red Hat Developer Blog
Consul Service Discovery with Docker :: Jeff Lindsay, Open source hacker
Docker DNS & Service Discovery with Consul and Registrator – Art+Tech
Docker launches Security Scanning to test containers for known vulnerabilities - TechRepublic
Docker for Mac and Windows Beta: the simplest way to use Docker on your laptop | Docker Blog


Part 1: Getting started with CI/CD and Docker
Part 2: Moving to Compose blueprints
Part 3: Adding Rancher for Orchestration






Tuesday, May 24, 2016

Container Days schedule 2016

Tuesday, May 24th

Time

Description

Comment

09:00-09:30am

Breakfast & registration

Welcome to ContainerDays Boston 2016!

09:30-09:45am

Welcome, introduction, OpenSpaces explanation

09:45-09:50am

Sponsor shout-out

09:50-10:35am

From Hello World to Real World: Creating a Production-Grade Container Environment

Bill Maxwell & Shannon Williams, Rancher Labs

10:35-10:40am

Sponsor shout-out

10:40-11:10am

OpenSpaces "ignite-off"

Quick intro to OpenSpaces and pitches for topics

11:10-11:55am

Docker For the Developer

Borja Burgos, Docker

11:10-11:55am

Hiding in Plain Sight: Managing Secrets in a Container Environment

Jeff Mitchell, HashiCorp

11:55-12:40pm

ComplianceOps: Containers in Regulated Environments

Elliot Murphy, Kindly Ops

11:55-12:40pm

Layer-X: Globally-aware, Pluggable Scheduling for Mesos

Scott Weiss, EMC

12:40-01:20pm

Lunch & exhibitor time

Also time to think about OpenSpaces &workshop topic suggestions!

01:20-02:05pm

Understanding Cluster Schedulers, and Why You'll Want a Better One

Ionel Gog, University of Cambridge & Malte Schwarzkopf, MIT

02:05-02:40pm

OpenSpaces session

02:40-03:15pm

OpenSpaces session

03:15-03:30pm

Coffee break & exhibitor time

Also time to think about OpenSpaces &workshop topic suggestions!

03:30-04:15pm

Introduction to Overlay Networks: The Good, The Bad and The Ugly

Andrey Sibiryov, Uber Technologies

03:30-04:15pm

Introducing Service Fabric: A Distributed Services Platform

James Sturtevant, Microsoft

04:15-04:50pm

OpenSpaces session

04:50-05:00pm

Day 1 wrap-up

05:00pm

Day 1 close

evening

Tuesday night drinks

At Meadhall around the corner

Wednesday, May 25th

Time

Description

Comment

09:00-09:30am

Breakfast

09:30-09:40am

Day 2 kickoff

09:40-09:45am

Sponsor shout-out

09:45-10:30am

Kubernetes in Production, From the Ground Up

Mike Splain, Barkly

09:45-10:30am

Your Database and Other Stateful Applications in Containers? You Betcha!

Jonas Rosland, EMC {code}

10:30-10:50am

Workshop "ignite-off"

Quick intro to "unworkshops" and pitches for topics

10:50-11:35am

Yelp's Journey Into Containers

Chris Kuehl, Yelp

10:50-11:35am

Clair: The Container Image Security Analyzer

Joey Schorr, CoreOS

11:35-12:20pm

Autopilot: Running Real-world Applications in Containers

Tim Gross, Joyent

11:35-12:20pm

Managing Your IT Stack to Assure Performance of Containerized Applications

Endre Sara & Enlin Xu, VMTurbo

12:20-01:00pm

Lunch & exhibitor time

Last chance to think about workshop topic suggestions!

01:00-02:50pm

Workshops

Multi-machine Docker: Getting to Grips With Docker Compose and Swarm
UniK: Unikernels Made Simple
Any other tool/framework of your choice!

02:50-03:05pm

Coffee break & exhibitor time

03:05-04:50pm

Workshops ctd.

Kubernetes Deep Dive: From Basics to Advanced Features
Using and Extending Apache Mesos
Any other tool/framework of your choice!

04:50-05:00pm

Thanks & closing discussion

 

Monday, May 23, 2016

Last Week on Channel 9: May 16th - May 22nd, 2016 | C9Team | Channel 9

Last Week on Channel 9: May 16th - May 22nd, 2016 | C9Team | Channel 9



Defrag: We're back with new tools, old tools, partition tips, Hibernate vetoing and more....

Build 2016: Key Highlights and Announcements for Windows Developers

Designing Identity solutions with AAD

Tech Futures: How can we achieve more? 

Microsoft Cognitive Services - My free subscriptions

Microsoft Cognitive Services - My free subscriptions



  1.    Bing Autosuggest – Preview     10,000 transactions per month, 10 per second.
     
  2.     Bing Search – Preview     Across all Bing Search APIs (Web, Image, Video, News): 1,000 transactions per month, 5 per second.
     
  3.     Bing Spell Check – Preview     5,000 transactions per month, 7 per minute.
     
  4.     Academic - Preview     10,000 transactions per month, 3 per second for interpret, 1 per second for evaluate, 6 per minute for calcHistogram.
     
  5.     Computer Vision - Preview     5,000 transactions per month, 20 per minute.
     
  6.     Emotion - Preview     30,000 transactions per month, 20 per minute.
     
  7.     Entity Linking - Preview     1000 transactions per day, 10KB text limit.
     
  8.     Face - Preview     30,000 transactions per month, 20 per minute.
     
  9.     Linguistic Analysis - Preview     5,000 transactions per month, 2 per second.
     
  10.     Speaker Recognition - Preview     10,000 transactions per month, 20 per minute.
     
  11.     Speech - Preview     5,000 transactions per month, 20 per minute for each feature for a total of 60 per minute.
     
  12.     Video - Preview     getResult: 1,200 transactions per month, 1 per minute. getOperation: 12,000 transactions per month, 5 per minute Each feature has 100MB file limit; 300 transactions per month per feature, 1 per minute.
     
  13.     WebLM - Preview     100,000 transactions per month, 1,000 per minute.
     
  14.     Recommendations - Preview     10,000 transactions per month.
     
  15.     Text Analytics - Preview     5,000 transactions per month. 

TWC9: .NET Core RC2 and Friends, Build Tour Registration, Desktop to UWP, 1 Billion Downloads and more... | This Week On Channel 9 | Channel 9

TWC9: .NET Core RC2 and Friends, Build Tour Registration, Desktop to UWP, 1 Billion Downloads and more... | This Week On Channel 9 | Channel 9



ReEntry [Petri Wilhelmsen]

Live from Build 2016: Cognitive Services | FWTV on 9 | Channel 9

Live from Build 2016: Cognitive Services | FWTV on 9 | Channel 9



Jennifer Marsman talks with Anna Roth about Microsoft Cognitive services.

links:

Episode 206: Bot Framework with Mike Hall | Microsoft Azure Cloud Cover Show | Channel 9

Episode 206: Bot Framework with Mike Hall | Microsoft Azure Cloud Cover Show | Channel 9