23 May 2017
|
Python
I was working on some python stuff when I stumbled upon the term decorator; not knowing what it was, a quick google search led to a relatively confusing explanation of what I believe to be a simple idea.
A decorator is simply a function that extends another function. As in, it executes that original function, but also does more. Take the function below:
def print_my_name:
print(“Neil Patel”)
Now take this decorator below:
def my_decorator(some_function):
def wrapper():
print(“Something is happening before some_function() is called.”)
some_function()
print(“Something is happening after some_function() is called.”)
return(wrapper)
If we were to execute this line of code:
my_decorator(print_my_name)
This is what would happen:
Something is happening before some_function() is called.
Neil Patel
Something is happening after some_function() is called.
You can call a decorator on a function using the @ sign! Take a look at the timing function below:
import time
def timing_function(some_function):
def wrapper():
t1 = time.time()
some_function()
t2 = time.time()
return “Time it took to run the function: “ + str((t2 - t1)) + “\n”
return(wrapper)
@timing_function
def my_function():
num_list = []
for num in (range(0, 10000)):
num_list.append(num)
print(“\nSum of all the numbers: “ + str((sum(num_list))))
print(my_function())
basically, what the @timing_function line does is say : my_function=timing_function(my_function)!
17 May 2017
|
AWS
Building Serverless Web Applications ?Randall Hunt
Technical Evangelist at AWS, of SpaceX and NASA
randhunt@amazon.com
@jrhunt on twitter
Concepts, Design Patterns, Tooling, Demo
AWS Lambda
No servers to provision or manage
Scales with Usage
Never pay for idle
Availability and fault tolerance built in
60TB/s throughput between S3 and Lambda
Never pay for Idle (GB/s)
Event driven, continuous Scaling
Function versioning and aliases
Versions = immutable copies of code+configurations
Aliases=mutable pointers to versons
Lambda Environment Variables
Key-Value pairs
Available via standard env variables, such as process.env or os.environ
Can be encrypted via AWS KMS
Web Applications, Backends, Data Processing, Chatbots, Amazon Alexa,
Amazon API Gateway Concepts
API gateway Variables
Sam : Serverless Application Model (helps define entire Serverless application)
is lot easier then cloud formation template
Framework is zappa for python is best
Serverless Apps with AWS Step Functions
Trevor Roberts Jr, Solutions architect
Modern Serverless Apps -> Lambda functions
Take discrete units of work and doing them well and packing them into lambda functions
12 factor application
AWS step functions: Application Lifecycle
Json defined, Visualize in console, Monitor Execution
Seven State types: Task, Choice, Parallel, Wait, Fail, Succeed, Pass
16 May 2017
|
AWS
Docker With ECS (Elastic Container Service)
Why containers?
OS virtualization, Process Isolation, Images, Automation
Portable Flexible Fast Efficient
Hard to switch over to batch jobs unless you have a cluster manager
Amazon ECS is a cluster management tool
Mix and match the different EC2 instances for your different container images
ECS scheduling will do everything for you
Optimistic, and strict scheduling
Optimistic tries to reserve, if fails, then fail
Other waits for completion before it works // block it out
No built in queue in ECS
Fully managed elastic service
Shared state optimistic scheduling
Deep integration with other AWS services
Load balancing, block storage, etc. etc.
Routing via Application Load balancer
Path based routing
Dynamic port mapping
HTTP/2
WebSockets
Detailed Logging
Task Definition
Deep Dive on microservices and Docker/Containers
Microservices Architecture, ECS, Placement, and 12factor App
What is a microservice?
Small independent stateless process communicating with API?s. Small and focused on doing a small task which lets you have a modular approach to system building
Monolithic vs Microservices
Build it all inside a package and put it on a webserver (but makes it hard for new features or bugs)
Make a change in the singular service you need
Black boxed
1Codebase
2Dependencies
3Config
4Backing Services
5Build,Release Run
6Processes
7 Port Binding
8 Concurrencies
9 Disposability
10 Dev/Prod parity
11 Logs
12Admin Processes
Container Orchestration with Amazon ECS & Blox
New Cluster Query Language
Lot of different things to filter by
Deep Dives on CI/CD and Docker
What is CI/CD,using Docker and ECS/ECR for this
Deployment strategies
Frequncies reduce difficultiy
Consistency improves confidence
Automation over toil
Empowered developers make happier teams
Smaller batch sizes are easier to debug
Faster delivery improves software development practices
Version control, branching, code review,
Compilation, unit tests, static analysis, packaging
Integration tests, load tests, security tests, acceptance test
Deployment, monitoring, measuring, validation
16 May 2017
|
AWS
Deploy a Deep Learning Framework on amazon ECS and EC2 Spot Instances
Introduce MXnet
Containers
Overview of ECS and ECR
Overview of AWS CloudFormation
Overview of EC2 Spot Instances
ECS and EC2 Spot Instances Together
Hands-on, Self paced workshop
What is MX net ?open source deep learning framework
Define train and deploy deep neural networks. Highly scalable, multihost, cpu/gpu
Why containers? Increase infrastructure utilization, environment isolation and fidelity, run diverse applications on shared hardware, changes are tracked, easy to deploy. Increase portability, and flexibility, speed and efficiency.
ECR is EC2 container registry that you use to store your docker images
It becomes a task definition and then reference an ECR container in the task
AWS Cloudformation
Great for infrastructure as Code. USE JSON or Yaml files as templates, make a service and build the stack
Stack replication and infrastructure scale out
Workshop: Image classification
Lab 1: Setup the workshop environment on AWS
Lab 2: Build an MXNet Docker Image
Lab 3: Launch MXnet with ECS
Lab 4: Image classification Demo
Lab 5: Wrap Image Classification in an ECS Task
15 May 2017
|
AWS
Introduction to Spot
Learn about Spot and best practices
customer examples
Tools and partners to help manage spot instances
See a demo
Bid for unused EC2 capacity lying around in data centers
AWS has millions of customers; 2300 government agencies, 7k schools, 22k nonprofit
Spot market is a real time market where the price changes based on supply and demand
Never pay more then what you bid; if price goes higher then that, you get 2 minutes
Applications should be as stateless as possible
Pick a bunch of types and zones and etc. etc. for more uptime
21 percent run < 1 hour
35 percent < 2 hours
40 percent < 3 hours
50 percent < less then 6 hours
spot blocks : only get a spot instance if you can have for dedicated number of hours
spot fleet: get lots of different kinds of spot instances
EC2 Spot Console
EC2 Spot Bid advisor
EC2 Spot Labs on Github : Python scripts that do cool things
How is market price set? Amazon sets it based on availability and bidding
AWS batch
Batch computing, overview, use cases, Demo, Q&A
Execute a bunch of programs on a bunch of inputs
2007 NYT processed 130 years in 36 hours for 900 dollars in 2007 ; 11 million articles and 4TB
Scale it
Job: Unit of work executed by AWS batch as containerized application running on Amazon EC2; reference container, command parameters, or .zip(run the .zip on amazon linux container)
Array jobs: Coming soon; run many copies of an application against an array of elements
Job definitions: how to run the jobs; mount points etc. etc.
Job queue: where job lives until assigned to compute environment
Compute environment: the EC2 instances these jobs are going to run on managed(they do) unmanaged (you take care of the EC2)
Job states: submitted (in queue), pending( waiting on dependencies), runnable(Ready to run), starting (your job is in process of being scheduled), succeeded, failed,
Introduction to AWS Lightsail
Everything you need for a low predictable price
Mem, processor, SSD, Transfer (prorated as well)
Launch a VPS with a single click
Pick your own preconfigured images with the stacks installed and applications