Amazon Services - hackpad.com

Amazon Services

I was researching Amazon's cloud offerings for a new project, and took these notes. There are a lot of gotchas in their products and I thought could be a lot more clearly described. 

Please feel free to contribute.

Really Common

Databases

A fast (~5ms) NoSQL key/value store that can scale up. Data replicates between data centres and availibility zonesfor redundancy. Has consistent reads, unlike many non-relational dbs.

A key/value store that scales up and is cheap.  Not as fast as DynamoDb. If you want consistency, it is slower.

MySQL hosted for you. Can get pretty big (3TB). Handles backups, snapshots, and replication. Sounds like adding more space and cpu is easy.

Storage 

When EC2 instances are terminated, the hard drive is wiped. Put data you want to keep on this.  You can take snapshots of these and store them on S3. You can also use these as the root partition a and create fast EBS AMIs. Standard is 100 IOPS, can order up to 1000 IOPS

A CDN. Great to put in front of S3 if you want your photos to load even faster.

Operates as a reverse proxy (PULL) similar to the way CloudFlare works.

Like S3 but deep storage. Cheaper but less easy to get the data back.

Ephemeral Storage (local to ec2 instances)

Local Storage; gets wiped on termination.  Good for logs, swap, etc.

Hosted Services

Hosted Hadoop. For map reducing large data sets.

Anycast DNS service.

Memcache like service.

Recipes / Bootstrap / Scaling

Something you need a bunch of computers doing a stardard thing, like Rails + Mysql. And you want a recipe for that. 

Heroku done by Amazon. Host here and it will scale up with you. Wraps a bunch of other Amazon services like load balancing + ec2 + cloudwatch. Supports Java, Ruby, Python, Node, and PHP.

You can write scripts to setup application servers and they will be run when you get lots of traffic. 

Balance traffic across multiple EC2 instances. Even if they are in different parts of the world. Can notice if a particular computer is handling badly.

Gotchas - no logging, little visibility into what's going on, traffic from a given IP seems to stick with whatever system it's routed to, no ability to rate-limit.

Notifications, Emails, and Queues

Notification service for alerting or message notification via SMS, email or HTTP end points.

Message queuing service, you can send messages and other computers can read and act on them.  It has fairly different characteristics compared to most other queues, requires careful thought before implementing workers.

Email service. Like sendgrid or mailgun.

Search

Upload your data and it searchable. Scales as you need it to.

Monitoring

Use this to watch your computers to make sure they haven't gone down or are overloaded. Supports custom metrics (OS, page load, query).

Sort of like pingdom + top.

Networking

A connection between your data center and AWS.

You want a bunch of computers to have a specific networking and subnet configuration. You want your database to have absolutely not be connected to the internet.

Payments

Vaguely like google payments, braintree, or stripe. If you want to sell things and accept money using amazon.

A way to wrap amazon services and collect money for them. You can meter people's usage and charge.

Other

Amazon Simple Workflow Service (Amazon SWF) is a workflow service for building scalable, resilient applications. Whether automating business processes for finance or insurance applications, building sophisticated data analytics applications, or managing cloud infrastructure services, Amazon SWF reliably coordinates all of the processing steps within an application. 

Put referral links on your site.

Someone who you can email if you don't get what is going on. 

When you can't program something and you want to pay another human to do it.

Convert videos from one format to another.

Data warehousing service, with a pay-as-you-go model.  

On premise appliance which connects to AWS cloud storage. Limit to 8 * 32TB volumes.

Identity and access mgmt; federation btwn corp and AMZ.