mooc-notes

Notes from online courses

View on GitHub

Introduction

Agenda

Understanding EFS

What is the Amazon Elastic File System?

Where does EFS fit in the storage scene?

How are people traditionally used to access network files and resources?

Cloud based, specifically within AWS and EFS

Properties of EFS

Summary

Storage Classes and Performance Options

Storage Classes

Amazon EFS offers 2 different storage classes

  1. Standard (Default)
    1. Access: Anytime
    2. Cost: Standard cost
    3. Performance: Standard latency
  2. Infrequent Access (IA) (Cheaper)
    1. Access: Infrequent
    2. Cost: Reduced
    3. Performance: Higher first-byte latency

Cost

EFS IA is cheaper than standard storage EFS IA also charges for read and write EFS standard class charges on the amount of storage used each month

Availability

Both provide same level of availability and durability

EFS Lifecycle Management

Performance Modes

EFS has 2 different performance modes

  1. General purpose
    1. When to use
      1. Default performance mode and is typically used for most use cases. For example, home directories and general file-sharing environments. It offers an all-round performance and low latency file operation,
    2. Throughput: Standard
    3. IOPS: <7K
    4. Latency: Low latency
  2. Max I/O
    1. When to use
      1. for many 1000s of EC2 instances concurrently
    2. Throughput: Unlimited
    3. IOPS: >7K
    4. Latency: Higher latency (The downside is, however, that your file operation latency will take a negative hit over that of General Purpose.)

The best way to determine which performance option that you need is to run tests alongside your application. If your application sits comfortably within the limit of 7,000 operations per second, then General Purpose will be best suited, with the added plus point of lower latency. However, if your testing confirms 7,000 operations per second may be reached or exceeded, then select Max I/O.

When using the General Purpose mode of operations, EFS provides a CloudWatch metric percent I/O limit, which will allow you to view operations per second as a percentage of the top 7,000 limit. This allows you to make the decision to migrate and move to the Max I/O file system, should your operations be reaching that limit. 

Throughput Modes

Data throughput patterns on file systems generally go through periods of relatively low activity with occasional spikes in burst usage, and EFS provisions throughput capacity to help manage this random activity of high peaks.

Throughput is measured by the rate of mebibytes. The 2 modes offered are

  1. Bursting Throughput (Default)
    1. The amount of throughput scales as your filesystem grows. So the more you store, the more throughput is available to you.
    2. The default throughput available is capable of bursting to 100 mebibytes per second, however, with the standard storage class, this can burst to 100 mebibytes per second per tebibyte of storage used within the file system.
    3. So, for example, presume you have five tebibytes of storage within your EFS file system. Your burst capacity could reach 500 mebibytes per second.
    4. The duration of throughput bursting is reflected by the size of the file system itself. Through the use of credits, which are accumulated during periods of low activity, operating below the baseline rate of throughput set at 50 mebibytes per tebibyte of storage used, which determines how long EFS can burst for. Every file system can reach its baseline throughput 100% of the time. By accumulating, getting credits, your file system can then burst above your baseline limit.
    5. The number of credits will dictate how long this throughput can be maintained for, and the number of burst credits for your file system can be viewed by monitoring the CloudWatch metric of BurstCreditBalance.
    6. If you’re finding that you’re running out of credits too often, then you might consider using the Provisioned Throughput mode
  2. Provisioned Throughput
    1. Provisioned Throughput allows you to burst above your allocated allowance, which is based upon your file system size.
    2. So if your file system was relatively small but the use case for your file system required a high throughput rate, then the default bursting throughput options may not be able to process your request quick enough. In this instance, you would need to use provisioned throughput.
    3. However, this option does incur additional charges, and you’ll pay additional costs for any bursting above the default option of bursting throughput. That brings me to the end of this lecture, now I want to shift my focus on creating and connecting to an EFS file system from a Linux based instance.

EFS In Practice

Creating an EFS File System

Mounting methods

  1. Linux NFS - https://docs.aws.amazon.com/efs/latest/ug/mounting-fs-old.html
  2. EFS mount helper (demo focuses on this)
    1. Simplifies the mount process
    2. Log to /var/log/amazon/efs
    3. Automatically connects to EFS at startup by editing /etc/fstab

Prerequisites before connecting to EFS Filesystem from your EC2 Instances

  1. Create EFS filesystem and EFS mount targets
  2. Be running an EC2 instance with EFS Mount Helper
  3. Needs an EC2 in a VPC configured to use Amazon DNS Servers with DNS hostnames
  4. A security group allowing NFS filesystem access your EC2
  5. Be able to connect to your Linux EC2 instance

Demo

Covers

TODO: insert image from Evernote

Managing EFS Security

Agenda

Access Control

Before you can create and manage your EFS file system, you need to ensure that you have the correct permissions to do so.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid" : "PermissionToCreateEFSFileSystem",
      "Effect": "Allow",
      "Action": [
        "elasticfilesystem:CreateFileSystem",
        "elasticfilesystem:CreateMountTarget"
      ],
      "Resource": "arn:aws:elasticfilesystem:region-id:file-system/*"
    },
    {
     "Sid" : "PermissionsRequiredForEC2",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeSubnets",
        "ec2:CreateNetworkInterface",
        "ec2:DescribeNetworkInterfaces"
      ],
      "Resource": "*"
    }
  ]
}

In addition to the these policies, you’ll also need the following permissions to manage EFS using the AWS management console:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid" : "Stmt1AddtionalEC2PermissionsForConsole",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeVpcs",
        "ec2:DescribeVpcAttribute"
      ],
      "Resource": "*"
    }
    {
     "Sid" : "Stmt2AdditionalKMSPermissionsForConsole",
      "Effect": "Allow",
      "Action": [
        "kms:ListAliases",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    }
  ]
}

The above permissions allow the console to

Encryption

EFS supports both encryption at rest and in transit

At rest

In transit

Importing Data

How to import on-premise data into EFS

Use cases

Summary

What is EFS and what it does

Storage classes and performance options

Creating an EFS Filesystem

Refer demo video

Managing EFS Security

Import data