Some notes I took whilst doing the "Google Cloud Fundamentals: Core Infrastructure" course.

Cloud Fundamentals - Core Infrastructure

IaaS - infra as a service Raw compute, storage, networking organised as virtual resources e.g. compute engine Pay for allocated resources ahead of time

PaaS - platform as a service Bind code to libs that give access to infra e.g. app engine Pay for used resources

Serverless - no infra management e.g. cloud run (containerized microservices) cloud run functions (payg)

SaaS - an entire cloud based application e.g. gmail

Google cloud network

100s of caching nodes worldwide for data locality exploitation

Based in 5 geographic locations

Each location is divided into regions and zones

Regions - geographic areas made of zones e.g. region europe-west-2 (40 regions)

Zone - where cloud resources are deployed within a region (121 zones across 40 regions)

Multi-region: replicate across multiple zones in multiple regions

Security

Hardware infra

• Hardware design and provenance - custom hardware and chips
• Secure boot stack - crypto signatures on BIOS, kernel, OS image
• Premises security - design and build own data centres with physical security, add own physical measures in 3rd party DCs

Service deployment layer

• Encryption of inter-service communications over RPC
• Infra auto encrypts RPC traffic between DCs - use hardware crypto accelerators to extend this to all RPC traffic

User Identity layer

• Risk factors like device and login location
• Secondary factors like U2F standards

Storage at rest

• Encryption of data at rest, hardware enryption on SSDs and HDs

Internet communication layer

• Register using GFE - ensures TLS, protection against DoS attacks
• Multi layer DoS protection to protect apps behind a GFE

Operational security

• Intrusion detection and red team exercises
• Reduces insider risk through monitoring of employees with access
• Employee U2F use - prevent phishing against google employees
• Software dev practices - 2 party review, strict use of provided libs, vulnz program

Resources and Access in the Cloud

Resources

resources - VMs, buckets, tables in big query

Organised into projects

Projects organised into folders

Org node - projects, and folders for a single org

Can add policies at project, folder and org level - some policies at the resource level.

Policies applied downwards e.g. to all projects in a folder.

Projects

Basis for using google cloud services e.g. using apis, billing, adding collaborators

Resources belong to a single project

projectId - google assigned id, immutable

project name - user created, mutable

project number - google assigned, immutable

Resource manager tools

API to manage projects e.g. list, create new, update, delete

Folders

Assign policies to groups of projects or other folders.

Group projects in an org under a hierarchy.

e.g. legal folder, finance folder, hr folder

Delegation of admin rights.

Resources inherit policies and permissions from the folder they're in.

Org node

Top most resource needed to use folders. Can have a top level org policy admin, can also assign project creator roles.

IAM

Admin policies on who can do what on resources.

The who, the principal - an account, a google group, service account or cloud identity domain

The what - the role, a collection of permissions

Assign a role to a principal. You give a role to a principal on an element in the resource hierarchy - you get all the resources lower down in the resource tree. Can define deny policies which are checked before checking for allowed.

Basic IAM role - broad in scope, owner, editor, viewer, billing admin

Predefined IAM roles - offered by particular google cloud services e.g. by compute engine.

Custom IAM roles - least privileged model where you get minimal privs to do your job. Define custom roles e.g. to start and stop but not to remove. Applied to project or org level, not the folder level.

Service Accounts

A type of resource that defines an account by an email address (uses keys as the password) that allows you to assign permissions at the VM level so services can access other cloud resources in a secure manner.

Because it is a resource, it also has it's own roles e.g. some users can edit service accounts and others can only view them.

Cloud identity

Users and user groups managed through cloud identity to manage policies. Links to existing LDAP systems.

Interacting with Google cloud

• Google cloud console - web based GUI
• Cloud SDK and shell - google cloud cli, bq cli for big query
• APIs - cloud client libs and API libs provided
• Google cloud app - start, stop and use ssh

VMs and Networks in the Cloud

Define a virtual private cloud inside a project.

VPC - secure, private cloud model within a public cloud.

Run code, store data etc inside your own VPC, hosted by a public provider.

Data isolation of private cloud with scale of public cloud.

VPC networks connects cloud resources to each other and to the internet. Allow for segmenting of networks. Uses firewalls to restrict access plus static routes for forwarding traffic.

VPC networks are global and can have segmented subnets which span zones within a region. Can define network layouts with global scope.

Compute Engine

Create and run VMs.

Pre-emptible VMs - max 24 hours runtime, cheaper, can be pre-empted by Google.

Can choose the best machine properties using predefined or custom machine types.

Scaling VMs

Autoscaling - add or remove VMs based on metrics, also balance incoming traffic across VMs Number of cores is dependent on machine family, zone dependent user quota.

VPC Compatability

VPC routing tables built in for traffic forwarding without ip addresses Globally distributed firewall Can tag instances e.g. tag web servers with "web" and use a firewall rule referencing the "web" tag

VPCs belong to projects. Sometimes VPCs need to communicate across projects e.g. different VPCs in different projects.

VPC Peering - relationship between 2 VPCs SHared VPC - who and what in one project can interact with who/what in another, use IAM

Cloud Load Balancing

Distribute application traffic to reduce risk of performance issues. FUlly distributed, software defined managed service. Can put cloud load balancing infront of all traffic like HTTP, UDP, SSL etc

Provides cross region load balancing with multi-region failover, gently moving traffic in fractions.

Load balance based on Application (layer 7, http/https), content based routing - a reverse proxy, distributing incoming traffic to multiple backend services.

Network load balancer (layer 4, TCP, UDP, other IP protocols) - transport layer. Proxy load balancers, terminate client connection and establish new ones to backend services.

Passthough network LBs - do not modify or terminate connections. Forward traffic to the backend, preserving source IP. Good for applications that need direct server return or need to handle a wider range of IP protocols.

Cloud DNS and Cloud CDN

Managed DNS service. Programmable DNS - can use the API to publish DNS zones.

Edge caching - content closer to users. Accelerate content delivery via cloud CDN.

Connecting networks to Google VPC

• VPN connection over the internet, use cloud VPN. Uses cloud router to enable BGP to exchange info. Security and bandwidth concerns.
• Direct peering - your own router in a google pop.
• Carrier peering - peering via an on-prem network, not covered by google SLA
• Dedicated interconnect - direct connect to google with sla of 99.99%, backed up via a VPN
• Partner interconnect - on prem and VPC connect via a supported service provider, can be covered by an 99.99% sla if they meet google specs.
• Cross cloud interconnect - dedicated connectivity between google cloud and another cloud provider. 10 or 100 Gbps.

Storage in the cloud

Object Storage - manage data as objects File storage - files Chunks on disk - block storage

Object storage has metadata to allow it to be references like a URL.

Cloud storage mainly used for BLOBS Organized into buckets with a globally unique name, geographic location

Storage objects are immutable - they are versioned on changes. Can allow over-writing or versioning.

Control access to data via IAM roles and ACLs.

Roles inherited from project to bucket to objects.

ACLs give finer grained control - contain scope (who) and permissions (what can be performed)

ALso offers lifecycle management policies e.g. delete old objects.

Storage Classes and Data Transfer

• Standard - frequent or hot data, brief access
• Nearline - infrequently accessed e.g. once per month
• Coldline - low cost, infrequent access e.g. once per 3 months
• Archive - lowest cost, once per year, DR and online backup. Higher cost for accesss and operations, minimum storage duration of 365 days

Autoclass

Automatically transitions objects to different storage classes based on access patterns.

Encryption

HTTPS/TLS between DCs and encrypted before written to disk.

Storage transfer service

Can upload via browser, console. For large amounts the storage transfer service allows scheduled batched transfer e.g from a HTTPS endpoint.

There is also a transfer appliance - you rack it, put the data, then ship it for google to upload to the cloud.

Cloud SQL

Managed SQL solutions - mysql, postgresql, sql server

Managed backup - 7 backups. Encrypted on network and files. Includes a network firewall. Accessible by other google cloud services like app engine and compute engine.

Spanner

Managed relational db, scales horizontally. Suited for joins, secondary indices, ha, global consistency, high read and write ops.

Firestore

NoSQL, stored in docs, organised into collections. Use firestore queries to get documents. Indexed by default. Uses data synch to update data on any connected device. The app caches data so can be used whilst the app is offline - once it comes backonline, firestore syncs the data. Strong consistency, atomic batch.

BIgTable

NoSQL, massive workloads with consistent latency e.g. analytical apps. Can work with more than 1TB semi-structured or structure data, data changes rapidly, low semantic requirement in the data, data is time series, running ML, async batch or realtime processing.

Comparing Storage Options

Containers in the cloud

K8s - to manage and scale containers, orchestrate containers, rollouts and rollbacks.

GKE - google kubernetes engine

Primary components - control plane Set of nodes - run containers

A node in k8s is a machine, not a VM.

Pod - a running process on the cluster. One container per pod generally. Can also package multiple containers on the same pod.

Pods provide ip, ports.

Deployment - a group of replicas of the same pod to provide HA.

K8s creates a service with fixed IP for pods. Controller attached an external LB with a public IP.

The LB is a network LB in GKE - routes out to a service abstraction which defined a set of pods and access policy.

A deployment creates and destroys pods so IP changes over time. A service group is a set of pods - the service group provides a fixed ip for that set of pods.

e.g. frontend and backend pods which are each behind their own service - the frontend references the backend via the service, so the backend pods can be going up and down and the frontend doesn't need to know their IP.

kubectl scale

To scale a deployment.

e.g. increase number of pods based on cpu limit.

Config file

You provide a config file that tells k8s how you want the state of your deployment to look like and k8s determines how to do it for you. This is a deployment config file.

Updating app

kubectl rollout ALlows for things like rolling updates e.g. roll out an update on each pod, waiting for it to come up before destroying the old pod.

GKE

Google cloud hosted k8s. Compute engines clustered.

Manages control plane components for us.

Node config and management depends on the type of GKE mode.

Autopilot -

GKE manages infra like autoscaling, upgrades, baseline security and networking

Strong security posture, operational efficiency, optimized for efficiency.

In standard mode you're reponsible for config, management and optimization.

Can create a cluster using gcloud.

• LOad balancing of google compute engines
• Node pools to create subsets of nodes withina cluster
• Auto scaling
• Auto upgrades cluster node software
• Node auto-repair
• GOogle cloud observability logging and monitoring

Starting up a cluster in GKE:

Applications in the cloud

Cloud run - stateless, managed containers, serverless, runs via web requests Built on knative

Charges at 100ms intervals.

Dev workflow

1. Write app - listen for web requests
2. Make a container image
3. Push the image to artifact registry for cloudrun to run

On deployment you get a https url back

All incoming requests are handled by dynamically adding and removing containers.

You can also use a source based workflow - deploy source code instead of the image - cloudrun then takes care of the packaging and containerization using buildpacks.

Price depends on cpu and memory - you specify this at the container level.

Apps compiled for linux 64bit.

Cloudrun functions

Single purpose functions - lightweight, event based, async compute solution

Construct functions to create application workflows from small pieces of logic.

Can connect and extend cloud services

Events from cloud storage and pubsub, or http can trigger cloudrun functions.

Lab Notes

Make a storage bucket for your exoported location with an ID which is the project ID

gcloud storage buckets create -l $LOCATION gs://$DEVSHELL_PROJECT_ID

Copy to storage bucket

gcloud storage cp my-excellent-blog.png gs://$DEVSHELL_PROJECT_ID/my-excellent-blog.png

Modify access controls to make readable for all users

gsutil acl ch -u allUsers:R gs://$DEVSHELL_PROJECT_ID/my-excellent-blog.png

List the active account name

gcloud auth list

List the project ID

gcloud config list project

Enable the cloud run API

gcloud services enable run.googleapis.com

Set the compute region

gcloud config set compute/region us-east4

Use gcloud builds to build a container image

gcloud builds submit --tag gcr.io/$GOOGLE_CLOUD_PROJECT/helloworld

LIst all images for current project

gcloud container images list

Register gcloud as creds helper for all google supported docker registries

gcloud auth configure-docker

Run the tagged image

docker run -d -p 8080:8080 gcr.io/$GOOGLE_CLOUD_PROJECT/helloworld

Deploy the container to cloudrun

gcloud run deploy --image gcr.io/$GOOGLE_CLOUD_PROJECT/helloworld --allow-unauthenticated --region=$LOCATION

Delete the image you created

gcloud container images delete gcr.io/$GOOGLE_CLOUD_PROJECT/helloworld

Delete the associated cloudrun service

gcloud run services delete helloworld --region=us-east4 gcloud run services delete helloworld --region=$LOCATION