Everyone relies on data for work and decision-making. There are various cloud-enabling technologies but we will concentrate on. Virtualization is responsible for the abstraction, isolation, and hoarding of applications. There are different levels of virtualization and different approaches.
Virtualization and containerization: Making applications easily portable to new environments eg in the cloud. This will allow us to easily spin up additional instances of the application eg when the load increases.
Storage technologies: Applications rely on some sort of data storage. AWS, GCP, and Azure.
Networks and restful services: They allow for abstraction, hiding inner complexities and easily connecting system components to a more extensive cloud system.
After reading this article, you should be able to:
Describe key features of virtualization and containerization technologies
Apply the key features of Docker.
Explain the benefits of characteristics of different cloud storage technologies
Explain how restful APIs work.
How different technologies and services can be used to manage
restful APIs.
Virtualization and Containerization: Containerization with docker.
Containerization is central to cloud computing because it allows us to wrap applications in self-contained capsules and thereby make them easily portable to the environment.
In cloud computing, we do that all the time for example, we might develop an application on our local machine, but for better reliability and scalability, we then want to execute the application on a more extensive cluster of devices in the cloud. Also working with containerized applications, we can easily add additional machines to a cluster running that container and thereby scale horizontally when the cloud increases.
Main components of containerization with docker.
- Dockerfile: A recipe containing the instructions to be executed when the container starts. Based on that docker file, we create. It describes how a containerized application is built in the form of a text file. Eg we specify the operating system which data is available within the container and which programs are to be executed.
- Docker Image: a blueprint or design to create instances of this container. These images are usually registered and distributed in the registry.
- Docker Image Registry: We can then store that image in a centralized image registry. there are private and public registries. The best-known open docker registry is the docker hub. Before we build a container from scratch, you can look at the docker hub to see if there is not a freely available image that we can adapt to our needs if necessary.
- Container Instance: create an instance from that container based on that image from everywhere we like. To launch an instance of an image, docker checks if the image already exists locally. If it doesn’t, the image is loaded from a registry. Then starting the containerized application is easy and it is done with a simple command. One of the advantages of this approach is that a container can be started independently of the host environment. it does not matter if we start a container on a local machine, a data center server, or the cloud.
- Docker Compose: To launch multiple containers that together form a larger application, we can use docker-compose. Docker compose is used to store an application containing several applications to work together. In a simple YAML document, we specify which containers should be launched with which configurations. Then we can use a simple command to spine up all the containers at once.
- Docker swarm: Used to run one or many containers on a cluster of several machines. A concept central to cloud computing. Here, we can run containers not on individual machines but distributed across multiple compute units in parallel. Docker swarm provides better scalability while abstracting internal complexity. From the outside, we see one logical unit running one or more containers.
Getting started with Docker: Hands-on docker
In your Integrated development environment/terminal,
Install docker.
Create a dockerfile. We write the instructions to be executed in the container with many commands.
.from: to start the container not from scratch but from an already existing image in an image registry.
.run: to execute a terminal command
.copy- to copy data into the container.
After creating a docker file, docker commands can be used to do several things.
Docker Build - will create a docker image based on the docker file.
Docker run- will create and run a container based on an image.
Docker images- gives us the list of all the images available in a running machine.
Docker ps: Shows us the running containers.
Rm: Remove a container from the machine.
start: Start a container.
stop: Stop a container.
commit: Commit changes to an image.
Tag: Add a name to an image.
Cloud deployment
Once done with development, we need to deploy the container to the cloud. We use the deployment manifest. It is a JSON file specifying which container to deploy, how ports of a container should be marched to the port of the hosting machine, and so on.
Storage Technologies.
There are so many storage solutions available. However, we will concentrate on the major three. AWS, Azure, and GCP.
AWS
- Simple Storage Service(S3): Object data store that can be used for unstructured data. It is highly reliable and its service level agreement concerning durability is about 99.999%. It is very reliable. It has an S3 access point and also has different tiering classes. S3 intelligent, S3 standard, S3 standard-infrequent access(S3-IA).
- Elastic Block Store (EBS) and Elastic File Store (EFS): They are cloud-based block storage and file systems that can be attached to EC2 instances. Amazon cloud virtual machines. It is a managed file storage for EC2. it has two classes standard and infrequent access (IA).
- RDS: Relational database services: It is used to host relational databases like aurora in the cloud.
- DynamoDB: Multimodal no SQL database on AWS and we can use a document-oriented API to work with it.
There are so many more solutions on AWS cloud storage. You can check their documentation for more.
FLASK in Practice
In many cases like cloud-related scenarios, we want to implement a program logic as a callable service. For example, imagine we trained a machine model and now is the time to provide this model as a service that we can call from another application or web page to obtain predictions by the model.
Flask is a popular lightweight web application framework for Python to achieve exactly this. here, we learn to set up and run a simple flask app with a rest API.
We want to offer the predictions made by this model as a service callable by our rest API. we need to import the model and flask to provide the rest API. we have to also tell the application which route to choose if we send data to that API.
You can use a graphical interface like Postman and paste the URL to which you deployed your API. then you choose the route and send the data you want to make predictions for.
You can use the model to make predictions of types, for example, what can of bag is this, etc.
It is easy to use Flask to provide a restful API for a machine learning model and other programming/business logic and provide this program with a restful API to be callable by other applications and web pages.
Google Cloud Platform(GCP)
Cloud storage: Object data store that can be attached to the app engine. Google’s cloud-based virtual machines. It is the cloud storage in GCP for unstructured object data. It is highly reliable and comes in 4 different tiers. Standard, Nearline tiering, coldline tiering, archive tiering.
Cloud SQL: A cloud-based relational database engine. it is fully managed and there is support for MySQL, PostgreSQL, and standard SQL
Firestore: no SQL data stores. This is also a document-oriented one. It is organized into collections of documents and sub-collections.
Bigtable: Also a no SQL data storage service. It is however column oriented one that is table-like data.
Microsoft Azure
Storage Account: used in many ways. A family of storage services consisting of object files, column-oriented NoSQL, and huge storage services. The four different storage types here are:
Blob storage: It is intended for storing unstructured binary data. It is organized in containers of which there are three types depending on the intended data usage.
File Storage:
Table Storage:
Queues:
Data Lake: An extension of storage accounts and can be used for data storage for an entire organization.
SQL Database: A cloud-based version of Microsoft SQL server. A relational database. It is a managed platform as a service. It comes with two resource usage models
Single database and elastic database pool.
Synapse: Azure’s cloud-based data warehouse solution.
Cosmos DB: globally distributed multi-modal NoSQL database. It allows data to geographically reside close to its intended usage it is also a multi-model NoSQL database. It has gradual consistency models.
Networks and Restful Services.
Rest APIs:
They are essential to cloud computing because they allow us to connect individual services in a standardized way without knowing about the inner complexities of each.
This allows for microservice architectures and makes it comfortable for us to develop applications on the cloud.
What are restful APIs? They are intermediate between the cloud and the server.
The client can be an application that needs data from a service such as a storage solution.
Instead of learning to communicate with each of the hundreds of available storage services, we can rely on a standardized intermediate. we will talk to the API in the language we know because they are standardized. The intermediate will speak to the servers to retrieve the resources we are asking for and this hides the servers/serves inner complexity leaving us with APIs that we can work with. Imagine we want to talk to a database service in the cloud and we use restful APIs for this. We can send an HTTP request. This request will start with a method.
- Get: to retrieve a resource
- Post: to create a resource and send data
- Put: to update a resource
- Delete: to delete a resource
Endpoint: A unique URL in which the API is reachable.
It contains
Header: Used for authentication as access for most APIs is managed by role-based access control and we have to ensure that all the authorized accesses are granted.
Body: Used to send additional data; the API will then process the request and respond to us. The response can come in various formats like JSON, XML, YAML, CSV, and HTML but JSON is the most popular format.
APIs make communication between services straightforward. This is done by several restful API characteristics.
Restful Paradigms
Client-server autonomy: means the client and the server are independent of each other. here, our application does not depend on how the database works and vice versa.
Uniform Interface between the services
Layered architecture: An intermediate between client and server can be interpreted as an additional architectural layer.
Caching: this can be disabled and enabled depending on the API. when enabled, we can allow clients to store resources on their machines thereby taking load from the server.
- Stateless interactions: restful APIs are stateless. This means no request and no resources are stored and they do not influence each other.
- Code on Demand: the requested resources can also be coded. This is called code on demand. example, we can request a short javascript code snippet to be executed on our client machine. This takes load from the servers and supports flexibility in developing and using applications.
Cloud Networks.
Transfer between cloud services: Cloud computing highly depends on interconnected services. This means that there is a considerable transfer between cloud services.
Low Latency: To ensure applications run smoothly, we depend on low latency as well as
Security: Secure communication. All cloud providers offer solutions for this. Eg, Azure virtual network(VNet) to establish a virtual network where our services can communicate securely and with low latency.
Cloud computing is all about distributing storage and processing across many machines while virtual networks can be used for secure communication between services.
Load Balancers: They distribute the load between servers. Every cloud provider offers load-balancing services in one way or another.
Managed Identities: As far as authentication and authorization are concerned, many different solutions are depending on the cloud provider and service. For example, in Azure, using managed identity, we can sign privileges to people, groups, applications, or services. For the latter, we can use a service principle and manage its privileges in the Azure active directory.
Note: This is a 4-fold series on cloud computing.
To read the first, here is the link:
https://dev.to/michellebuchiokonicha/cloud-computing-for-data-and-software-developers-43ce
it focuses on cloud services, models and roles.
Top comments (0)