Hello everyone, hope you are doing good. This tutorial is for all who (in beginning) feels working with cloud storage is burdensome. As a beginner, reading the official documentation full of many high level code examples is quite difficult to grasp. When started exploring Azure storage, I felt the same. This is a small effort to help beginners understand and get started with Azure.
We will make a mini express app to demonstrate uploading, downloading & listing PDF files from Azure Storage.
So lets begin...
Before diving into the hows, lets understand the whats
What is Azure Storage ?
Azure Blob storage is Microsoft's object storage solution for the cloud. Blob storage is optimized for storing massive amounts of unstructured data. Unstructured data is data that doesn't adhere to a particular data model or definition, such as text or binary data.
What it can do ?
- Serving images or documents directly to a browser
- Storing files for distributed access.
- Streaming video and audio.
- Writing to log files.
- Storing data for backup and restore, disaster recovery, and archiving.
- Storing data for analysis by an on-premises or Azure-hosted service.
Get to know more at Azure Storage Docs
Prerequisites
To begin you should have an Azure subscription. If you don't already have a subscription, create a free account.
There's one more thing. In order to use Azure Storage, you have to create a Storage Account on the portal. Go to Dashboard , search for Storage Account and create.
For help creating a storage account, see Create a storage account.
Copy your credentials from the Azure portal : When the sample application makes a request to Azure Storage, it must be authorized. To authorize a request, add your storage account credentials to the application as a connection string. View your storage account credentials by following these steps:
In the Settings section of the storage account overview, select Access keys. Here, you can view your account access keys and the complete connection string for each key.
Find the Connection string value under key1, and select the Copy button to copy the connection string. You will add the connection string value to an environment variable later.
Blob Storage
Azure has four types of storage, Blob Storage, Table Storage, Queue Storage and File Storage. In this tutorial we'll learn to use Blob Storage.
Blob storage offers three types of resources:
- The storage account
- A container in the storage account
- A blob in a container
Storage Account : A storage account provides a unique namespace in Azure for your data. Every object that you store in Azure Storage has an address that includes your unique account name. The combination of the account name and the Azure Storage blob endpoint forms the base address for the objects in your storage account.
Containers : It is similar to a folder in File system. Containers allows you to store multiple blobs inside them. You can add as many containers but it should follow certain naming convention.
Blobs : Blobs are used to store a text or binary data. You can store any type of data in blobs such as images, videos, pdf, etc.
That was a lot for now. Take a nap before we jump on the code part.
Welcome back !!
Setting up project
Make sure you have NodeJS installed on your system.
Create a project folder named "node-azure-tutorial".
Open this folder with your favorite code editor.
Create app.js file & leave it empty for now.
Initialize Node Packager Manager for your project by opening terminal and run
npm init -y
Now install few packages which we'll be working with
npm i --save express dotenv ejs multer into-stream @azure/storage-blob
Lets have a breakdown of these minions:
express : Its a nodejs framework.
dotenv : To manage enviroment variables.
ejs : Its a templating engine.
multer : Used for multipart form data (handling file uploads).
into-stream : to convert buffer into stream.
@azure/storage-blob : Azure storage SDK for Javascript.
Do you remember the connection string I told you to copy & save. Now is the time to use it.
Create a file named .env. Its an environment file where you can store your apps environment variables. Save your connection string as shown.
AZURE_STORAGE_CONNECTION_STRING = <yourconnectionstring>
Create two folders named "views" & "public" in the root directory. Inside views create a new file "home.ejs and copy the below content in it.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<!-- Bootstrap CSS -->
<link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta3/dist/css/bootstrap.min.css" rel="stylesheet"
integrity="sha384-eOJMYsd53ii+scO/bJGFsiCZc+5NDVN2yr8+0RDqr0Ql0h+rP48ckxlpbzKgwra6" crossorigin="anonymous">
<!-- Custom CSS -->
<link rel="stylesheet" href="style.css">
<title>Azure Storage</title>
</head>
<body>
<main>
<div class="wrapper row ">
<div class="col-12 my-2 text-center">
<h2>Azure Blob Storage</h2>
</div>
<form action="/upload" method="POST" enctype="multipart/form-data" class="col-12 row text-center">
<div class="mb-3">
<label for="formFile" class="form-label">Upload PDF file</label>
<input class="form-control" type="file" name='pdf' id="pdf">
</div>
<div class="col-12 text-center">
<button type="submit" class="btn btn-primary">Upload</button>
</div>
</form>
<ul class="list">
<% blobs.forEach((blob)=> { %>
<li class="item">
<p>
<%= blob.name %>
</p>
<button class="btn btn-success"><a href="/download/<%= blob.name %>">View</a></button>
<button class="btn btn-danger"><a href="/delete/<%= blob.name %>">Delete</a></button>
</li>
<% }) %>
</ul>
</div>
</main>
<!-- Bootstrap JS -->
<script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta3/dist/js/bootstrap.bundle.min.js"
integrity="sha384-JEW9xMcG8R+pH31jmWH6WWP0WintQrMb4s7ZOdauHnUtxwoG2vI5DkLtS3qm9Ekf"
crossorigin="anonymous"></script>
</body>
</html>
The above file is the view of our home page. It has a file input field and list of blobs. All the blobs which are uploaded will be shown below with View & Delete options. Add a bit styling by creating a style.css file in public folder.
* {
box-sizing: border-box;
margin: 0;
padding: 0;
}
main {
width: 100vw;
display: flex;
justify-content: center;
align-items: center;
}
.wrapper {
margin-top: 24px;
width: 500px;
}
form {
text-align: center;
}
.list {
margin-top: 24px;
padding: 4px;
}
.list .item {
list-style: none;
text-align: left;
display: flex;
flex-direction: row;
justify-content: left;
align-items: center;
padding: 2px 8px;
margin: 8px 8px;
box-shadow: 2px 2px 4px 2px rgb(243, 240, 240);
border-radius: 12px;
}
.list .item:hover {
cursor: pointer;
box-shadow: 2px 2px 4px 2px rgb(233, 227, 227);
}
.list p {
width: 300px;
overflow-x: hidden;
}
.list button {
margin: 4px 8px;
}
.list button a {
text-decoration: none;
color: #fff;
}
p {
margin-bottom: 0;
}
Your folder structure should look like this.
We begin with app.js.
//imports
const express = require("express");
const multer = require("multer");
const path = require("path");
const ejs = require("ejs");
require("dotenv").config();
// express config
const app = express();
const PORT = process.env.PORT || 3000;
app.use(express.static(path.join(__dirname, "public")));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.set("view engine", "ejs");
app.set("views", "views");
//multer config
var storage = multer.memoryStorage();
var upload = multer({ storage: storage });
Above lines are basic configurations to get our app started. We are using multer memory storage method.
Azure Javascript SDK provides some classes with which we can interact our storage account, containers and blobs.
- BlobServiceClient: The BlobServiceClient class allows you to manipulate Azure Storage resources and blob containers.
- ContainerClient: The ContainerClient class allows you to manipulate Azure Storage containers and their blobs.
- BlobClient: The BlobClient class allows you to manipulate Azure Storage blobs.
Continue further in app.js.
//azure config
const { BlobServiceClient } = require("@azure/storage-blob");
const getStream = require("into-stream");
const ONE_MEGABYTE = 1024 * 1024;
const uploadOptions = { bufferSize: 4 * ONE_MEGABYTE, maxBuffers: 20 };
const AZURE_STORAGE_CONNECTION_STRING =
process.env.AZURE_STORAGE_CONNECTION_STRING;
const blobServiceClient = BlobServiceClient.fromConnectionString(
AZURE_STORAGE_CONNECTION_STRING
);
We imported BlobServiceClient class from the SDK and created an instance blobServiceClient. Here are some functions we get form blobServiceClient.
Some more utility functions are also needed.
Now is the time to create a container in our storage. For simplicity we are creating our own little function createMyContainer. Remember that we are creating a single container named "pdfcontainer", so we need to run this once.
Below in app.js :
//container variable
var myContainer;
const createMyContainer = async (text) => {
const containerName = `${text}`;
myContainer = blobServiceClient.getContainerClient(containerName);
const createContainerResponse = await myContainer.create();
console.log(
`Create container ${containerName} successfully`,
createContainerResponse
);
};
//Immediately Invoked Async Function (IIAF)
(async function () {
await createMyContainer("pdfcontainer");
})();
Run this once by command node app.js
Below is the log.
We now have successfully created a container named pdfcontainer. Just comment out the IIAF. Get hold of the containerClient reference as myContainer .
//reference to containerClient
myContainer = blobServiceClient.getContainerClient("pdfcontainer");
// (async function () {
// await createMyContainer("pdfcontainer");
// })();
Lets write some of our own custom functions to get list of blobs,upload blob , download blob and delete blob. Add these functions at the very bottom of app.js
//GET all blobs
const getAllBlobs = async () => {
let allBlobsList = [];
for await (const blob of myContainer.listBlobsFlat()) {
allBlobsList.push(blob);
}
return allBlobsList;
};
//UPLOAD blob to Azure
const uploadToAzure = async (buffer, req) => {
const blobName = `${req.file.originalname}-${new Date().getTime()}`;
const myBlob = myContainer.getBlockBlobClient(blobName);
const stream = getStream(buffer);
try {
const uploadResponse = await myBlob.uploadStream(
stream,
uploadOptions.bufferSize,
uploadOptions.maxBuffers,
{
blobHTTPHeaders: { blobContentType: "application/pdf" },
}
);
console.log(`file uploaded...`, uploadResponse);
} catch (e) {
console.log(e);
throw new Error("uploadToAzure function throwed !");
}
};
//DELETE a blob
const deleteBlob = async (blobName) => {
const myBlob = myContainer.getBlockBlobClient(blobName);
try {
const deleteResponse = await myBlob.deleteIfExists();
console.log(`blob deleted ....`, deleteResponse);
} catch (e) {
console.log(e);
throw new Error("deleteBlob function throwed !");
}
};
//DOWNLOAD blob
const downloadBlob = async (blobName) => {
const myBlob = myContainer.getBlockBlobClient(blobName);
try {
const downloadBlockBlobResponse = await myBlob.download();
const downloaded = await streamToBuffer(
downloadBlockBlobResponse.readableStreamBody
);
return downloaded;
} catch (e) {
console.log(e);
throw new Error("downloadBlob function throwed !");
}
};
//stream to buffer utilty function
async function streamToBuffer(readableStream) {
return new Promise((resolve, reject) => {
const chunks = [];
readableStream.on("data", (data) => {
chunks.push(data instanceof Buffer ? data : Buffer.from(data));
});
readableStream.on("end", () => {
resolve(Buffer.concat(chunks));
});
readableStream.on("error", reject);
});
}
getAllBlobs : In this custom function we are fetching all the blobs form our myContainer reference. Azure provides a function called blobServiceClient.listContainers().
uploadToAzure : In this custom function we are uploading the buffer from req to our Azure container. In the process we are generating stream form getStream() method and using the SDK function blockBlobClient.uploadStream().
downloadBlob : In this custom function we are downloading a blob by its name and converting into buffer. To download blobs Azure SDK provides blockBlobClient.download().
deleteBlob : In this custom function we are deleting a blob by its name. To delete blobs Azure SDK provides blockBlobClient.deleteIfExists().
Lets write some express middlewares to use these functions upon called. In app.js above all the functions we just declared, write these middlewares.
app.post("/upload", upload.single("pdf"), async (req, res, next) => {
const pdfFile = req.file;
try {
//upload
await uploadToAzure(pdfFile.buffer, req);
} catch (e) {
console.log(e);
}
res.redirect("/");
});
app.get("/delete/:name", async (req, res, next) => {
const name = req.params.name;
try {
await deleteBlob(name);
} catch (e) {
console.log(e);
}
res.redirect("/");
});
app.get("/download/:name", async (req, res, next) => {
const name = req.params.name;
try {
const pdfBuffer = await downloadBlob(name);
res.writeHead(200, {
"Content-Type": "application/pdf",
"Content-Disposition":
"inline; filename = case study rohan.pdf-1618136609114.pdf",
"Content-Length": pdfBuffer.length,
});
res.end(pdfBuffer);
} catch (e) {
console.log(e);
}
});
app.get("/", async (req, res, next) => {
try {
//get all
const allBlobsList = await getAllBlobs();
res.render("home", { blobs: allBlobsList });
} catch (e) {
console.log(e);
return res.render("home", { blobs: [] });
}
});
app.listen(PORT, () => {
console.log("listening on PORT 3000");
});
Go ahead & start the server by running node app.js
Our local address is http://localhost:3000
We are rendering the file upload field and list of all the blobs. Currently there isn't any files in the container hence an empty list.
Select a pdf file and upload it. It will upload and redirect to the same page & you'll now see a single pdf with delete & view options. Also you can checkout the Azure Portal to see the file in storage container.
Click on View to view the pdf file. You can also customize it inline or as an attachment.
Delete removes the file from the Azure storage.
Github Repo : https://github.com/jaiswalrohan8796/node-azure-tutorial
Feel free to explore more about it.
This is my first article. Open to improvements/suggestions and other ideas to share.
