Master Node.js From Scratch

Clear, interactive, and structured Node.js lessons designed for beginners and developers.

COURSE SYLLABUS
1. Introduction to Node.js
Introduction to Node.js What Is Node.js? Node.js Architecture Features of Node.js Node.js vs Browser JavaScript Node.js Use Cases
2. Installation and Setup
Installation Introduction LTS and Current Versions npm Setup Environment Setup Verify Installation Node.js REPL
3. Node.js Fundamentals
Fundamentals Introduction Running JavaScript Global Objects Process Object Command-Line Arguments Environment Variables
4. JavaScript Basics
JavaScript Basics Introduction Variables: var, let and const JavaScript Data Types Comparison and Logical Operators If, Else If, Else and Switch Loops and Iteration Functions and Arrow Functions JavaScript Array Methods
5. Node.js Modules
Modules Introduction CommonJS Modules ES Modules Export and Import Built-in Modules Creating Custom Modules Path Module OS Module
6. Working with JSON
JSON Introduction JSON Syntax JSON.stringify() JSON.parse() JSON Array of Objects JSON User Database
7. Node Package Manager
npm Introduction package.json Installing Packages Dependencies and DevDependencies npm Scripts Package Versions and Semantic Versioning npm init and Project Setup
8. File System
File System Introduction Reading Files Writing Files Appending Files Deleting Files Working with Directories
9. CLI Applications
CLI Applications Introduction Command-Line Input Task Manager Project Structure Reading todos.json Adding, Updating and Deleting Tasks Saving Tasks to JSON
10. Events and EventEmitter
Events Introduction EventEmitter Registering Events Emitting Events Removing Event Listeners Custom Events
11. Asynchronous Programming
Async Programming Introduction Callbacks Promises async and await setTimeout() and setInterval() Event Loop and Non-Blocking I/O
12. Native HTTP Module
HTTP Module Introduction Creating an HTTP Server Handling Requests Sending Responses HTTP Request Methods Status Codes and Headers Handling POST Request Body Parsing Incoming JSON Products JSON API
13. Express.js
Express.js Introduction Installing Express Creating an Express Server Request and Response Objects Route Parameters and Query Strings Serving Static Files
14. Express Middleware
Middleware Introduction Built-in Middleware Application-Level Middleware Router-Level Middleware Custom Middleware Error-Handling Middleware
15. REST API Development
REST API Introduction GET Endpoints POST Endpoints PUT and PATCH Endpoints DELETE Endpoints Request Validation and API Responses
16. Database Integration
Database Introduction MySQL Integration MongoDB Integration Database CRUD Operations Database Queries and Filtering Connection Management
17. Authentication and Authorization
Authentication Introduction Password Hashing User Registration User Login JWT Authentication Authorization and Protected Routes
18. Advanced Node.js
Advanced Node.js Introduction Error Handling Debugging Node.js Applications Streams Buffers Performance and Security

Streams in Node.js

Streams are an important feature of Node.js that allow applications to process data in smaller chunks instead of loading the entire data into memory at once. They are commonly used for file operations, HTTP requests, HTTP responses, and processing large amounts of data efficiently.

1. What Are Streams in Node.js?

A stream is an interface for working with data that is read or written over time. Instead of waiting for all the data to become available, a program can process chunks as they arrive.

For example, when reading a very large file, a stream can provide a small portion of the file at a time. This can reduce memory usage compared with reading the entire file into a single buffer.

JavaScript
const fs = require("fs");

const readableStream = fs.createReadStream("large.txt");

readableStream.on("data", (chunk) => {
    console.log("Received a chunk");
    console.log("Chunk size:", chunk.length);
});

readableStream.on("end", () => {
    console.log("File reading completed");
});

readableStream.on("error", (error) => {
    console.error("Reading failed:", error.message);
});

This program reads the file in chunks. The number and size of chunks depend on the file, stream configuration, and runtime behavior.

2. Why Do We Use Streams?

  • To process large files without loading all their contents at once.
  • To transfer data efficiently between sources and destinations.
  • To handle HTTP request and response bodies.
  • To process data as it becomes available.
  • To connect multiple data-processing operations together.
  • To control data flow using backpressure.
Remember: Streams do not automatically make every operation faster, but they can reduce memory pressure and let applications process data incrementally.

3. Types of Streams in Node.js

Node.js provides four commonly used stream types.

Stream Type Purpose Example
Readable Provides data that can be consumed. Reading a file.
Writable Accepts data to be written. Writing to a file.
Duplex Supports reading and writing independently. A TCP socket.
Transform Reads data, transforms it, and produces output. Compressing data.

4. Readable Streams

A Readable stream provides data for a consumer. Examples include file reading streams and incoming HTTP request bodies.

Example: Reading a File

Create a text file named input.txt with the following content:

input.txt
Hello Node.js!
Learning Readable Streams.

Now create the following JavaScript file:

JavaScript — readable.js
const fs = require("fs");

const readable = fs.createReadStream("input.txt", {
    encoding: "utf8"
});

readable.on("data", (chunk) => {
    console.log("Received data:");
    console.log(chunk);
});

readable.on("end", () => {
    console.log("Reading completed");
});

readable.on("error", (error) => {
    console.error("Error:", error.message);
});

Run the Program

Terminal
node readable.js

Expected Output

Terminal
Received data:
Hello Node.js!
Learning Readable Streams.
Reading completed

The file's content is delivered through the data event. When no more data remains, the stream emits the end event.

5. Writable Streams

A Writable stream accepts data from a program and sends it to a destination. For example, a file-writing stream writes data to a file.

JavaScript — writable.js
const fs = require("fs");

const writable = fs.createWriteStream("output.txt");

writable.on("error", (error) => {
    console.error("Writing failed:", error.message);
});

writable.on("finish", () => {
    console.log("Writing completed");
});

writable.write("Hello Node.js!\n");
writable.write("This is a Writable Stream.\n");

writable.end("File writing is complete.\n");

Run the Program

Terminal
node writable.js

Output

Terminal
Writing completed

The program creates or replaces output.txt and writes the supplied content. The finish event is emitted after end() has been called and all written data has been processed.

Important: The finish event does not guarantee that data has been physically persisted to storage. For durability-sensitive operations, consider the relevant filesystem guarantees and synchronization requirements.

6. Using pipe() with Streams

The pipe() method connects a Readable stream to a Writable stream. It transfers data and manages backpressure between the streams.

JavaScript — pipe.js
const fs = require("fs");

const readable = fs.createReadStream("input.txt");
const writable = fs.createWriteStream("copy.txt");

readable.on("error", (error) => {
    console.error("Read error:", error.message);
});

writable.on("error", (error) => {
    console.error("Write error:", error.message);
});

writable.on("finish", () => {
    console.log("File copied successfully");
});

readable.pipe(writable);

Run the Program

Terminal
node pipe.js

Output

Terminal
File copied successfully

The program copies the contents of input.txt into copy.txt. The Writable stream finishes after receiving the data and completing its writes.

Note: For more complex pipelines, use stream.pipeline() or its Promise-based version. It helps propagate errors and clean up streams when a pipeline fails.

7. Using pipeline() for Safer Stream Processing

The pipeline() function connects streams and handles errors across the pipeline. It is often preferable for production data-processing flows because it coordinates completion and error propagation.

JavaScript — pipeline.js
const fs = require("fs");
const { pipeline } = require("stream");

pipeline(
    fs.createReadStream("input.txt"),
    fs.createWriteStream("pipeline-copy.txt"),
    (error) => {
        if (error) {
            console.error("Pipeline failed:", error.message);
            return;
        }

        console.log("Pipeline completed successfully");
    }
);

Run the Program

Terminal
node pipeline.js

Output

Terminal
Pipeline completed successfully

This example copies a file using a stream pipeline. If the source or destination fails, the callback receives the error.

8. Transform Streams

A Transform stream is a special Duplex stream. It reads input, transforms it, and produces output. Examples include text conversion, compression, and data formatting.

Example: Convert Text to Uppercase

JavaScript — transform.js
const { Transform } = require("stream");

const uppercase = new Transform({
    transform(chunk, encoding, callback) {
        callback(null, chunk.toString().toUpperCase());
    }
});

uppercase.on("data", (chunk) => {
    console.log(chunk.toString());
});

uppercase.on("error", (error) => {
    console.error("Transform failed:", error.message);
});

uppercase.on("end", () => {
    console.log("Transformation completed");
});

uppercase.write("hello ");
uppercase.write("node.js");
uppercase.end();

Output

Terminal
HELLO
 NODE.JS
Transformation completed

The output may be emitted in separate chunks. The important point is that each chunk is converted to uppercase before it is delivered to the consumer.

9. Duplex Streams

A Duplex stream supports both reading and writing. Its readable side and writable side operate independently. A TCP socket is a common example.

You can also create a Duplex stream for learning purposes:

JavaScript — duplex.js
const { Duplex } = require("stream");

const customStream = new Duplex({
    read() {
        this.push("Data from readable side");
        this.push(null);
    },

    write(chunk, encoding, callback) {
        console.log("Received:", chunk.toString());
        callback();
    }
});

customStream.on("data", (chunk) => {
    console.log("Read:", chunk.toString());
});

customStream.on("end", () => {
    console.log("Readable side completed");
});

customStream.write("Hello from writable side");
customStream.end();

Expected Output

Terminal
Received: Hello from writable side
Read: Data from readable side
Readable side completed

The writable side accepts the input message, while the readable side produces a separate message. The two sides do not have to return the same data.

10. What Is Backpressure?

Backpressure occurs when data is produced faster than the destination can consume it. Node.js streams provide mechanisms to slow the producer down and prevent excessive buffering.

For example, when writing to a Writable stream, write() returns false when the internal buffer has reached its threshold. The producer should wait for the drain event before continuing to write more data.

JavaScript
const fs = require("fs");

const writable = fs.createWriteStream("large-output.txt");

writable.on("error", (error) => {
    console.error("Write failed:", error.message);
});

writable.on("finish", () => {
    console.log("All data written");
});

let index = 0;

function writeData() {
    while (index < 100000) {
        const canContinue = writable.write(`Line ${index}\n`);

        index++;

        if (!canContinue) {
            writable.once("drain", writeData);
            return;
        }
    }

    writable.end();
}

writeData();

This example pauses when the Writable stream signals backpressure. It resumes when the stream emits drain, avoiding an uncontrolled loop that keeps filling the buffer.

11. How Stream Processing Works

Data Source

A file, request, or another readable source.

↓
Readable Stream

Produces data in chunks.

↓
Transform Stream (Optional)

Processes or modifies the data.

↓
Writable Stream

Writes the processed data to its destination.

12. Stream Best Practices

  • Use streams when working with large files or continuous data.
  • Handle stream errors appropriately.
  • Prefer pipeline() for complex stream pipelines.
  • Respect backpressure when writing data.
  • Use end() to signal that no more data will be written.
  • Do not assume all stream data arrives in one chunk.
  • Use Transform streams for incremental data processing.
  • Consider file permissions and destination paths when writing files.

Important Points to Remember

  • Streams process data incrementally.
  • Readable streams provide data.
  • Writable streams accept data.
  • Duplex streams support reading and writing.
  • Transform streams modify data while processing it.
  • pipe() connects a Readable stream to a destination.
  • pipeline() helps coordinate errors and completion.
  • Backpressure prevents uncontrolled buffering.

Summary

Streams are a powerful Node.js feature for processing data incrementally. Readable, Writable, Duplex, and Transform streams serve different purposes. Methods such as pipe() and pipeline(), together with proper error handling and backpressure management, help developers build efficient file-processing and data-transfer applications.