Streams in Node.js
Streams are an important feature of Node.js that allow applications to process data in smaller chunks instead of loading the entire data into memory at once. They are commonly used for file operations, HTTP requests, HTTP responses, and processing large amounts of data efficiently.
1. What Are Streams in Node.js?
A stream is an interface for working with data that is read or written over time. Instead of waiting for all the data to become available, a program can process chunks as they arrive.
For example, when reading a very large file, a stream can provide a small portion of the file at a time. This can reduce memory usage compared with reading the entire file into a single buffer.
const fs = require("fs");
const readableStream = fs.createReadStream("large.txt");
readableStream.on("data", (chunk) => {
console.log("Received a chunk");
console.log("Chunk size:", chunk.length);
});
readableStream.on("end", () => {
console.log("File reading completed");
});
readableStream.on("error", (error) => {
console.error("Reading failed:", error.message);
});
This program reads the file in chunks. The number and size of chunks depend on the file, stream configuration, and runtime behavior.
2. Why Do We Use Streams?
- To process large files without loading all their contents at once.
- To transfer data efficiently between sources and destinations.
- To handle HTTP request and response bodies.
- To process data as it becomes available.
- To connect multiple data-processing operations together.
- To control data flow using backpressure.
3. Types of Streams in Node.js
Node.js provides four commonly used stream types.
| Stream Type | Purpose | Example |
|---|---|---|
| Readable | Provides data that can be consumed. | Reading a file. |
| Writable | Accepts data to be written. | Writing to a file. |
| Duplex | Supports reading and writing independently. | A TCP socket. |
| Transform | Reads data, transforms it, and produces output. | Compressing data. |
4. Readable Streams
A Readable stream provides data for a consumer. Examples include file reading streams and incoming HTTP request bodies.
Example: Reading a File
Create a text file named input.txt with the following content:
Hello Node.js!
Learning Readable Streams.
Now create the following JavaScript file:
const fs = require("fs");
const readable = fs.createReadStream("input.txt", {
encoding: "utf8"
});
readable.on("data", (chunk) => {
console.log("Received data:");
console.log(chunk);
});
readable.on("end", () => {
console.log("Reading completed");
});
readable.on("error", (error) => {
console.error("Error:", error.message);
});
Run the Program
node readable.js
Expected Output
Received data:
Hello Node.js!
Learning Readable Streams.
Reading completed
The file's content is delivered through the data event.
When no more data remains, the stream emits the end event.
5. Writable Streams
A Writable stream accepts data from a program and sends it to a destination. For example, a file-writing stream writes data to a file.
const fs = require("fs");
const writable = fs.createWriteStream("output.txt");
writable.on("error", (error) => {
console.error("Writing failed:", error.message);
});
writable.on("finish", () => {
console.log("Writing completed");
});
writable.write("Hello Node.js!\n");
writable.write("This is a Writable Stream.\n");
writable.end("File writing is complete.\n");
Run the Program
node writable.js
Output
Writing completed
The program creates or replaces output.txt and writes the
supplied content. The finish event is emitted after
end() has been called and all written data has been processed.
finish event does not guarantee
that data has been physically persisted to storage. For durability-sensitive
operations, consider the relevant filesystem guarantees and synchronization
requirements.
6. Using pipe() with Streams
The pipe() method connects a Readable stream to a Writable
stream. It transfers data and manages backpressure between the streams.
const fs = require("fs");
const readable = fs.createReadStream("input.txt");
const writable = fs.createWriteStream("copy.txt");
readable.on("error", (error) => {
console.error("Read error:", error.message);
});
writable.on("error", (error) => {
console.error("Write error:", error.message);
});
writable.on("finish", () => {
console.log("File copied successfully");
});
readable.pipe(writable);
Run the Program
node pipe.js
Output
File copied successfully
The program copies the contents of input.txt into
copy.txt. The Writable stream finishes after receiving
the data and completing its writes.
stream.pipeline() or its Promise-based version. It helps
propagate errors and clean up streams when a pipeline fails.
7. Using pipeline() for Safer Stream Processing
The pipeline() function connects streams and handles errors
across the pipeline. It is often preferable for production data-processing
flows because it coordinates completion and error propagation.
const fs = require("fs");
const { pipeline } = require("stream");
pipeline(
fs.createReadStream("input.txt"),
fs.createWriteStream("pipeline-copy.txt"),
(error) => {
if (error) {
console.error("Pipeline failed:", error.message);
return;
}
console.log("Pipeline completed successfully");
}
);
Run the Program
node pipeline.js
Output
Pipeline completed successfully
This example copies a file using a stream pipeline. If the source or destination fails, the callback receives the error.
8. Transform Streams
A Transform stream is a special Duplex stream. It reads input, transforms it, and produces output. Examples include text conversion, compression, and data formatting.
Example: Convert Text to Uppercase
const { Transform } = require("stream");
const uppercase = new Transform({
transform(chunk, encoding, callback) {
callback(null, chunk.toString().toUpperCase());
}
});
uppercase.on("data", (chunk) => {
console.log(chunk.toString());
});
uppercase.on("error", (error) => {
console.error("Transform failed:", error.message);
});
uppercase.on("end", () => {
console.log("Transformation completed");
});
uppercase.write("hello ");
uppercase.write("node.js");
uppercase.end();
Output
HELLO
NODE.JS
Transformation completed
The output may be emitted in separate chunks. The important point is that each chunk is converted to uppercase before it is delivered to the consumer.
9. Duplex Streams
A Duplex stream supports both reading and writing. Its readable side and writable side operate independently. A TCP socket is a common example.
You can also create a Duplex stream for learning purposes:
const { Duplex } = require("stream");
const customStream = new Duplex({
read() {
this.push("Data from readable side");
this.push(null);
},
write(chunk, encoding, callback) {
console.log("Received:", chunk.toString());
callback();
}
});
customStream.on("data", (chunk) => {
console.log("Read:", chunk.toString());
});
customStream.on("end", () => {
console.log("Readable side completed");
});
customStream.write("Hello from writable side");
customStream.end();
Expected Output
Received: Hello from writable side
Read: Data from readable side
Readable side completed
The writable side accepts the input message, while the readable side produces a separate message. The two sides do not have to return the same data.
10. What Is Backpressure?
Backpressure occurs when data is produced faster than the destination can consume it. Node.js streams provide mechanisms to slow the producer down and prevent excessive buffering.
For example, when writing to a Writable stream, write()
returns false when the internal buffer has reached its
threshold. The producer should wait for the drain event
before continuing to write more data.
const fs = require("fs");
const writable = fs.createWriteStream("large-output.txt");
writable.on("error", (error) => {
console.error("Write failed:", error.message);
});
writable.on("finish", () => {
console.log("All data written");
});
let index = 0;
function writeData() {
while (index < 100000) {
const canContinue = writable.write(`Line ${index}\n`);
index++;
if (!canContinue) {
writable.once("drain", writeData);
return;
}
}
writable.end();
}
writeData();
This example pauses when the Writable stream signals backpressure.
It resumes when the stream emits drain, avoiding an
uncontrolled loop that keeps filling the buffer.
11. How Stream Processing Works
A file, request, or another readable source.
Produces data in chunks.
Processes or modifies the data.
Writes the processed data to its destination.
12. Stream Best Practices
- Use streams when working with large files or continuous data.
- Handle stream errors appropriately.
- Prefer
pipeline()for complex stream pipelines. - Respect backpressure when writing data.
- Use
end()to signal that no more data will be written. - Do not assume all stream data arrives in one chunk.
- Use Transform streams for incremental data processing.
- Consider file permissions and destination paths when writing files.
Important Points to Remember
- Streams process data incrementally.
- Readable streams provide data.
- Writable streams accept data.
- Duplex streams support reading and writing.
- Transform streams modify data while processing it.
pipe()connects a Readable stream to a destination.pipeline()helps coordinate errors and completion.- Backpressure prevents uncontrolled buffering.
Summary
Streams are a powerful Node.js feature for processing data incrementally. Readable, Writable, Duplex, and Transform streams serve different purposes. Methods such as pipe() and pipeline(), together with proper error handling and backpressure management, help developers build efficient file-processing and data-transfer applications.