Buffers, Streams and Compression

By Syncfusion®, Inc. · Last updated November 13, 2026

Buffers and raw bytes (alloc, allocUnsafe, from, encodings, concat, subarray, compare, equals, indexOf, typed arrays, string_decoder), why streams save memory, readable streams (flowing and paused modes, events, async iteration, highWaterMark, Readable.from, createReadStream), writable streams and backpressure, duplex and transform streams, object mode, composing streams with pipe and pipeline, the readline module, gzip, deflate and Brotli compression with zlib, and web streams.

Buffers, Streams and Compression

Buffer = a box of raw bytes, such as the bytes of a photo or a file.

Stream = data that arrives or leaves a piece at a time, like water through a pipe.

• Files, network replies and uploads are all bytes.
• Small data fits in memory at once, in a Buffer.
• Big data is handled in small pieces, with a stream.
• Compression with zlib makes data smaller, and works great with streams.

Simple lesson: a Buffer holds bytes, and a stream moves bytes a piece at a time.

A Real-World Example

Think of filling the school water tank.

• One way: bring the whole tank of water on a lorry. You need a huge lorry.
• Better way: connect a pipe. Water flows bucket by bucket into the tank.
• If the tank fills too fast, you close the tap a little until it catches up.

The pipe is a stream. Each bucket is a chunk. Closing the tap is called backpressure.

What You Will Learn

In this lesson, you will learn:

1. What a Buffer is and how to make one
2. Text encodings: utf8, hex, base64 and more
3. Buffer tools: concat, subarray, compare and friends
4. Why streams save memory
5. Readable and writable streams
6. Backpressure
7. Transform streams, object mode and pipeline
8. Reading files line by line with readline
9. Compression with zlib, and web streams

Buffers

Byte = a number from 0 to 255. Computers store everything as bytes.

• Buffer.from("text") = the bytes of a text, using UTF-8.
• Buffer.alloc(n) = n bytes, all set to zero.
• Buffer.allocUnsafe(n) = n bytes, not cleared. Faster, but may contain old data from memory. Fill it before use.
• buf.length = the number of bytes, which can be more than the number of letters.

Encodings

Encoding = a rule for turning bytes into text and back.

• utf8 = normal text in every language. The default.
• hex = each byte as two characters, 0 to f. Good for showing raw bytes.
• base64 = bytes as safe letters. Used in emails and data URLs.
• base64url = base64 that is safe inside web addresses and file names.
• latin1 = one byte per character, for old Western text.

Buffer Tools

• Buffer.concat([a, b]) = join buffers into one.
• buf.subarray(start, end) = a view of part of the buffer. It shares the same memory.
• a.equals(b) and Buffer.compare(a, b) = compare bytes.
• buf.indexOf("text") = where a value starts, or -1.
• buf.write(text, offset) and buf.readUInt8(i) = write and read single values.
• Typed arrays: a Buffer is a Uint8Array. buf.buffer is the ArrayBuffer underneath.

• Changing section changed label, because subarray does not copy.
• Use Buffer.from(part) when you need a separate copy.

string_decoder and Buffer Pooling (Advanced)

• A UTF-8 letter like ₹ takes 3 bytes. A chunk may end in the middle of it.
• StringDecoder from node:string_decoder keeps the half letter until the rest arrives.
• Buffer pool = Node.js hands out small allocUnsafe buffers from one shared 8 KB block, for speed.
• That is why allocUnsafe can show old bytes. Never send an allocUnsafe buffer before filling it.

Simple lesson: a Buffer is bytes. Pick the right encoding when you turn it into text.

Stream Fundamentals

Why Streams?

Animation: readFile fills memory with the whole file, while a stream keeps only one small chunk in memory at a time readFile: the whole file at once stream: one chunk at a time

memory used: 2 GB file = 2 GB

memory used: 64 KB, again and again

readFile needs memory for the whole file. A stream needs memory for one chunk.

• readFile loads the whole file into memory. A 2 GB file needs 2 GB of memory.
• A stream reads 64 KB, hands it on, and reads the next 64 KB.
• Streams start working at once. You do not wait for the whole file.
• Many Node.js objects are streams: files, HTTP requests and responses, process.stdin.

Stream Types

• Readable = you read from it. Example: fs.createReadStream.
• Writable = you write to it. Example: fs.createWriteStream, process.stdout.
• Duplex = both, separately. Example: a network socket.
• Transform = a duplex that changes data on the way through. Example: gzip compression.

Simple lesson: streams keep memory low and start working early.

Readable Streams

• Flowing mode = data is pushed to you through the data event.
• Paused mode = you pull data with read().
• Events: data (a chunk), end (no more data), error (a problem), close (resources freed).
• highWaterMark = the chunk size the stream aims for, in bytes. 64 KB for files.
• Async iteration = for await (const chunk of stream). The easiest way to read.
• Readable.from(list) = make a readable stream from an array or a generator.

Simple lesson: read a stream with for await, or listen to data and end.

Writable Streams and Backpressure

• write(chunk) = send a chunk. It returns true or false.
• end(chunk) = send the last chunk and close.
• finish event = all data has been handled.
• fs.createWriteStream(p) = a writable stream into a file.

Backpressure = when the writer is slower than the reader, the reader must wait.

• write() returns false when the stream's buffer is full.
• Then stop writing and wait for the drain event.
• If you ignore false, data piles up in memory, and the stream saved you nothing.

Simple lesson: when write() returns false, wait for drain before writing more.

Duplex and Transform Streams

• Duplex = a readable side and a writable side in one object.
• Transform = what you write in comes out changed.
• PassThrough = a transform that changes nothing. Handy for testing.
• Object mode = chunks are any JavaScript values, not just bytes.
• Write a transform with a transform(chunk, encoding, callback) function. Call callback(null, newChunk).

Composing Streams

• a.pipe(b) = send everything from a into b. It handles backpressure for you.
• pipe does not pass errors on, and does not clean up when one stream fails.
• pipeline(a, b, c) from node:stream/promises = join streams, forward errors, and destroy all of them on failure. Use this.
• finished(stream) = wait until a stream is done or failed.
• stream.destroy(err) = stop a stream and free its resources.

Animation: chunks flow from a file read stream through an uppercase transform into a write stream Readablemarks.csv Transformmake report line Writablereport.txt

pipeline(read, transform, write): one error stops and cleans up all three

pipeline joins streams and handles errors and cleanup for you.

• This simple transform expects whole lines in each chunk. For big files, use readline, below, which handles lines split across chunks.

Simple lesson: build transforms to change data on the way, and join streams with pipeline.

The readline Module

• readline.createInterface({ input }) = split any readable stream into lines.
• Lines that are cut across chunks are joined for you.
• Read with the line event, or with for await (const line of rl).
• Reading from stdin works the same way, with input: process.stdin.
• readline/promises gives rl.question("Your name? "), which returns a promise, for interactive prompts.

// ask.mjs: run it in your own terminal with "node ask.mjs"
import { createInterface } from "node:readline/promises";

const rl = createInterface({ input: process.stdin, output: process.stdout });

const name = await rl.question("What is your name? ");
const grade = await rl.question("Which class? ");
console.log(`Welcome, ${name} of Class ${grade}!`);
rl.close();

• Each await rl.question(...) prints the question and waits for you to type a line.
• Run this one on your own computer. It needs a person typing, so it has no lab here.

Simple lesson: use readline whenever you need lines, from a file or from a person.

Compression with zlib

Compression = make data smaller by removing repeats. Decompression = get the original back.

• gzip = the most common format on the web and for .gz files.
• deflate = the format inside gzip, without the header.
• Brotli = newer, often smaller than gzip for text. Used by modern browsers.
• Sync (gzipSync) = simple, but blocks the event loop. Fine for small data and scripts.
• Async and streams (createGzip) = use them in servers and for big files.

// main.mjs
import { gzipSync, gunzipSync, brotliCompressSync } from "node:zlib";

const text = "Asha,91\n".repeat(1000);
const gz = gzipSync(text);
const br = brotliCompressSync(text);

console.log("Original:", Buffer.byteLength(text), "bytes");
console.log("gzip:", gz.length, "bytes");
console.log("Brotli:", br.length, "bytes");
console.log("Back again?", gunzipSync(gz).toString() === text);

Output:

Original: 8000 bytes
gzip: 60 bytes
Brotli: 27 bytes
Back again? true

Streaming Compression

// main.mjs
import { createReadStream, createWriteStream } from "node:fs";
import { writeFile, stat } from "node:fs/promises";
import { pipeline } from "node:stream/promises";
import { createGzip } from "node:zlib";

await writeFile("attendance.log", "Asha present\nRavi present\n".repeat(5000));
await pipeline(createReadStream("attendance.log"), createGzip(), createWriteStream("attendance.log.gz"));

console.log("Log size:", (await stat("attendance.log")).size);
console.log("Gzip size:", (await stat("attendance.log.gz")).size);

Output:

Log size: 130000
Gzip size: 377

• The file is compressed chunk by chunk. Memory stays small even for huge logs.
• The lab runtime on this page does not include zlib. These two examples ran on real Node.js.

Simple lesson: compress with streams in servers, and use the Sync versions only in small scripts.

Web Streams (Advanced)

• Browsers have their own stream type: ReadableStream. Node.js has it too, as a global.
• fetch gives you response.body as a web ReadableStream.
• Readable.toWeb(nodeStream) and Readable.fromWeb(webStream) convert between the two kinds.

// main.mjs
import { Readable } from "node:stream";

const web = Readable.toWeb(Readable.from(["Asha ", "Ravi ", "Meena"]));
let text = "";
for await (const chunk of web) text += chunk;
console.log("From a web stream:", text);

Output:

From a web stream: Asha Ravi Meena

Simple lesson: web streams and Node.js streams can be converted into each other.

Common mistakes

TL;DR

1. Buffer = raw bytes. Buffer.from(text), Buffer.alloc(n).
2. Encoding = utf8, hex, base64, base64url, latin1.
3. subarray = shares memory. Buffer.from(part) = copies.

4. Stream = data in pieces. Low memory, starts early.

   • Readable, Writable, Duplex, Transform
5. Backpressure = write() returns false. Wait for drain.
6. pipeline = join streams with error handling and cleanup.
7. readline = lines from any stream, including stdin.
8. zlib = gzip, deflate, Brotli. Use streams in servers.

Check Yourself

Exercises

What to Remember

• A Buffer is raw bytes. Choose the encoding when you turn it into text.
• subarray shares memory. Buffer.from copies.
• Streams handle big data in small pieces and keep memory low.
• When write() returns false, wait for drain.
• Join streams with pipeline, not pipe.
• Use readline for lines, and zlib streams for compression.

Next: HTTP and Networking. You will build a web server with no framework and call other servers.