# Buffers, Streams and Compression
Course: [Node.js](https://www.devtutorials.ai/learn/nodejs) · Section: System and Files
Source: https://www.devtutorials.ai/learn/nodejs/system-files/buffers-streams-and-compression
> Buffers and raw bytes (alloc, allocUnsafe, from, encodings, concat, subarray, compare, equals, indexOf, typed arrays, string_decoder), why streams save memory, readable streams (flowing and paused modes, events, async iteration, highWaterMark, Readable.from, createReadStream), writable streams and backpressure, duplex and transform streams, object mode, composing streams with pipe and pipeline, the readline module, gzip, deflate and Brotli compression with zlib, and web streams.
## Buffers, Streams and Compression
**Buffer** = a box of raw bytes, such as the bytes of a photo or a file.
**Stream** = data that arrives or leaves a piece at a time, like water through a pipe.
- Files, network replies and uploads are all bytes.
- Small data fits in memory at once, in a Buffer.
- Big data is handled in small pieces, with a stream.
- Compression with `zlib` makes data smaller, and works great with streams.
**Simple lesson:** a Buffer holds bytes, and a stream moves bytes a piece at a time.
### A Real-World Example
Think of filling the school water tank.
- One way: bring the whole tank of water on a lorry. You need a huge lorry.
- Better way: connect a pipe. Water flows bucket by bucket into the tank.
- If the tank fills too fast, you close the tap a little until it catches up.
The pipe is a stream. Each bucket is a chunk. Closing the tap is called **backpressure**.
## What You Will Learn
In this lesson, you will learn:
1. What a Buffer is and how to make one
2. Text encodings: utf8, hex, base64 and more
3. Buffer tools: `concat`, `subarray`, `compare` and friends
4. Why streams save memory
5. Readable and writable streams
6. Backpressure
7. Transform streams, object mode and `pipeline`
8. Reading files line by line with `readline`
9. Compression with `zlib`, and web streams
## Buffers
**Byte** = a number from 0 to 255. Computers store everything as bytes.
- **`Buffer.from("text")`** = the bytes of a text, using UTF-8.
- **`Buffer.alloc(n)`** = `n` bytes, all set to zero.
- **`Buffer.allocUnsafe(n)`** = `n` bytes, not cleared. Faster, but may contain old data from memory. Fill it before use.
- **`buf.length`** = the number of bytes, which can be more than the number of letters.
### Encodings
**Encoding** = a rule for turning bytes into text and back.
- **`utf8`** = normal text in every language. The default.
- **`hex`** = each byte as two characters, `0` to `f`. Good for showing raw bytes.
- **`base64`** = bytes as safe letters. Used in emails and data URLs.
- **`base64url`** = base64 that is safe inside web addresses and file names.
- **`latin1`** = one byte per character, for old Western text.
### Buffer Tools
- **`Buffer.concat([a, b])`** = join buffers into one.
- **`buf.subarray(start, end)`** = a view of part of the buffer. It **shares** the same memory.
- **`a.equals(b)`** and **`Buffer.compare(a, b)`** = compare bytes.
- **`buf.indexOf("text")`** = where a value starts, or -1.
- **`buf.write(text, offset)`** and **`buf.readUInt8(i)`** = write and read single values.
- **Typed arrays:** a Buffer is a `Uint8Array`. `buf.buffer` is the `ArrayBuffer` underneath.
- Changing `section` changed `label`, because `subarray` does not copy.
- Use `Buffer.from(part)` when you need a separate copy.
A Buffer is a `Uint8Array`, so indexes and `for...of` work on it.
### string_decoder and Buffer Pooling (Advanced)
- A UTF-8 letter like `₹` takes 3 bytes. A chunk may end in the middle of it.
- **`StringDecoder`** from `node:string_decoder` keeps the half letter until the rest arrives.
- **Buffer pool** = Node.js hands out small `allocUnsafe` buffers from one shared 8 KB block, for speed.
- That is why `allocUnsafe` can show old bytes. Never send an `allocUnsafe` buffer before filling it.
**Simple lesson:** a Buffer is bytes. Pick the right encoding when you turn it into text.
## Stream Fundamentals
### Why Streams?
readFile needs memory for the whole file. A stream needs memory for one chunk.
- **`readFile`** loads the whole file into memory. A 2 GB file needs 2 GB of memory.
- **A stream** reads 64 KB, hands it on, and reads the next 64 KB.
- Streams start working at once. You do not wait for the whole file.
- Many Node.js objects are streams: files, HTTP requests and responses, `process.stdin`.
### Stream Types
- **Readable** = you read from it. Example: `fs.createReadStream`.
- **Writable** = you write to it. Example: `fs.createWriteStream`, `process.stdout`.
- **Duplex** = both, separately. Example: a network socket.
- **Transform** = a duplex that changes data on the way through. Example: gzip compression.
**Simple lesson:** streams keep memory low and start working early.
## Readable Streams
- **Flowing mode** = data is pushed to you through the `data` event.
- **Paused mode** = you pull data with `read()`.
- **Events:** `data` (a chunk), `end` (no more data), `error` (a problem), `close` (resources freed).
- **`highWaterMark`** = the chunk size the stream aims for, in bytes. 64 KB for files.
- **Async iteration** = `for await (const chunk of stream)`. The easiest way to read.
- **`Readable.from(list)`** = make a readable stream from an array or a generator.
**Simple lesson:** read a stream with `for await`, or listen to `data` and `end`.
## Writable Streams and Backpressure
- **`write(chunk)`** = send a chunk. It returns `true` or `false`.
- **`end(chunk)`** = send the last chunk and close.
- **`finish` event** = all data has been handled.
- **`fs.createWriteStream(p)`** = a writable stream into a file.
**Backpressure** = when the writer is slower than the reader, the reader must wait.
- `write()` returns **`false`** when the stream's buffer is full.
- Then stop writing and wait for the **`drain`** event.
- If you ignore `false`, data piles up in memory, and the stream saved you nothing.
`pipe` and `pipeline` handle backpressure for you. Handle `drain` by hand only when you write chunks yourself.
**Simple lesson:** when `write()` returns `false`, wait for `drain` before writing more.
## Duplex and Transform Streams
- **Duplex** = a readable side and a writable side in one object.
- **Transform** = what you write in comes out changed.
- **`PassThrough`** = a transform that changes nothing. Handy for testing.
- **Object mode** = chunks are any JavaScript values, not just bytes.
- Write a transform with a `transform(chunk, encoding, callback)` function. Call `callback(null, newChunk)`.
## Composing Streams
- **`a.pipe(b)`** = send everything from `a` into `b`. It handles backpressure for you.
- `pipe` does **not** pass errors on, and does not clean up when one stream fails.
- **`pipeline(a, b, c)`** from `node:stream/promises` = join streams, forward errors, and destroy all of them on failure. Use this.
- **`finished(stream)`** = wait until a stream is done or failed.
- **`stream.destroy(err)`** = stop a stream and free its resources.
pipeline joins streams and handles errors and cleanup for you.
- This simple transform expects whole lines in each chunk. For big files, use `readline`, below, which handles lines split across chunks.
Always handle stream errors. Use `pipeline`, or listen for `error` on every stream.
**Simple lesson:** build transforms to change data on the way, and join streams with `pipeline`.
## The readline Module
- **`readline.createInterface({ input })`** = split any readable stream into lines.
- Lines that are cut across chunks are joined for you.
- Read with the `line` event, or with `for await (const line of rl)`.
- **Reading from stdin** works the same way, with `input: process.stdin`.
- **`readline/promises`** gives `rl.question("Your name? ")`, which returns a promise, for interactive prompts.
```js
// ask.mjs: run it in your own terminal with "node ask.mjs"
import { createInterface } from "node:readline/promises";
const rl = createInterface({ input: process.stdin, output: process.stdout });
const name = await rl.question("What is your name? ");
const grade = await rl.question("Which class? ");
console.log(`Welcome, ${name} of Class ${grade}!`);
rl.close();
```
- Each `await rl.question(...)` prints the question and waits for you to type a line.
- Run this one on your own computer. It needs a person typing, so it has no lab here.
**Simple lesson:** use `readline` whenever you need lines, from a file or from a person.
## Compression with zlib
**Compression** = make data smaller by removing repeats. **Decompression** = get the original back.
- **gzip** = the most common format on the web and for `.gz` files.
- **deflate** = the format inside gzip, without the header.
- **Brotli** = newer, often smaller than gzip for text. Used by modern browsers.
- **Sync** (`gzipSync`) = simple, but blocks the event loop. Fine for small data and scripts.
- **Async and streams** (`createGzip`) = use them in servers and for big files.
```js
// main.mjs
import { gzipSync, gunzipSync, brotliCompressSync } from "node:zlib";
const text = "Asha,91\n".repeat(1000);
const gz = gzipSync(text);
const br = brotliCompressSync(text);
console.log("Original:", Buffer.byteLength(text), "bytes");
console.log("gzip:", gz.length, "bytes");
console.log("Brotli:", br.length, "bytes");
console.log("Back again?", gunzipSync(gz).toString() === text);
```
Output:
```text
Original: 8000 bytes
gzip: 60 bytes
Brotli: 27 bytes
Back again? true
```
### Streaming Compression
```js
// main.mjs
import { createReadStream, createWriteStream } from "node:fs";
import { writeFile, stat } from "node:fs/promises";
import { pipeline } from "node:stream/promises";
import { createGzip } from "node:zlib";
await writeFile("attendance.log", "Asha present\nRavi present\n".repeat(5000));
await pipeline(createReadStream("attendance.log"), createGzip(), createWriteStream("attendance.log.gz"));
console.log("Log size:", (await stat("attendance.log")).size);
console.log("Gzip size:", (await stat("attendance.log.gz")).size);
```
Output:
```text
Log size: 130000
Gzip size: 377
```
- The file is compressed chunk by chunk. Memory stays small even for huge logs.
- The lab runtime on this page does not include `zlib`. These two examples ran on real Node.js.
**Simple lesson:** compress with streams in servers, and use the Sync versions only in small scripts.
## Web Streams (Advanced)
- Browsers have their own stream type: **`ReadableStream`**. Node.js has it too, as a global.
- `fetch` gives you `response.body` as a web `ReadableStream`.
- **`Readable.toWeb(nodeStream)`** and **`Readable.fromWeb(webStream)`** convert between the two kinds.
```js
// main.mjs
import { Readable } from "node:stream";
const web = Readable.toWeb(Readable.from(["Asha ", "Ravi ", "Meena"]));
let text = "";
for await (const chunk of web) text += chunk;
console.log("From a web stream:", text);
```
Output:
```text
From a web stream: Asha Ravi Meena
```
**Simple lesson:** web streams and Node.js streams can be converted into each other.
## Common mistakes
| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| `readFile` on a 4 GB log | Memory is unlimited | Use `createReadStream` and `readline` | The program runs out of memory |
| Ignoring `write()` returning `false` | Writes are always accepted | Wait for `drain` | Memory fills up with waiting chunks |
| `a.pipe(b).pipe(c)` with no error handling | `pipe` forwards errors | Use `pipeline` | Errors are lost and files stay open |
| Splitting chunks by `"\n"` yourself | Chunks end on line breaks | Use `readline` | Lines cut in half between chunks |
| Changing a `subarray` and expecting the original to stay | `subarray` copies | It shares memory. Use `Buffer.from(part)` | The original buffer changes too |
| Sending `Buffer.allocUnsafe(n)` unfilled | It is empty | Fill it, or use `Buffer.alloc` | Old memory may leak out |
| `gzipSync` in a busy server | Sync is fine everywhere | Use `createGzip` or the async functions | Every request waits |
## TL;DR
1. **Buffer** = raw bytes. `Buffer.from(text)`, `Buffer.alloc(n)`.
2. **Encoding** = utf8, hex, base64, base64url, latin1.
3. **`subarray`** = shares memory. **`Buffer.from(part)`** = copies.
4. **Stream** = data in pieces. Low memory, starts early.
- Readable, Writable, Duplex, Transform
5. **Backpressure** = `write()` returns `false`. Wait for `drain`.
6. **`pipeline`** = join streams with error handling and cleanup.
7. **`readline`** = lines from any stream, including stdin.
8. **zlib** = gzip, deflate, Brotli. Use streams in servers.
## Check Yourself
## Exercises
## What to Remember
- A Buffer is raw bytes. Choose the encoding when you turn it into text.
- `subarray` shares memory. `Buffer.from` copies.
- Streams handle big data in small pieces and keep memory low.
- When `write()` returns `false`, wait for `drain`.
- Join streams with `pipeline`, not `pipe`.
- Use `readline` for lines, and `zlib` streams for compression.
Next: **HTTP and Networking**. You will build a web server with no framework and call other servers.