BASALT · JOURNAL
Reading a PDF in chunks instead of loading it
Every application that opens a document makes one early decision that sets its ceiling forever: read the file into memory, or read through it.
The two designs
Buffering reads the whole file into a block of memory and works there. It is simpler, faster for small documents, and its peak memory is the file size. A 2 GB production needs 2 GB before anything begins.
Streaming reads a fixed size piece, handles it, discards it, and moves on. Peak memory follows the piece. Reading in 4 MB chunks costs about 4 MB whether the document is 40 MB or 4 GB.
Why the difference is invisible in testing
The documents developers test with are small. A ten page invoice behaves identically under both designs, and the ceiling only appears when somebody opens a real scanned production, which usually happens after release and in front of a customer.
This is why large file failures are so common in otherwise careful software. Nothing was done wrong; the case was never exercised.
Verifying it yourself
Open the largest document you have and watch the application in Activity Monitor.
Memory rising towards the file size means buffering. Memory staying flat and small means streaming. In a measured run on a 273 MB scanned document, peak resident memory was 89 MB, which is far below the file size and therefore evidence the document was never held whole.
The cost of streaming
Streaming usually means a working copy on disk instead of in memory. That is a good trade, since disk is plentiful and running low on it degrades gracefully rather than killing the process, but it is a real cost worth knowing about. Opening a 2 GB document may need a similar amount of free disk space, and a well behaved application should release it when the document closes.
Where buffering still wins
For small documents, buffering is faster, and there is no reason to stream a 200 KB form. The sensible design chooses by size rather than dogma, and the observable outcome is what matters: memory that does not track file size on the documents you actually work with.
Frequently asked questions
What does it mean to stream a PDF?
Reading the file in fixed size pieces and processing each one before moving on, rather than loading the whole document into memory first. Peak memory then follows the piece size instead of the file size, so a 4 MB chunk costs about 4 MB regardless of whether the document is 40 MB or 4 GB.
How can I tell if an application streams or buffers?
Open your largest document and watch the application in Activity Monitor. Memory climbing to roughly the file size means the document is being buffered. Memory staying flat and modest means it is being streamed.
Is streaming always better?
Not for small documents, where buffering is simpler and faster and the memory cost is trivial. It matters when documents are large, which is where buffering fails outright rather than merely being slower.
Does streaming use disk instead of memory?
Usually. A streamed document generally has a working copy written to disk, so opening a 2 GB file can require similar free space. That is a favourable trade because disk exhaustion degrades gracefully where memory exhaustion does not.
Doing it in Basalt
Basalt is a native macOS PDF toolkit with eighteen tools in one window. It opens large documents without loading them into memory, renders pages on demand, and copies files into its engine in fixed-size chunks, so peak memory follows the chunk size rather than the file size. Redaction destroys content rather than covering it, and an independent verifier re-opens every written file to prove the material is gone before the file is saved. A one time $29 licence covers up to three Macs, it is free for the first 24 hours, and the engine holds no network entitlement at all, which macOS enforces at the code-signature level. Download it at basaltformac.com.
Redaction that proves itself
Basalt destroys the content you mark, then re-opens the file it wrote and proves the content is gone before it saves anything. Your documents never leave your Mac.