Caching is one of the oldest functions of a proxy: instead of downloading the same content again and again, you keep a copy and answer later requests locally. As HTTPS became universal, the scope of this function narrowed, but it did not disappear entirely — and it is still a serious source of savings in your own data collection pipeline.
The Basic Flow
On a cache hit, the origin server is never contacted. That is a win for speed, for bandwidth and for the load on the target server alike.
Who Decides What Gets Stored?
The server makes the decision and announces it with the Cache-Control header:
| Directive | Meaning | Proxy behavior |
|---|---|---|
public | Shared caches may store it | Stores |
private | Only the browser may store it | Does not store |
no-store | Must not be stored at all | Does not store |
no-cache | May be stored, but validated on every use | Makes a conditional request |
max-age=3600 | Considered fresh for 3600 seconds | Does not contact the origin during that period |
s-maxage=600 | A separate lifetime for shared caches | Gives this value precedence |
no-cachedoes not mean "do not store at all" — it means "store it, but validate before each use". The equivalent of "do not store at all" is no-store.
Conditional Requests: 304 Not Modified
When the cached copy goes stale, the proxy does not have to download the content from scratch. It asks the server, "is the version I have still valid?":
A 304 response contains no body; only headers are returned. That means turning a 500 KB page into 300 bytes.
In your own data collection pipeline, using this mechanism cuts bandwidth costs noticeably.
How Did HTTPS Affect Caching?
Once a CONNECT tunnel is established, the proxy cannot see the content — and therefore cannot cache it either. Because almost the entire web has moved to HTTPS, classic proxy caching has largely been left without a job.
That is why modern caching has moved out of the proxy layer and into the CDN and client layers.
Building Your Own Cache
If proxy caching is unavailable, you can achieve the same gain inside your own application. This is very effective in data collection pipelines:
This simple layer keeps you from downloading rarely changing pages over and over. With GB-based resources such as residential proxies, the savings show up directly on your bill.
When Is Caching the Wrong Choice?
Caching is a good fit
- Product and category pages that rarely change.
- Static reference data.
- Jobs that fetch the same page several times a day.
- Development and testing (it preserves your quota).
Caching is the wrong choice
- Real-time price and stock tracking.
- Personalized content.
- Pages that require a session.
- SEO jobs that track ranking changes.
Working with stale cached data can produce worse results than collecting no data at all. For time-sensitive fields such as price and stock, keep the cache lifetime very short or do not cache at all.
Summary
With the spread of HTTPS, proxy caching has largely lost its classic function, because the proxy cannot see the content inside a CONNECT tunnel. Building the same logic into your own application layer, however, is still possible and very valuable. ETag-based conditional requests eliminate almost all traffic for content that rarely changes. For time-sensitive data, on the other hand, caching should be avoided. For consumption math, see our bandwidth article .