Proxy for Flickr: Layers, API Quota and Shared Use

Coverage in a script environment: environment variables and exceptions

In the browser the proxy is given from a settings screen; in a script you set up the coverage yourself from the start. The most common way is environment variables: HTTP_PROXY and HTTPS_PROXY routes outgoing requests, NO_PROXY leaves certain addresses outside the rule. The case sensitivity of the names varies by tool; one tool reads only the lowercase form while another accepts both.

The scheme distinction is the second detail and is where errors most often come from. Since Flickr requests go over HTTPS, the decisive value is HTTPS_PROXY ; in an environment where only HTTP_PROXY is defined, the calls go out directly without ever seeing the proxy. Because the process does not throw an error, this situation proceeds silently: the job completes, but no request has gone through the exit you expected.

The third piece is the exception list. If you do not include internal network addresses and local service names in NO_PROXY , the connection to your own database or cache server is also routed to the external exit; the result is timeouts whose source is hard to identify. There is a risk in the opposite direction too: writing too broad a pattern causes the very requests you want to route to get caught by the list and go out directly.

For the system-level counterparts, the Ubuntu and Linux proxy settings article shows step by step where the variables are written permanently. On Windows the same job is handled either by the system-wide network setting or by variables defined per process; browser profiles, on the other hand, never read these variables and look at their own configuration. Forgetting this distinction while running a script and a browser on the same machine leads you to notice too late that the two go out through different exits.

Role and access separation in teams managing an archive

If a corporate archive is managed by more than one person, everyone working with the same access details is practical in the short term and problematic in the long term. When a problem arises you cannot tell which session caused it, and for one person leaving the team you have to renew all the access details.

A role-based separation is more sustainable. The table below summarises common roles and the access they need.

The second benefit of separation is on the measurement side. When every role uses its own entry point, volume and error counts also become readable per role. When a month's invoice comes in higher than expected, you are not left guessing whether it came from editor uploads or from a loop on the developer side.

The choice of authentication method is the technical counterpart of this separation; the options are compared in the proxy authentication methods article. For stable session behaviour you need to know the difference between a rotating proxy and a static exit: archive management does not want rotation, it wants stability.

Access from a corporate network: allowlists and continuity

In a corporate environment the demand for a proxy often arises from management rather than privacy. Traffic leaving the company network is required to appear from a single known address; this both makes the audit logs readable and creates a fixed identity that can be referenced on the other side. In archive work this identity makes it traceable afterwards who touched a collection and in what scope.

Allowlists are the most concrete form of this need. If you access a partner's system or your own internal service from outside, an exit whose address does not change simplifies rule management on the other side. A rotating pool has exactly the opposite effect here: every new address falls outside the list and access is cut off at unpredictable moments.

The choice of access method depends on where the team works. IP authorisation is practical inside the office, because there is a single address to define; for someone working from home, the username-password method is more suitable since access would drop when the subscriber address changes. Binding the two methods to separate entry points on the same account puts both groups at ease; the architecture of entry points is gateway architecture article.

• Keep a record in one place of the exit address used and which team uses it.
• Inform every counterparty you have given an allowlist entry to before an address change.
• Define separate entry points for in-office and remote workers.

If you are planning a collection job that runs through the night, clarify the continuity commitment upfront as well: whether the job should stop, wait or resume where it left off when the exit is cut must be decided in advance. How to read provider commitments is uptime and SLA is explained in that article.

Static media addresses and bandwidth accounting

Flickr stores photos as files derived in different sizes and serves them from a static media address separate from the main domain. This separation makes sense for performance: static content is suitable for caching, a dynamic page is not.

It has two consequences for the proxy. The first is coverage: a rule that covers only the main address does not route the images. The second is cost: on an exit charged by volume, what grows the bill is not the pages but the downloaded photos. When crawling a high-resolution archive this difference grows quickly; for the calculation method see bandwidth calculation article.

A third consequence is on the logging side. Static media requests are numerous: a single gallery page can produce dozens of image requests. Looking at the line count in the proxy logs and concluding "too many requests were made" is therefore misleading; the meaningful metric is not the line count but the volume transferred and the number of API calls made.

Size derivatives are your strongest lever here. The volume difference between a thumbnail and a full-resolution file is large; downloading the largest derivative for a job intended for preview or classification produces unnecessary cost. Setting up the workflow as "smallest derivative first, the larger one if needed" delivers the most noticeable saving in most projects.

Planning the difference between reading metadata and downloading files upfront is equally effective. In most workflows the metadata is collected first and images are downloaded only for the records needed; this ordering protects both the quota and the volume.

The order of setup and verification

In setup, the order determines the reliability of the outcome. First you decide where the rule will be written, then the exit is verified, then leakage is measured, and finally the quota and error behaviour are monitored.

On the browser side DNS leak test and WebRTC leak test should be run together; the two look at different layers. You can see that the exit address and its country are as you expect my IP address with the tool.

Client libraries

If you make the API calls from a script, the proxy setting is usually given via an environment variable or the library's own parameter. Most libraries do not read environment variables automatically, or read them only for certain schemes; that is why giving the setting explicitly is the safest way. Verify that the setting is actually applied by printing the exit address the script sees once. The example connection details format is proxy.example.com, port 8080, username username and password password ; the real values are in your panel.

Verification is not a one-off job. Repeat the same four steps when you change the exit type, move to a new location or update the library. The most common surprise is a setup that has worked flawlessly for months quietly starting to bypass the proxy after a small update.

When is a proxy really not necessary?

If you manage your own archive from your own country with a single account, putting a proxy in between gains you nothing; it only adds a stop and a point of failure.

The cases where a proxy becomes meaningful are clear: exiting a corporate network with a fixed and recorded address, verifying how a collection appears in different regions, separating team members' access, and collecting publicly available metadata at regular intervals. What these scenarios have in common is that the proxy is used as an identity and access tool rather than a speed tool.

In these scenarios the decision comes down to two topics: how many separate exit addresses are really needed and which transport protocol covers the job. In archive management the answer is usually few and simple: a single fixed exit with an HTTPS tunnel meets most teams' needs. One question is enough when making the decision: is there something concrete you cannot do without a proxy? If the answer is no, not adding the layer is the best configuration.