Both of them change your IP address. But how they change it, and how much of it they change, are completely different. Understanding this difference answers the question "which one should I use" by itself.
For a general proxy–VPN comparison, see our the difference between a proxy and a VPN article; here we focus specifically on SOCKS5.
The Fundamental Difference: Scope
A VPN adds a virtual interface to the operating system's network layer; all traffic passes through it. SOCKS5, on the other hand, only affects the applications routed to it.
Comparison Table
The most critical row is "multiple identities": running five applications with five different exit IPs on the same machine is easy with SOCKS5 and almost impossible with a VPN.
When SOCKS5?
If you need per-application routing
If you want only one browser profile or one script to go through the proxy. The rest of your traffic keeps flowing normally.
If you need multiple identities at the same time
If you want to manage five different accounts with five different exit IPs. You assign a separate SOCKS5 proxy to each profile.
If maximum speed matters
Because there is no encryption layer, CPU load and latency are lower.
If you will collect data from a specific exit country
In data collection and automation work the proxy is the standard tool; a VPN was not designed for this scenario.
When a VPN?
If you are working on an untrusted network
On public Wi-Fi all your traffic needs to be encrypted. SOCKS5 does not provide this.
If you want to protect the whole device
If you want everything — including background applications, updaters and system services — to go through the tunnel.
If you want to hide from your internet provider
A VPN prevents your provider from seeing which site you visit. SOCKS5 alone does not do this.
If you need access to a corporate network
This is exactly what a VPN was designed for: secure access to a remote private network.
If you are on an untrusted network and also need multiple identities, you can combine the two: encrypt all traffic with a VPN and assign separate SOCKS5 proxies to individual applications. This gives you security and flexibility at the same time.
Closing the Encryption Gap
How much of a problem is SOCKS5's lack of encryption in practice? The answer depends on the content being carried:
As long as you use HTTPS, the content is protected everywhere. There are two things that are not protected: which site you visit and your SOCKS5 credentials. An SSH tunnel closes this gap.
how to set up encrypted SOCKS5 with an SSH tunnel a separate article in detail.
Performance Comparison
The difference comes from the encryption process and from the VPN tunnelling all traffic. Geographic distance is the dominant factor in both cases.
Summary
SOCKS5 and VPNs solve different problems. SOCKS5 is per-application, fast and allows multiple identities to be managed on the same machine; however, it offers no encryption. A VPN covers the whole system and encrypts traffic; however, it offers a single exit identity and is slower. For data collection, automation and multi-account work SOCKS5 is the right tool; for general protection on an untrusted network, a VPN is. For a product that suits your needs, see SOCKS5 proxy page.