A proxy in cURL refers to a server that functions as an intermediary between the client and the destination server, enabling the client to access resources with enhanced anonymity and without network restrictions. This article includes details on how to use cURL with a proxy, as well as a step-by-step guide on how to install it on each OS.
What is cURL?
The cURL tool supports a wide array of protocols, including HTTP, HTTPS, FTP, FTPS, SFTP, POP3, POP3S, IMAP, IMAPS, LDAP, LDAPS, SCP, and more. Consequently, curl has become the most widely used and popular command-line HTTP client globally.
Speaking of using cURL, cURL is highly versatile and serves as a common solution for various tasks, such as making HTTP requests, downloading files, uploading data, and interacting with APIs. It offers numerous features, ranging from request customization to proxy support.
How to Install cURL on Windows, MacOS, and Linux
Regardless of the operating system you’re on, setting up cURL is fairly easy. However, it does require some knowledge of handling command lines.
Starting with the easiest, Mac OS users don’t need any additional setup when it comes to cURL. This operating system already integrated the cURL tool, so you can use it natively via the Terminal.
On a similar note, Windows also comes with a copy of cURL. However, since this copy automatically runs the Invoke-Request behind the scenes, you’ll need to replace cURL with ‘curl.exe’. This way, the Windows PowerShell will run cURL. To make sure that cURL is set up properly, you can check the version by typing curl.exe –version into the CMD.
Lastly, on Linux, the procedure for installing cURL changes depending on the specific distribution you are using. Popular Linux distributions, such as Ubuntu and Fedora, come with it by default. Therefore, you can use cURL directly from the terminal.
In other Linux distributions, however, cURL may not be included. In this case, you can add it using the distribution package manager. For example, on a Debian-based OS, you can install curl by typing sudo apt-get install curl into the Terminal.
How to use cURL with a Proxy in PHP
Here is an example of how to set up a proxy with cURL with coding, in this case, in the PHP programming language.
Start with setting up the proxy:
$url = ‘http://yourwebsite.com/ip.php’; // replace the URL with the real one
$proxy = ‘x.x.x.x:xxxx’; // replace the x with the real proxy server numbers
$proxyauth = ‘user:password’;
Next, here are the basic cURL commands in PHP that you can use in your PHP function:
$ch = curl_init(); // initiating the cURL
curl_setopt($ch, CURLOPT_URL, $url); // URL for cURL call
curl_setopt($ch, CURLOPT_PROXY, $proxy); // PROXY details with port
curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxyauth); // Use if proxy requires authentication (username & password)
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5); // If expected to call with specific PROXY type
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); // If the URL has redirects then go to the final redirected URL.
curl_setopt($ch, CURLOPT_HEADER, 1); // If you want Header information of response, if not then replace 1 with 0
$curl_scraped_page = curl_exec($ch); // executing the function
curl_close($ch); // closing the session
echo $curl_scraped_page; // return command of the function you’re creating
curl_setopt($ch, CURLOPT_PROXY, null); // disabling the proxy
How to use cURL with a Proxy in Python
Similarly to the previous example, it’s possible to use cURL with a proxy in the Python language.
The first thing to do is to set up PycURL and certifi tools, both of which you’ll need to set up the Proxy connection in Python and make requests further on. To do so, simply type this command:
pip install pycurl
pip install certifi
Now that it’s installed, import it in Python and use these commands:
- Proxy Setup in Python:
c.setopt(c.PROXY_SSL_VERIFYHOST, 0) # Enable insecure proxies for easier handling
c.setopt(c.PROXY_SSL_VERIFYPEER, 0)
c.setopt(pycurl.PROXY, <YOUR_HTTPS_PROXY_SERVER>) # Set proxy server
c.setopt(pycurl.PROXYUSERPWD, f”{<YOUR_USERNAME>}:{<YOUR_PASSWORD>}”) # Complete authentication
c.setopt(pycurl.PROXYTYPE, 2) # Set proxy type to https
- GET Requests:
import pycurl
import certifi
from io import BytesIO
buffer = BytesIO() # Set buffer and Curl object.
c = pycurl.Curl()
c.setopt(c.URL, ‘http://pycurl.io/’) # Set request options and the request destination.
c.setopt(c.WRITEDATA, buffer) # Set the buffer as the destination of the request’s response.
c.setopt(c.CAINFO, certifi.where()) # Validate the SLL Certificate
c.perform() # Execute the request
c.close() # Close the request
body = buffer.getvalue() # Print the buffer’s content.
data = body.decode(‘iso-8859-1’)
print(data)
- POST Requests:
import pycurl
import certifi
from io import BytesIO
buffer = BytesIO() # Set buffer and Curl object.
c = pycurl.Curl()
c.setopt(c.URL, ‘http://pycurl.io/’) # Set request options and the request destination.
post_body = {‘greeting’: ‘hello’} ## Set the request’s body.
postfields = urlencode(post_body)
c.setopt(c.POSTFIELDS, postfields)
c.setopt(c.WRITEDATA, buffer) ## Set the buffer as the destination of the request’s response
c.setopt(c.CAINFO, certifi.where()) # Validate the SLL Certificate
c.perform() # Execute the request
c.close() # Close the request
# Print the buffer’s content with a Latin1 (iso-8859-1) encoding.
body = buffer.getvalue()
print(body.decode(‘iso-8859-1’))
- PUT Requests:
import pycurl
import certifi
from io import BytesIO
c = pycurl.Curl() # Initiate the cURL.
c.setopt(c.URL, ‘http://pycurl.io/’) # Set request options and the request destination.
c.setopt(c.UPLOAD, 1) # Set data for the PUT request.
data = ‘{“greeting”: “hello”}’
buffer = BytesIO(data.encode(‘utf-8’))
c.setopt(c.READDATA, buffer)
c.setopt(c.CAINFO, certifi.where()) # Validate the SLL Certificate
c.perform() # Execute the request
c.close() # Close the request
How to Get a Valid Proxy?
Proxies serve to separate end-user clients from the websites they browse, providing multiple levels of functionality and security.
For instance, proxy servers can compress and encrypt network traffic, prevent intrusion into a local network from the Internet, and monitor employee Internet use during working hours. When utilizing proxy servers, all requests to the Internet are initially directed to the proxy server, which assesses the request, applies a set of rules to it, and then, if necessary, forwards the request to the Internet.
The key step in using cURL with a proxy is obtaining a valid and secured HTTP proxy. There are several websites that offer free proxies online. A free proxy is a good choice for beginners and students who are inexperienced with using proxies. However, since these free proxies are prone to errors, usually not compatible with the HTTPS, and are often poorly protected, the reliability of a free proxy cannot always be guaranteed.
In instances where a more stable and expedient connection is imperative, opting for a high-quality proxy service is a more prudent choice in order to evade potential blocking. Therefore, if you’re looking for a proxy that is cost-effective and reliable, we’ve got you covered. Web Data Extraction Services offers highly protected private proxies that you can customize to cater to your business needs.
Starting at $insert cents$, a single private proxy can elevate your marketing strategy and increase the success rate of your company by allowing you to conduct research and analysis from multiple locations and IP addresses and, therefore, gain leverage against your competitors. In order to get your hands on a premium-quality proxy, reach out to us by completing the contact us form.
