How To Use A Proxy With Requests Module In Python

Guides, How to's, Proxies, Mar-16-20225 mins read

Every computer gets a unique Internet Protocol (IP) address that identifies the computer and its geographic location when connected to the Internet. Your computer sends out a request whenever it needs any information from the Internet. The request is sent to a target computer that checks the type of information being asked for. The target computer sends the information back if it is allowed to give it to our IP address. At times, the computer wants to get the information from the Internet without being identified. That information is usually blocked, but we can get it using a proxy that acts as an intermediary between the client and the server machine.

Each proxy server has its IP address, so when a user requests via a proxy to access a website, the website sends the data to the proxy server IP that forwards it to the user.

Proxies hide the identity of web scrapers and make their traffic look like regular user traffic.
Proxies provide additional security to websites and balance the internet traffic.
Proxies protect web users’ data or help access websites blocked by a country’s censorship mechanism.

Using A Proxy With Requests Module in Python

For using proxies with the Python requests, you need to follow the steps below.

Import requests

Import the requests package that is a simple HTTP library. You can easily send requests through this package without manually adding query strings to your URLs. You can import requests using the below command.

import requests

Create a Dictionary

You need to create a proxies dictionary defining the HTTP and HTTPS connections. You can give the dictionary variable any name like “proxies” that map a protocol to the proxy URL. Further, you have to make the URL variable set to the website you have to scrape from.

proxies = {
  "http":'http://203.190.46.62:8080',
  "https":'https://111.68.26.237:8080'
}
url = 'https://httpbin.org/ip'

Here the dictionary defines the URL of the proxy for two separate protocols i-e HTTP and HTTPS.

Create a Response Variable

You have to create a response variable that uses any of the requests methods. This method takes two arguments:

The URL that you created
The dictionary you defined

response = requests.get(url,proxies = proxies)
print(response.json())

The output is as:

You can also use the requests module in Python to rotate the IP addresses as shown below.

You have to import the requests module by using the below command.

import requests

Create sending_request function

You have to create a sending_request function and prefer using the http proxy as most free proxies don’t use the https protocol. We used try-except block because most of the free proxies don’t work.

def sending_request(session, proxy):
    try:
        response = session.get('http://httpbin.org/ip', proxies={'http': f"http://{proxy}"})
        print(response.json())
    except:
        pass

Read proxies_txt

You have to read the list_proxies.txt file containing the free proxies list and save it to the variable named proxies.

if __name__ == "__main__":
    with open('list_proxies.txt', 'r') as file:
        proxies = file.readlines()

Create Session

You have to create a session from the requests module and follow the below steps.

Loop through the proxy
Pass the proxy and the session to the sending_request function

with requests.Session() as session:
    for proxy in proxies:
        sending_request(session, proxy)

You will get the list of proxies as shown below.

These are some proxies, and the list goes on with more proxies, but all of them don’t work.

Need of Proxies

Every business needs to know the five vital corporate reasons for using proxies mentioned below.

Anonymously Carrying Out Sensitive Tasks

Proxies are well-known for their ability to anonymize web traffic. But most people fail to understand their importance in the business industry. Proxy servers allow the security officers and reporters to protect themselves, companies, sources, clients, and partners.

You can also use proxies to protect the development and current research and other company activities. Suppose your company uses a proxy and a potential spy to track the web traffic to determine what your business is developing. In that case, it won’t be able to track your employees easily.

Improving Corporate and Institutional Security

You know that data breaches are costly both in terms of public image and monetary loss. So, companies are worried about hackers. But proxies can help you as they reduce the chance of data breach. They add an additional layer of security between your servers and the outside traffic. The proxy servers also act as a buffer as they face the internet and relay requests from computers outside the network.

If hackers have access to your proxy servers, they will still face trouble reaching the server that runs the web software where the data is stored.

Controlling Employee Internet Usage

If hackers have access to your proxy servers, they will still face trouble reaching the server that runs the web software where the data is stored.

Saving Bandwidths and Achieving Faster Speeds

Some people assume that proxy servers slow down internet speeds due to the large amount of work they accomplish in the background. But it isn’t always true. The proxy servers can be used to save bandwidth and increase speeds by:

Caching web pages and files accessed by multiple users
Compressing traffic
Stripping ads from websites

Which Proxies To Use?

You might think there is the only type of proxy that provides all benefits to businesses, such as:

Preventing data breaches
Setting competitive prices
Collecting valuable data on social media
Building an effective SEO strategy

In reality, there are many types of proxies available, and the one to use depends on your requirements or use case.

Given below are the most common types of proxies.

Data Center Proxies

Data center proxies are the most common proxies used by businesses worldwide. Data centers produce and manage these proxies. You can use these proxies if you have to improve the security of your system as they are cheap and easy to acquire. But some websites ban their use as they associate them with bot-like activity.

Residential Proxies

The residential proxies are associated with physical residences and use the IP addresses of actual people provided by Internet Service Providers (ISPs). When you use them to connect to a website, you look like an everyday user. Thus, you are less likely to be detected and banned. You can scrape a large amount of web data using residential proxies and achieve improved anonymity and security.

Conclusion

So far, we discussed that a proxy acts as a relay between the client and the server machine. Whenever you request information, your computer sends this request to the proxy, which then sends the information to the target computer using a different IP address. Thus your IP address remains confidential. Further, you can use proxies with requests module in Python and perform various actions depending on your need. If you need a static IP with the speed of datacenter proxies and the high anonymity of residential proxies, then static proxies are the way to go as the IP address remains unchanged with each new request. On the contrary, the rotating proxies provide benefits in testing and scraping.

By: ProxyScrape