প্রোগ্রামিং: পাইথন

Scrap Google Related searche by Python (without Api)

ByAsrafulporag December 29, 2022December 30, 2022

1st step, you will need to install these four modules:-

To scrape related searches from Google by Python, you can use the Python libraries:

bs4,
requests,
time,
pandas

Beautiful Soup to parse the HTML of the Google search page and extract the related searches.

2nd step, Import the necessary libraries:

You can install these modules with the following command in your terminal or command prompt.

pip install bs4
pip install requests
pip install python-time
pip install pandas

Once these modules are installed, you can use the following code to scrape the results.

from bs4 import BeautifulSoup
import requests
import pandas as pd

Use the requests library to send a GET request to the Google search page for the term you want to get related searches for. For example:

search_term = “How to Test for air in brake lines”
url = f”https://www.google.com/search?q={search_term}”
response = requests.get(url)

Parse the HTML of the page using Beautiful Soup:

soup = BeautifulSoup(response.text, “html.parser”)

Now, Find the element on the page that contains the related searches. Extract the related searches from the element, You can use the “params queries” method to find all the related searches, and then extract the text from each search. This should give you a list of related searches for the search term you provided.

Note that the specific HTML element and class names may change in the future, so you may need to update your code accordingly. For example:

p = soup.select(“.y6Uyqe .AB4Wff”)

I hope this helps! Let me know if you have any questions.

Final Code:

import time
import requests
from bs4 import BeautifulSoup
import pandas as pd

headers = {
    "User-Agent": "Mozilla/5.0 (X11; CrOS x86_64 14526.89.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.133 Safari/537.36"
}

queries = ["How to Test for air in brake lines"]

df2 = []

# get_current_related_searches
for query in queries:
    params = {"q": query}
    response = requests.get("https://google.com/search", params=params, headers=headers)

    soup = BeautifulSoup(response.text, "html.parser")

    p = soup.select(".y6Uyqe .AB4Wff")

    d = pd.DataFrame(
        {"loop": 1, "source": query, "from": query, "to": [s.text for s in p]}
    )

    terms = d["to"]
    df2.append(d)

    time.sleep(3)

df = pd.concat(df2).reset_index(drop=False)
# print(df)
# df.to_csv("related_searches.csv")

col_list = df.to.values.tolist()
# print(col_list)

finalvaluee = ", ".join(col_list)
print(finalvaluee)