[웹크롤링] 구글맵 크롤링하기 (Selenium)

구글맵의 데이터를 크롤링하기 위해 필요한 라이브러리인 selenium과 BeautifulSoup을 설치합니다.

conda install selenium
conda install BeautifulSoup

 

chrome driver 설치

아래의 웹에서 크롬 버전에 맞는 크롬 드라이버를 설치합니다.

https://sites.google.com/chromium.org/driver/

ChromeDriver - WebDriver for Chrome

 

코드

필요한 패키지들을 불러옵니다.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
import pandas as pd
import time

 

 

크롬창을 호출합니다.

#크롬창 호출
driver = webdriver.Chrome() 

# 옵션 생성
options = webdriver.ChromeOptions()

# 옵션 추가
options.add_argument('disable-gpu') # GPU를 사용하지 않도록 설정
options.add_argument('headless')

 

크롬창에 띄울 url을 입력합니다.

driver.get('https://www.google.com/maps/') #띄워진 크롬창에 불러오고 싶은 주소 입력

 

 

검색할 키워드를 넣습니다.

searchbox = driver.find_element_by_css_selector('input#searchboxinput')
searchbox.send_keys('경주 맛집') # 키워드 입력

searchbutton = driver.find_element_by_css_selector("button#searchbox-searchbutton")
searchbutton.click()

 

 

크롬창에서 원하는 위치의 xpath를 추출하여 넣습니다.

#첫 번째 리스트의 가게 클릭
driver.find_element_by_xpath('//*[@id="QA0Szd"]/div/div/div[1]/div[2]/div/div[1]/div/div/div[2]/div[1]/div[3]/div').click()

 

 

상호명과 주소를 크롤링하기 위해 각 xpath를 입력하였습니다.

title = driver.find_element_by_xpath('//*[@id="QA0Szd"]/div/div/div[1]/div[3]/div/div[1]/div/div/div[2]/div[2]/div/div[1]/div[1]/h1').text
address = driver.find_element_by_xpath('//*[@id="QA0Szd"]/div/div/div[1]/div[2]/div/div[1]/div/div/div[9]/div[3]/button/div/div[3]/div[1]').text

 

검색해서 나온 상호명이 표출되는 검색창의 스크롤 다운 함수입니다.

SCROLL_PAUSE_TIME = 2

# Get scroll height
last_height = driver.execute_script("return document.body.scrollHeight")
number = 0

while True:
    number = number+1

    # Scroll down to bottom
    scroll = driver.find_element_by_xpath('//*[@id="QA0Szd"]/div/div/div[1]/div[2]/div/div[1]/div/div/div[2]/div[1]')
    driver.execute_script('arguments[0].scrollBy(0, 5000);', scroll)

    # Wait to load page
    time.sleep(SCROLL_PAUSE_TIME)

    # Calculate new scroll height and compare with last scroll height
    print(f'last height: {last_height}')
    scroll = driver.find_element_by_xpath('//*[@id="QA0Szd"]/div/div/div[1]/div[2]/div/div[1]/div/div/div[2]/div[1]')
    new_height = driver.execute_script("return arguments[0].scrollHeight", scroll)

    print(f'new height: {new_height}')

    if number == 3:
        break

    if new_height == last_height:
        break

    print('cont')
    last_height = new_height

 

 

검색으로 노출된 가게 리스트의 정보를 크롤링하는 함수입니다.

title_ = []
address_ = []

for i in range(3, 100, 2):  
    time.sleep(1)
    SCROLL_PAUSE_TIME = 5
    
    global last_height
    global new_height
    
    # Get scroll height
    last_height = driver.execute_script("return document.body.scrollHeight")
    number = 0
    
    while True:
        number = number+1 
    
        # Scroll down to bottom
        scroll = driver.find_element_by_xpath('//*[@id="QA0Szd"]/div/div/div[1]/div[2]/div/div[1]/div/div/div[2]/div[1]')
        driver.execute_script('arguments[0].scrollBy(0, 1000);', scroll)

        # Wait to load page
        time.sleep(SCROLL_PAUSE_TIME)

        # Calculate new scroll height and compare with last scroll height
        print(f'last height: {last_height}')
        scroll = driver.find_element_by_xpath('//*[@id="QA0Szd"]/div/div/div[1]/div[2]/div/div[1]/div/div/div[2]/div[1]')
        new_height = driver.execute_script("return arguments[0].scrollHeight", scroll)

        print(f'new height: {new_height}')
        
        if number == i:
            break

        if new_height == last_height:
            break

        print('cont')
        last_height = new_height

    
    
    driver.find_element_by_xpath(f'//*[@id="QA0Szd"]/div/div/div[1]/div[2]/div/div[1]/div/div/div[2]/div[1]/div[{i}]/div/a').click()
    time.sleep(2)
    
    title = driver.find_element_by_xpath('//*[@id="QA0Szd"]/div/div/div[1]/div[3]/div/div[1]/div/div/div[2]/div[2]/div/div[1]/div[1]/h1').text
    
    try:
        address = driver.find_element_by_xpath('//*[@id="QA0Szd"]/div/div/div[1]/div[3]/div/div[1]/div/div/div[2]/div[9]/div[3]/button/div/div[3]/div[1]').text
       
    except:
        address = 'na'
        rating = 'na'
        price = 'na'

    title_.append(title)
    address_.append(address)
   
    driver.back()     
    print('complete')

 

 

각 정보를 pandas를 이용해 데이터 프레임으로 생성합니다.

import pandas as pd
data = pd.DataFrame(data=[], columns = ['title', 'address'])

data['title'] = title_
data['address'] = address_

 

 

 

 

참조

https://connie-n.tistory.com/36

[Python] Selenium 이용한 Googlemap 크롤링