크롤링 - 교보문고 도서 리스트 추출하기

교보문고 검색어로 나온 도서 리스트에 대해 책 제목, 기본 가격, 판매 가격, 도서 썸네일 이미지 링크 주소, 그리고 상세페이지 링크 주소를 파이썬 크롤링으로 추출하는 연습을 해본다.

 

 

파이썬 크롤링 - 교보문고 검색 도서 리스트

• 교보문고 검색어와 추출 희망 페이지수를 입력받는다.
• 검색 결과로 나온 전체 페이지수를 찾아서 for 문에 1페이지부터 전체 페이지수를 대상으로 순환하게 하되, 추출 희망 페이지수가 전체 페이지수보다 작으면 그 숫자만큼만 추출하고 중지(break), 크면 전체 페이지수를 대상으로 추출한다.
• 추출하는 항목은 책 제목, 책 가격(기본, 세일), 책 썸네일 이미지 링크, 그리고 상세페이지 링크 주소를 대상으로 한다.

 

[크롤링 소스 코드]

import requests
from bs4 import BeautifulSoup
from requests_html import HTMLSession
import time, random


def book(search_word, extract_pages):
    _url = f'https://search.kyobobook.co.kr/web/search?vPstrKeyWord={search_word}&orderClick=LAG'  # 페이지 없이
    headers = {
        'user_agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.75 Safari/537.36'
    }    

    ## 페이징 처리 부분의 <span id="totalpage">0,000</span> 부분의 숫자가 javascript로 되어있고,
    # 이를 rendering 위해 requests_html 모듈을 사용하여 숫자를 추출함
    s = HTMLSession()
    r = s.get(_url, headers=headers)
    r.html.render(sleep=1)

    if r.ok:
        tot_page = r.html.find('span#totalpage', first=True)
        # tot_page = r.html.xpath('//*[@id="totalpage"]', first=True)  # xpath도 가능
        total_page = int(tot_page.text.split("페이지")[0].strip().replace(',', ''))
        print(f'total_page:{total_page}')

    if extract_pages > total_page:
        extract_pages = total_page

    for page in range(1, total_page + 1):
        if page > extract_pages:
            break
        url = f'https://search.kyobobook.co.kr/web/search?vPstrKeyWord={search_word}&vPstrTab=PRODUCT&searchPcondition=1&currentPage={page}&orderClick=LAG#container'
        response = requests.get(url, headers=headers)
        soup = BeautifulSoup(response.text, 'html.parser')
        book_trs = soup.select('#search_list tr')
        print(len(book_trs))

        contents = []
        for tr in book_trs:
            element = []
            title = tr.select_one('td.detail > div.title > a')  # 도서 제목
            element.append(title.text.strip())
            try:
                org_price = tr.select_one('td.price > div.org_price')  # 기본가격
                element.append(org_price.text.strip())
            except:
                element.append(0)
            try:
                sel_price = tr.select_one('td.price > div.org_price')  # 판매가격
                element.append(sel_price.text.strip())
            except:
                element.append(0)
            img = tr.select_one('td.image > div.cover > a > img')  # 도서 이미지
            element.append(img.get('src'))
            a_link = tr.select_one('td.image > div.cover > a')  # 도서 상세 링크
            element.append(a_link.get('href'))
        
            contents.append(element)

        print(contents)
        print(f'\n{"="*10} {page}페이지 추출 완료 {"="*10}\n')
        time.sleep(random.uniform(0.5, 1))


book('그로스해킹', 2)

• 검색된 도서 목록의 페이지를 대상으로 추출하기 위해 전체 페이지 수를 추출하려고 보니 <span id="totalpage">0,000</span>의 숫자 부분이 javascript로 되어있다.
• 이를 추출하기 위해서 requests_html 모듈을 사용하여 rendering 처리 후 숫자를 추출했다.
• 각 도서 리스트의 추출 항목을 element 리스트 변수에 append 한 후 다시 contents 리스트에 append 하였다.

 

 

 

 

[추출 결과]

total_page:5
20
[['[컴퓨터/IT]  그로스 해킹(Growth Hacking) :  스타트업을 위한 실용주의 마케팅', '13,800원', '13,800원', 'https://simage.kyobobook.co.kr/images/book/large/311/l9788966189311.jpg', 'http://www.kyobobook.co.kr/product/detailViewKor.laf?ejkGb=KOR&mallGb=KOR&barcode=9788966189311&orderClick=LAG&Kc='], ['[컴퓨터/IT] 성장의 시대를 위한 
안내서 그로스 해킹(Growth Hacking)', '15,000원', '15,000원', 'https://simage.kyobobook.co.kr/images/book/large/969/l9788960881969.jpg', 'http://www.kyobobook.co.kr/product/detailViewKor.laf?ejkGb=KOR&mallGb=KOR&barcode=9788960881969&orderClick=LAG&Kc='], ['[경제/경영]  그로스 해킹 :  데이터와 실험을 통해 성장하는 서비스를 만드는  
방법', '15,000원', '15,000원', 'https://simage.kyobobook.co.kr/images/book/large/369/l9791158392369.jpg', 'http://www.kyobobook.co.kr/product/detailViewKor.laf?ejkGb=KOR&mallGb=KOR&barcode=9791158392369&orderClick=LAG&Kc='], ['[경제/경영]  진화된 마
케팅 그로스 해킹 :  프로세스와 실행 전략 바이블 [양장]', '19,000원', '19,000원', 'https://simage.kyobobook.co.kr/images/book/large/026/l9791188225026.jpg', 'http://www.kyobobook.co.kr/product/detailViewKor.laf?ejkGb=KOR&mallGb=KOR&barcode=9791188225026&orderClick=LAG&Kc='], ['[경제/경영]  마켓4.0 그로스해킹 :  애자일/데이터 기반으
로 진화한 디지털 마케팅 가이드', '18,000원', '18,000원', 'https://simage.kyobobook.co.kr/images/book/large/507/l9791196752507.jpg', 'http://www.kyobobook.co.kr/product/detailViewKor.laf?ejkGb=KOR&mallGb=KOR&barcode=9791196752507&orderClick=LAG&Kc='], ['[IT/프로그래밍]  그로스 해킹(Growth Hacking) : 스타트업을 위한 실용주의 마케팅', 0, 0, 'https://simage.kyobobook.co.kr/images/book/large/311/l4808966189311.jpg', 
'http://digital.kyobobook.co.kr/digital/ebook/ebookDetail.ink?selectedLargeCategory=001&barcode=4808966189311&orderClick=LAG&Kc='], ["[경제/경영] Hacking Growth : How Today's Fastest-Growing Companies Drive Breakout Success [Hardcover]", '43,150원', '43,150원', 'https://simage.kyobobook.co.kr/images/book/large/215/l9780451497215.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9780451497215&orderClick=LAG&Kc='], ["[경제/경영] Hacking Growth : How Today's Fastest-Growing Companies Drive Breakout Success [Paperback]", '46,370원', '46,370원', 'https://simage.kyobobook.co.kr/images/book/large/007/l9781524760007.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9781524760007&orderClick=LAG&Kc='], ['[컴퓨터/IT]  마케터를 위한 구글 애널리틱스 :  ( 
그로스해킹, 구글태그매니저, 퍼포먼스마케)', '22,000원', '22,000원', 'https://simage.kyobobook.co.kr/images/book/large/703/l9788960883703.jpg', 'http://www.kyobobook.co.kr/product/detailViewKor.laf?ejkGb=KOR&mallGb=KOR&barcode=9788960883703&orderClick=LAG&Kc='], ['[경제/경영] Growth Marketing and Hacking [Paperback]', '25,280원', '25,280원', 'https://simage.kyobobook.co.kr/images/book/large/982/l9798549411982.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9798549411982&orderClick=LAG&Kc='], ['[경제/경영] Youtube Growth Hacking : How to Drive Massive Traffic to Your Channel and Turn People Into Rabid Fans ( Social Media Marketing ) [Paperback]', '19,910원', '19,910원', 'https://simage.kyobobook.co.kr/images/book/large/020/l9781985089020.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9781985089020&orderClick=LAG&Kc='], ['[과학/기술] Growth Hacking : Mehr Wachstum, mehr Kunden, mehr Erfolg ( Rheinwerk Computing ) [페이퍼백]', '71,660원', '71,660원', 'https://simage.kyobobook.co.kr/images/book/large/354/l9783836259354.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9783836259354&orderClick=LAG&Kc='], ['[경제
경영]  마켓4.0 그로스해킹 : 애자일/데이터 기반으로 진화한 디지털 마케팅 가이드', 0, 0, 'https://simage.kyobobook.co.kr/images/book/large/507/l9791196752507.jpg', 'http://digital.kyobobook.co.kr/digital/ebook/ebookDetail.ink?selectedLargeCategory=001&barcode=4801196752508&orderClick=LAG&Kc='], ['[경제경영]  그로스 해킹 : 데이터와  
실험을 통해 성장하는 서비스를 만드는 방법', 0, 0, 'https://simage.kyobobook.co.kr/images/book/large/369/l9791158392369.jpg', 'http://digital.kyobobook.co.kr/digital/ebook/ebookDetail.ink?selectedLargeCategory=001&barcode=4801158392360&orderClick=LAG&Kc='], ['[북모닝CEO(PDF)]  新경제개념사전 시즌. 1: 혁신기업의 마케팅기법, 그로스해
킹', 0, 0, 'https://simage.kyobobook.co.kr/images/book/large/490/l480D171023490.jpg', 'http://digital.kyobobook.co.kr/digital/ebook/ebookDetail.ink?selectedLargeCategory=001&barcode=480D171023490&orderClick=LAG&Kc='], ['[경제경영]  진화된 마케팅 그 
로스 해킹', 0, 0, 'https://simage.kyobobook.co.kr/images/book/large/026/l9791188225026.jpg', 'http://digital.kyobobook.co.kr/digital/ebook/ebookDetail.ink?selectedLargeCategory=001&barcode=4801188225027&orderClick=LAG&Kc='], ['[경제/경영] Growth Hacker Marketing : A Primer on the Future of Pr, Marketing, and Advertising [Reprint edition | Paperback]', '23,810원', '23,810원', 'https://simage.kyobobook.co.kr/images/book/large/380/l9781591847380.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9781591847380&orderClick=LAG&Kc='], ['[과학/ 
기술] Big Data For Beginners : Understanding SMART Big Data, Data Mining & Data Analytics For improved Business Performance, Life Decisions & More! ( Data Analytics, 
Big Data, Computer Programming, Growth Hacking, Itil ) [Paperback]', '14,870원', '14,870원', 'https://simage.kyobobook.co.kr/newimages/apps/b2c/product/Noimage_l.gif', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9781530412044&orderClick=LAG&Kc='], ["[경제/경영] Hacking Growth : How Today's Fastest Growing Companies Drive Breakout Success [Hardcover]", '35,910원', '35,910원', 'https://simage.kyobobook.co.kr/newimages/apps/b2c/product/Noimage_l.gif', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9780753545379&orderClick=LAG&Kc='], ['[경제/경영] El Metodo Hacking Growth : Que Hacen Companias Explosivas Como Facebook, Airbnb y Walmart Para Ser Lideres En El Mercado/ 
Hacking Growth [Paperback]', '29,690원', '29,690원', 'https://simage.kyobobook.co.kr/images/book/large/060/l9786073161060.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9786073161060&orderClick=LAG&Kc=']]   

========== 1페이지 추출 완료 ==========

20
[['[경제/경영] Growth Hacking Mit Strategie : Wie Erfolgreiche Startups Und Unternehmen Mit Growth Hacking Ihr Wachstum Beschleunigen [1. Aufl. 2017 edition | Paperback]', '59,510원', '59,510원', 'https://simage.kyobobook.co.kr/images/book/large/306/l9783658162306.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9783658162306&orderClick=LAG&Kc='], ['[경제/경영] Guide To Growth Hacking : Guide To Use Growth Hacking Strategy To Grow Faster And Smarter: How To Apply Growth Hacking [Paperback]', '13,380원', '13,380원', 'https://simage.kyobobook.co.kr/images/book/large/944/l9798546173944.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9798546173944&orderClick=LAG&Kc='], ['[경제/경영] Growth Hacking 101 : How You Should Apply Growth Hacking To Achieve Great Success: Growth Hacking Mindset [Paperback]', '13,380원', '13,380원', 
'https://simage.kyobobook.co.kr/images/book/large/990/l9798546163990.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9798546163990&orderClick=LAG&Kc='], ['[경제/경영] Growth Hacking Plans : How I create Growth Hacking Plans for startups for $10,000 + TOP 300 growth hacks you can put into practice right away [Paperback]', '296,110원', '296,110원', 'https://simage.kyobobook.co.kr/images/book/large/928/l9781534840928.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9781534840928&orderClick=LAG&Kc='], ['[과학/기술] Growth Hacking LinkedIn(TM) : Moeglichkeiten zur Erweiterung des persoenlichen Netzwerks und Steigerung der professionellen Sichtbarkeit auf LinkedIn. [1 edition | Paperback]', '39,370원', '39,370원', 'https://simage.kyobobook.co.kr/images/book/large/079/l9783347218079.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9783347218079&orderClick=LAG&Kc='], ["[경제/경영] Growth Hacking : Silicon Valley's Best Kept Secret [Paperback]", '22,250원', '22,250원', 'https://simage.kyobobook.co.kr/newimages/apps/b2c/product/Noimage_l.gif', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9781619616004&orderClick=LAG&Kc='], ['[경제/경영] Growth Hacking - konsequent umsetzen : Prozesse, Instrumente und Mindset fuer ein schnelles und nachhaltiges Wachstum - mit Roadmap und Checklisten [1. Aufl. 2021 edition | Paperback]', '83,910원', '83,910원', 'https://simage.kyobobook.co.kr/images/book/large/563/l9783658332563.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9783658332563&orderClick=LAG&Kc='], ['[과학/기술] Growth Hacking fuer Dummies ( fuer Dummies ) [Paperback]', '41,960원', '41,960원', 'https://simage.kyobobook.co.kr/images/book/large/927/l9783527717927.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9783527717927&orderClick=LAG&Kc='], ['[과학/기술] Growth Hacking : Mehr Wachstum, mehr Kunden, mehr Erfolg. Der 
Praxisratgeber fuer Durchstarter im Online-Marketing! ( Rheinwerk Computing ) [2. Auflage edition | Paperback]', '73,220원', '73,220원', 'https://simage.kyobobook.co.kr/images/book/large/182/l9783836270182.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9783836270182&orderClick=LAG&Kc='], ['[경제/경영] Growth Hacking for Hair Salons : Transform Your Hair Salon, Day Spa, or Service Business Into a BADASS Money Machine! ( Salon Success ) [Paperback]', '22,250원', '22,250원', 'https://simage.kyobobook.co.kr/images/book/large/944/l9780982327944.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9780982327944&orderClick=LAG&Kc='], ['[경제/경영] Growth Hacking for Beginners : Transform Your Business Into a Badass Money Machine [Paperback]', '22,250원', '22,250원', 'https://simage.kyobobook.co.kr/newimages/apps/b2c/product/Noimage_l.gif', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9780996689533&orderClick=LAG&Kc='], ['[경제/경영] Introduction to Growth 
Hacking [Paperback]', '10,400원', '10,400원', 'https://simage.kyobobook.co.kr/newimages/apps/b2c/product/Noimage_l.gif', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9781652085577&orderClick=LAG&Kc='], ["[경제/ 
경영] The Growth Hacking Book : Most Guarded Growth Marketing Secrets The Silicon Valley Giants Don't Want You To Know [Paperback]", '29,720원', '29,720원', 'https://simage.kyobobook.co.kr/images/book/large/556/l9781733095556.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9781733095556&orderClick=LAG&Kc='], ['[경제/경영] Growth Hacking im Online Marketing von agilen Start-ups [Paperback]', '52,820원', '52,820원', 'https://simage.kyobobook.co.kr/images/book/large/956/l9783346091956.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9783346091956&orderClick=LAG&Kc='], ['[경제/경 
영] Growth Hacking : Innovative Marketing Tactics to grow faster and smarter [Paperback]', '25,280원', '25,280원', 'https://simage.kyobobook.co.kr/images/book/large/560/l9781922300560.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9781922300560&orderClick=LAG&Kc='], ['[경제/경영] Powering 
Startups from 1 to 10^6X by Growth Hacking : Exponential Growth Driver for Startups & Digital Marketers [Paperback]', '8,930원', '8,930원', 'https://simage.kyobobook.co.kr/images/book/large/149/l9798648764149.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9798648764149&orderClick=LAG&Kc='], ['[경제/경영] 10 Disziplinen mit zahlreichen Tipps und Tricks fuer Growth Hacking : Mit Strategie zu mehr Wachstum und Erfolg [Paperback]', '42,650원', '42,650원', 
'https://simage.kyobobook.co.kr/images/book/large/747/l9783951992747.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9783951992747&orderClick=LAG&Kc='], ['[경제/경영] Growth Hacking Your First Startup : A Simple Guide to Marketing for Entrepreneurs [Paperback]', '16,350원', '16,350원', 'https://simage.kyobobook.co.kr/images/book/large/719/l9781646507719.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9781646507719&orderClick=LAG&Kc='], ['[경제/경영] Growth Hacking : Innovative Marketing Tactics to grow faster and smarter [Hardcover]', '44,630원', '44,630원', 'https://simage.kyobobook.co.kr/images/book/large/062/l9781922301062.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9781922301062&orderClick=LAG&Kc='], ['[경제/경영] Entrepreneur Voices on Growth Hacking ( Entrepreneur Voices ) [Paperback]', '29,750원', '29,750원', 'https://simage.kyobobook.co.kr/images/book/large/276/l9781599186276.jpg', 'http://www.kyobobook.co.kr/product/detailViewEng.laf?ejkGb=BNT&mallGb=ENG&barcode=9781599186276&orderClick=LAG&Kc=']]

========== 2페이지 추출 완료 ==========