쿠팡 파트너스 상품 링크 상세정보 URL로 들어가기 - 오류 발생

 

쿠팡 파트너스 상품 링크 상세정보 URL로 들어가기

 

쿠팡 파트너스로 링크된 URL(쿠팡 파트너스 API로 상품 추출 결과)을 selenium으로 클릭하고 들어가서 상품 내용 중 일부를 스크래핑하는 기능을 만들어 보려고 했다.

 

그러나, 셀레니움 사용 시 쿠팡 사이트는 거부되는 듯... (coupang.com 사이트는 어떤지 확인 안함)
셀레니움을 한 번 쓰면 그 다음부터는 아래와 같은 메시지가 나오면서 막힘 

※ 오류 :

MaxRetryError(_pool, url, error or ResponseError(cause)

NewConnectionError : Failed to establish a new connection: [WinError 10061] 대상 컴퓨터에서 연결을 거부했으므로 연결하지 못했습니다.

  File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\util\retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='xxx.xxx.xxx.xxx', port=xxxxx): Max retries exceeded with url: /session/01174078163188a6bb12156b88d9e66f/window (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000234797F63A0>: Failed to establish a new connection: [WinError 10061] 대상 컴퓨터에서 연결을 거부했으므로 연결하지 못했습 
니다'))

 

여러 가지로 해봤으나 접속 불가!!

할 수 없이 구글링을 통해 fake_useragent.UserAgent도 해보고 했으나 이 또한 쉽지 않았다. (초보자의 한계...)

 

requests로 다시 시도해 보았으나, 이것도 또 아래와 같은 메시지만 나왔다.

※ 오류 :

InvalidSchema("No connection adapters were found for {!r}".format(url) )

No connection adapters were found for '<a xxxx </a>'

  File "C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py", line 649, in send
    adapter = self.get_adapter(url=request.url)
  File "C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py", line 742, in get_adapter
    raise InvalidSchema("No connection adapters were found for {!r}".format(url))
requests.exceptions.InvalidSchema: No connection adapters were found for '<a xxxx </a>'

 

 

구글에서 다시 여러 사이트를 찾아 다니다가 힌트를 얻었는데,

 

다른 사람이 올려 놓은 에러 메시지를 보고...

short_url = "t.co/NHBbLlfCaa"
r = requests.get(short_url)
if r.status_code == 200:
    print("Actual url:%s" % r.url)
-----------------------------------------
(결과)
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='www.fatlossadvice.pw', port=80): Max retries exceeded with url: /TIPS/KILLED-THAT-TREADMILL-WORKOUT-WORD-TO-TIMMY-GACQUIN.ASP (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 11004] getaddrinfo failed',))
-----------------------------------------
(다시 시도)
user_agent_list = [
        'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:25.0) Gecko/20100101 Firefox/25.0',
        'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0',
        'Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36',
    ]

r = requests.get(short_url, headers={'User-Agent': user_agent_list[np.random.randint(0, len(user_agent_list))]})
if r.status_code == 200:
    print("Actual url:%s" % r.url)
(성공 못함)

이에 대한 답변으로,

You can prevent the error by adding allow_redirects=False to requests.get() method to prevent redirecting to page that doesn't exist (and thus raising the error). You have to examine the header sent by server yourself (replace XXXX by https, remove spaces):

import requests

short_url = ["XXXX t.co /namDL4YHYu",
 'XXXX t.co /MjvmV',
 'XXXX t.co /JSjtxfaxRJ',
 'XXXX t.co /xxGSANSE8K',
 'XXXX t.co /ZRhf5gWNQg']

for url in short_url:
    r = requests.get(url, allow_redirects=False)
    try:
        print(url, r.headers['location'])
    except KeyError:
        print(url, "Page doesn't exist!")

 

※ 참고사이트
https://stackoverflow.com/questions/56809085/expand-short-urls-in-python-using-requests-library
https://stackoverflow.com/questions/63394684/invalidschemano-connection-adapters-were-found-for-r-formaturl-while-us

 

단축 URL 문제??

그래, URL과 param 분리로 풀면 될 수도.... 

 

결론은 해결했다.

....

for url in coups:
    url_default, url_params = url['href'].split('?')
    params_lst = url_params.split('&')
    param = {}
    param['lptag'] = params_lst[0].split('=')[1]
    param['pageKey'] = params_lst[1].split('=')[1]
    param['itemId'] = params_lst[2].split('=')[1]
    param['vendorItemId'] = params_lst[3].split('=')[1]
    param['traceid'] = params_lst[4].split('=')[1]
    # print(param)
    res = requests.get(url_default, allow_redirects=False, params=param)
    # print(res.status_code)
    try:
        # print(res.headers['location'])
        coup_url = res.headers['location']
    except KeyError:
        print("Page doesn't exist!")
    # print(coup_url)
    if coup_url:
        result = product_detail(coup_url, headers)
        print(result)
 
 ....

쿠팡 파트너스의 URL 부분은 단축 URL이었으며, 이것에 뒤에 있는 params를 더하여 requests를 하면 정상적인 결과가 나왔다.

 

 

▷ 관련 자료 더보기>>

API를 잘 모르면서 해본 '쿠팡 파트너스 API' 실습

 

 

[파이썬 크롤링 연습] 쿠팡(Coupang) 상품 리스트 가져오기