Issue

This Content is from Stack Overflow. Question asked by Abhishek Kumar

I am learning web scraping using selenium. In a task to scrape all the comments on a YouTube video I need to scroll down to the last comment. I have tried the following code but it’s not working –

# imports used

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
import time

driver = webdriver.Chrome()
url = 'https://www.youtube.com/watch?v=etzmAZ7oiz0'

driver.get(url)
time.sleep(3)

last_height = driver.execute_script("return document.body.scrollHeight")
html = driver.find_element(By.TAG_NAME, 'html')

while True:
    # Scroll down to bottom
    html.send_keys(Keys.PAGE_DOWN)

   # Wait to load the page
    time.sleep(5)


    --- Code to scrape comments ---


   # Calculate new scroll height and compare with last scroll height

    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    else:
        last_height = new_height

Will be grateful for any help

Solution

Get the locator of the last comment and build the webelement, pass it on to the below function, where ‘args’ is the Webelement built using the last comment’s locator(css/xpath etc). Below method will scroll to the Webelement which you will pass to in the place of ‘args’

driver.execute_script("arguments[0].scrollIntoView()", args)

This Question was asked in StackOverflow by Abhishek Kumar and Answered by Nischitha K N It is licensed under the terms of CC BY-SA 2.5. - CC BY-SA 3.0. - CC BY-SA 4.0.