This Content is from Stack Overflow. Question asked by Abhishek Kumar
I am learning web scraping using selenium. In a task to scrape all the comments on a YouTube video I need to scroll down to the last comment. I have tried the following code but it’s not working –
# imports used from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.common.keys import Keys import time
driver = webdriver.Chrome() url = 'https://www.youtube.com/watch?v=etzmAZ7oiz0' driver.get(url) time.sleep(3) last_height = driver.execute_script("return document.body.scrollHeight") html = driver.find_element(By.TAG_NAME, 'html') while True: # Scroll down to bottom html.send_keys(Keys.PAGE_DOWN) # Wait to load the page time.sleep(5) --- Code to scrape comments --- # Calculate new scroll height and compare with last scroll height new_height = driver.execute_script("return document.body.scrollHeight") if new_height == last_height: break else: last_height = new_height
Will be grateful for any help
Get the locator of the last comment and build the webelement, pass it on to the below function, where ‘args’ is the Webelement built using the last comment’s locator(css/xpath etc). Below method will scroll to the Webelement which you will pass to in the place of ‘args’
This Question was asked in StackOverflow by Abhishek Kumar and Answered by Nischitha K N It is licensed under the terms of CC BY-SA 2.5. - CC BY-SA 3.0. - CC BY-SA 4.0.