Issue

This Content is from Stack Overflow. Question asked by Goune Kenfack

I have created a search engine for web pages in Django and would like to perform queries on these pages so that I can find out the value of their various X-Frame-Options headers. So I wrote the following code:

def search(request):
            . . .
            final_result = []
            for page_num in range(1, max_pages_to_scrap+1):
                url = "https://www.ask.com/web?q=" + search + "&qo=pagination&page=" + str(page_num)
                res = requests.get(url)
                soup = bs(res.text, 'lxml')
                result_listings = soup.find_all('div', {'class': 'PartialSearchResults-item'})
    
                for result in result_listings:
                    result_title = result.find(class_='PartialSearchResults-item-title').text
                    result_url = result.find('a').get('href')
                    result_desc = result.find(class_='PartialSearchResults-item-abstract').text
                    final_result.append((result_title, result_url, result_desc))
    
                    for header in final_result[1]: #the error is generated here
                     response = requests.get(header)
                    if response['X-Frame-Options'] in ["DENY", "SAMEORIGIN"]:
                        head = True
                        notAccept = bool(head)
                    else:
                        notAccept = bool(False)

But when I test, I get in the terminal the following errors:

Internal Server Error: /search
Traceback (most recent call last):
  File "C:Python310libsite-packagesdjangocorehandlersexception.py", line 55, in inner
    response = get_response(request)
  File "C:Python310libsite-packagesdjangocorehandlersbase.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "C:UsersuserDocumentsAAprojectsWhelpsgroups1searchEnginesearchviews.py", line 29, in search
    for header in final_result[1]:
IndexError: list index out of range
[17/Sep/2022 23:28:40] "GET /search?csrfmiddlewaretoken=t57w6QZSzkgsKoNJWXrjotBRdKSNxBBxzfzPnUC6N7kXfs5pMyGWcjQUYcrT13Ds&search=moto HTTP/1.1" 500 88593

I don’t understand why I also hope to rely on the support of the community. Thanks!

Solution

The error is because in the first iteration the list final_result has only one item and the way to access to it would be final_result[0], python index start at 0.

I think you’re trying to access the first element of the tuple, aren’t you? Thus, use final_result[0][0] or use directly one of the variables already defined (result_title, result_url, result_desc)

This Question was asked in StackOverflow by Goune Kenfack and Answered by irvoma It is licensed under the terms of CC BY-SA 2.5. - CC BY-SA 3.0. - CC BY-SA 4.0.