[SOLVED] How to scrape a table and convert it into CSV with Python?

Issue

This Content is from Stack Overflow. Question asked by Agung

I’d like to gather some lecturer info and export it into csv. I read some article and tutorial using python. The code looks like this

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
import pandas as pd
from bs4 import BeautifulSoup

driver = webdriver.Chrome(ChromeDriverManager().install())

url = "https://uin-suka.ac.id/id/page/detil_dosen/197606110000002301"

driver.get(url)

soup = BeautifulSoup(driver.page_source, 'lxml')

df = pd.read_html(str(soup))[0]
print(df)

and the output look like this

                             0  1                                   2
0                         Nama  :     Dr. Nina Mariani Noor, SS., MA.
1                Program Studi  :   Interdisciplinary Islamic Studies
2                     Fakultas  :                        Pascasarjana
3       Jenis Pegawai | Status  :  Pegawai Tetap BLU | Aktif Mengajar
4  Jabatan Akademik | Golongan  :                      Lektor | III/C
5                        Email  :                                   -
6          Pendidikan Terakhir  :                                  S3

Process finished with exit code 0

the csv table i created look like this

table

the problem is, how can i pull the second column and put it into csv as row?
thankyou



Solution

without using df and assuming input is dict

keys = res.keys()
with open("output.csv", "w", newline="") as f:
    dict_writer = csv.DictWriter(f, keys)
    dict_writer.writeheader()
    dict_writer.writerows(res) 

using df I think you need to transpose it first.

df = df.drop('1',axis=1)
df = df.set_index('col1').T
df.to_csv('output.csv')


This Question was asked in StackOverflow by Agung and Answered by Solly It is licensed under the terms of CC BY-SA 2.5. - CC BY-SA 3.0. - CC BY-SA 4.0.

people found this article helpful. What about you?