Power BI and Python - Incomplete Script Result

Question

Hello,

I am trying to return the link list found in HTML with the python script below. When run in the Python IDE, the result returns all found occurrences. When running on Power Bi Desktop, the result returns only 1 line.

Script:

import requests
from bs4 import BeautifulSoup
import pandas as pd

req = requests.get("https://python.org")
html = req.text
soup = BeautifulSoup(html, 'html.parser')
for lnk in soup.select('link'):
    l = lnk.get('href')
    if len(l) > 0:
        base = pd.DataFrame({'link':[l]}, index=['link'])

Result in Power BI

Output in Python IDE

My question is: does the script have to be modified to bring the results through Power Bi? In Power Bi, the "for" loop does not work as in the IDE ?

Answer 1

Hi, Carine

The error in this case is that you are confusing print with Return.

Power BI can only see the data if you send it the Dataframe, and in your case you are creating a df inside the loop, it means that every time the loop runs it Zera the df and creates a new one with a line, so its result.

To fix it you can create an out-of-loop dataframe and append inside the loop

import requests
from bs4 import BeautifulSoup
import pandas as pd
base_total = pd.DataFrame()
req = requests.get("https://python.org")
html = req.text
soup = BeautifulSoup(html, 'html.parser')

for lnk in soup.select('link'):
    l = lnk.get('href')
    if len(l) > 0:
        base = pd.DataFrame({'link':[l]}, index=['link'])
        base_total = base_total.append(base)

If you want you can solve even in other simpler ways

import requests
from bs4 import BeautifulSoup
import pandas as pd
req   = requests.get("https://python.org")
html  = req.text
soup  = BeautifulSoup(html, 'html.parser')
links = [l.get('href') for l in soup.select('link')]
base  = pd.DataFrame({"links":links})