MMO P&S Selenium help required , extension and headless browser...

[zuanchakma]

Hello,

I've been attempting to create a headless scraper with an extension. The scraping process functions smoothly when the browser is not in headless mode. Both browsers are configured with a profile that includes the installed extension. I'd consider abandoning the extension, but the challenge lies in dynamic variables affecting the elements. Identifying the right element to extract data from the site has proven elusive.

Chromedriver: It appears that extensions are not supported in headless mode with Chrome, or at least I haven't discovered a solution yet.

Geckodriver: While extensions work with headless mode, the issue arises when text is placed on the clipboard, and it seems the clipboard is not shared from headless mode to the host machine.

Are there alternative solutions on Windows to enable scraping in headless mode? Any insights would be appreciated.

 

[Upilword]

Hello,
Are there alternative solutions on Windows to enable scraping in headless mode? Any insights would be appreciated.

Why use browser emulation when you can make direct HTTP requests with PHP, Python, and other languages?

 

[zuanchakma]

Why use browser emulation when you can make direct HTTP requests with PHP, Python, and other languages?

I’m not skilled enough to get the data because the element names keep changing, and I always end up with the wrong results.

I found a browser extension that works every time and gives me exactly what I need from the page. But I’ll try using requests again since I know that would be a cleaner solution.

 

[SummersBase]

Why does the browser need to be in headless mode? I usually use Selenium when I need to scrape data. If you share some details with me, I might be able to help.

 

[zuanchakma]

Why does the browser need to be in headless mode? I usually use Selenium when I need to scrape data. If you share some details with me, I might be able to help.

I use it on my regular computer for other work, and it’s annoying when the browser popups. I was hoping to run it separately from the clipboard, because if I or the script copy something, it can mess things up. So, I’ve just been leaving it running at night so it doesn’t bother me.

 

[SummersBase]

I use it on my regular computer for other work, and it’s annoying when the browser popups. I was hoping to run it separately from the clipboard, because if I or the script copy something, it can mess things up. So, I’ve just been leaving it running at night so it doesn’t bother me.

Ah, I get it now. So you can run Selenium in headless mode—what do you need the extension for then?

 

[zuanchakma]

Ah, I get it now. So you can run Selenium in headless mode—what do you need the extension for then?

The extension extracts data from SERPs because I can't do it with find_elements. There are so many dynamic variables, and I'm not sure how to find the right elements. I know it's possible with XPath axes like following-sibling or descendant, but those are too advanced for me, and I haven’t had time to learn them.

 

[Oakwordue]

Extensions in Chrome don't work with Selenium in headless mode. I’d suggest not using headless if you need the extensions.