Month: November 2019

Dismissing Firefox popups using Selenium and Python

In my new journey to figure out how to automate stuff in my life, one of the time-sucking adventures was to try to stop browser popups such as geolocation and notifications. I just wanted to click ‘ok’ or ‘dismiss’ and move on since this was my ‘bot’ instance of the browser.

The issue with these browser-based pop ups is that they are handled by the browser, not by selenium so it seems selenium can’t dismiss them or dismiss them easily.

The solution workflow that I found to work well for me is as follows:

  1. Switch to your Selenium Firefox profile
  2. Do your browser setting that you need
  3. Save the change
  4. Start your Selenium script with new changes

Here are the details on how to do this:

Create a new Firefox Profile

  1. in browser, type about:profiles
  2. Create a new profile
  3. Take a note of the location of the root directory of that profile (ie paste to your Selenium script for now)
  4. ‘Launch profile in new browser’ (this will launch your new profile in a new instance nicely)
  5. Make your browser changes and save

Do your browser setting changes

The following two changes were the main two that bothered me so perhaps I’ll highlight those here:

  1. Don’t allow websites to send you notifications
  • Preferences/privacy and security / Notifications
  1. Don’t let websites ask for geolocation stuff
  • Preferences/privacy and security / Permissions – Locations (Settings button)

TIP! It’s very useful to actually do a dry run as a human before you let your bot run free so that you can deal with these popups logged in, one time, as this profile. So, workflow is to switch to your Selenium profile, launch it, do a dry run on with real human clicks, deal with any popups or browser setting stuff, save changes, go run the bot script.

Setting up your Python Script to use your Selenium Profile

Now that you’ve got your new profile, let’s actually use it in your script, instead of what will always be a fresh browser instance

Here is my code block which you can add in your project:

## SETUP SELENIUM TO USE CUSTOM FIREFOX PROFILE

#Pretty sure you need to import this to use 'FirefoxProfile but I'm too lazy to confirm - feel free
from selenium.webdriver.firefox.options import Options 

#Root directory copied from Step 3 above
profilePath = '/home/user/.mozilla/firefox/cjda7321.Selena'

#Directs profile selector thing to the right path created above
profile = webdriver.FirefoxProfile(profilePath) 

#Tells Selenium to use the custom profile
driver = webdriver.Firefox(firefox_profile=profile)

Hope that helps!

Tags : , ,

HOW TO CLICK AN ITEM IN A SUBMENU (UL) LIST USING PYTHON, SELENIUM

I spent literally 3 days trying to simply click a logout link with Selenium. I searched every stackoverflow post I could find until I found this one.

I had tried pretty much everything I could to try to click the logout link which was the fifth item down in the move-over list.

I tried find_element_by_xpath, find_element_by_id, find this, find that, blah blah blah

Finally, it was indeed the find_element_by_css_selector that worked.

The only thing is I still do not know WHY the xpath option didn’t work for this list while the others did. Hmm. Whatever. Probably will figure it out later….HOW TO CLICK AN ITEM IN A SUBMENU (UL) LIST USING PYTHON, SELENIUMThe only thing is I still do not know WHY the xpath option didn’t work for this list while the others did. Hmm. Whatever. Probably will figure it out later….

so, here is a code block of what worked for me to move to a mouse-over menu and click the menu’s submenu item using Selenium and python (my css selector is just a fake example of course so paste your correct one in):

Edit: I also realized I had to click on the button *above* the actual unordered list (UL) element to trigger the drop down. This tip for the ‘main_menu’ element below might also help someone.

main_menu = driver.find_element_by_xpath("//*[@id='with-label']")

actions.move_to_element(main_menu).perform()

time.sleep(2)

WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".css-selector-thing > ul:nth-child(2) > li:nth-child(6) > a:nth-child(1)")))

driver.find_element_by_css_selector(".css-selector-thing > ul:nth-child(2) > li:nth-child(6) > a:nth-child(1)").click()
Tags : ,

OPENING A NEW TAB WITH SELENIUM + PYTHON + FIREFOX + UBUNTU

So this was super duper hard and I believe the reason why is because technically the browser settings and the user choices impact whether or not a clicked link opens in a fresh new tab or a fresh new window. Keep that in mind in case you have other headaches. It may be impossible (so I’ve read online) to perfectly control whether a tab opens on your users’ browser, however, since web automation is probably your browser you should be able to work with this solution and your browser settings to get it going.

Another important thing before you begin here. This assumes you are ‘right clicking an element on a page and opening in a new tab’ kind of thing. If you are trying to make a new, unrelated link open in a new tab, this doesn’t work. I’ll figure that out maybe in another blog post. This one takes a page element already in focus and then opens that into a new tab.

Final note: I’m using Ubuntu so not sure if you need to adjust for yourself…

Anyway, here comes the code.

# Import your stuffs
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

# Set path to your executable Firefox` 
ffpath = '/home/wt/Documents/Waynes_Python_Education_Directory/Automation_Browser_Drivers'

# Turn your Firefox browser into a usable object (or something like that)
driver = webdriver.Firefox(ffpath)

# Come up with a creative new URL to go to (this one is great, btw)
driver.get("https://www.engrish.com/")

# Wait for a while to enjoy it..
time.sleep(5)

# Come up with a creative new URL to go to (this one is great, btw)
new_url = driver.find_element_by_xpath("/html/body/div[1]/div[2]/div/div/div[2]/div[1]/div[1]/ul/li[6]/a")

# And now, the magic!  Open fun URL in a new Firefox tab!
new_url.send_keys(Keys.CONTROL + Keys.RETURN)

# Wait for a while to enjoy it..
time.sleep(5)

# Shut 'er down and go home for the day...
driver.quit()
Tags : ,

Overcoming Painful Setup for Selenium with Python on Ubuntu

EDIT 191118
I realized that I need to execute the script below every time my computer reboots so my current workflow is to leave the command commented out in script and then run it before I begin. You can probably run it with the script too? This one: export PATH=$PATH:/home/user/path/to/browser_drivers_for_automation

I had the following continual painfull errors:

  1. selenium.common.exceptions.WebDriverException: Message: ‘my_folder_name’ executable may have wrong permissions.
  2. selenium.common.exceptions.WebDriverException: Message: ‘Automation_Browser_Drivers’ executable needs to be in PATH.

To be transparent, I’m not sure which if any of these steps happened first, or whether it matters, but I’ll give you both in case it helps and in the order I did:

For my setup, since I plan to mess around with automation long term I wanted to have all my different browser drivers in one folder so I can kind of ‘containerize’ them and always know where they are, and update them accordingly. So, I made a directory called ‘browser_drivers_for_automation”

In that, I downloaded all the executables (go figure out how to do that if you want from Selenium page)

The following is the seemingly standard ‘trial run’ to make sure you are setup and working. I definitely recommend not moving forward until you have this trial run going, as well!

from selenium import webdriver

ffpath = '/home/username/path/to/browser_drivers_for_automation'
browser = webdriver.Firefox(ffpath)

browser.get("http://www.python.org")

First, try that. If it runs, awesome, but if you’re like me probably it won’t, HA

So then first, make sure this ‘browser_drivers_for_automation’ directory is listed in your Ubuntu PATH. If you’re like me I didn’t even know what that was but it seems (short version of the story) that this is the part of Ubuntu that says ‘any directory path in here, if you execute a file, I will allow it”

So do this in a terminal:

export PATH=$PATH:/home/user/path/to/browser_drivers_for_automation

not sure what this next one does but I did it so you might as well join me!

source ~/.profile

So now if you do echo $PATH in your terminal you should see your folder listed.

Now, try the above Selenium trial run again. If it works, great. If not, like me, then do this step:

Navigate to your Browser_drivers_for_automation directory and make your driver executable. In my example here i’m using the firefox geckodriver, but you can do this to any/all drivers in here.

sudo chmod +x geckodriver

See if that works. If it does, the browser will open and you should be good to go now.

Hope this helps

Thanks to all those people answering questions in stackoverflow as usual! 🙂

Tags : , ,