Categories
Technology Tutorial

Dismissing Firefox popups using Selenium and Python

In my new journey to figure out how to automate stuff in my life, one of the time-sucking adventures was to try to stop browser popups such as geolocation and notifications. I just wanted to click ‘ok’ or ‘dismiss’ and move on since this was my ‘bot’ instance of the browser.

The issue with these browser-based pop ups is that they are handled by the browser, not by selenium so it seems selenium can’t dismiss them or dismiss them easily.

The solution workflow that I found to work well for me is as follows:

  1. Switch to your Selenium Firefox profile
  2. Do your browser setting that you need
  3. Save the change
  4. Start your Selenium script with new changes

Here are the details on how to do this:

Create a new Firefox Profile

  1. in browser, type about:profiles
  2. Create a new profile
  3. Take a note of the location of the root directory of that profile (ie paste to your Selenium script for now)
  4. ‘Launch profile in new browser’ (this will launch your new profile in a new instance nicely)
  5. Make your browser changes and save

Do your browser setting changes

The following two changes were the main two that bothered me so perhaps I’ll highlight those here:

  1. Don’t allow websites to send you notifications
  • Preferences/privacy and security / Notifications
  1. Don’t let websites ask for geolocation stuff
  • Preferences/privacy and security / Permissions – Locations (Settings button)

TIP! It’s very useful to actually do a dry run as a human before you let your bot run free so that you can deal with these popups logged in, one time, as this profile. So, workflow is to switch to your Selenium profile, launch it, do a dry run on with real human clicks, deal with any popups or browser setting stuff, save changes, go run the bot script.

Setting up your Python Script to use your Selenium Profile

Now that you’ve got your new profile, let’s actually use it in your script, instead of what will always be a fresh browser instance

Here is my code block which you can add in your project:

## SETUP SELENIUM TO USE CUSTOM FIREFOX PROFILE

#Pretty sure you need to import this to use 'FirefoxProfile but I'm too lazy to confirm - feel free
from selenium.webdriver.firefox.options import Options 

#Root directory copied from Step 3 above
profilePath = '/home/user/.mozilla/firefox/cjda7321.Selena'

#Directs profile selector thing to the right path created above
profile = webdriver.FirefoxProfile(profilePath) 

#Tells Selenium to use the custom profile
driver = webdriver.Firefox(firefox_profile=profile)

Hope that helps!

Categories
Technology Tutorial

HOW TO CLICK AN ITEM IN A SUBMENU (UL) LIST USING PYTHON, SELENIUM

I spent literally 3 days trying to simply click a logout link with Selenium. I searched every stackoverflow post I could find until I found this one.

I had tried pretty much everything I could to try to click the logout link which was the fifth item down in the move-over list.

I tried find_element_by_xpath, find_element_by_id, find this, find that, blah blah blah

Finally, it was indeed the find_element_by_css_selector that worked.

The only thing is I still do not know WHY the xpath option didn’t work for this list while the others did. Hmm. Whatever. Probably will figure it out later….HOW TO CLICK AN ITEM IN A SUBMENU (UL) LIST USING PYTHON, SELENIUMThe only thing is I still do not know WHY the xpath option didn’t work for this list while the others did. Hmm. Whatever. Probably will figure it out later….

so, here is a code block of what worked for me to move to a mouse-over menu and click the menu’s submenu item using Selenium and python (my css selector is just a fake example of course so paste your correct one in):

Edit: I also realized I had to click on the button *above* the actual unordered list (UL) element to trigger the drop down. This tip for the ‘main_menu’ element below might also help someone.

main_menu = driver.find_element_by_xpath("//*[@id='with-label']")

actions.move_to_element(main_menu).perform()

time.sleep(2)

WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".css-selector-thing > ul:nth-child(2) > li:nth-child(6) > a:nth-child(1)")))

driver.find_element_by_css_selector(".css-selector-thing > ul:nth-child(2) > li:nth-child(6) > a:nth-child(1)").click()
Categories
Technology Tutorial

OPENING A NEW TAB WITH SELENIUM + PYTHON + FIREFOX + UBUNTU

So this was super duper hard and I believe the reason why is because technically the browser settings and the user choices impact whether or not a clicked link opens in a fresh new tab or a fresh new window. Keep that in mind in case you have other headaches. It may be impossible (so I’ve read online) to perfectly control whether a tab opens on your users’ browser, however, since web automation is probably your browser you should be able to work with this solution and your browser settings to get it going.

Another important thing before you begin here. This assumes you are ‘right clicking an element on a page and opening in a new tab’ kind of thing. If you are trying to make a new, unrelated link open in a new tab, this doesn’t work. I’ll figure that out maybe in another blog post. This one takes a page element already in focus and then opens that into a new tab.

Final note: I’m using Ubuntu so not sure if you need to adjust for yourself…

Anyway, here comes the code.

# Import your stuffs
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

# Set path to your executable Firefox` 
ffpath = '/home/wt/Documents/Waynes_Python_Education_Directory/Automation_Browser_Drivers'

# Turn your Firefox browser into a usable object (or something like that)
driver = webdriver.Firefox(ffpath)

# Come up with a creative new URL to go to (this one is great, btw)
driver.get("https://www.engrish.com/")

# Wait for a while to enjoy it..
time.sleep(5)

# Come up with a creative new URL to go to (this one is great, btw)
new_url = driver.find_element_by_xpath("/html/body/div[1]/div[2]/div/div/div[2]/div[1]/div[1]/ul/li[6]/a")

# And now, the magic!  Open fun URL in a new Firefox tab!
new_url.send_keys(Keys.CONTROL + Keys.RETURN)

# Wait for a while to enjoy it..
time.sleep(5)

# Shut 'er down and go home for the day...
driver.quit()
Categories
Technology Tutorial Ubuntu

Overcoming Painful Setup for Selenium with Python on Ubuntu

EDIT 191118
I realized that I need to execute the script below every time my computer reboots so my current workflow is to leave the command commented out in script and then run it before I begin. You can probably run it with the script too? This one: export PATH=$PATH:/home/user/path/to/browser_drivers_for_automation

I had the following continual painfull errors:

  1. selenium.common.exceptions.WebDriverException: Message: ‘my_folder_name’ executable may have wrong permissions.
  2. selenium.common.exceptions.WebDriverException: Message: ‘Automation_Browser_Drivers’ executable needs to be in PATH.

To be transparent, I’m not sure which if any of these steps happened first, or whether it matters, but I’ll give you both in case it helps and in the order I did:

For my setup, since I plan to mess around with automation long term I wanted to have all my different browser drivers in one folder so I can kind of ‘containerize’ them and always know where they are, and update them accordingly. So, I made a directory called ‘browser_drivers_for_automation”

In that, I downloaded all the executables (go figure out how to do that if you want from Selenium page)

The following is the seemingly standard ‘trial run’ to make sure you are setup and working. I definitely recommend not moving forward until you have this trial run going, as well!

from selenium import webdriver

ffpath = '/home/username/path/to/browser_drivers_for_automation'
browser = webdriver.Firefox(ffpath)

browser.get("http://www.python.org")

First, try that. If it runs, awesome, but if you’re like me probably it won’t, HA

So then first, make sure this ‘browser_drivers_for_automation’ directory is listed in your Ubuntu PATH. If you’re like me I didn’t even know what that was but it seems (short version of the story) that this is the part of Ubuntu that says ‘any directory path in here, if you execute a file, I will allow it”

So do this in a terminal:

export PATH=$PATH:/home/user/path/to/browser_drivers_for_automation

not sure what this next one does but I did it so you might as well join me!

source ~/.profile

So now if you do echo $PATH in your terminal you should see your folder listed.

Now, try the above Selenium trial run again. If it works, great. If not, like me, then do this step:

Navigate to your Browser_drivers_for_automation directory and make your driver executable. In my example here i’m using the firefox geckodriver, but you can do this to any/all drivers in here.

sudo chmod +x geckodriver

See if that works. If it does, the browser will open and you should be good to go now.

Hope this helps

Thanks to all those people answering questions in stackoverflow as usual! 🙂

Categories
Technology

How to Do Some Code Changes in Gitlab

I’m not a developer but was encouraged to try a one character change to some software. I felt that I couldn’t break too much so I did my first tiny change. Literally one character text edit. However, it was a bit scary so I’m just posting this to show the basic workflow. This assumes you already have a gitlab account and username and can log in.

1. Find it.

probably you found or someone sent you a link to where the code is and you are on the gitlab page and can see the line of code you’d like to change. If not, get there.

Also, make sure you’re on the right branch. I tried to click ‘edit’ but I was not allowed because I was not on the branch called ‘master’. Your branch that you can edit may have a different name but just be aware that you have to be on the authorized branch that accepts changes. the left side has ‘branches’ and you can look at them there and select the right one.

2. Fork it

When you click ‘edit’ it seems to automatically force you to ‘fork’ the code. This makes sense. You wouldn’t want to disrupt the main code until the overseeing person has reviewed your changes and authorized them. So you ‘fork’ it and do your changes there. I think a fork is basically a copy/paste of the whole block of code for that file

3. Change it.

On the page where you change there is a box where you can comment. I was a bit worried about this but it wasn’t so bad. it’s just a comment box but it seems a bit more important than that. I put my personal comment about the change I made and then saved it.

4. Merge it.

Once you save the changes it becomes a ‘Merge Request’ also known as MR.
I think this part depends on how the project is set up. It might merge automatically into the main software or, it might require a ‘stamp of approval’. Regardless, once you send the MR it’s out of your hands unless you are the developer overseeing the project, also known as the ‘maintainer’.

Hopefully this helps someone else make their first change and thanks to the folks in free software who helped me do this!

Categories
Life Skills Technology

CONVERT HARVESTED BULK EMAILS INTO USABLE SPREADSHEET IN A FEW STEPS

Did someone send you a bulk email and foolishly leave all the recipients exposed? Do you want to grab those recipients and use for your own great purposes? Good news, it’s not so hard to do!

Here’s what you’ll need:

  • A text editor (gedit on ubuntu)
  • Spreadsheet software (Libre Office is the one I’ll use here since it’s both free and awesome)

1. Copy and paste the emails into a text editor

Grab the group of email addresses starting from right after the ‘to: ‘ but don’t actually include the ‘to:’. Copy this into your clipgboard.

2. Paste this to a fresh text editor window

3. Clean up the Paste

Make sure there is a trailing semi-colon after the last entry and no other gobbly gook before or after, other than just the names of the recipients and the actual email addresses. So like this stuff:

John Doe johndoe@johndoe.com; Jane Doe janedoe@janedoe.com;

4. Save file as a .csv file

Click ‘save as’ and give the file a handy name and make sure it has a trailing .csv as the file type

5. Go and open the file.

In my case it opens the file with LibreOffice – this is good. In your case…how could I know your case?

6. Select ‘semi-colon’ as the separation type

When Libre Office opens the file, uncheck comma, etc, and just leave the semi-colon because that’s what you’ve got.

7. Reformat the Orientation (optional)

After importing thus far, Libre Office seems to make the orientation horizontal putting one email address in each column and all the entries into a row spanning many horizontal columns which I didn’t like. I want to convert this to a vertical orientation with the entries spanning downwards across many rows.

Yippee yay. Someone alrady figured this out here

But in short, just do this:

  1. copy all your horizontal entries
  2. copy them to clipboard (control c)
  3. control + shift + V into the cell you want
  4. Paste (which does the ‘paste special’ command)
  5. Delete the original row you just copied from

Voila. Clean vertical list of harvested emails!

Hope that helps.

My Comments

My only improvement would be to add a script that pulls the name out and adds to the column on the right. I know thunderbird can do this when you right click and either ‘copy email address’ or ‘copy name and email address’. So it’s definitely possible to do this. In fact, how cool would that be if Thunderbird could just do all this in an add-on … hmm. Too bad I can’t program…

Categories
Life Skills Nextcloud Technology Tutorial

How to Install Video Convertor App on Nextcloud

Someone complaining that they can’t view an Apple .MOV file on your Nextcloud server? No surprise considering ‘apple is apple’.

Heard about the cool app ‘Video Convertor’ for Nextcloud? Tried to install it and use it and got that ‘requires ffmpeg error?

Same.

Turns out it’s pretty easy to to get it going and here is how:

Let’s do it.

  1. ssh into your Nextcloud box. In my case it’s a nextcloudpi box if that matters…
  2. type this into the command line: sudo apt install ffmpeg
  3. Hit your enter key
  4. Wait a long time while a seriously large amount of stuff installs (and probably this is why it’s not installed by default)
  5. Go back to your video file in the Nextcloud in the browser user interface and note the three dots on the right side of the file name. Click that
  6. Select the video icon ‘convert into’
  7. Choose your options for speed and format
  8. Do it.
  9. Wait again (a lot of waiting here…)
  10. Create a new share of the newly created video file
  11. Test the file (good practice) by sending the share to yourself in a browser that you don’t use so you are seeing what your recipient will see without your login credentials affecting things.
  12. All good? Ship the new link to your recipient

Hope that helps

Categories
Freedom and Privacy Technology Tutorial yunohost

INSTALLING ROUNDCUBE 1.4 VERSION ON YUNOHOST

Thanks to the work of Brian we can now install the newest version of Roundcube on Yunohost.

Why is this so exciting?

  • Newer, fresher UI
  • PGP encryption Functionality with the enigma plugin
  • Mobile friendly skin

In short, it makes your self-hosted email awesome on a mobile too, regardless of whether you have an email app that works or not with PGP

Normally, if this app was an approved app in the Yunohost app list you would be able to simply search it and install it from the app list. Until then we have to install it by the command line but it’s not that scary at all so let’s begin.

Installing it

  1. ssh into your Yunohost box
  2. run this command to install it:
    sudo yunohost app install https://github.com/bhdouglass/roundcube_ynh/tree/testing

Configuring the Install

A bunch of questions will start. Here is how I answered mine but you can adjust as you like. Note, if you press the enter key it will choose the default option for quick installation. Critical step is choosing ‘yes’ when you are presented with the Enigma opion.

This first question is just a warning. You’re brave. Take that risk!

WARNING! Installing 3rd party applications may compromise the integrity and security of your system. You should > probably NOT install it unless you know what you are doing. Are you willing to take that risk? [Y/N] : y

This next step shows all the domains you have configured in your Yunohost box and will ask you which one you want this Roundcube to be associated with:

Available domains:
-domainone.com
-domaintwo.com

Choose a domain for Roundcube (default: domainone.com:

This next question lets you choose which URL folder you want for the mail. I changed mine from the default since I have other things running but you can leave default if you don’t have something already using ‘/webmail’:

Choose a path for Roundcube (default: /webmail): /pgpmail

I’m using Nextcloud calendar stuff so i don’t need CardDav stuff now but feel free to install it if you need it and want to use it:

Install CardDAV synchronization plugin? [yes | no] (default: no):

This is an important one if you plan to encrypt your emails!

Install Enigma messages encryption plugin? [yes | no] (default: no): yes

Testing it out

Once complete, you should now be able to go to ‘domainone.com/pgpmail’ (or whatever your options are) and hit the new Roundcube installation and log in with your Yunhohost email server settings. I recommend doing the following tests each time you get setup with something like this since desktop browser and mobile browser use different skins (sometimes) in Roundcube.

Before beginning, make sure that the public key of each email address has been sent/imported into each side of the email transaction. You can use the ‘import key’ feature in Roundcube’s desktop mode quite easily. At point of this tutorial I haven’t tried importing keys with Mobile browser, so I can’t confirm if that works or works well.

  • Send totally plain text test message from desktop browser to test email address
  • Send totally plain text test message from Mobile browser to test email address

All good? Now with public keys attached:

  • Send new email with just public key attached using Roundcube’s ‘attach public key’ feature – from Desktop browser
  • Send new email with just public key attached using Roundcube’s ‘attach public key’ feature – from Mobile browser

Still good? Now encrypt it!

  • Send new email fully encrypted using Roundcube’s ‘encrypt this email’ feature – from Desktop browser
  • Send new email with just public key attached using Roundcube’s ‘encrypt this email’ feature – from Mobile browser

Everything still good? You should be ‘in business’

Hope you enjoyed and found this useful.

Categories
Freedom and Privacy Life Skills Nextcloud Tutorial Ubuntu Touch

How to Set up a Calendar with Ubuntu Touch and Nextcloud

1. Set up online account in Ubuntu Touch

You will need all your calendar info from your Nextcloud instance before beginning. Looks something like this: https://yourdomain.com/nextcloudserverlocation/remote.php/dav/calendars/username

  1. open calendar
  2. middle icon that shows grid calendar
  3. ‘add online calendar’
  4. choose NC
  5. enter credentials

2. Sync

UT calendar ‘should’ start syncing right away and you’ll see a little icon appear showing that sync started and stopped. Sometimes, however, it seems like a manual sync is required. You can do this by going to the calendar page and selecting the top right hamburger menu and then the circular arrow ‘refresh’ icon.

Categories
Freedom and Privacy Mesh Networks Technology Tutorial

Converting a Zsun Wifi Card Reader into a Wifi Range Extender

This post is a follow-up post to my original where I successfully flashed OpenWRT onto these Zsun devices. Be sure to see that post first if you haven’t already flashed OpenWRT onto the device.

Note that this tutorial should also work for any device upon which you can put OpenWRT (ie any compatible old router you have kicking around).

With this project, what I was really trying to do was create a legitimate ‘mesh network’ but my skills and time ran out so I resolved to have a ‘half victory’ which was to be able to use these little devices to expand our home wifi with small size footprint and lower energy usage, even if it was just on demand, as needed. For example, if I needed wifi to reach outside while gardening so I could listen to streamed music, etc, I could plug one of these in nearby and extend the range instantly.

Before beginning, it’s important to note that this process may need your critical thinking to build on what I’ve done, and if you have further progress, it would be appreciated by all to know, if you could write them in the comments. For full disclosure I fried two Zsun devices while learning so make sure to heed my advice in the other blog if you are using this device.

Oh, one last useful statement: I recommend turning off the wifi in your master-router so that you (you) don’t get confused by which wifi radio device you are connecting to since both devices will, by the end of the tuturial, be sharing the exact same SSID. It reduces confusion and headaches to turn this off (just the wifi, you can use wired connection if you have access). Also, while you are turning it off, take note as to what channel it is broadcasting on since you will want to choose a new channel that is far away from this one on the new device.

Ok, let’s get started.

Setting up the Device as an Access Point

For full credit I pulled the methods for this process from this video, but the video wasn’t super helpful because it required an internet connection to do the changes and I needed a static page with text instructions! These are those:

Step 1: Set up the Interface

  1. Go to ‘network’ and ‘interfaces’ in the sub-servient (new word I made, enjoy, GNU license word..gnucabulary…) device (in my case the zsun).
  2. If you have any other interfaces besides ‘LAN’, remove them as they won’t be used
  3. Edit the LAN
  4. Change the IPv4 field to the static IP address that this device will have on your main home network.
    If your main router is 192.168.1.1 for example, then you could set this to 192.168.1.5 if it’s available. If not, find one that is and set it. And don’t lose it! You will need it to log back into the router after making the change.
  5. Change the gateway IP address to the master (gnucabulary…) routers (ie. 192.168.1.1 if that’s your router’s admin login page)
  6. In the “DHCP Server” settings below on the same page, there is a checkbox called ‘ignore interface’. Check that box which will disable DHCP (the thing that sends out IP addresses to all your devices) since you won’t need it
  7. “Save and Apply’ button at the bottom

Reminder note: your device will no longer be found at 192.168.1.1 if that’s where you just logged in. It will now be accessible at the address you chose in step 4 above. I always forget this one, ha. Go find it and log back in…

Step 2: Disable the Firewall

  1. Go to ‘System’ and then ‘Startup’
  2. Scroll down until you see ‘firewall’
  3. Disable it by clicking on the ‘enabled’ button
  4. ‘Submit’ button

Step 3: Adjust the Wifi settings

  1. Go to ‘Network’ then ‘wifi’
  2. Edit the active wifi entry
  3. Change the channel (1 to 11) of the device to one that is fairy far away from that of your main router so there is a nice gap between the frequencies
  4. In ‘Interface Configuration’ section, change the mode to ‘access point’if it isn’t already
  5. change the SSID to exactly the same one as your main router (if it’s slightly different it won’t work)
  6. Change the WPA2/psk password to exactly the same one as your main router is outputting. If you don’t it won’t work
  7. ‘Save & Apply’ button

Some Follow up Notes

As hinted at at the very beginning of this tutorial, from this point on you will not (or may not?) be able to access your subservient device while the wifi of the master router is on. The reason for this is because probably your computer will find the master router’s wifi device and connect to that. I had big struggles trying to find this device again. If you need to access it, either unplug your master router (honestly this is the easiest way if no one will be angry at you for killing their internet) or go into the master router’s settings and disable the wifi transmit. For me, I recommend turning off the master router’s wifi transmit until it’s all setup on the subservient first.

I had quite a bit of problems, even though my master router wasn’t transmitting wifi, connecting to my newly-IP’d subservient device. After I cleared my browsers cache it did re-appear but I’m not sure that’s why. You might need to mess around with your browser to be able to hit the admin page again. I think my problem might be because I have multiple devices running OpenWRT and the browser gets confused…

Special thanks to all the contributors at OpenWRT!