In my new journey to figure out how to automate stuff in my life, one of the time-sucking adventures was to try to stop browser popups such as geolocation and notifications. I just wanted to click ‘ok’ or ‘dismiss’ and move on since this was my ‘bot’ instance of the browser.
The issue with these browser-based pop ups is that they are handled by the browser, not by selenium so it seems selenium can’t dismiss them or dismiss them easily.
The solution workflow that I found to work well for me is as follows:
- Switch to your Selenium Firefox profile
- Do your browser setting that you need
- Save the change
- Start your Selenium script with new changes
Here are the details on how to do this:
Create a new Firefox Profile
- in browser, type
- Create a new profile
- Take a note of the location of the root directory of that profile (ie paste to your Selenium script for now)
- ‘Launch profile in new browser’ (this will launch your new profile in a new instance nicely)
- Make your browser changes and save
Do your browser setting changes
The following two changes were the main two that bothered me so perhaps I’ll highlight those here:
- Don’t allow websites to send you notifications
- Preferences/privacy and security / Notifications
- Don’t let websites ask for geolocation stuff
- Preferences/privacy and security / Permissions – Locations (Settings button)
TIP! It’s very useful to actually do a dry run as a human before you let your bot run free so that you can deal with these popups logged in, one time, as this profile. So, workflow is to switch to your Selenium profile, launch it, do a dry run on with real human clicks, deal with any popups or browser setting stuff, save changes, go run the bot script.
Setting up your Python Script to use your Selenium Profile
Now that you’ve got your new profile, let’s actually use it in your script, instead of what will always be a fresh browser instance
Here is my code block which you can add in your project:
## SETUP SELENIUM TO USE CUSTOM FIREFOX PROFILE #Pretty sure you need to import this to use 'FirefoxProfile but I'm too lazy to confirm - feel free from selenium.webdriver.firefox.options import Options #Root directory copied from Step 3 above profilePath = '/home/user/.mozilla/firefox/cjda7321.Selena' #Directs profile selector thing to the right path created above profile = webdriver.FirefoxProfile(profilePath) #Tells Selenium to use the custom profile driver = webdriver.Firefox(firefox_profile=profile)
Hope that helps!
I spent literally 3 days trying to simply click a logout link with Selenium. I searched every stackoverflow post I could find until I found this one.
I had tried pretty much everything I could to try to click the logout link which was the fifth item down in the move-over list.
I tried find_element_by_xpath, find_element_by_id, find this, find that, blah blah blah
Finally, it was indeed the
find_element_by_css_selector that worked.
The only thing is I still do not know WHY the xpath option didn’t work for this list while the others did. Hmm. Whatever. Probably will figure it out later….HOW TO CLICK AN ITEM IN A SUBMENU (UL) LIST USING PYTHON, SELENIUMThe only thing is I still do not know WHY the xpath option didn’t work for this list while the others did. Hmm. Whatever. Probably will figure it out later….
so, here is a code block of what worked for me to move to a mouse-over menu and click the menu’s submenu item using Selenium and python (my css selector is just a fake example of course so paste your correct one in):
Edit: I also realized I had to click on the button *above* the actual unordered list (UL) element to trigger the drop down. This tip for the ‘main_menu’ element below might also help someone.
main_menu = driver.find_element_by_xpath("//*[@id='with-label']") actions.move_to_element(main_menu).perform() time.sleep(2) WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".css-selector-thing > ul:nth-child(2) > li:nth-child(6) > a:nth-child(1)"))) driver.find_element_by_css_selector(".css-selector-thing > ul:nth-child(2) > li:nth-child(6) > a:nth-child(1)").click()
So this was super duper hard and I believe the reason why is because technically the browser settings and the user choices impact whether or not a clicked link opens in a fresh new tab or a fresh new window. Keep that in mind in case you have other headaches. It may be impossible (so I’ve read online) to perfectly control whether a tab opens on your users’ browser, however, since web automation is probably your browser you should be able to work with this solution and your browser settings to get it going.
Another important thing before you begin here. This assumes you are ‘right clicking an element on a page and opening in a new tab’ kind of thing. If you are trying to make a new, unrelated link open in a new tab, this doesn’t work. I’ll figure that out maybe in another blog post. This one takes a page element already in focus and then opens that into a new tab.
Final note: I’m using Ubuntu so not sure if you need to adjust for yourself…
Anyway, here comes the code.
# Import your stuffs from selenium import webdriver from selenium.webdriver.common.keys import Keys import time # Set path to your executable Firefox` ffpath = '/home/wt/Documents/Waynes_Python_Education_Directory/Automation_Browser_Drivers' # Turn your Firefox browser into a usable object (or something like that) driver = webdriver.Firefox(ffpath) # Come up with a creative new URL to go to (this one is great, btw) driver.get("https://www.engrish.com/") # Wait for a while to enjoy it.. time.sleep(5) # Come up with a creative new URL to go to (this one is great, btw) new_url = driver.find_element_by_xpath("/html/body/div/div/div/div/div/div/div/ul/li/a") # And now, the magic! Open fun URL in a new Firefox tab! new_url.send_keys(Keys.CONTROL + Keys.RETURN) # Wait for a while to enjoy it.. time.sleep(5) # Shut 'er down and go home for the day... driver.quit()
I realized that I need to execute the script below every time my computer reboots so my current workflow is to leave the command commented out in script and then run it before I begin. You can probably run it with the script too? This one:
I had the following continual painfull errors:
- selenium.common.exceptions.WebDriverException: Message: ‘my_folder_name’ executable may have wrong permissions.
- selenium.common.exceptions.WebDriverException: Message: ‘Automation_Browser_Drivers’ executable needs to be in PATH.
To be transparent, I’m not sure which if any of these steps happened first, or whether it matters, but I’ll give you both in case it helps and in the order I did:
For my setup, since I plan to mess around with automation long term I wanted to have all my different browser drivers in one folder so I can kind of ‘containerize’ them and always know where they are, and update them accordingly. So, I made a directory called ‘browser_drivers_for_automation”
In that, I downloaded all the executables (go figure out how to do that if you want from Selenium page)
The following is the seemingly standard ‘trial run’ to make sure you are setup and working. I definitely recommend not moving forward until you have this trial run going, as well!
from selenium import webdriver
ffpath = '/home/username/path/to/browser_drivers_for_automation'
browser = webdriver.Firefox(ffpath)
First, try that. If it runs, awesome, but if you’re like me probably it won’t, HA
So then first, make sure this ‘browser_drivers_for_automation’ directory is listed in your Ubuntu PATH. If you’re like me I didn’t even know what that was but it seems (short version of the story) that this is the part of Ubuntu that says ‘any directory path in here, if you execute a file, I will allow it”
So do this in a terminal:
not sure what this next one does but I did it so you might as well join me!
So now if you do
echo $PATH in your terminal you should see your folder listed.
Now, try the above Selenium trial run again. If it works, great. If not, like me, then do this step:
Navigate to your Browser_drivers_for_automation directory and make your driver executable. In my example here i’m using the firefox geckodriver, but you can do this to any/all drivers in here.
sudo chmod +x geckodriver
See if that works. If it does, the browser will open and you should be good to go now.
Hope this helps
Thanks to all those people answering questions in stackoverflow as usual! 🙂
I’m not a developer but was encouraged to try a one character change to some software. I felt that I couldn’t break too much so I did my first tiny change. Literally one character text edit. However, it was a bit scary so I’m just posting this to show the basic workflow. This assumes you already have a gitlab account and username and can log in.
1. Find it.
probably you found or someone sent you a link to where the code is and you are on the gitlab page and can see the line of code you’d like to change. If not, get there.
Also, make sure you’re on the right branch. I tried to click ‘edit’ but I was not allowed because I was not on the branch called ‘master’. Your branch that you can edit may have a different name but just be aware that you have to be on the authorized branch that accepts changes. the left side has ‘branches’ and you can look at them there and select the right one.
2. Fork it
When you click ‘edit’ it seems to automatically force you to ‘fork’ the code. This makes sense. You wouldn’t want to disrupt the main code until the overseeing person has reviewed your changes and authorized them. So you ‘fork’ it and do your changes there. I think a fork is basically a copy/paste of the whole block of code for that file
3. Change it.
On the page where you change there is a box where you can comment. I was a bit worried about this but it wasn’t so bad. it’s just a comment box but it seems a bit more important than that. I put my personal comment about the change I made and then saved it.
4. Merge it.
Once you save the changes it becomes a ‘Merge Request’ also known as MR.
I think this part depends on how the project is set up. It might merge automatically into the main software or, it might require a ‘stamp of approval’. Regardless, once you send the MR it’s out of your hands unless you are the developer overseeing the project, also known as the ‘maintainer’.
Hopefully this helps someone else make their first change and thanks to the folks in free software who helped me do this!
The Background – The Feelings
The background to this one is that I was really amazed by the technology and functionality of QR codes. QR codes brought the convenience of UPC retail bar codes to the average person without the crazy costs associated. So, even from a business perspective you could have your own scanning system if you wanted.
On a personal level, it was mind blowing that I could have a business card with a QR code on it that would ‘hide’ data within visually and then the smart phone or web cam could see the code, unpack the data and leave it with you in a digital format – with no typing needed and no errors (unless the creator goofed). This allowed me to turn a business card into an immediately usable tool, instead of what would happen before where the paper card would get buried or lost before I got to use it, plus it would force me to type more keystrokes than I wanted. I mean.. It could even do a .vcf format something which could be loaded direct into your contact books, from what I recall (would need to review this one but 99% sure I did that…)
Further to the personal benefits, let’s say you are at a social gathering. Someone approaches you and wishes to get your business card, or, some form of contact informaton that might normally be on a business card. But, you don’t really want to share certain parts, or any parts with said person. These days normally you would fumble and figure it out on the fly. What if the person says ‘text me’? Typically I say “I don’t text” and then I have to have a long discussion about why I don’t text and how to reach me, and here’s my email, and here’s my number, but don’t call me, because I don’t like phone calls, blah blah blah… It would be nice to have a standardized way of transmitting data – line of sight – no internet – no wifi – no bluetooth. Just transmit basic data. No worrying about whether they are on apple, or android or Ubuntu Touch (wait, that’s no problem!) or whatever. No worrying about whether they have a cell phone or a cell phone plan, etc. It’s also more inclusive that way.
I can hear the voices now:
- “But you can email all that!” – No! I want faster. No typing. No thinking.
- “But you can text all that!” – No! I said I don’t text. Don’t ask me to text you.
- “But you can ‘bump’ with Android” – I don’t bump. I don’t Android.
- “This is making something new that we don’t need!” – Ok… maybe… And that’s the point of this blog. Let’s figure that out…
Why QR has failed thus far (In my opinion)
After all these thoughts above, I got to thinking about the limitations of QR. You can read some of my feelings in this bug report I filed at UBports for the Camera app. Of course, I completely understand the security need here and I can’t argue it. But this has always been the issue. For QR codes to flourish, the scanner must be one button away.
And, even if you had one button there is still the annoyance of having to focus the camera. I thought about creating a tiny keychain which had one button and a camera. You would point at the QR and push/hold the ‘shutter’ button. It would focus, scan the QR code and beep when it was logged. Later you could sync that data with whatever device you want, somehow securely.
And that’s when I started thinking that it would be cool to have an ‘audio QR code’. So I searched that and found Chirp. They basically have figured it out with proprietary software – not typically a good solution if you want awesome security and privacy… But, their stuff apparently works. It sends hypersonic sounds and data.
In my opinion, this direction could work, and here are the first pros/cons:
- No focusing of cameras (faster scan?)
- No camera at all (cheaper hardware?)
- Audio? Maybe not as radioactive as radio??? (just throwing it out there for the healthy people?)
- Other cool pros?
- Apparently dogs don’t like hypersonic?
- Would these things have to transmit all the time or could the ‘chirp’ be triggered with some kind of ‘hey! any birds out there who want to chirp me anything?”
- Security? Could anyone chirp anything? I haven’t really looked deeply at that part yet as you can see.
Some suggestions to the conversation were:
- A watch (with a cam – full Dick Tracey style)
Sure, except I would never wear a watch, and I’m not a Dick…Tracey…
- Near Field Communication (NFC) which is explained well in this old 2014 vid
Then, whatever the solution is, would be able to securely/safely transmit the stored data into the device of your choice for reviewing, handling.
I feel like I don’t want to add to the smartphone in my pocket, but, it seems like there is no other way. Security people say you can’t really have a one-button something without compromising security of smartphone. SO…
That’s what I’ve got to say.
I hope some kind of cheap and useful solution lands here…
Did someone send you a bulk email and foolishly leave all the recipients exposed? Do you want to grab those recipients and use for your own great purposes? Good news, it’s not so hard to do!
Here’s what you’ll need:
- A text editor (gedit on ubuntu)
- Spreadsheet software (Libre Office is the one I’ll use here since it’s both free and awesome)
1. Copy and paste the emails into a text editor
Grab the group of email addresses starting from right after the ‘to: ‘ but don’t actually include the ‘to:’. Copy this into your clipgboard.
2. Paste this to a fresh text editor window
3. Clean up the Paste
Make sure there is a trailing semi-colon after the last entry and no other gobbly gook before or after, other than just the names of the recipients and the actual email addresses. So like this stuff:
4. Save file as a .csv file
Click ‘save as’ and give the file a handy name and make sure it has a trailing .csv as the file type
5. Go and open the file.
In my case it opens the file with LibreOffice – this is good. In your case…how could I know your case?
6. Select ‘semi-colon’ as the separation type
When Libre Office opens the file, uncheck comma, etc, and just leave the semi-colon because that’s what you’ve got.
7. Reformat the Orientation (optional)
After importing thus far, Libre Office seems to make the orientation horizontal putting one email address in each column and all the entries into a row spanning many horizontal columns which I didn’t like. I want to convert this to a vertical orientation with the entries spanning downwards across many rows.
Yippee yay. Someone alrady figured this out here
But in short, just do this:
- copy all your horizontal entries
- copy them to clipboard (control c)
- control + shift + V into the cell you want
- Paste (which does the ‘paste special’ command)
- Delete the original row you just copied from
Voila. Clean vertical list of harvested emails!
Hope that helps.
My only improvement would be to add a script that pulls the name out and adds to the column on the right. I know thunderbird can do this when you right click and either ‘copy email address’ or ‘copy name and email address’. So it’s definitely possible to do this. In fact, how cool would that be if Thunderbird could just do all this in an add-on … hmm. Too bad I can’t program…
Someone complaining that they can’t view an Apple .MOV file on your Nextcloud server? No surprise considering ‘apple is apple’.
Heard about the cool app ‘Video Convertor’ for Nextcloud? Tried to install it and use it and got that ‘requires ffmpeg error?
Turns out it’s pretty easy to to get it going and here is how:
Let’s do it.
- ssh into your Nextcloud box. In my case it’s a nextcloudpi box if that matters…
- type this into the command line:
sudo apt install ffmpeg
- Hit your enter key
- Wait a long time while a seriously large amount of stuff installs (and probably this is why it’s not installed by default)
- Go back to your video file in the Nextcloud in the browser user interface and note the three dots on the right side of the file name. Click that
- Select the video icon ‘convert into’
- Choose your options for speed and format
- Do it.
- Wait again (a lot of waiting here…)
- Create a new share of the newly created video file
- Test the file (good practice) by sending the share to yourself in a browser that you don’t use so you are seeing what your recipient will see without your login credentials affecting things.
- All good? Ship the new link to your recipient
Hope that helps
Thanks to the work of Brian we can now install the newest version of Roundcube on Yunohost.
Why is this so exciting?
- Newer, fresher UI
- PGP encryption Functionality with the enigma plugin
- Mobile friendly skin
In short, it makes your self-hosted email awesome on a mobile too, regardless of whether you have an email app that works or not with PGP
Normally, if this app was an approved app in the Yunohost app list you would be able to simply search it and install it from the app list. Until then we have to install it by the command line but it’s not that scary at all so let’s begin.
- ssh into your Yunohost box
- run this command to install it:
sudo yunohost app install https://github.com/bhdouglass/roundcube_ynh/tree/testing
Configuring the Install
A bunch of questions will start. Here is how I answered mine but you can adjust as you like. Note, if you press the enter key it will choose the default option for quick installation. Critical step is choosing ‘yes’ when you are presented with the Enigma opion.
This first question is just a warning. You’re brave. Take that risk!
WARNING! Installing 3rd party applications may compromise the integrity and security of your system. You should > probably NOT install it unless you know what you are doing. Are you willing to take that risk? [Y/N] : y
This next step shows all the domains you have configured in your Yunohost box and will ask you which one you want this Roundcube to be associated with:
Choose a domain for Roundcube (default: domainone.com:
This next question lets you choose which URL folder you want for the mail. I changed mine from the default since I have other things running but you can leave default if you don’t have something already using ‘/webmail’:
Choose a path for Roundcube (default: /webmail): /pgpmail
I’m using Nextcloud calendar stuff so i don’t need CardDav stuff now but feel free to install it if you need it and want to use it:
Install CardDAV synchronization plugin? [yes | no] (default: no):
This is an important one if you plan to encrypt your emails!
Install Enigma messages encryption plugin? [yes | no] (default: no): yes
Testing it out
Once complete, you should now be able to go to ‘domainone.com/pgpmail’ (or whatever your options are) and hit the new Roundcube installation and log in with your Yunhohost email server settings. I recommend doing the following tests each time you get setup with something like this since desktop browser and mobile browser use different skins (sometimes) in Roundcube.
Before beginning, make sure that the public key of each email address has been sent/imported into each side of the email transaction. You can use the ‘import key’ feature in Roundcube’s desktop mode quite easily. At point of this tutorial I haven’t tried importing keys with Mobile browser, so I can’t confirm if that works or works well.
- Send totally plain text test message from desktop browser to test email address
- Send totally plain text test message from Mobile browser to test email address
All good? Now with public keys attached:
- Send new email with just public key attached using Roundcube’s ‘attach public key’ feature – from Desktop browser
- Send new email with just public key attached using Roundcube’s ‘attach public key’ feature – from Mobile browser
Still good? Now encrypt it!
- Send new email fully encrypted using Roundcube’s ‘encrypt this email’ feature – from Desktop browser
- Send new email with just public key attached using Roundcube’s ‘encrypt this email’ feature – from Mobile browser
Everything still good? You should be ‘in business’
Hope you enjoyed and found this useful.
So, you want to not wait for RoundCube to release 1.4 to cpanel, or, you have Roundcube on your cpanel setup and for whatever reason the Enigma plugin that makes the PGP stuff work – isn’t working. Whatever your reason is, the solution is not insanely hard, but it took me about a week and a lot of hours to figure out how to get it going. Hopefully this will save you many hours that I lost! 🙂
Probably this page will work for future stuff too for future releases so I’ll leave the download pages more general.
Before we begin, quick thanks to everyone at Roundcube for really improving the look, feel and security of everything. Really nice upgrades in the UI for version 1.4 RC!
STEP ONE – DOWNLOAD
- Get your version (in this case 1.4-rc) from this Roundcube download page
- Download compressed Roundcube file to your computer
- Upload compressed Roundcube it via FTP (or whatever method you like)
- In your cpanel File Manager, Right click on the compressed Roundcube file and ‘extract’ – the file will decompress and extract the directory with the same name in the same directory where you clicked ‘extract’
- Locate the newly extracted file and rename to something you like. This will form part of the URL when you log into your email. If you leave it as is the URL will be too long and annoying so changing it to something like ‘mail’ or ’roundcube’ would be smart. Maybe don’t call it ‘webmail’ because most shared hosting uses that? Not sure, I didn’t test that but something more unique would probably be smart.
- go to the URL of your domain, with the new directory folder and add ‘installer’ at the end and this will start installer. You can read details on the Roundcube installation page and we are now at step ‘Configuring Roundcube’
Create your MYSQL database for Roundcube
You’ll need a database to make Roundcube work. Although this kind of stuff seems scary, it’s not that scary as long as you don’t delete stuff that’s already there that you don’t know about. In short, all you are gong to do is create a database, create a user for the database, and then glue the user to the database. That’s it.
Before you begin this part you should have some kind of password manager software setup, I think, so you can create strong passwords and not lose them. Otherwise, ‘do it the way you like’.
- in your Cpanel home panel, go to ‘MYSQL Databases’
- in the top section, Create New Database, give your new database a decent name like ’roundcube_abc’ (no one sees this stuff, it’s just back end). You will note that it will automatically append something to the front. That will be part ofyour database name, not just the part you are typing in the field.
- Click ‘Create Database’ button
- Scroll down to the MYSQL Users section and in the ‘Add New User’ section at the top, create a new username with a strong password. You can use the password generator and then make sure to save it safely. You will note that it will automatically append something to the front. That will be part ofyour username name, not just the part you are typing in the field.
- Click ‘Create user’ button
Now you have both a database and a username. the last steps are to glue them together.
- Scroll down to ‘Add User to Database’
- From the ‘User’ dropdown, select the user you just created
- From the Database dropdown, select the database you just created above
- Click the ‘Add’ button
It will take you to another screen where it asks what permissions you want to give this user in this database. You will give it all permissions which should be the default (all boxes selected).
Confirm these changes.
Really you should read this entire page and learn a bit as your setup might be different and it’s good to review the items they mention, although most or much of it is for people who have full access to their servers, not cpanel people.
Once you start the installation process, there really isn’t too much documentation on how to actually set it up. It will start a kind of ‘installation wizard’ but not a lot of help is there during the process. You’ll need to have the following items near you before you begin:
- The email credentials that will be using Roundcube including: server names, type of email (ie. SMTP, IMAP), security protocol of each, ports for each. You can get all this from your email provider.
- The database name, user name, and user name password for the MYSQL stuff you created above
Once all this is in hand simply walk through all the fields and do your best. The details of that part is out of scope of this tutorial but there is one vital point which you absolutely must seclect in order to use encryption and PGP and that is, in the plugin section you must select the Enigma plugin which will add all the functionality.
Again, be sure to install the the Enigma plugin.
Fixing the enigma_pgp_homedir not specified error
After installation is complete and everything appears working, if you were to go to the settings and to PGP keys you will be greeted with a warning that enigma_pgp_homedir is not specified, or, if you tried to simply put the path in there as if it were a full control server, it probably won’t work. At least that’s what happened to me and the purpose of this tutorial.
The solution to properly point Roundcube to a secure folder was pretty hard to figure out but now that it’s figured out should be pretty easy for you.
You should apparently not put your pgp key folder into the enigma plugin folder, or in the document root folder. Frankly I”m not skilled enough to know all the whys but I’ve learned to trust people smarter than I. So, I put the directory in the main directory of my domain where roundcube is. You can mirror this for simplicity and probably should. So, here go the steps:
- In cpanel File manager go to: public_html/yourdomain.com
- Create a folder for your keys. For this tutorial I’ll call it ‘keez’ but you can call it whatever you want. Make sure permissions are 0755 (they should be after you create it)
- In File manager, navigate to this location: /yourRoundCubeInstallationFolder//plugins/enigma/
- Locate the config.inc.php.dist file
- Touch it, right click on it, and rename and remove the ‘.dist’ from the end of the filename and save the change
- Touch the newly renamed file again and right click and ‘edit’. This will open the Cpanel text editor.
- It will give you a warning that by editing you can break everything, which of course we know, so, click edit again and reall stick it to the man!
- Scroll down in the code stuff until you see this section:
// REQUIRED! Keys directory for all users.
// Must be writeable by PHP process, and not in the web server document root
$config[‘enigma_pgp_homedir’] = null;
Now is the fun part. All you have to do is enter in the path that points to your keys folder that you created in step 2 above.
Mine now looks like this:
$config['enigma_pgp_homedir'] = realpath(__DIR__.'/..'.'/..'.'/..').'/keez';
As long as you put your keys folder in the main directory of your domain and named it ‘keez’ this line shoud work. If you have your folder somewhere else you’ll have to adjust accordingly. What I learned was each instance of
'/..' moves up one level directory. So this command is saying ‘you will find the ‘keez’ folder by going up 3 levels from where you are now”.
Once you have adjusted this one line of code, click the ‘save changes’ button in your cpanel editor.
Go to your Roundcube email, go to ‘settings’ and go to ‘PGP Keys’ and you should now have working PGP functionality working in your Cpanel Roundcube, plus, you are running the new mobile-friendly version 1.4 now that all the paths are working.
I hope this helps!