Category: Tutorial

Categories
Technology Tutorial

Dismissing Firefox popups using Selenium and Python

In my new journey to figure out how to automate stuff in my life, one of the time-sucking adventures was to try to stop browser popups such as geolocation and notifications. I just wanted to click ‘ok’ or ‘dismiss’ and move on since this was my ‘bot’ instance of the browser.

The issue with these browser-based pop ups is that they are handled by the browser, not by selenium so it seems selenium can’t dismiss them or dismiss them easily.

The solution workflow that I found to work well for me is as follows:

  1. Switch to your Selenium Firefox profile
  2. Do your browser setting that you need
  3. Save the change
  4. Start your Selenium script with new changes

Here are the details on how to do this:

Create a new Firefox Profile

  1. in browser, type about:profiles
  2. Create a new profile
  3. Take a note of the location of the root directory of that profile (ie paste to your Selenium script for now)
  4. ‘Launch profile in new browser’ (this will launch your new profile in a new instance nicely)
  5. Make your browser changes and save

Do your browser setting changes

The following two changes were the main two that bothered me so perhaps I’ll highlight those here:

  1. Don’t allow websites to send you notifications
  • Preferences/privacy and security / Notifications
  1. Don’t let websites ask for geolocation stuff
  • Preferences/privacy and security / Permissions – Locations (Settings button)

TIP! It’s very useful to actually do a dry run as a human before you let your bot run free so that you can deal with these popups logged in, one time, as this profile. So, workflow is to switch to your Selenium profile, launch it, do a dry run on with real human clicks, deal with any popups or browser setting stuff, save changes, go run the bot script.

Setting up your Python Script to use your Selenium Profile

Now that you’ve got your new profile, let’s actually use it in your script, instead of what will always be a fresh browser instance

Here is my code block which you can add in your project:

## SETUP SELENIUM TO USE CUSTOM FIREFOX PROFILE

#Pretty sure you need to import this to use 'FirefoxProfile but I'm too lazy to confirm - feel free
from selenium.webdriver.firefox.options import Options 

#Root directory copied from Step 3 above
profilePath = '/home/user/.mozilla/firefox/cjda7321.Selena'

#Directs profile selector thing to the right path created above
profile = webdriver.FirefoxProfile(profilePath) 

#Tells Selenium to use the custom profile
driver = webdriver.Firefox(firefox_profile=profile)

Hope that helps!

Categories
Technology Tutorial

HOW TO CLICK AN ITEM IN A SUBMENU (UL) LIST USING PYTHON, SELENIUM

I spent literally 3 days trying to simply click a logout link with Selenium. I searched every stackoverflow post I could find until I found this one.

I had tried pretty much everything I could to try to click the logout link which was the fifth item down in the move-over list.

I tried find_element_by_xpath, find_element_by_id, find this, find that, blah blah blah

Finally, it was indeed the find_element_by_css_selector that worked.

The only thing is I still do not know WHY the xpath option didn’t work for this list while the others did. Hmm. Whatever. Probably will figure it out later….HOW TO CLICK AN ITEM IN A SUBMENU (UL) LIST USING PYTHON, SELENIUMThe only thing is I still do not know WHY the xpath option didn’t work for this list while the others did. Hmm. Whatever. Probably will figure it out later….

so, here is a code block of what worked for me to move to a mouse-over menu and click the menu’s submenu item using Selenium and python (my css selector is just a fake example of course so paste your correct one in):

Edit: I also realized I had to click on the button *above* the actual unordered list (UL) element to trigger the drop down. This tip for the ‘main_menu’ element below might also help someone.

main_menu = driver.find_element_by_xpath("//*[@id='with-label']")

actions.move_to_element(main_menu).perform()

time.sleep(2)

WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".css-selector-thing > ul:nth-child(2) > li:nth-child(6) > a:nth-child(1)")))

driver.find_element_by_css_selector(".css-selector-thing > ul:nth-child(2) > li:nth-child(6) > a:nth-child(1)").click()
Categories
Technology Tutorial

OPENING A NEW TAB WITH SELENIUM + PYTHON + FIREFOX + UBUNTU

So this was super duper hard and I believe the reason why is because technically the browser settings and the user choices impact whether or not a clicked link opens in a fresh new tab or a fresh new window. Keep that in mind in case you have other headaches. It may be impossible (so I’ve read online) to perfectly control whether a tab opens on your users’ browser, however, since web automation is probably your browser you should be able to work with this solution and your browser settings to get it going.

Another important thing before you begin here. This assumes you are ‘right clicking an element on a page and opening in a new tab’ kind of thing. If you are trying to make a new, unrelated link open in a new tab, this doesn’t work. I’ll figure that out maybe in another blog post. This one takes a page element already in focus and then opens that into a new tab.

Final note: I’m using Ubuntu so not sure if you need to adjust for yourself…

Anyway, here comes the code. 

# Import your stuffs
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

# Set path to your executable Firefox` 
ffpath = '/home/wt/Documents/Waynes_Python_Education_Directory/Automation_Browser_Drivers'

# Turn your Firefox browser into a usable object (or something like that)
driver = webdriver.Firefox(ffpath)

# Come up with a creative new URL to go to (this one is great, btw)
driver.get("https://www.engrish.com/")

# Wait for a while to enjoy it..
time.sleep(5)

# Come up with a creative new URL to go to (this one is great, btw)
new_url = driver.find_element_by_xpath("/html/body/div[1]/div[2]/div/div/div[2]/div[1]/div[1]/ul/li[6]/a")

# And now, the magic!  Open fun URL in a new Firefox tab!
new_url.send_keys(Keys.CONTROL + Keys.RETURN)

# Wait for a while to enjoy it..
time.sleep(5)

# Shut 'er down and go home for the day...
driver.quit()
Categories
Technology Tutorial Ubuntu

Overcoming Painful Setup for Selenium with Python on Ubuntu

EDIT 191118
I realized that I need to execute the script below every time my computer reboots so my current workflow is to leave the command commented out in script and then run it before I begin. You can probably run it with the script too? This one: export PATH=$PATH:/home/user/path/to/browser_drivers_for_automation

I had the following continual painfull errors:

  1. selenium.common.exceptions.WebDriverException: Message: ‘my_folder_name’ executable may have wrong permissions.
  2. selenium.common.exceptions.WebDriverException: Message: ‘Automation_Browser_Drivers’ executable needs to be in PATH.

To be transparent, I’m not sure which if any of these steps happened first, or whether it matters, but I’ll give you both in case it helps and in the order I did:

For my setup, since I plan to mess around with automation long term I wanted to have all my different browser drivers in one folder so I can kind of ‘containerize’ them and always know where they are, and update them accordingly. So, I made a directory called ‘browser_drivers_for_automation”

In that, I downloaded all the executables (go figure out how to do that if you want from Selenium page)

The following is the seemingly standard ‘trial run’ to make sure you are setup and working. I definitely recommend not moving forward until you have this trial run going, as well!

from selenium import webdriver

ffpath = '/home/username/path/to/browser_drivers_for_automation'
browser = webdriver.Firefox(ffpath)

browser.get("http://www.python.org")

First, try that. If it runs, awesome, but if you’re like me probably it won’t, HA

So then first, make sure this ‘browser_drivers_for_automation’ directory is listed in your Ubuntu PATH. If you’re like me I didn’t even know what that was but it seems (short version of the story) that this is the part of Ubuntu that says ‘any directory path in here, if you execute a file, I will allow it”

So do this in a terminal:

export PATH=$PATH:/home/user/path/to/browser_drivers_for_automation

not sure what this next one does but I did it so you might as well join me!

source ~/.profile

So now if you do echo $PATH in your terminal you should see your folder listed.

Now, try the above Selenium trial run again. If it works, great. If not, like me, then do this step:

Navigate to your Browser_drivers_for_automation directory and make your driver executable. In my example here i’m using the firefox geckodriver, but you can do this to any/all drivers in here.

sudo chmod +x geckodriver

See if that works. If it does, the browser will open and you should be good to go now.

Hope this helps

Thanks to all those people answering questions in stackoverflow as usual! 🙂

Categories
Life Skills Nextcloud Technology Tutorial

How to Install Video Convertor App on Nextcloud

Someone complaining that they can’t view an Apple .MOV file on your Nextcloud server? No surprise considering ‘apple is apple’.

Heard about the cool app ‘Video Convertor’ for Nextcloud? Tried to install it and use it and got that ‘requires ffmpeg error?

Same.

Turns out it’s pretty easy to to get it going and here is how:

Let’s do it.

  1. ssh into your Nextcloud box. In my case it’s a nextcloudpi box if that matters…
  2. type this into the command line: sudo apt install ffmpeg
  3. Hit your enter key
  4. Wait a long time while a seriously large amount of stuff installs (and probably this is why it’s not installed by default)
  5. Go back to your video file in the Nextcloud in the browser user interface and note the three dots on the right side of the file name. Click that
  6. Select the video icon ‘convert into’
  7. Choose your options for speed and format
  8. Do it.
  9. Wait again (a lot of waiting here…)
  10. Create a new share of the newly created video file
  11. Test the file (good practice) by sending the share to yourself in a browser that you don’t use so you are seeing what your recipient will see without your login credentials affecting things.
  12. All good? Ship the new link to your recipient

Hope that helps

Categories
Freedom and Privacy Technology Tutorial yunohost

INSTALLING ROUNDCUBE 1.4 VERSION ON YUNOHOST

Thanks to the work of Brian we can now install the newest version of Roundcube on Yunohost.

Why is this so exciting?

  • Newer, fresher UI
  • PGP encryption Functionality with the enigma plugin
  • Mobile friendly skin

In short, it makes your self-hosted email awesome on a mobile too, regardless of whether you have an email app that works or not with PGP

Normally, if this app was an approved app in the Yunohost app list you would be able to simply search it and install it from the app list. Until then we have to install it by the command line but it’s not that scary at all so let’s begin.

Installing it

  1. ssh into your Yunohost box
  2. run this command to install it:
    sudo yunohost app install https://github.com/bhdouglass/roundcube_ynh/tree/testing

Configuring the Install

A bunch of questions will start. Here is how I answered mine but you can adjust as you like. Note, if you press the enter key it will choose the default option for quick installation. Critical step is choosing ‘yes’ when you are presented with the Enigma opion.

This first question is just a warning. You’re brave. Take that risk!

WARNING! Installing 3rd party applications may compromise the integrity and security of your system. You should > probably NOT install it unless you know what you are doing. Are you willing to take that risk? [Y/N] : y

This next step shows all the domains you have configured in your Yunohost box and will ask you which one you want this Roundcube to be associated with:

Available domains:
-domainone.com
-domaintwo.com

Choose a domain for Roundcube (default: domainone.com:

This next question lets you choose which URL folder you want for the mail. I changed mine from the default since I have other things running but you can leave default if you don’t have something already using ‘/webmail’:

Choose a path for Roundcube (default: /webmail): /pgpmail

I’m using Nextcloud calendar stuff so i don’t need CardDav stuff now but feel free to install it if you need it and want to use it:

Install CardDAV synchronization plugin? [yes | no] (default: no):

This is an important one if you plan to encrypt your emails!

Install Enigma messages encryption plugin? [yes | no] (default: no): yes

Testing it out

Once complete, you should now be able to go to ‘domainone.com/pgpmail’ (or whatever your options are) and hit the new Roundcube installation and log in with your Yunhohost email server settings. I recommend doing the following tests each time you get setup with something like this since desktop browser and mobile browser use different skins (sometimes) in Roundcube.

Before beginning, make sure that the public key of each email address has been sent/imported into each side of the email transaction. You can use the ‘import key’ feature in Roundcube’s desktop mode quite easily. At point of this tutorial I haven’t tried importing keys with Mobile browser, so I can’t confirm if that works or works well.

  • Send totally plain text test message from desktop browser to test email address
  • Send totally plain text test message from Mobile browser to test email address

All good? Now with public keys attached:

  • Send new email with just public key attached using Roundcube’s ‘attach public key’ feature – from Desktop browser
  • Send new email with just public key attached using Roundcube’s ‘attach public key’ feature – from Mobile browser

Still good? Now encrypt it!

  • Send new email fully encrypted using Roundcube’s ‘encrypt this email’ feature – from Desktop browser
  • Send new email with just public key attached using Roundcube’s ‘encrypt this email’ feature – from Mobile browser

Everything still good? You should be ‘in business’

Hope you enjoyed and found this useful.

Categories
Freedom and Privacy Technology Tutorial yunohost

HOW TO INSTALL NEW ROUNDCUBE VERSION ON SHARED HOST CPANEL WITH PGP KEYS WORKING

So, you want to not wait for RoundCube to release 1.4 to cpanel, or, you have Roundcube on your cpanel setup and for whatever reason the Enigma plugin that makes the PGP stuff work – isn’t working. Whatever your reason is, the solution is not insanely hard, but it took me about a week and a lot of hours to figure out how to get it going. Hopefully this will save you many hours that I lost! 🙂

Probably this page will work for future stuff too for future releases so I’ll leave the download pages more general.

Before we begin, quick thanks to everyone at Roundcube for really improving the look, feel and security of everything. Really nice upgrades in the UI for version 1.4 RC!

STEP ONE – DOWNLOAD

  1. Get your version (in this case 1.4-rc) from this Roundcube download page
  2. Download compressed Roundcube file to your computer
  3. Upload compressed Roundcube it via FTP (or whatever method you like)
  4. In your cpanel File Manager, Right click on the compressed Roundcube file and ‘extract’ – the file will decompress and extract the directory with the same name in the same directory where you clicked ‘extract’
  5. Locate the newly extracted file and rename to something you like. This will form part of the URL when you log into your email. If you leave it as is the URL will be too long and annoying so changing it to something like ‘mail’ or ’roundcube’ would be smart. Maybe don’t call it ‘webmail’ because most shared hosting uses that? Not sure, I didn’t test that but something more unique would probably be smart.
  6. go to the URL of your domain, with the new directory folder and add ‘installer’ at the end and this will start installer. You can read details on the Roundcube installation page and we are now at step ‘Configuring Roundcube’

Create your MYSQL database for Roundcube

You’ll need a database to make Roundcube work. Although this kind of stuff seems scary, it’s not that scary as long as you don’t delete stuff that’s already there that you don’t know about. In short, all you are gong to do is create a database, create a user for the database, and then glue the user to the database. That’s it.

Before you begin this part you should have some kind of password manager software setup, I think, so you can create strong passwords and not lose them. Otherwise, ‘do it the way you like’.

Here’s how:

  1. in your Cpanel home panel, go to ‘MYSQL Databases’
  2. in the top section, Create New Database, give your new database a decent name like ’roundcube_abc’ (no one sees this stuff, it’s just back end). You will note that it will automatically append something to the front. That will be part ofyour database name, not just the part you are typing in the field.
  3. Click ‘Create Database’ button
  4. Scroll down to the MYSQL Users section and in the ‘Add New User’ section at the top, create a new username with a strong password. You can use the password generator and then make sure to save it safely. You will note that it will automatically append something to the front. That will be part ofyour username name, not just the part you are typing in the field.
  5. Click ‘Create user’ button

Now you have both a database and a username. the last steps are to glue them together.

  1. Scroll down to ‘Add User to Database’
  2. From the ‘User’ dropdown, select the user you just created
  3. From the Database dropdown, select the database you just created above
  4. Click the ‘Add’ button

It will take you to another screen where it asks what permissions you want to give this user in this database. You will give it all permissions which should be the default (all boxes selected).

Confirm these changes.

Installing Roundcube

Really you should read this entire page and learn a bit as your setup might be different and it’s good to review the items they mention, although most or much of it is for people who have full access to their servers, not cpanel people.

Once you start the installation process, there really isn’t too much documentation on how to actually set it up. It will start a kind of ‘installation wizard’ but not a lot of help is there during the process. You’ll need to have the following items near you before you begin:

  • The email credentials that will be using Roundcube including: server names, type of email (ie. SMTP, IMAP), security protocol of each, ports for each. You can get all this from your email provider.
  • The database name, user name, and user name password for the MYSQL stuff you created above

Once all this is in hand simply walk through all the fields and do your best. The details of that part is out of scope of this tutorial but there is one vital point which you absolutely must seclect in order to use encryption and PGP and that is, in the plugin section you must select the Enigma plugin which will add all the functionality.

Again, be sure to install the the Enigma plugin.

Fixing the enigma_pgp_homedir not specified error

After installation is complete and everything appears working, if you were to go to the settings and to PGP keys you will be greeted with a warning that enigma_pgp_homedir is not specified, or, if you tried to simply put the path in there as if it were a full control server, it probably won’t work. At least that’s what happened to me and the purpose of this tutorial.

The solution to properly point Roundcube to a secure folder was pretty hard to figure out but now that it’s figured out should be pretty easy for you.

You should apparently not put your pgp key folder into the enigma plugin folder, or in the document root folder. Frankly I”m not skilled enough to know all the whys but I’ve learned to trust people smarter than I. So, I put the directory in the main directory of my domain where roundcube is. You can mirror this for simplicity and probably should. So, here go the steps:

  1. In cpanel File manager go to: public_html/yourdomain.com
  2. Create a folder for your keys. For this tutorial I’ll call it ‘keez’ but you can call it whatever you want. Make sure permissions are 0755 (they should be after you create it)
  3. In File manager, navigate to this location: /yourRoundCubeInstallationFolder//plugins/enigma/
  4. Locate the config.inc.php.dist file
  5. Touch it, right click on it, and rename and remove the ‘.dist’ from the end of the filename and save the change
  6. Touch the newly renamed file again and right click and ‘edit’. This will open the Cpanel text editor.
  7. It will give you a warning that by editing you can break everything, which of course we know, so, click edit again and reall stick it to the man!
  8. Scroll down in the code stuff until you see this section:

// REQUIRED! Keys directory for all users.
// Must be writeable by PHP process, and not in the web server document root
$config[‘enigma_pgp_homedir’] = null;

Now is the fun part. All you have to do is enter in the path that points to your keys folder that you created in step 2 above.

Mine now looks like this:

$config['enigma_pgp_homedir'] = realpath(__DIR__.'/..'.'/..'.'/..').'/keez';

As long as you put your keys folder in the main directory of your domain and named it ‘keez’ this line shoud work. If you have your folder somewhere else you’ll have to adjust accordingly. What I learned was each instance of '/..' moves up one level directory. So this command is saying ‘you will find the ‘keez’ folder by going up 3 levels from where you are now”.

Once you have adjusted this one line of code, click the ‘save changes’ button in your cpanel editor.

Go to your Roundcube email, go to ‘settings’ and go to ‘PGP Keys’ and you should now have working PGP functionality working in your Cpanel Roundcube, plus, you are running the new mobile-friendly version 1.4 now that all the paths are working.

I hope this helps!

Categories
Freedom and Privacy Life Skills Nextcloud Technology Tutorial Ubuntu

HOW TO MAKE A NEXTCLOUD PI BOX WORK AS REVERSE PROXY TO YUNOHOST

Background

The situation was that I wanted to test out the very cool project Yunohost but I already had Nextcloudpi (another awesome project!) running on my local network. I already had a DDNS service (No-ip) running which was pointing to my Nextcloudpi (“NCP” moving forward) box, and a second DNS service that I set up which pointed to my router for the purpose of Yunohost (“YH” moving forward). You can read about that cool DNS solution in my other blog post, by the way, as it works really well and gives a bit more power.. and it’s free.

The problem was that ports 443 and 80 were being used by NCP but YH needed them as well. The only options appeared to be:

a) change the ports of one of the machines (complicated because clients outside of the LAN in the world webs won’t know those ports) or
b) figure out what a ‘reverse proxy’ is and then make it work

The challenge was that NCP was using Apache whilst YH uses NGINX – both of which are capable of reverse proxy. So, in order to do this I ended up doing some learning of both although it turns out it wasn’t really needed after all. C’est la vie…at least I learned some things!

At the end of the journey of trying about 10,000 different settings in the Apache default configuration file that comes with NCP (and other Apache installs) called “000-default.conf” it started working after adding just two lines to my configuration which seemed not to be in any other tutorial online for some reason. The key two lines that were needed were:

SSLEngine On
SSLProxyEngine On

Without those two lines it would just never work even though the rest of my settings were right.

Ok, enough of my hard journey story, let’s log the actual configuration and steps so that anyone who wants to do the same setup can save the pain!

Assumptions

Before we begin, I will assume that you already have the following set up:

  1. Server A (in my case NCP) running Apache which is already successfully reachable and working from the outside world. Through this machine Server B will be reached.
  2. Server B (in my case YH) running whatever (I think) but in my case it’s running NGINX and this box is the one we are trying to make visible to the outside world through ports 80 and 443
  3. You have a domain (nameofyourdomain.com in this tutorial) which you own and which is already successfully hitting your router (You can test by pinging the domain and seeing the IP address of your router show up). You can do this with my other tutorial mentioned above as well. You can also get a free ‘domain’ from services like No-ip if you don’t care what the domain looks like.
  4. You have full access to SSH into both machines, but in this case Server A is the critical one.
  5. You are using an Ubuntu environment and have know how to open a Terminal and use it (roughly)
  6. You are willing to learn and try things if this doesn’t perfectly work as per this specific example. I’ll give you a few resource links as well to help you in case your set up needs tweaking.

Let’s Begin – Setting up Apache Default Config on Server A

  1. ssh into Server A (format ssh username@your.IP.Address )
  2. Change directory (cd) to your Apache2 sites-available directory. In my case it looks like this but if you aren’t using NCP it might be different
    cd /etc/apache2/sites-available
  3. Type this command to back up your Server A apache settings. If you mess anything up you can restore this one and delete the default and rename it back to original name.

sudo cp 000-default.conf 000-default.backup

  1. Check to make sure the new file with .backup is showing up by typing ‘ls’. If it’s there then proceed.
  2. Copy the sample configuration below into your clipboard
  3. Open the default Apache config file with this command (if you haven’t used nano before probably good to do a quick online overview) for editting:
    sudo nano 000-default-conf
  4. you may have some settings already in this file (you should) at the top. Scroll down to the bottom of whatever is there and then paste in the sample you have copied from below with the control + shift + v (If you don’t hold shift it won’t paste)
  5. Go through the newly-pasted configs and adjust to your settings changing domain names and ip addresses to yours.
  6. Control x to save and exit, ‘y’ to save modified buffer and ‘enter’ key to write your changes
  7. Restart apache with this command to see if it works (this will shut down whatever stuff is running on Server A so probably good idea to do this wisely if the server is currently being used by others…:

sudo systemctl restart apache2

If you get nice silence from your terminal, and no ‘journalctl’ messages, then things are going the right direction.

Run Let’s Encrypt Manually for SSL certs on Server A

For this step, to be honest, I’m not sure if you need to do it because certs are already on both boxes for NCP and YH. But you might not have that so I’ll provide the steps since after I did them nothing was worse and everything was working… I would love to get some feedback on this step.

  1. Install Let’s Encrypt tools:
    sudo apt-get install python-certbot-apache
  2. Run it
    sudo certbot --apache -d example.com -d www.example.com

Let’s Finish – Test Server B

Go to your domain from outside your LAN (just to make sure you are getting a real test) and try to hit Server B. I find mobile phone data plans are good for this kind of testing, otherwise, call your grandma and ask her what happens when she goes to nameofyourdomain.com…

If it works, you’re done.

If it doesn’t you might need to tweak your settings.

Sample Configuration – copy this and adjust to your set up

Your IP address will obviously be changed to the correct one where your Server B is. Copy everything in the code block below.

 <VirtualHost *:80>
    ServerAdmin name@nameofyourdomain.com
    ServerName nameofyourdomain.com
    ServerAlias www.nameofyourdomain.com

   ProxyPreserveHost on
   ProxyPass / http://192.168.1.37:80/
   ProxyPassReverse / http://192.168.1.37:80/

</VirtualHost>

#Listen 443

<VirtualHost *:443>

    SSLEngine On
    SSLProxyEngine On

     ServerAdmin name@nameofyourdomain.com
     ServerName nameofyourdomain.com
     ServerAlias www.nameofyourdomain.com

     ProxyPreserveHost on
     ProxyPass / https://192.168.1.37:443/
     ProxyPassReverse / https://192.168.37:443/
</VirtualHost>

FULL Sample Configuration Reference (DO NOT COPY THIS ONE)

This is what my config looked like when everything was done and working.

The ‘Rewrite engine’ stuff here was added by Lets Encrypt when it was run so it ‘should’ appear in your config after you run it after initial settings have been added. Same with the ‘Include’ stuff and the SSL certificate stuff at the bottom of the second entry.

<VirtualHost *:80>
    ServerAdmin name@nameofyourdomain.com
    ServerName nameofyourdomain.com
    ServerAlias www.nameofyourdomain.com

   ProxyPreserveHost on
   ProxyPass / http://192.168.1.37:80/
   ProxyPassReverse / http://192.168.1.37:80/

RewriteEngine on
RewriteCond %{SERVER_NAME} =nameofyourdomain.com [OR]
RewriteCond %{SERVER_NAME} =www.nameofyourdomain.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

</VirtualHost>

#Listen 443

<VirtualHost *:443>

    SSLEngine On
    SSLProxyEngine On

     ServerAdmin name@nameofyourdomain.com
     ServerName nameofyourdomain.com
     ServerAlias www.nameofyourdomain.com

     ProxyPreserveHost on
     ProxyPass / https://192.168.1.37:443/
     ProxyPassReverse / https://192.168.37:443/

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/nameofyourdomain.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/nameofyourdomain.com/privkey.pem
</VirtualHost>

Random Keywords and messy spam from the Journey

This next section is merely a copy/paste of all the steps I was trying to try to get this working. The purpose is not to follow any of these instructions but merely to leave as keywords in hopes that other people trying the same things will end up finding this blog and save themself the pain! 🙂 So, don’t use the next section for any form of tutorial but feel free to read and learn.

  1. set up individual virtual host conf files on box 1 else:

We were unable to find a vhost with a ServerName or Address of mydomain.ca.
Which virtual host would you like to choose?

1: nextcloud.conf | mydomain.hopto.org | HTTPS | Enabled
2: ncp.conf | | HTTPS | Enabled
3: 000-default.conf | | | Enabled

Select the appropriate number [1-3] then [enter] (press ‘c’ to cancel):

Select the appropriate number [1-3] then [enter] (press ‘c’ to cancel): c
No vhost exists with servername or alias of mydomain.ca. No vhost was selected. Please specify ServerName or ServerAlias in the Apache config.
No vhost selected

hmm.

finding apache config…

seems like one shouldn’t mess with this… and that lets encxrypt probably does it for you

  1. sudo apt-get install python-certbot-apache (apparently not installed on ncp somehow..)
  2. created basic conf file in /sites-available
  3. restarted apache – worked
  4. added symlink to sites-enabled, restarted apache, breaks
  5. run certbot without enabled…with usual
    sudo certbot –apache -d example.com -d www.example.com

pi@nextcloudpi:/etc/apache2 $ sudo certbot –apache -d mydomain.ca -d www.mydomain.ca
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Cert not yet due for renewal

You have an existing certificate that has exactly the same domains or certificate name you requested and isn’t close to expiry.
(ref: /etc/letsencrypt/renewal/mydomain.ca.conf)

What would you like to do?

1: Attempt to reinstall this existing certificate
2: Renew & replace the cert (limit ~5 per 7 days)

choosing option 2

fail. same error above

now trying to go back to simply 443 config in 000-default but wtihout ssl engine stuff.

now running:
sudo certbot --apache -d mydomain.ca -d www.mydomain.ca

this is something… progress….

the bad part:

Failed redirect for mydomain.ca
Unable to set enhancement redirect for mydomain.ca
Unable to find corresponding HTTP vhost; Unable to create one as intended addresses conflict; Current configuration does not support automated redirection

the good part

IMPORTANT NOTES:

  • We were unable to set up enhancement redirect for your server,
    however, we successfully installed your certificate.
  • Congratulations! Your certificate and chain have been saved at:
    /etc/letsencrypt/live/mydomain.ca/fullchain.pem
    Your key file has been saved at:
    /etc/letsencrypt/live/mydomain.ca/privkey.pem
    Your cert will expire on 2019-09-14. To obtain a new or tweaked
    version of this certificate in the future, simply run certbot again
    with the “certonly” option. To non-interactively renew all of
    your certificates, run “certbot renew”
Categories
Freedom and Privacy Life Skills Technology Tutorial

SETTING UP EMAIL WITH YUNOHOST AND CLOUDFLARE

In a previous blog post I set up a Yunohost (“YH” moving forward) box with a script so that it would report it’s location back to Cloudflare (“CF” moving forward) automatically using a cron job entry on the box and a cool piece of free software called ddns-cloudflare. That blog was to make sure the website stuff (ie. WordPress blog, Nextcloud, etc) would work. The other neat part about setting up your YH box this way, I was thinking during the process, is that (I guess but haven’t tested yet), you could just unplug it and plug it in at another physical location (with the right ports open at that location, of course) and it should just start ‘magically working’. This would be a real selling feature for getting ‘off the grid’.

Now to attack the part that most people like me avoid – EMAIL!

We have all heard that email servers are complicated and stressful, but, with the CF-YH combo – once I figured it out – it now seems much easier than I had expected. But there weren’t any specific blogs out there for me to follow so I decided it would be super helpful to write one to help others avoid what I just went through.

This tutorial will connect CF to your YH email and give you a few tips to test as you go until it’s all working, since there are a few things in both CF and in YH that are a bit ‘weird’ I discovered. My hope is that this tutorial helps you get setup faster and easier.

This tutorial assumes you already have a CF account setup with the settings from the previous tutorial (www and A record stuff).

KNOWING WHERE YOUR YUNOHOST SETTINGS ARE

You will be able to find the private and unique details for your own Yunohost installation in the following section of your user interface:

Domains / nameofyourdomain.com / DNS Configuration

When you click this it will open up a pane that has all your records from the previous tutorial but also the recommended email settings. If you are like me, none of it will make sense at all.

The parts you are going to need to match up to CF are:

MX, DKIM and DMARC

The way in which you input them into CF is more than half of the battle, and the part where this tutorial should save you about 3 days of messing around.

First, let me give you a link to Cloudflare’s own support page on this topic. This will also give you a list of pretty much any kind of entry you might need in your own setup, if it’s more advanced than this tutorial. It also shows you how to create a records in your CF DNS settings. Here’s the link.

Now that you know how to enter a record in general, let’s enter them.

I’m going to display this like this:

MX RECORD

  • WHAT YH SHOWS IN DNS CONFIG PANE: @ 3600 IN MX 10 mylataylor.ca
  • HOW TO ENTER AND PASTE IT INTO CF
  • TYPE: MX
  • NAME: nameofyourdomain.com
  • VALUE: SERVER: nameofyourdomain.com PRIORITY: 10
  • TTL: AUTOMATIC

DKIM RECORD

  • WHAT YH SHOWS IN DNS CONFIG PANE: mail._domainkey 3600 IN TXT “v=DKIM1; h=sha256; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA…super_duper_long_long_thing”
  • HOW TO ENTER AND PASTE IT INTO CF
  • TYPE: TXT
  • NAME: mail._domainkey
  • VALUE: v=DKIM1; h=sha256; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA…super_duper_long_long_thing (NO quotations)
  • TTL: AUTOMATIC

DMARC RECORD

  • WHAT YH SHOWS IN DNS CONFIG PANE: _dmarc 3600 IN TXT “v=DMARC1; p=none”
  • HOW TO ENTER AND PASTE IT INTO CF
  • TYPE: TXT
  • NAME: _dmarc
  • VALUE: v=DMARC1; p=none
  • TTL: AUTOMATIC

It was explained to me that I also need to check RDNS, but I have not had any problems yet, and I’m not sure what this is nor how to do it. If you want to add this instruction in the comments that would be great.

If you refresh your page in CF and notice that an orange cloud has re-appeared from grey status, you may not have updated your .yml zone file correctly from previous tutorial. Your script might be updating the DNS records and accidentally forcing it back on. And this will stop your emails from working (the orange cloud). Go back to that tutorial and review the script yml config file setup and make sure you got he hashtags on the right lines…

At this point, it’s the moment of truth: will you be able to send and receive emails?

CREATE ACCOUNTS (AND EMAIL ACCOUNTS) IN YUNOHOST

This part created some issues for me because there may (or may not) be either a bug or an interface issue in the YH account setup. It appears, as you create the YH user that the email can be separate, however, from my experience, you should keep the username and the ’email name’ to be exactly the same – let YH auto-fill it and keep that as your email name. So, if you want your email to be johndoe@nameofyourdomain.com, make your YH username as johndoe at the top and let that auto-fill into the email field below. It seems like YH can’t take periods/dots in the username so john.doe won’t work. There appears to be email aliases that are supposed to work so probably you can figure this out but for me, for this tutorial, I would just avoid dots/periods, keep a simple username and make sure it auto-fills into the email field.

TEST YOUR SETUP

Once your username / email is set up in YH, now move on to test the email, in the client of your choice, but I strongly recommend Thunderbird to at least test to make sure things are working because it definitely works, I can confirm. Once this test is confirmed and you can send, receive emails with a basic thunderbird setup, then can feel confidence about all your settings above.

THUNDERBIRD SETUP

Literally, just follow this link exactly. If your settings are right, it will work. If they aren’t, they won’t. Also, be sure to wait at least 30 minutes for your cron jobs (if you are continuing from the Cloudflare tutorial) to run because the cron job can mess up your settings as well, I discovered. Once you have run a cycle or two of cron jobs and all is well, go wild with the hottest new technology – email…

TIP: when you get to the manual config, Thunderbird puts a period / dot before the serverhostname which is easy to miss. If your email is in the main domain root, then make sure to remove these dots.

Now send a test email to another account you have access to. One important thing to check is that you aren’t ending up in spam folders…

TIP: If you press send on your test email and it hangs on sending, go into your account settings of Thunderbird and you might have some old Outoing (SMTP) servers from previous YH email tests in there. I found after I deleted these old test servers and tried again, it worked perfectly, but if there were other ones in there it hung and failed.

Assuming you got your test email, now send a reply back to it and make sure you get that too. If you’re excited and bored or both, do this step over and over again perhaps with nursery rhyme verses…but make sure no one is watching you… I can’t emphasize enough that you need to wait at least a cycle or two of your cron jobs running to make sure it’s not messing things up over at Cloudflare.

All good? Nice. Another consideration now that you are a warrior hosting your own email is that by using Thunderbird you can back up your emails easily enough by simply connecting and synching your emails across multiple devices.

UBUNTU TOUCH DEKKO SETUP

Now let’s set this up on our Ubuntu Touch device with Dekko.

  1. Select the left hamburger menu
  2. Select the top right settings cogwheel on dark panel
  3. Select ‘mail’
  4. Select ‘accounts’
  5. Select the top right + plus sign
  6. Select IMAP
  7. Enter ‘whatever you want’ for the first two name options
  8. IMAP hostname: overwrite example with your yunhost server email location
  9. ENCRYPTION: should already default to this: ‘force encryption (SSL/TLS)’. if not, do it.
  10. username/password: auto-filled from first step
  11. authentication: change to ‘login’ (defaults to ‘plain’)
  12. SMTP server: overwrite example with your yunohost server email location
  13. SMTP port: 587
  14. ENCRYPTION: ‘use encryption (STARTTLS)
  15. AUTHENTICATION: change to ‘login’ (it defaults to ‘plain’)

Last and final and very important or your outbound email will literally break for this account and, I think, all your email accounts. You need to go back into the settings for this new email address and to this step:

  1. Select top left hamburger menu
  2. Select top right settings cogwheel on dark panel
  3. Select ‘mail’
  4. Select ‘accounts’
  5. Select Your newly-created yunohost email account
  6. Select Outgoing Server
  7. Scroll down under the LOGIN field and turn on the switch that says ‘Authenticate from server capabilities’
  8. Press left arrow at top of screen to save settings

CONCLUSION

Now, you should be in business sending and receiving emails from a Yunohost server, in your house, using Dekko email client on your Ubuntu Touch device.

Categories
Freedom and Privacy Life Skills Nextcloud Technology Tutorial Ubuntu

HOW TO SET UP YUNOHOST WITH YOUR OWN DOMAIN (USING CLOUDFLARE)

EDIT 19/06/12 – made some tweaks to this after realizing a few small errors. Sorry if you followed before June 12 🙁

I really wanted to self-host a kind of ‘family box’ which would allow me to have self-hosted email, Nextcloud, websites, and a few other basic things and not have it running on someone else’s server. During the process of searching I came across Yunohost (Pronounced “Why You No Host?”). I installed it on an old test box (super old) with their own documentation and it was really quite simple, especially if you have done any kind of operating system installation before.

My specific goal was to make it all work with a domain that I own (nameofyourdomain.com for this tutorial). I feel that having a strange email address (the default Yunohost email setup looks weird and is awkward) is of little value for most people so this step must be overcome to become a viable solution for myself and other people I know.

OPTION 1 – DIRECT WITH REGISTRAR (EASIEST)

If your registrar gives you full control of your DNS, CNAME, MX etc settings you might not even need this Cloudflare-Yunohost tutorial. My registrar didn’t allow me to do what I needed so I went to the next step. I don’t have enough experience to speak about the different registrars and their settings so research that yourself if you want. Otherwise, move on to this exciting Cloudflare-Yunohost setup…

OPTION 2 – WITH A CLOUDFLARE SCRIPT

As mentioned above, my registrar didn’t make it clear how to do CNAME stuff and mess with MX records, etc, so I ended up searching high and low for an open, free and reliable solution. Thanks to the free software community, I was pointed towards Cloudflare. People I trust and like consider Cloudflare to be ‘good guys’ and that was enough for me to trust and try.

Forgive my limited understanding and description, but I’ll do my best here: Cloudflare is a super robust ‘web traffic controller’ which gives the website admin person (since you are installing Yunohost that is you now!) really powerful control over how data moves to/from the domain/servers. They have cool controls and a nice interface too. Anyway, they have a free account you can start which allows you to do everything in this tutorial and through the process you’ll get a chance to see how nice Cloudflare (“CF” moving forward) is too.

Note: this tutorial assumes you are using Ubuntu or at least have the same terminal commands.

PRE-FLIGHT BULLET POINTS
We’re going to do this:

  1. Tell your domain registrar to point traffic to CF
  2. Tell your Yunohost (“YH” moving forward) box to point to CF
  3. Stick a free software script on your YH box that automatically tells CF where your YH box is every 30 minutes (in case your IP address changes)(replaces dynamic dns service need…)

THE FLIGHT

  1. Do the YH setup as per yunohost. Make sure your router’s ports are open! Check this page about ports and note that some ISPs will (unethically?) block you from using port 25 (email) and in this case you might be completely out of luck or have to change your ISP so you should check that first on this page. If port 25 is blocked you should be able to use everything except email (nextcloud, wordpress, etc should work) so it’s not completely without hope… TIP! If you get your domain setup first in YH sometimes Cloudflare will be able to magically import all your stuff automatically helping you avoid the manual inputs over at CF
  2. Get a Cloudflare account
  3. Log into your domain registrar and change nameservers to the ones shown in your CF account. This guy’s video is pretty good if you haven’t done it before.
  4. Take note of this project, which is the script which will automate the DNS updates stuff. Special thanks to the programmer!
  5. ssh into your yunohost box by typing (where 123 stuff is the local IP address of your YH box):
    ssh admin@123.123.123.12
    This will get you into your YH box where you can stick the script files into your home directory.
  6. Clone the cloudflare-ddns project files above into your YH box by typing this into your terminal (TIP! do NOT use ‘sudo’ here!):
    git clone https://github.com/adrienbrignon/cloudflare-ddns.git
  7. Then change to your new directory:
    cd cloudflare-ddns
  8. Then change to the zones directory within:
    cd zones
  9. Then copy the example yml file so that it duplicates and is named to your own domain:
    cp example.com.yml nameofyourdomain.com.yml
  10. Now open the file so you can edit the contents:
    sudo nano nameofyourdomain.com.yml
  11. Now edit the ‘admin@example.com’ line and change to the email you registered your CF account with
  12. Change whatever it says to the right of cp_api_key: to your cloudflare api key. There is a link of how to find that right in the terminal window but in case it stresses you out and you miss it here is the link
  13. Change zone name (cf_zone:) to: nameofyourdomain.com
  14. Set all the DNS stuff so that the file looks like this where the # signs are ‘comments’ telling the script to forget about this part:

Only write the subdomain (‘ddns’ for ‘ddns.example.com’)

cf_records:
– ‘@’:
type: A
# proxied: true
log: ERROR

  • ‘www’:
    type: A
    # – ‘ddns’:
    # type: AAAA
    # ttl: 300
    # proxied: false
    # log: INFO

If you compare to the example file you can see the changes.

I just commented-out with hashtags the AAAA stuff since apparently I don’t need it (a great contributer told me) as well as the smallest but most painful one, the ‘Proxied: true” line! This one, if you don’t put a hashtag in front, will, every time your cron job runs, tell CF to make CF the controller of the DNS and then basically shut down your websites and your email will also stop working. Then you have to go in and turn the orange cloud back to grey again.

  1. Save and exit the nano editor with control x and ‘yes’
  2. Then move back up one level in the directory so you can run next command:
    cd ..
  3. Run a one-off test to see if it’s working as per the usage docs with this command
    python cloudflare-ddns.py -z nameofyourdomain.com

if it’s ‘working’ you should:

a) see a success report back from your terminal that looks something like this:

2019-05-31 05:16:15,165 | INFO | The record 'www.nameofyoudomain.ca' (A) is already up to date

and

b) should be able to now go to your CF account and see the IP address of where your YUNO-box is / public-facing router listed in the DNS area. You can check this part by going into your router (or one of those ‘what’s my IP address?” websites) and compare your router’s IP address with the IP address in Cloudflare – they should be the same now. If not, assure that the orange cloud in CF is turned off by clicking. It will change to a grey colour when off and this is what you want.

If everything is looking good, let’s move on to making this update process happen automated in the backgroun since it would not be fun to have to run this test script every day or a few times a day!

Now that your config file for this script is all good, let’s go and do the cron job thing

  1. Type:
    crontab -e

Probably if you haven’t done this already it will say it’s blank and give you two options. choose ‘nano’ because it’s easier (option 1) and not VIM because it’s brutal and hurts

  1. The script provided in the usage page for doing this next step assumes you know what you are doing, that you understand Linux file structures and paths and even cron jobs. I didn’t. So, I’m going to spare you the pain here (you can read the pain below in the bonus section(s) if you are bored or like learning) and tweak this script so that you have a higher chance of this working. First, this is what was provided from the usage page:

Every 30 minutes, update my Cloudflare records.

*/30 * * * * python /path/to/cloudflare-ddns.py -z example.com

If you just simply ran this tutorial that means your Cloudflare directory that you git-cloned in step 6 above is in your home directory. However, you need to add in the user into the path for this to work properly. Also, until you know this thing is working, I would advise you add in the MAILTO option above the script so that you can get a few emails for a few hours confirming it is or isn’t working. You can go back in and remove or comment out with a # the MAILTO line (or comment it out with a hashtag) after you are sure everything is working.

So, here is what I did that finally made it work

Every 30 minutes, update my Cloudflare records.

MAILTO=myemail@myreliablemail.ca
*/30 * * * * python /home/admin/cloudflare-ddns/cloudflare-ddns.py -z nameofyourdomain.com

If everything is working, you’ll keep getting ‘success’ emails that look like this everytime the cron job runs:

2019-05-31 05:30:05,942 | INFO | The record ‘www.mylataylor.ca’ (A) is already up to date

At this point I went back in (see step 17 above) in and hashtagged out the MAILTO= line so the emails stopped coming every thirty minutes.

ADJUST YOUR CLOUDFLARE SETTINGS!

  1. CRYPTO/SSL SETTING
    This one took me an addition day to figure out. I was getting continual TOO_MANY_REDIRECTS errors when trying to access my Yunobox. The problem was solved by the click of one box in my Cloudflare settings as follows:

crypto / SSL : change to ‘FULL’ in the dropdown.

  1. TURN YOUR ORANGE CLOUDS TO GREY
    If you don’t do this step, your email and a few other things won’t work. Just click the orange clounds in your DNS section so they turn grey. That’s it.

DO YOUR LETS ENCRYPT SSL CERTIFICATE ON YOUR YUNHOST BOX

If you try to do your letsencrypt SSL cert before these steps are done, it won’t let you (from my experience) But at this point it should all work. In your YH admin interface, just go to domains, nameofyourdomain.com, SSL certificate, and then ‘install lets encrypt’

CONCLUSIONS

Now your Yunobox should be automatically reporting back your router’s IP address to Cloudflare and Cloudflare is routing your website traffic through its nameservers, etc. As long as your ports and certificates are working, you should now be able to start using it with your own domain.

Now that this is done and you go to your new domain and nothing is there that’s because… there is nothing there. Go figure. So you have to install an ‘app’ (ie wordpress, nextcloud) through the Yunohost app area of admin. I’ll do a separate blog on that probably, but it’s pretty easy.