Nice long title. Thankfully the speed of getting your first big upload to your new Nextcloudpi (NCP) server won’t be so long, thanks to this tutorial! By spending 10 minutes and doing this tutorial you will be uploading 95% faster (that was my experience).
Assumptions Before Beginning
- You have full 100% admin access to your NCP (ie. you are the master admin and probably created the server and installed it, or are close friends with the person who did)
- You have SSH access to your NCP, and you know how to SSH into your NCP. If you don’t… you’ll need to research that first.
1. Confirm the Username in NCP Who Will receive the Big File Shipment
This major upload will need to be associated with a username. In my case, I have created a ‘master-master user’ for this kind of reason. So I will be shipping this big upload to my ‘master-master user’ so that after it’s done that user can assign which files are to be shared with whom (and how). I think this is the right way to do it, even if you are the admin yourself. Topic is open to discussion, but that’s how I roll…
Make sure this user exists in NCP is the key point.
2. SSH into your NCP
NOTE! Apparently you can do steps 2, 3, and 4 via the NCP web admin so this means you might not need SSH, plus it might be easier. I won’t have a chance to test myself for a while but try that out first maybe! Otherwise, learn SSH and do the next few steps the way I write.
3. Setup Samba in NCP
The first screen is informative and the ‘yes/no’ answers don’t make sense if you read it for grammatical sense. Just chose ‘yes’ which means “I understand that I have to force NCP to ‘scan’ the files when I’m done putting files on the box” This has now (Dec 2018) been fixed by the developers and it now says ‘I understand’ in the option box. Nice and thanks!
You will see:
Type ‘yes’ (no quotes) overtop of ‘no’ in ‘active’ and type in a strong password. You will use this password later and probably you don’t want to give this to anyone else because if you do, that user can go in and mess with other people’s files (I think) Use the Tab key until you arrive at ‘yes’ and press ‘enter’
It should automatically create the ‘samba shares’ for each username you have already put into the system. This means that every user in your box can also access files on the cloud this way and not just with a nextcloud client user features. But the main point is that we’ll be able to move files quickly across the Local Area Network (LAN)
Once this part is done, press any key it will bring you back to your ncp-config screen.
Tab twice until you hit ‘finish’ and then press enter. That will bring you back to your terminal.
4. Do your Samba Shares (and I don’t mean the dance….)
In this example I will be using Ubuntu desktop, so if you are using some other operating system – tough bananas – and you’ll have to search some other tutorial about how to connect your computer to your NCP using Samba.
First, open Nautilus (also called ‘Files” if you mouse-over it. The thing that lets you browser your files on your computer and looks like a file cabinet.
Next, go down to ‘Other Locations’ on the left panel and click it. In this case we want to use the LAN IP address because that’s the whole point of this exercse – fast transfer across LAN instead of going through the internets…
As soon as you enter it in, assuming your box is on, it will find all those usenames and folders for them automatically. Double click the one you want to dump all the files into (probably your master-master admin account). The next part, although it seems easy – is not! But the reward is great so let’s do it.
I realized that what you need here for the username is indeed the NCP username but, but, but.. the password is the one you created in step 3 above! So tricky but alas…
5. “You like to Move it, Move it.”
Let’s move the files now. In Nautilus, middle mouse click wherever your main dump of files are. That will open up a new Nautilius tab from where you can drag, drop stuff into the other tab you just logged into. I just find this a nice and easy way but you can drag drop files there however you like.
Now, select everything you want to move and move them into your NCP user’s Nautilus tab.
Note: You should consider doing this piece by piece unlike me who tried to move 13 GB at a time. You don’t have an easy way to check the progress in this way so consider doing these moves small pieces at a time so you can see progress more easily.
While this is file move is happening, read on to the next section because you’ll have to tell NCP to scan the files when it’s done.
6. Scan the Files
After the move is completely finished from step __ above, in your Nextcloudpi web admin area, scroll down to the ‘nc-scan’ section and run it. It took much less time than I expected. It quickly scans all the files associated with all the users and (I guess) says ‘hey, there is a file here connected to this user’. After running the scan NCP is ready to roll.
7. Start your sharing
Log into the account you just put the files in and start sharing as you like and normally do.
We here at wayne(outthere) hope this made your day shiny and bright. Have a nice day.
The premise of this tutorial is to get a nextcloud instant back up and running while throwin away all the user data and nextcloud customizations of your last install but while keeping all the file stuff for a new upload. In my case, we have a ‘small operation’ and it’s not a big deal to just do a completely fresh install and put the files back. It annoys a few people because they have to do a brand new big sync, but it’s workable and assures that the install is truly ‘fresh’. This might work for other small business and families so I figured I would just document the workflow since it’s helpful for me to not forget something or lose something.
1. Notify others to backup
This step is important because the other people might have put some personal files in their user account that may no longer be on their local machine. If you do this wipe, they might lose the only copy of the file. Notify them to download to their local machine every file they might want letting them know if they don’t, it will be wiped
2. Backing Up
If you are using a local client, this will be easy: just copy/paste contents of the sync folder onto a big hard drive. If you are using the online version, download the files to a local machine.
3. Notify Others to Clear out their Local Sync Folder
If there are others using the NC instance, tell them to clear out the drive they are using now. They can do this after they have confirmed the files are safe in the step above ‘backing up’. If you are the main admin and are uploading a big chunk of files for others to share (ie. company directory) just leave everything where it is and it will make it faster to get everything back up
4. Clear Out the Hard Drive Upon Which Your Nextcloud Will Store Files
This is a bit more complicated than we can write here but depending on how you setup your NC your database files may be on the hard drive itself. To assure everything ‘just works’ (better) access the drive, clear out absolutely everything on it. Remember, make sure your drive is backed up (easy to forget) before doing this step! If this is your second install (or more) you may, like me, find it easier to SSH into the cloud and remove the files that way, if you have access. This allows me to just leave the drive as it is, mounted, and wiped clean. Again, depends on your setup. The key is that you will wipe all the ‘ncdata’ and ‘ncdatabase’ directories from the drive if they are stored there.
5. Do a Completely Fresh Nextcloud Install
Self explanatory, but there are many ways to do this depending on your setup. This is where you jump to your ‘nextcloud installation tutorial’…
6. Create a Nextcloud ‘Master Master Admin’ Account.
Welcome back! How was the install?? I do this step because I want an account over top of my own user account to do higher level admin stuff. I like to do this so I can share stuff with people on a more overarching manner – including my own user account. Now, with your NC ‘master master’ admin account create a ‘working admin’ user/password which will be used for high level controls and also creating new users, etc. Just type ‘admin’ in the groups box and save the user. Once created, log out.
So just to review you will have:
- Main Nextcloud master-master admin (for doing stuff on a purely technical basis for the cloud)
- Master-Master account (for doing permission stuff over files and users)
- Your account (created by Master-Master)
Log Back in as New Admin User
Recreate All the User Accounts with the Admin Account You Just Created
Recreate all the user name accounts as they were before. They will have new passwords when created, or, if you are family and have their passwords you can even imput their passwords for them as they were before.
Upload the Files
If you are doing a massive upload (ie. small business with lots of files) it’s better to use a desktop client app I have found. The web interface seems to crap out if the upload is insanely large and this can cost you a lot of time. I recommend getting a laptop/desktop client setup and syncing that way.
Useful Note about super huge uploads
In the past I have found out that I was accidentally uploading to the wrong hard drive due to a setup issue. I recommend using SSH and going into your NC instance and looking at the hard drive once in a while to see if there is data going on while it’s syncing. On Ubuntu I just run this command every 3 minutes for the first 10 minutes to make sure everything is alright:
sudo du -sh /media
It will output something like:
Then run same command again in about 2 minutes and hopefully that 501M is much bigger. That means it’s working. If not, well then. Stop your sync and fix whatever is wrong 😉
Hope this helps and have a great day out there.
The following tutorial is how you can setup an encrypted hard drive to work with Nextcloudpi. Please note that there are a few steps you will have to perform every time your pi goes down because the drive will require decrypting. Basic understanding of the command line will be required for this so if you don’t have these skills locate someone who does. One step that should be complete before beginning is formatting your encrypted drive. We recommend following this tutorial for setting up your drive.
1. Flashing Nextcloudpi onto the SD Card using Etcher
2. Download the appropriate NCP image
Here is the repository for the NCP downloads. Make sure to get the right one as there are different ‘flavours’ of raspberry pi’s out there. Consider asking a community member. Generally it will be the generic RPi version if you are on a raspberry pi.
3. Extract the image from the downloaded archive
This extraction of the downloaded archive takes a bit more time than I expected so maybe get a coffee or play with your cat. Just saying. The extracted version is what you’ll flash to the card in the next steps, however, I think Etcher can use the raw archive but I’m too lazy to research that…
4. Flash the NCP Image to the SD Card
The instructions are pretty hard to mess up with Etcher in terms of how to use it. Just do it, but read the next important note (seriously read it, that’s why i put it bold and I’m mentioning it before you even read it)
Important usefule note!! It’s very easy to create a tragedy when flashing an image onto an SD card since Etcher doesn’t care that much what you are flashing on. I recommend physically removing any drive you don’t want to screw up. If you don’t it’s possibe to accidentally flash this onto your drive and completely kill it. Again, physically remove the drives you don’t want to kill and you’ll be a happier person.
- Optional Step if you have previously attempted an Installation on this computer (clearly out your history)
If you have already accessed a nextcloud server from Firefox and accessed it via ssh. While image is flashing onto the SD, remove historical garbage that will screw things up:
- Remove cached stuff in Firefox (assuming Firefox)
By going to settings and preferences / privacy & security / Cookies & Site data-Manage Data, then search IP address of your box and ‘remove’ and then ‘save’. It will give a warning which you say ok to. Not doing this might prevent you from accessing your box on same IP address with new install
- Remove ‘known_hosts’ from SSH.
This makes sure your old SSH keys and such don’t get in the way of a new SSH setup. In terminal go to /home/user(whatever it is) / .ssh.
Now you are in the .ssh folder. Now type rm known_hosts.
- Remove cached stuff in Firefox (assuming Firefox)
5. Plug in Encrypted Drive
This step assumes you have already encrypted your drive. If you haven’t or aren’t sure if you have, don’t continue but instead refer to comment in pre-amble above.
6. Put newly-etched SD card with NCP image on it, into your Raspberry Pi and plug it in.
About 2 minutes later you should be able to move to next step. If it hangs, you’re too zealous… and chill. If you find the page won’t load, perhaps you already tried an installation and you need to follow the ‘optional steps’ above?
7. Go to IP address of your Pi in your Browser
If you don’t know the IP address of your Pi yet, you can get it from your router (if you know how) or you can use tools like nmap and zenmap to do this on your network. They scan to show what devices are there and their IP addresses. After entering your IP address into the browser URL (something like 192.168.x.xx), you will be prompted with an activation page. But righ before that you will be prompted to accept the not secure connection (which is fine for this part).
Save those passwords somewhere safe (note the convenient clipboard icon which automatically copies the long string to clipboard!) (I use KeepassX and ‘activate’ installation. Should take a minute or two. If it hangs on the activation page for more than 5 minutes, although unlikely, you may need to re-flash the image from Step 1 above as there could be a problem with the way the image flashed onto the card.
8. Enter user and password into the prompt box.
These are the passwords you saved from step 5. Specifically it will be the password for the top one (:4443). The user is ‘ncp’ and the password is that long string of gobbly gook you saved in Step 5 above. You may/will also need to confirm security exception here again (which is normal).
9. Skip the installation wizard when prompted
We are skipping this step since we are adding an encrypted drive. We’ll do part of it later.
10. (Optional) Make Static IP
You can skipt this step, but I think it’s smart for your future to make a static IP for your NCP at this point because some routers tend to change it etc, etc. Just go to the nc-static-IP option and type in what you like and what will work in your unique network config.
Power off and get back to this web admin area so that your router/network will have new static IP if you did this step. You can do this with the power button icon in the top right of NCP admin, too, but when it comes back remember you’ll need to change the URL to the new IP in your browser.
11. Activate SSH in NCP admin
- On the left hand column you will see the SSH option in the NCP admin page. Go there and click the activate checkbox and enter an easy password. You can enter something as simple as 1234 here since it won’t be your ‘actual password’.
- Go to your terminal and do ssh firstname.lastname@example.org where the x’s are your pi’s IP address discovered in step 5 above.
- At the first prompt you enter the 1234 (easy password) you just made in the NCP admin page. This next part is a bit ‘weird’ if you haven’t dont it because it will kick back a request for the same password again.
- Enter it again.
- NOW you enter a real and strong SSH password that you will use for actual access to your box. Make sure it’s strong and you don’t lose it.
- Once you enter that it will log you out of SSH again and force you to log in again with your new and real password.
Mastering this step is critical because you’ll need SSH access to do encrypted drive stuff (such as decrypting it every time the power goes off) if something ‘goes wrong’ usually you can access your pi via SSH to try to fix it. Note: if you are prompted for the key fingerprint (should be) then answer ‘yes’.
12. Update your Pi-kages
This is to make sure you have the packages required to do useful stuff such as encrypt your drive. The cryptsetup package is in here so if ou want to do steps 11 below you better run these two:
sudo apt update
sudo apt upgrade
9. Do an NCP Update
Log in again with ssh email@example.com and run this command below. This is to make sure that your packages includ the ‘cryptsetup’ package and also makes sure that your box is up to date:
10. Make Apache2 not start on boot.
Making apache2 not start on boot lets you decrypt your encrypted drive before the system starts up. If/when your pi goes down, you will need to later go in and manually mount the drive each time (instructions to follow):
sudo update-rc.d apache2 disable
Remember: when the power goes off your Nextcloud will not work until you go in with SSH, decrypt drive, and restart apache2. More on this later…
11. Pre-Mounting of the Encrypted Drive
From this point we assume your drive is already encrypted in Luks format. If it’s not refer to [this page](link to come) for those instructions
- a) Install the encryption toolset so you can decrypt your drive on NCP
sudo apt install cryptsetup
- b) Check your pi to make sure the drive is showing up at least
Mine shows up as ‘sda’ but yours might be different. Look at profile of it and make sure it’s at least there.
- c) Key step: –> make sure contents of encypted drive are EMPTY…..
- d) Decrypt the drive so it’s usable by Nextcloud. You’ll need your drive de-cryption password here (and every single time you reboot your NCP…so get used to this step…):
sudo cryptsetup luksOpen /dev/sda gcw2
- e) Check again to make sure drive is looking right
Mine looks like this:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 232.9G 0 disk
└─gcw2 254:0 0 232.9G 0 crypt
12. Start apache
This makes your nextcloud stuff work so you can reach it in a browser
sudo /etc/init.d/apache2 start
13. Run the NCP Installation Wizard to Move Files to Encrypted Drive
- Go to the address of your pi in your browser with :4443/wizard at the end to access the first run wizard in NCP https://xxx.xxx.x.xx:4443/wizard
- “Do you want to save Nextcloud data in a USB drive?” Yes.
- “Plug in the USB drive and hit continue.” –> it’s plugged in so ‘continue’
- “If you want to prepare the USB drive to be used with NextCloudPi hit Format USB. Skip if already formated as ext4 or BTRFS. Attention! This will format your USB drive as BTRFS and will destroy any current data.” –> Skip formatting of drive because it’s encrypted and you want to keep it that way
- Move data to USB –> click the button
- Go through the ‘external access’ wizard however you like. I do mine manually in router
- For DDNS, I skip and do mine manually in router as well with No-ip but you can try this if you want. This is not the point of this tutorial This should make your nc-datadir point to your drive meaning that your hard files will now save to the encrypted USB drive instead of to the stock SD card which is by default where they would go. You will know if this part was successful because nc-automount and nc-datadir should will change from an orange colour to a green colour in the bottom right side of your browser screen.
- Go back to web admin panel from there
14. Run the nc-database move feature in the NCP admin panel
Again, make sure the hard drive is completely clear at this point. It’s probably possible to move a previous existing database here, but it’s out of the scope of my ability or this tutorial. You can investigate it yourself but this is assuming you have a clear drive.
Bonus section you hopefully won’t need
If you got a green light above in the last step don’t even read this section and skip to Step 15. If you have a problem where you try to do the above step and it gives you a permission So what happens here with encryption is a ‘symlink’ is created so it’s this symlink that needs to get the right permissions or NCP can’t do it’s thing with the step above. This may be a bug that no one else sees, but I’m leaving a few hints here in case we need it later:
In the next steps you have to in your terminal go to your /media/ folder and correct a permission manually before you are able to use the NCP ncdatabase function. if you have done previous nextcloud installations with their default directories on this drive, you will need to wipe out whatever is there before you move forward.
sudo chmod o+xr /media/gcw-ssd
(gcw-ssd is the name of the symlink created on your drive that points to USBdrive in Nextcloud)
Now go back to your NCP web area and do the nc-database move and it should work.
Command to empty your folders complete are as follow (use with caution, of course because this will ruin your day if you do it to the wrong dir!)
(if it’s not empty run: sudo rm -rf /media/USBdrive/ncdatabase)
You might also like to keep this command handy to check permissions if someone asks:
sudo ls -ld
15. LetsEncrypt – nice and easy.
This is a good chance to relax and do some Lets Encrypt since it’s easy and satisfying. Go to the left panel of web admin find letsencrypt, fill in the blanks, and press go. Now you should be able to find your box from the internets with secure connection too. You’ll need your dynamic dns url at this point to make it all work so go and do that at no-ip.com or whatever you like. S
16. Reboot system to make sure things are working as they ought
- Shut down your box with command:
- To be sure it’s back up you can ping xxx.xxx.x.xx (your box). When it starts responding you should be ready to ssh in
- SSH in (see instructions above in Step 8) At this point, because you made apache2 not start on reboot, neither your NCP admin pages nor your nextcloud instance will be accessible. We will proceed with a new section now which will be your process to get it back up each time the power goes down or it’s rebooted.
17. Getting things back up after a reboot:
- Unlock/decrypt drive. Note: yours will not be ‘gcw2’ – that’s just my example. Can be whatever you like.
sudo cryptsetup luksOpen /dev/sda gcw2
- Enter your decryption password for drive
- Restart apache (see above)
sudo /etc/init.d/apache2 start
Celebrate if it’s working! Try again if it’s not!
Special thanks to Tobias, Nachoparker and Kevin for all your hard work with me getting it this far!