Solved Using Rclone from a server to backup multiple volumes from different SSH hosts

[DustinB3403]

This is a PoC I'm attempting to work on, and I'm looking for some guidance as I've not used RClone.

What I'd like to do:

Have a Fedora 30 server running the rclone process and connection to B2 storage, have it connect to multiple Apple Workstations with their own external storage and sync said external storage to B2. over individual SSH connections to each apple computer.

What isn't an option:

Allowing the end user to operate this process - in any fashion - they simply won't do it and it'll cause issues - the system needs to be automated.

What I can do:

I can connected to each workstation over ssh without issue.
I've also confirmed I can connect to B2 using rclone

Where is the problem:

How do I connect to each workstation using rclone and then backup to B2?

[DustinB3403]

Anyways I figured this out.

Having multiple rclone sources like
https://i.imgur.com/SKtlxrr.png

and running

Works

[marcinozga]

@DustinB3403 Install rclone on each workstation and setup cron jobs on each machine. Brew and Ansible make this trivial. Do you really need Fedora Server in this setup?

[black3dynamite]

Use this to rclone your Mac clients to Fedora Server
https://rclone.org/sftp/

Use this to rclone your Fedora Server to B2
https://rclone.org/b2/

[DustinB3403]

@marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 Install rclone on each workstation and setup cron jobs on each machine. Brew and Ansible make this trivial. Do you really need Fedora Server in this setup?

This I considered, but would involve tinkering with each workstation.

[DustinB3403]

@black3dynamite said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

Use this to rclone your Mac clients to Fedora Server
https://rclone.org/sftp/

Use this to rclone your Fedora Server to B2
https://rclone.org/b2/

So I've already gone through this, maybe I'm just missing it.. will rereview.

[marcinozga]

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 Install rclone on each workstation and setup cron jobs on each machine. Brew and Ansible make this trivial. Do you really need Fedora Server in this setup?

This I considered, but would involve tinkering with each workstation.

That's why I mentioned Ansible. Your tinkering is limited to workstation that runs playbooks.

[DustinB3403]

@marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 Install rclone on each workstation and setup cron jobs on each machine. Brew and Ansible make this trivial. Do you really need Fedora Server in this setup?

This I considered, but would involve tinkering with each workstation.

That's why I mentioned Ansible. Your tinkering is limited to workstation that runs playbooks.

That's not at all accurate, I would have to install rclone on every workstation, have custom cron jobs on every workstation and then would have to validate that every workstation sync'd.

[DustinB3403]

In addition to that, I would have to enable cron on every apple workstation, which while doable, is a pain in the rear should I ever need to make any changes to the backup operation.

Centrally managing this makes way more sense rather than the ad hoc approach.

[DustinB3403]

One option I considered is just installing Cyberduck on the workstations and having the user do the backup.

Of course that just won't be done and thus is off the table.

[marcinozga]

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 Install rclone on each workstation and setup cron jobs on each machine. Brew and Ansible make this trivial. Do you really need Fedora Server in this setup?

This I considered, but would involve tinkering with each workstation.

That's why I mentioned Ansible. Your tinkering is limited to workstation that runs playbooks.

That's not at all accurate, I would have to install rclone on every workstation, have custom cron jobs on every workstation and then would have to validate that every workstation sync'd.

And all that you can do with Ansible.
Install rclone? https://docs.ansible.com/ansible/latest/modules/homebrew_module.html#homebrew-module
Setup cron? https://docs.ansible.com/ansible/latest/modules/cron_module.html#cron-module

[marcinozga]

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

In addition to that, I would have to enable cron on every apple workstation, which while doable, is a pain in the rear should I ever need to make any changes to the backup operation.

Centrally managing this makes way more sense rather than the ad hoc approach.

And since when has cron been disabled in macOS?

[DustinB3403]

@marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

In addition to that, I would have to enable cron on every apple workstation, which while doable, is a pain in the rear should I ever need to make any changes to the backup operation.

Centrally managing this makes way more sense rather than the ad hoc approach.

And since when has cron been disabled in macOS?

For quite some time, it's still there, but you have to enable it, grant it root access etc.

[marcinozga]

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@marcinozga said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

In addition to that, I would have to enable cron on every apple workstation, which while doable, is a pain in the rear should I ever need to make any changes to the backup operation.

Centrally managing this makes way more sense rather than the ad hoc approach.

And since when has cron been disabled in macOS?

For quite some time, it's still there, but you have to enable it, grant it root access etc.

That's simply not true. You don't have to enable it. And you don't have to give it root access unless you're writing somewhere you shouldn't be in first place.

[DustinB3403]

Anyways I figured this out.

Having multiple rclone sources like
https://i.imgur.com/SKtlxrr.png

and running

Works

[dafyre]

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

Anyways I figured this out.

Having multiple rclone sources like
https://i.imgur.com/SKtlxrr.png

and running

Works

The /Volumes/<path to external> would change for each system, wouldn't it?

Could you back up to b2-rclone:/<username>/external ?

[DustinB3403]

@dafyre said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

@DustinB3403 said in Using Rclone from a server to backup multiple volumes from different SSH hosts:

Anyways I figured this out.

Having multiple rclone sources like
https://i.imgur.com/SKtlxrr.png

and running

Works

The /Volumes/<path to external> would change for each system, wouldn't it?

Could you back up to b2-rclone:/<username>/external ?

Possibly, but what I was thinking I would do is simply connect to the administrative user on each of these workstations, hit the volume that way. The Volume name may change (as spaces make everything more difficult). But yeah I think this'll work.

[DustinB3403]

To clarify for anyone else in the future, you need to have multiple apple-# sftp connections.

The b2 connections could also be custom as you can create multiple buckets and simply connect to that bucket via rclone.

[DustinB3403]

This is what the b2-rclone remote setting looks like for my needs, obviously this could be adjusted with the advanced configuration menu.

https://i.imgur.com/sj858rr.png

And this is the Apple workstation

https://i.imgur.com/jGytpzZ.png

[DustinB3403]

Ideally when this is all said and done, what I think would make the most sense is to create multiple buckets for each of these apple workstations.

So apple-bucket-1, apple-bucket-2 and so on.

From there, simply have cron run the sync commands for each of these and have it run hourly.

[DustinB3403]

Worth noting on this that while sync works, copy is the much safer option.

So

rclone copy source:/path dest:/path is what I'll likely end up using. In any case, the resulting solution still works.

Still need to tweak OSX a bit as it seems to kill the SSH session after some time for sporadic reasons - still testing but I may end up having to add ServerAliveInterval 120 to /etc/ssh/ssh_config to resolve that issue.