Getting my head around a new backup system

 

Warning .... this is just a brain dump really while I wade through the crap that is BackBlaze's confusing product + thirdparty software you need, the aftermath of crashplan and trying to narrow down what it is I need backing up where.

Crash plan is dead.... thank God for that. Despite the software being very good, very useful, sensibly thought out for normal people who have normal household backup needs, the cloud service sucked. The way they exited the consumer market too, it just shows how cloud services in general should not be 100% relied upon. ALWAYS HAVE A CLOUD EXIT STRATEGY.

Anyway, in the midst of everyone moaning about Cloudflare, BackBlaze, the only other real cost effective competitor, went on a gloating spree to say how good they are happy to accept new customers. So let's dive into my backup needs, what is available and what it costs.

My Backup responsibilities

Machines to backup (some needless detail too cos I'm a geek);

  • Laptop (My Lenovo X250 180GB SSD) dual boot Win10 (100GB) and Linux (80GB)
  • Laptop (Girlfriends MacBook Pro 80GB HDD) MacOS
  • Laptop (Girlfriends Lenovo T450 180GB SSD) Win10
  • Desktop (AMD A10 6800k 250GB SSD) Win10
  • NAS (HP N40L Microserver w/12TB Storage) Ubuntu LTS

Requirement

I want program that I can install on each machine/partition which backs up to both my NAS and a cloud service.

  • If I don't backup my girlfriends machines, they won't get backed up, so it needs to be done without her intervention
  • It shouldn't have to cost more than the family backup plan with CrashPlan ($150 per year total)

Previous Setup

We had a CrashPlan family plan and was installed on all the laptops (and partitions) and Desktop computer. CrashPlan was also installed in headless mode on the NAS. So all of the laptops and desktop backed up to both CrashPlan Cloud and my NAS.

The NAS itself has a more powerful twin (A N54L at my Dads house). The NAS has about 500GB of important data which I was backing up to the CrashPlan cloud (photos, emails, documents, previous work that kind of stuff) and the rest is stuff that I can re-download, re-encode or reprocess (sure it'd be a pain but it's not the end of the world, hence why I have the N54L twin).

So in all I have three operating systems, six instances that will have around 1TB max of data total to backup to the cloud, five instances (the laptops and desktop) which will back up to my NAS for local backup. 

The BackBlaze Offering

BackBlaze personal is $5 per month per PC or Mac for Unlimited backup.

Clearly this is very strict and they are trying to protect themselves against going the way of Crashplan. I understand this.

However for the following reasons, BackBlaze personal isn't an option;

  • To backup only the Windows and Mac instances, it'd be $20 per month to back up
  • I want to be able to sort all the backups in one account (like a family plan!)
  • I'd need a secondary piece of software on each machine to backup to my NAS
  • I'd need another solution Linux Laptop partition and however I deal with the important data on my NAS

To use BackBlaze with Linux use B2

Okay, so this isn't an unlimited plan, but since it's less that 1TB I'm trying to backup here then the cost shouldn't be so bad and I really do understand why BackBlaze are being an arse about this. However, here are the points here;

  • Simple pricing
    • Storage - $0.005/GB/Month
    • Download - $0.02/GB
    • Class "A" transactions – Free
    • Class "B" transactions - $0.004 per 10,000 with 2,500 free per day.
    • Class "C" transactions - $0.004 per 1,000 with 2,500 free per day.

Simple right, oh, here is the simple API Calls chart with all the transaction limits and what the transaction actually is that you need to be aware of https://www.backblaze.com/b2/b2-transactions-price.html .... Just as an aside, as much as I enjoy IT work, SysAdmin etc. when I get home I don't really want to have to do the same cloud costing bulls&*t that I have had to do at work.

So using the B2 Cost Calculator it looks like backing up 500GB with a monthly churn of say 30GB and no downloads (I assume this means recovery and not some sort of sync cost?) then it's looking at around $40 per year. This looks really reasonable, however in the estimator it doesn't not mention anything about "transactions", let's assume that I won't go over the daily limit, because nothing ever goes wrong when you assume you understand a pricing plan right? 

Pricing aside, how do you actually backup to B2?

Well, you need some software from somewhere else as BackBlaze only provide the storage and API to this. 

This is where I get quite annoyed with BackBlaze claiming to be a viable alternative to Crashplan. B2 is simply NOT a set once and forget backup solution. This is clearly designed as a cloud storage solution to rival S3 and in that it succeeds. But as an actual backup solution, this is 100% dependent on a now 3rd party in this backup solution.

I suppose we should look at some software

  • ArqBackup - $49 one-time fee per machine user (+$29.99 per licence for lifetime upgrade), Mac and Win only.
    • It can backup locally to SFTP so that's a plus
    • Looks very nice, well supported and easy to use
    • Two licences will cost me $196 $99.98 + ($59.98 for lifetime upgrades) + the backup cost
      • CORRECTION 11-09-2017 Licence improvement URL
      • Micheal Bennett kindly contacted me to let me know that ArqBackup now provide the licence per user rather than per machine. This greatly decreases this cost.
    • I still need a solution for the Linux part
  • duplicati duplicati.com - Free! - Linux, Mac, Win
    • This looks at the moment to be my best bet
    • Still in Beta, I've tried the Linux version and it seems okay for NAS backup so far
    • B2 support is quite new
    • If the Windows/Mac version is okay I may do this and donate the difference between the Crashplan family plan and the BackBlaze storage to the Duplicati developers
  • CyberDuck
    • Not a backup utility, it's a file transfer program like WinSCP
    • As a file browser though, this is awesome, just not for my needs today.
  • GoodSync $29.99 per computer
    • Quite cheap, no yearly cost
    • Might be okay for me but certainly not automated and invisible enough for my girlfriends needs
    • If I'm going to use something as complex as this, I may as well use duplicati
  • qBackup $29.99 per computer - $99.99 for a linux licence!!! WTF!?!
    • After seeing the Linux price, I stopped downloading. What a cheek!
    • So from the manual, it looks similar to GoodSync
    • The website doesn't look like it's been updated in a while, I don't trust it.
  • Cloudberry - $29.99 per computer
    • AGAIN! WTF!? They limit you to 1TB storage.... but they don't offer storage. What kind of crap is this!?
    • Sure I'm not using 1TB but why should the 3rd party vendor dictate how much I store when they're not even storing it?!
  • duplicacy.com - $20 per computer + $5 per year per computer
    • Not as much as Arq but adds a yearly cost per machine
    • Doesn't look as feature rich as Arq, but that could just be the lack of info on the website
      • I must admit that if I'm looking to pay for a backup solution, first appearances of the website really seem to matter to me
    • I may try this out but probably only if duplicati isn't suitable which I suspect won't be the case

 

Other sensible solutions?

Office 365?

This one isn't quite for me, but it maybe in the future. I'm an academic so I get a work copy of MS Office via 365. However my girlfriend doesn't have this. So, for £59.99 per year (£5 per month), you can get a personal Office 365 account giving you all the products as well as 1TB oneDrive storage, way more than needed.

This can be further extended to £79.99 per year for a family version and you can have up to 5 users each with 1TB storage on oneDrive.

This could be coupled with any of the same pieces of software you have to have for B2.

 

Conclusions?

I will finish this when I have some conclusions. In the mean time, I hate the cloud!

undefined

Getting Paypal Identity token for Woocommerce

 

Paypal has changed the layout of the website dashboard since most of the 'how to' guides have been written so here is my updated one.

So, for woocommerce you may wish to setup a Paypal Identity token which helps to solve some of the IPN errors you may encounter. Especially those where you have two or more email addresses linked to your paypal account and woocommerce places the order on hold due producing the following error:


IPN Response is for another account: [email protected] Your email is [email protected]

Annoyingly I've not been able to resolve this in a sensible fashion, in this example here, [email protected] is the primary address with paypal, so why does it try to send the IPN response using my login email? Anyway...

You can't see your Paypal Identity token UNTIL you've activated Payment Data Transfer. 

To find the Paypal Identity token, log into paypal.

  • Click Profile (top right)
  • Click Profile and Settings (from the drop down box)
  • Click My Selling preferences (left menu)
  • Click Update on the right of the Website preferences option
  • Turn on both Auto Return and Payment Data Transfer
  • Click Save at the bottom of the page
  • This will return you to the previous menu page
  • Click Update on the right of the Website preferences option again to take you back into the previous option
  • Scroll down and beneath the Payment Data Transfer option you should see your token

Talk about faffy!

 

 

Scaleway - ARMv8

 

Martin Rusev at Anom.cx wrote a great analysis of the Scaleway ARMv7 and Digtial Ocean droplets some time ago (not sure exactly when but around 2015 from the github comment). Yesterday I received an email from Scaleway announcing their ARMv8 machines so I thought I would give one a go. Martin Rusev noted that against a DO droplet there was certainly a drop in performance with the ARMv7. So let's see how the v8 gets on.

I booted up a 2C ARMv8 with 2GB RAM as well as a 4C ARMv8 with 4GB RAM. It took around 10 minutes to get a console on one of these in the Paris DC. I would imagine that since the Amsterdam DC is out of stock of a lot of these they are getting quite a lot of demand.

CPU Test

To begin with I run the single core test, along with 2C and 4C (because 4C was run on Martin Rusev's test too).

Single Core

[email protected]:~# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000

Test execution summary:
total time: 29.9082s
total number of events: 10000
total time taken by event execution: 29.9016
per-request statistics:
min: 2.94ms
avg: 2.99ms
max: 10.56ms
approx. 95 percentile: 3.02ms

Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 29.9016/0.00

Dual Core

[email protected]:~# sysbench --test=cpu --cpu-max-prime=20000 run --num-threads=2

sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 2

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000

Test execution summary:
total time: 14.9265s
total number of events: 10000
total time taken by event execution: 29.8405
per-request statistics:
min: 2.94ms
avg: 2.98ms
max: 3.42ms
approx. 95 percentile: 3.00ms

Threads fairness:
events (avg/stddev): 5000.0000/0.00
execution time (avg/stddev): 14.9203/0.00

Quad Core

[email protected]:~# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=4 run
sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 4

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
total time: 14.9632s
total number of events: 10000
total time taken by event execution: 59.8098
per-request statistics:
min: 2.94ms
avg: 5.98ms
max: 20.55ms
approx. 95 percentile: 11.01ms

Threads fairness:
events (avg/stddev): 2500.0000/1.22
execution time (avg/stddev): 14.9525/0.01


ARMv8 4Core 4GB RAM

 

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          29.8729s
    total number of events:              10000
    total time taken by event execution: 29.8657
    per-request statistics:
         min:                                  2.96ms
         avg:                                  2.99ms
         max:                                  8.31ms
         approx.  95 percentile:               3.00ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   29.8657/0.00

 

[email protected]:~# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=4 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 4

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          7.5198s
    total number of events:              10000
    total time taken by event execution: 30.0429
    per-request statistics:
         min:                                  2.94ms
         avg:                                  3.00ms
         max:                                 15.02ms
         approx.  95 percentile:               3.01ms

Threads fairness:
    events (avg/stddev):           2500.0000/5.79
    execution time (avg/stddev):   7.5107/0.00

 

I/O

[email protected]:~# sysbench --test=fileio --file-total-size=6G prepare
[email protected]:~# sysbench --test=fileio --file-total-size=6G --file-test-mode=rndrw --max-time=300 --max-requests=0 --file-extra-flags=direct run

Operations performed: 336060 Read, 224040 Write, 716822 Other = 1276922 Total
Read 5.1279Gb Written 3.4186Gb Total transferred 8.5464Gb (29.172Mb/sec)
1866.99 Requests/sec executed

Test execution summary:
total time: 300.0018s
total number of events: 560100
total time taken by event execution: 156.6031
per-request statistics:
min: 0.21ms
avg: 0.28ms
max: 10.57ms
approx. 95 percentile: 0.32ms

Threads fairness:
events (avg/stddev): 560100.0000/0.00
execution time (avg/stddev): 156.6031/0.00

ARMv8 4Core 4GB RAM

 Operations performed:  327880 Read, 218587 Write, 699392 Other = 1245859 Total
Read 5.0031Gb  Written 3.3354Gb  Total transferred 8.3384Gb  (28.462Mb/sec)
 1821.55 Requests/sec executed

Test execution summary:
    total time:                          300.0013s
    total number of events:              546467
    total time taken by event execution: 152.3080
    per-request statistics:
         min:                                  0.22ms
         avg:                                  0.28ms
         max:                                 17.58ms
         approx.  95 percentile:               0.34ms

Threads fairness:
    events (avg/stddev):           546467.0000/0.00
    execution time (avg/stddev):   152.3080/0.00


MySQL

mysql Ver 14.14 Distrib 5.7.18, for Linux (aarch64) using EditLine wrapper 

ARMv8 2Core 2GB RAM

[email protected]:~# sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=benchmark --mysql-user=root --mysql-password=password123 prepare

[email protected]:~# sysbench --test=oltp --oltp-table-size=1000000 --mysql-db=benchmark --mysql-user=root --mysql-password=password123 --max-time=60 --oltp-read-only=on --max-requests=0 --num-threads=8 run

sysbench 0.4.12: multi-threaded system evaluation benchmark

No DB drivers specified, using mysql
Running the test with following options:
Number of threads: 8

Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 7 times)
Done.

OLTP test statistics:
queries performed:
read: 276500
write: 0
other: 39500
total: 316000
transactions: 19750 (329.05 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 276500 (4606.75 per sec.)
other operations: 39500 (658.11 per sec.)

Test execution summary:
total time: 60.0206s
total number of events: 19750
total time taken by event execution: 479.8237
per-request statistics:
min: 16.70ms
avg: 24.29ms
max: 84.53ms
approx. 95 percentile: 26.47ms

Threads fairness:
events (avg/stddev): 2468.7500/37.75
execution time (avg/stddev): 59.9780/0.01

ARMv8 4Core 4GB RAM

No DB drivers specified, using mysql
Running the test with following options:
Number of threads: 8

Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 7 times)
Done.

OLTP test statistics:
    queries performed:
        read:                            537012
        write:                           0
        other:                           76716
        total:                           613728
    transactions:                        38358  (639.13 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 537012 (8947.79 per sec.)
    other operations:                    76716  (1278.26 per sec.)

Test execution summary:
    total time:                          60.0161s
    total number of events:              38358
    total time taken by event execution: 479.5496
    per-request statistics:
         min:                                  5.95ms
         avg:                                 12.50ms
         max:                                 33.54ms
         approx.  95 percentile:              13.75ms

Threads fairness:
    events (avg/stddev):           4794.7500/46.71
    execution time (avg/stddev):   59.9437/0.00

 

Results ARMv8 against published ARMv7

 

 

CPU Single Core

Total Time (s)

CPU Quad Core

Total Time (s)

Disk I/O

Mb/sec

MySQL Transactions

per sec

MySQL read/write

requests per sec

ARMv7 685.512 171.3962 16.347 467.07 6539.00
ARMv8 2C 2GB RAM 29.9082 14.9632 29.172 329.05 4606.75
ARMv8 4C 4GB RAM 29.8729 7.5198 28.462 639.13 8947.79


So the CPU and Disk I/O show performance increases (the ARMv8 CPU performs the prime calculations faster than the DO droplet Martin Rusev used. However as you can see there is a performance decrease in the MySQL results of the 2 core ARMv8 vs the ARMv7. I would imagine this is because the 4C ARMv7 used has a performance advantage over the faster clockspeed but lower number of CPU cores of the ARMv8 2 core.

Conclusion

Well after spending a week migrating my production webserver from a 1GB Single-Core DO droplet to a 2GB Single-Core Linode VM last month I am now wondering if I it would be beneficial to go for one of Scaleways 4C ARMv8 with 8GB RAM which costs the same price as the Linode VM.

My only real concern for production is the support and stability as I'm new Scaleway and they don't have the best reviews yet. My own experience hasn't been smooth sailing with the brief interactions I have had with them. They locked my account I'd set up a few years ago and faffed around telling me I couldn't use that email account anymore and I'd have to use a new one. After escalating the support thread to a manager it was resolved. I'm sure these customer service type issues will iron themselves out in the future. 

In the meantime I am going to stick with this small ARMv8 and play around with it. Personally I really hope these micro dedicated servers become the norm. No more noisy neighbours and great performance for the price. 

 

 

Home ← Older posts