How to dynamical exclude folders from MacOS Time Machine

11/22/2020

What is the problem?

MacOS includes a very handy backup system called Time Machine which does a snapshot of all changes every hour.

As a Developer relying on other packages it is quit common that a simple project is based on thousends packages which easily aggregate to hundredthousends of files. Each request for updating on package might change thousends of files leading to a huge backup for something which exists online on multiple servers and could be restored with a simple single command per project.

Existing Solutions

Searching for a solution if found this tools:

asimov - scanning your filesystem for known dependency directories (node_modules, vendors,..) and excludes them from backup, can run as cron job
tmignore - excludes files and directories matched by .gitignore files, can run as cron job
tmutil - official Time Machine utility, examine and manipulate Time Machine backups, restore data from backups, add or remove exclusions, and compare backups
npm wrapper - bash wrapper for npm to run on specific events

Experiance

asimov

good:

simple installation with brew
can be installed as a service with brew to run once a day

bad:

scanned my complete home directory and excluded node_modules folder in the .vscode setting folder :-(
no options to define the foldes which should be excluded / included
no command to list the files which have been excluded
no command to remove the excluded directories
no dryrun

tmignore

good:

simple installation with brew
can be installed as a service with brew to run once a day
has a configuration to define the search path and pattern to include files which are ignored by .gitignore but need to be backuped - e.g. secret keys
commands to list and reset

bad

no dryrun
to danger to miss something on whitelist and break emergency recover

npm wrapper

This is not working if you use any kind of viual client.

What API is used by this tools?

All this tools use the command line tool tmutil with the command tmutil addexclusion <path to file>.

Avoid the option -p will add the absolute path to the item to the global list of excluded items and requires root privileges.

Without option -p the item will be marked to not be backuped and no root privileges are required. Marking equals adding the extended attribute com.apple.metadata:com_apple_backup_excludeItem to the item. This will make it possible to move the item without loosing the "do not backup" flag.

It will not listed in the Time Machine Settings panel which makes this a perfect solution as there are no configurations which need to be cleaned up when marked items are removed.

Here is a simple example which shows what happens behind the command:

1mkdir "not_backuped"
2xattr "not_backuped"

there shoud be no output from xattr.

1tmutil addexclusion "$(pwd)/not_backuped"
2tmutil isexcluded "$(pwd)/not_backuped"
3xattr "not_backuped"

Output tmutil isexcluded: [Excluded] .../not_backuped Output xattr: com.apple.metadata:com_apple_backup_excludeItem

Note: Based on your current MacOS version there might be a request to allow the terminal app to get Full Disk Access.

you can remove the extended attribute: xattr -d "com.apple.metadata:com_apple_backup_excludeItem" not_backuped or with tmutil removeexclusion "$(pwd)/not_backuped"

Note: tmutil will allways require an absolute path to the item!

Is this item excluded?

tmutil isexcluded "$(pwd)/not_backuped/readme.txt" Output: [Excluded] .../not_backuped/readme.txt

The command will check if an item is excluded from backup based on the global list from Time Machine Settings, settings added by other apps, extended attributes on the iten and when any excluded parent folder idepend of the deep in hirachy.

Find excluded Folders

Now with the knowledge that excluded items have a specific extended attribute it is very easy to use the find command to find all items with the attribute:

1find ~/my-projects -xattrname "com.apple.metadata:com_apple_backup_excludeItem" -type d -prune

~/my-projects - the search folder
-xattrname "com.apple.metadata:com_apple_backup_excludeItem" - this is the specific attribute
-type d - here we only search for Folders
-prune - we are not interested to search inside folders we found
Note: _there are some sources which suggest to use sudo mdfind "com_apple_backup_excludeItem = 'com.apple.backupd'" for this task but using the spotlight index. On my compter I have excluded my source files to be indexed by Spotlight which makes this command fail to find all items.

Remove Items from being excluded

1) use tmutil with absolute path to the item tmutil removeexclusion "$(pwd)/not_backuped"

2) use xattr (not recommended) xattr -d "com.apple.metadata:com_apple_backup_excludeItem" not_backuped

3) use find for multiple items find ~/my-projects -xattrname "com.apple.metadata:com_apple_backup_excludeItem" -type d -prune -exec tmutil removeexclusion "{}" \; this will search in folder my-projects for folders which are excluded from backup and will remove them from exclusion.

Simple Solution

Create your own simple script which includes your rules with this draft:

1#!/bin/sh
2#set -x
3
4EXCLUDEFOLDERNAME=${3:-"node_modules"}
5
6get_realpath ()
7{
8    echo "$(cd "$(dirname "$1")"; pwd)/$(basename "$1")"
9}
10SEARCHPATH="$(get_realpath ${2:-"${HOME}"})"
11
12list_tm ()
13{
14    find "${SEARCHPATH}" -xattrname "com.apple.metadata:com_apple_backup_excludeItem" -name "${EXCLUDEFOLDERNAME}" -type d -prune
15}
16
17run_tm ()
18{
19    find "${SEARCHPATH}" -name "${EXCLUDEFOLDERNAME}" -type d -prune -print0 -exec tmutil addexclusion "{}" \;
20}
21
22reset_tm ()
23{
24    find "${SEARCHPATH}" -xattrname "com.apple.metadata:com_apple_backup_excludeItem" -name "${EXCLUDEFOLDERNAME}" -type d -prune -print0 -exec tmutil removeexclusion "{}" \;
25}
26
27case "${1}" in
28    list)
29        list_tm
30    ;;
31    run)
32        run_tm
33    ;;
34    reset)
35        reset_tm
36    ;;
37    *)
38        echo $"Usage: $0 {run|list|reset} [search path (Default: '${HOME}')] [exclude folder name (Default: 'node_modules')]"
39        exit 1
40esac