Merge pull request #470 from LFd3v/master

community repo was merged with extra, please refer to:

https://archlinux.org/news/git-migration-announcement/
https://archlinux.org/news/git-migration-completed/

.github/CONTRIBUTING.md

@@ -14,6 +14,7 @@ areas.
 
 To contribute, you can:
 
+* Star the yadm repo, the star count helps others discover yadm.
 * Report [bugs](#reporting-a-bug)
 * Request [features/enhancements](#suggesting-a-feature-or-enhancement)
 * Contribute changes to [code, tests](#contributing-code), and [documentation](#improving-documentation)
@@ -206,11 +207,16 @@ these principles when making changes.
     ```
 
 4. Verify you can run the test harness. _(This will require dependencies:
-   `make`, `docker`, and `docker-compose`)_.
+   `make` and `docker`)_.
 
     ```text
     $ make test
     ```
+    If you don't use `docker` but an OCI engine akin to `podman`, you can set it through the `OCI` switch for every target
+
+    ```text
+    $ make test OCI=podman
+    ```
 
 5. Create a feature branch, based off the `develop` branch.
 

.github/workflows/schedule.yml

@@ -0,0 +1,20 @@
+---
+name: Scheduled Site Tests
+on:  # yamllint disable-line rule:truthy
+  schedule:
+    - cron: "0 0 1 * *"  # Monthly
+jobs:
+  Tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          ref: gh-pages
+      - run: >-
+          docker create -t
+          --name yadm-website
+          --entrypoint test/validate
+          yadm/jekyll:2019-10-17;
+          docker cp ./ yadm-website:/srv/jekyll
+      - name: Test Site
+        run: docker start yadm-website -a

.github/workflows/stale.yml

@@ -0,0 +1,25 @@
+---
+name: Close Stale Issues
+on:  # yamllint disable-line rule:truthy
+  schedule:
+    - cron: "30 1 * * *"  # Daily
+jobs:
+  Stale:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/stale@v4
+        with:
+          close-issue-message: >-
+            This issue was closed because it has been labeled as stale for 7
+            days with no activity.
+          days-before-close: 7
+          days-before-stale: 60
+          exempt-all-assignees: true
+          exempt-issue-labels: in develop, 1, 2, 3
+          exempt-pr-labels: in develop, 1, 2, 3
+          stale-issue-label: stale
+          stale-issue-message: >-
+            This issue has been labeled as stale because it has been open 60
+            days with no activity. Remove stale label or comment or this will
+            be closed in 7 days.
+          stale-pr-label: stale

.github/workflows/test.yml

@@ -0,0 +1,13 @@
+---
+name: Tests
+on:  # yamllint disable-line rule:truthy
+  - push
+  - pull_request
+  - workflow_dispatch
+jobs:
+  Tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Tests
+        run: make test

.gitignore

@@ -2,5 +2,6 @@
 .jekyll-metadata
 .pytest_cache
 .sass-cache
+.testyadm
 _site
 testenv

.travis.yml

@@ -1,8 +0,0 @@
----
-language: minimal
-services:
-  - docker
-before_install:
-  - docker pull yadm/testbed:2019-09-25
-script:
-  - docker run -t --rm -v "$PWD:/yadm:ro" yadm/testbed:2019-09-25

CHANGES

@@ -1,3 +1,98 @@
+3.2.2
+  * Support spaces in distro/distro-family (#432)
+  * Fix zsh hanging when tab completing add/checkout (#417)
+  * Add yadm-untracked script to contributed files (#418)
+  * Fix documentation typos (#425)
+  * Support docker-like OCI engines for dev testing (#431)
+
+3.2.1
+  * Fix Bash 3 bad array subscript bug (#411)
+
+3.2.0
+  * Support architecture for alternates/templates (#202, #203, #393)
+  * Support distro_family for alternates/templates (#213)
+  * Support setting multiple classes (#185, #304)
+  * Support environment variables in default template processor (#347)
+  * Update version command to include Bash & Git versions (#377)
+
+3.1.1
+  * Fix clone support for older versions of Git (#348)
+  * Fix support for multiple GPG recipients (#342)
+  * Find symlinks in bootstrap-in-dir (#340)
+
+3.1.0
+  * Use `git clone` directly during clone (#289, #323)
+  * Fix compatibility bug with Git completions (#318, #321)
+  * Support relative paths for --yadm-* and -w (#301)
+  * Improve parsing of if-statement in default template (#303)
+  * Read files without running cat in subshells (#317)
+  * Improve portability of updating read-only files (#320)
+  * Various code improvements (#306, #307, #311)
+
+3.0.2
+  * Fix parsing by sh (#299)
+
+3.0.1
+  * Improve handling of submodules at upgrade (#284, #285, #293)
+  * Improve Zsh completions (#292, #298)
+  * Use stderr for error messages (#297)
+
+3.0.0
+  * Support encryption with OpenSSL (#138)
+  * Support "include" directive in built-in template processor (#255)
+  * Support extensions for alternate files and templates (#257)
+  * Improve support for default branches (#231, #232)
+  * Add --version and --help as yadm internal commands (#267)
+  * Improve support for XDG base directory specification
+  * Use XDG_DATA_HOME used for encrypted data and repository (#208)
+  * Default repo is now ~/.local/share/yadm/repo.git
+  * Default encrypted archive is now ~/.local/share/yadm/archive
+  * Improve shell completions (#238, #274, #275)
+  * Remove support for YADM_COMPATIBILITY=1 (#242)
+  * Remove deprecated option cygwin-copy
+  * Fix template mode inheritance on FreeBSD (#243, #246)
+  * Fix hook execution under MinGW (#150)
+  * Improve compatibility with Oil shell (#210)
+
+2.5.0
+  * Support for transcrypt (#197)
+  * Support ESH templates (#220)
+  * Preserve file mode of template (#193)
+  * Fish shell completions (#224)
+  * Fix alt processing when worktree is `/` (#198)
+  * Assert config directory if missing (#226, #227)
+  * Documentation improvements (#229)
+
+2.4.0
+  * Support multiple keys in `yadm.gpg-recipient` (#139)
+  * Ensure all templates are written atomically (#142)
+  * Add encrypt_with_checksums to the hooks collection (#188)
+  * Escape white space in YADM_HOOK_FULL_COMMAND (#187)
+  * Improve parsing of os-release (#194)
+  * Improve identification of WSL (#196)
+  * Fix troff warnings emitted by man page (#195)
+  * Write encrypt-based exclusions during decrypt
+
+2.3.0
+  * Support git-crypt (#168)
+  * Support specifying a command after `yadm enter`
+  * Expose GIT_WORK_TREE during `yadm enter` (#160)
+  * Support GNUPGHOME environment variable (#134)
+  * Assert private dirs, only when worktree = $HOME (#171)
+
+2.2.0
+  * Resolve hostname using `uname -n` (#182)
+  * Use /etc/os-release if lsb_release is missing (#175)
+  * Issue warning for any invalid alternates found (#183)
+  * Add support for gawk (#180)
+
+2.1.0
+  * Use relative symlinks for alternates (#100, #177)
+  * Support double-star globs in .config/yadm/encrypt (#109)
+  * Improve bash completion (#136)
+  * Update docs about using magit (#123)
+  * Note exception for WSL (#113)
+
 2.0.1
   * Fix bug with worktree permissions (#174)
 

CONTRIBUTORS

@@ -1,21 +1,52 @@
 CONTRIBUTORS
 
 Tim Byrne
+Erik Flodin
+Martin Zuther
+Jan Schulz
+Ross Smith II
+Jonathan Daigle
+Luis López
+Tin Lai
 Espen Henriksen
 Cameron Eagans
-Ross Smith II
 Klas Mellbourn
-Jan Schulz
-Satoshi Ohki
-Siôn Le Roux
-Sébastien Gross
-Thomas Luzat
+James Clark
+Glenn Waters
+Nicolas signed-log FORMICHELLA
 Tomas Cernaj
-Uroš Golja
-Brayden Banks
-japm48
-Daniel Wagenknecht
-Franciszek Madej
-Mateusz Piotrowski
+Joshua Cold
+jonasc
+Nicolas stig124 FORMICHELLA
+Chad Wade Day, Jr
+Sébastien Gross
+David Mandelberg
+Paulo Köch
+Oren Zipori
+Daniel Gray
 Paraplegic Racehorse
+japm48
+Siôn Le Roux
+Mateusz Piotrowski
+Uroš Golja
+Satoshi Ohki
+Jonas
+Franciszek Madej
+Daniel Wagenknecht
+Stig Palmquist
 Patrick Hof
+con-f-use
+Samisafool
+Bram Ceulemans
+Travis A. Everett
+Sheng Yang
+Jared Smartt
+Adam Jimerson
+dessert1
+addshore
+Tim Condit
+Thomas Luzat
+Russ Allbery
+Brayden Banks
+Alexandre GV
+Felipe S. S. Schneider

Dockerfile

@@ -1,52 +0,0 @@
-FROM ubuntu:18.04
-MAINTAINER Tim Byrne <sultan@locehilios.com>
-
-# No input during build
-ENV DEBIAN_FRONTEND noninteractive
-
-# UTF8 locale
-RUN apt-get update && apt-get install -y locales
-RUN locale-gen en_US.UTF-8
-ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8'
-
-# Convenience settings for the testbed's root account
-RUN echo 'set -o vi' >> /root/.bashrc
-
-# Install prerequisites
-RUN \
-  apt-get update && \
-  apt-get install -y \
-    curl \
-    expect \
-    git \
-    gnupg1 \
-    lsb-release \
-    make \
-    python3-pip \
-    shellcheck=0.4.6-1 \
-    vim \
-  ;
-RUN pip3 install \
-      envtpl \
-      j2cli \
-      flake8==3.7.8 \
-      pylint==2.4.1 \
-      pytest==5.1.3 \
-      yamllint==1.17.0 \
-    ;
-
-# Force GNUPG version 1 at path /usr/bin/gpg
-RUN ln -fs /usr/bin/gpg1 /usr/bin/gpg
-
-# Create a flag to identify when running inside the yadm testbed
-RUN touch /.yadmtestbed
-
-# /yadm will be the work directory for all tests
-# docker commands should mount the local yadm project as /yadm
-WORKDIR /yadm
-
-# Create a Makefile to be used if no /yadm volume is mounted
-RUN echo "test:\n\t@echo 'The yadm project must be mounted at /yadm'\n\t@echo 'Try using a docker parameter like -v \"\$\$PWD:/yadm:ro\"'\n\t@false" > /yadm/Makefile
-
-# By default, run all tests defined
-CMD make test

Makefile

@@ -1,4 +1,6 @@
 PYTESTS = $(wildcard test/test_*.py)
+IMAGE = docker.io/yadm/testbed:2022-01-07
+OCI = docker
 
 .PHONY: all
 all:
@@ -22,7 +24,7 @@ usage:
 	@echo
 	@echo '  make testhost [version=VERSION]'
 	@echo '    - Create an ephemeral container for doing adhoc yadm testing. The'
-	@echo '      HEAD revision of yadm will be used unless "version" is'
+	@echo '      working copy version of yadm will be used unless "version" is'
 	@echo '      specified. "version" can be set to any commit, branch, tag, etc.'
 	@echo '      The targeted "version" will be retrieved from the repo, and'
 	@echo '      linked into the container as a local volume.'
@@ -32,7 +34,7 @@ usage:
 	@echo '      exiting the shell, a log of the commands used to illustrate the'
 	@echo '      problem will be written to the file "script.txt". This file can'
 	@echo '      be useful to developers to make a repeatable test for the'
-	@echo '      problem.'
+	@echo '      problem. The version parameter works as for "testhost" above.'
 	@echo
 	@echo 'LINTING'
 	@echo
@@ -81,7 +83,7 @@ usage:
 # Make it possible to run make specifying a py.test test file
 .PHONY: $(PYTESTS)
 $(PYTESTS):
-	@$(MAKE) test testargs="-k $@ $(testargs)"
+	@$(MAKE) test testargs="$@ $(testargs)"
 %.py:
 	@$(MAKE) test testargs="-k $@ $(testargs)"
 
@@ -92,65 +94,75 @@ test:
 		cd /yadm && \
 		py.test -v $(testargs); \
 	else \
-		if command -v "docker-compose" >/dev/null 2>&1; then \
-			docker-compose run --rm testbed make test testargs="$(testargs)"; \
-		else \
-			echo "Sorry, this make test requires docker-compose to be installed."; \
-			false; \
-		fi \
+		$(MAKE) -s require-docker && \
+		$(OCI) run \
+			--rm -t$(shell test -t 0 && echo i) \
+			-v "$(CURDIR):/yadm:ro" \
+			$(IMAGE) \
+			make test testargs="$(testargs)"; \
 	fi
 
+.PHONY: .testyadm
+.testyadm: version ?= local
+.testyadm:
+	@rm -f $@
+	@if [ "$(version)" = "local" ]; then \
+		ln -sf yadm $@; \
+		echo "Using local yadm ($$(git describe --tags --dirty))"; \
+	else \
+		git show $(version):yadm > $@; \
+		echo "Using yadm version $$(git describe --tags $(version))"; \
+	fi
+	@chmod a+x $@
+
 .PHONY: testhost
-testhost: require-docker
-	@version=HEAD
-	@rm -rf /tmp/testhost
-	@git show $(version):yadm > /tmp/testhost
-	@chmod a+x /tmp/testhost
-	@echo Starting testhost version=\"$$version\"
-	@docker run \
+testhost: require-docker .testyadm
+	@echo "Starting testhost"
+	@$(OCI) run \
 		-w /root \
 		--hostname testhost \
 		--rm -it \
-		-v "/tmp/testhost:/bin/yadm:ro" \
-		yadm/testbed:2019-09-25 \
+		-v "$(CURDIR)/.testyadm:/bin/yadm:ro" \
+		$(IMAGE) \
 		bash -l
 
 .PHONY: scripthost
-scripthost: require-docker
-	@version=HEAD
-	@rm -rf /tmp/testhost
-	@git show $(version):yadm > /tmp/testhost
-	@chmod a+x /tmp/testhost
-	@echo Starting scripthost version=\"$$version\" \(recording script\)
+scripthost: require-docker .testyadm
+	@echo "Starting scripthost \(recording script\)"
 	@printf '' > script.gz
-	@docker run \
+	@$(OCI) run \
 		-w /root \
 		--hostname scripthost \
 		--rm -it \
-		-v "$$PWD/script.gz:/script.gz:rw" \
-		-v "/tmp/testhost:/bin/yadm:ro" \
-		yadm/testbed:2019-09-25 \
+		-v "$(CURDIR)/script.gz:/script.gz:rw" \
+		-v "$(CURDIR)/.testyadm:/bin/yadm:ro" \
+		$(IMAGE) \
 		bash -c "script /tmp/script -q -c 'bash -l'; gzip < /tmp/script > /script.gz"
 	@echo
-	@echo "Script saved to $$PWD/script.gz"
+	@echo "Script saved to $(CURDIR)/script.gz"
 
 
 .PHONY: testenv
 testenv:
 	@echo 'Creating a local virtual environment in "testenv/"'
 	@echo
-	virtualenv --python=python3 testenv
+	@rm -rf testenv
+	python3 -m venv --clear testenv
 	testenv/bin/pip3 install --upgrade pip setuptools
-	testenv/bin/pip3 install --upgrade \
-		flake8==3.7.8 \
-		pylint==2.4.1 \
-		pytest==5.1.3 \
-		yamllint==1.17.0 \
-	;
+	testenv/bin/pip3 install --upgrade -r test/requirements.txt;
+	@for v in $$(sed -En -e 's:.*/yadm-([0-9.]+)$$:\1:p' test/Dockerfile); do \
+		git show $$v:yadm > testenv/bin/yadm-$$v; \
+		chmod +x testenv/bin/yadm-$$v; \
+	done
 	@echo
 	@echo 'To activate this test environment type:'
 	@echo '  source testenv/bin/activate'
 
+.PHONY: image
+image:
+	@$(OCI) build -f test/Dockerfile . -t "$(IMAGE)"
+
+
 .PHONY: man
 man:
 	@groff -man -Tascii ./yadm.1 | less
@@ -164,21 +176,26 @@ man-ps:
 	@groff -man -Tps ./yadm.1 > yadm.ps
 
 yadm.md: yadm.1
-	@groff -man -Tascii ./yadm.1 | col -bx | sed 's/^[A-Z]/## &/g' | sed '/yadm(1)/d' > yadm.md
+	@groff -man -Tutf8 -Z ./yadm.1 | grotty -c | col -bx | sed 's/^[A-Z]/## &/g' | sed '/yadm(1)/d' > yadm.md
 
 .PHONY: contrib
+contrib: SHELL = /bin/bash
 contrib:
-	@echo "CONTRIBUTORS\n" > CONTRIBUTORS
-	@git shortlog -ns master gh-pages develop dev-pages | cut -f2 >> CONTRIBUTORS
+	@echo -e "CONTRIBUTORS\n" > CONTRIBUTORS
+	@IFS=$$'\n'; for author in $$(git shortlog -ns master gh-pages develop dev-pages | cut -f2); do \
+		git log master gh-pages develop dev-pages \
+			--author="$$author" --format=tformat: --numstat | \
+			awk "{sum += \$$1 + \$$2} END {print sum \"\t\" \"$$author\"}"; \
+	done | sort -nr | cut -f2 >> CONTRIBUTORS
 
 .PHONY: install
 install:
 	@[ -n "$(PREFIX)" ] || { echo "PREFIX is not set"; exit 1; }
 	@{\
 		set -e                                               ;\
-		bin="$(PREFIX)/bin"                                  ;\
-		doc="$(PREFIX)/share/doc/yadm"                       ;\
-		man="$(PREFIX)/share/man/man1"                       ;\
+		bin="$(DESTDIR)$(PREFIX)/bin"                        ;\
+		doc="$(DESTDIR)$(PREFIX)/share/doc/yadm"             ;\
+		man="$(DESTDIR)$(PREFIX)/share/man/man1"             ;\
 		install -d "$$bin" "$$doc" "$$man"                   ;\
 		install -m 0755 yadm "$$bin"                         ;\
 		install -m 0644 yadm.1 "$$man"                       ;\
@@ -188,11 +205,11 @@ install:
 
 .PHONY: sync-clock
 sync-clock:
-	docker run --rm --privileged alpine hwclock -s
+	$(OCI) run --rm --privileged alpine hwclock -s
 
 .PHONY: require-docker
 require-docker:
-	@if ! command -v "docker" >/dev/null 2>&1; then \
-		echo "Sorry, this make target requires docker to be installed."; \
+	@if ! command -v $(OCI) > /dev/null 2>&1; then \
+		echo "Sorry, this make target requires docker to be installed, to use another docker-compatible engine, like podman, re-run the make command adding OCI=podman"; \
 		false; \
 	fi

README.md

@@ -3,50 +3,85 @@
 [![Latest Version][releases-badge]][releases-link]
 [![Homebrew Version][homebrew-badge]][homebrew-link]
 [![OBS Version][obs-badge]][obs-link]
-[![Arch Version][aur-badge]][aur-link]
+[![Arch Version][arch-badge]][arch-link]
 [![License][license-badge]][license-link]<br />
 [![Master Update][master-date]][master-commits]
 [![Develop Update][develop-date]][develop-commits]
 [![Website Update][website-date]][website-commits]<br />
-[![Master Status][master-badge]][travis-ci]
-[![Develop Status][develop-badge]][travis-ci]
-[![GH Pages Status][gh-pages-badge]][travis-ci]
-[![Dev Pages Status][dev-pages-badge]][travis-ci]
+[![Master Status][master-badge]][workflow-master]
+[![Develop Status][develop-badge]][workflow-develop]
+[![GH Pages Status][gh-pages-badge]][workflow-gh-pages]
+[![Dev Pages Status][dev-pages-badge]][workflow-dev-pages]
 
 [https://yadm.io/][website-link]
 
-[**yadm**][website-link] is a tool for managing [dotfiles][].
+**yadm** is a tool for managing [dotfiles][].
 
 * Based on [Git][], with full range of Git's features
-* Supports system-specific alternative files
-* Encryption of private data using [GnuPG][]
+* Supports system-specific alternative files or templated files
+* Encryption of private data using [GnuPG][], [OpenSSL][], [transcrypt][], or
+  [git-crypt][]
 * Customizable initialization (bootstrapping)
+* Customizable hooks for before and after any operation
 
-Features, usage, examples and installation instructions can be found on the
-[website][website-link].
+Complete features, usage, examples and installation instructions can be found on
+the [yadm.io][website-link] website.
+
+## A very quick tour
+
+    # Initialize a new repository
+    yadm init
+
+    # Clone an existing repository
+    yadm clone <url>
+
+    # Add files/changes
+    yadm add <important file>
+    yadm commit
+
+    # Encrypt your ssh key
+    echo '.ssh/id_rsa' > ~/.config/yadm/encrypt
+    yadm encrypt
+
+    # Later, decrypt your ssh key
+    yadm decrypt
+
+    # Create different files for Linux vs MacOS
+    yadm add path/file.cfg##os.Linux
+    yadm add path/file.cfg##os.Darwin
+
+If you enjoy using yadm, consider adding a star to the repository on GitHub.
+The star count helps others discover yadm.
 
 [Git]: https://git-scm.com/
 [GnuPG]: https://gnupg.org/
-[aur-badge]: https://img.shields.io/aur/version/yadm-git.svg
-[aur-link]: https://aur.archlinux.org/packages/yadm-git
-[dev-pages-badge]: https://img.shields.io/travis/TheLocehiliosan/yadm/dev-pages.svg?label=dev-pages
-[develop-badge]: https://img.shields.io/travis/TheLocehiliosan/yadm/develop.svg?label=develop
+[OpenSSL]: https://www.openssl.org/
+[arch-badge]: https://img.shields.io/archlinux/v/extra/any/yadm
+[arch-link]: https://archlinux.org/packages/extra/any/yadm/
+[dev-pages-badge]: https://img.shields.io/github/actions/workflow/status/TheLocehiliosan/yadm/test.yml?branch=dev-pages
+[develop-badge]: https://img.shields.io/github/actions/workflow/status/TheLocehiliosan/yadm/test.yml?branch=develop
 [develop-commits]: https://github.com/TheLocehiliosan/yadm/commits/develop
 [develop-date]: https://img.shields.io/github/last-commit/TheLocehiliosan/yadm/develop.svg?label=develop
 [dotfiles]: https://en.wikipedia.org/wiki/Hidden_file_and_hidden_directory
-[gh-pages-badge]: https://img.shields.io/travis/TheLocehiliosan/yadm/gh-pages.svg?label=gh-pages
+[gh-pages-badge]: https://img.shields.io/github/actions/workflow/status/TheLocehiliosan/yadm/test.yml?branch=gh-pages
+[git-crypt]: https://github.com/AGWA/git-crypt
 [homebrew-badge]: https://img.shields.io/homebrew/v/yadm.svg
 [homebrew-link]: https://formulae.brew.sh/formula/yadm
 [license-badge]: https://img.shields.io/github/license/TheLocehiliosan/yadm.svg
 [license-link]: https://github.com/TheLocehiliosan/yadm/blob/master/LICENSE
-[master-badge]: https://img.shields.io/travis/TheLocehiliosan/yadm/master.svg?label=master
+[master-badge]: https://img.shields.io/github/actions/workflow/status/TheLocehiliosan/yadm/test.yml?branch=master
 [master-commits]: https://github.com/TheLocehiliosan/yadm/commits/master
 [master-date]: https://img.shields.io/github/last-commit/TheLocehiliosan/yadm/master.svg?label=master
-[obs-badge]: https://img.shields.io/badge/OBS-v2.0.1-blue
+[obs-badge]: https://img.shields.io/badge/OBS-v3.2.2-blue
 [obs-link]: https://software.opensuse.org//download.html?project=home%3ATheLocehiliosan%3Ayadm&package=yadm
 [releases-badge]: https://img.shields.io/github/tag/TheLocehiliosan/yadm.svg?label=latest+release
 [releases-link]: https://github.com/TheLocehiliosan/yadm/releases
-[travis-ci]: https://travis-ci.org/TheLocehiliosan/yadm/branches
+[transcrypt]: https://github.com/elasticdog/transcrypt
+[travis-ci]: https://travis-ci.com/TheLocehiliosan/yadm/branches
 [website-commits]: https://github.com/TheLocehiliosan/yadm/commits/gh-pages
 [website-date]: https://img.shields.io/github/last-commit/TheLocehiliosan/yadm/gh-pages.svg?label=website
 [website-link]: https://yadm.io/
+[workflow-dev-pages]: https://github.com/thelocehiliosan/yadm/actions?query=workflow%3a%22test+site%22+branch%3adev-pages
+[workflow-develop]: https://github.com/TheLocehiliosan/yadm/actions?query=workflow%3ATests+branch%3Adevelop
+[workflow-gh-pages]: https://github.com/thelocehiliosan/yadm/actions?query=workflow%3a%22test+site%22+branch%3agh-pages
+[workflow-master]: https://github.com/TheLocehiliosan/yadm/actions?query=workflow%3ATests+branch%3Amaster

bootstrap

@@ -35,7 +35,7 @@ REPO_URL=""
 
 function _private_yadm() {
   unset -f yadm
-  if command -v yadm >/dev/null 2>&1; then
+  if command -v yadm &> /dev/null; then
     echo "Found yadm installed locally, removing remote yadm() function"
     unset -f _private_yadm
     command yadm "$@"
@@ -57,7 +57,7 @@ function remote_yadm() {
 }
 
 function ask_about_source() {
-  if ! command -v yadm >/dev/null 2>&1; then
+  if ! command -v yadm &> /dev/null; then
     echo
     echo "***************************************************"
     echo "yadm is NOT currently installed."

completion/README.md

@@ -1,36 +1,47 @@
 # Installation
 
-## Bash completions
 ### Prerequisites
-**yadm** completion only works if Git completions are also enabled.
 
-### Homebrew
-If using `homebrew` to install **yadm**, completions should automatically be handled if you also install `brew install bash-completion`. This might require you to include the main completion script in your own bashrc file like this:
-```
+Bash and Zsh completion only works if Git completions are also enabled.
+
+## Homebrew
+
+If using `homebrew` to install yadm, Bash, Zsh, and Fish completions should
+automatically be installed. For Bash and Zsh, you also must install
+`bash-completion` or `zsh-completions`. This might require you to include the
+main completion script in your own shell configuration like this:
+
+```bash
 [ -f /usr/local/etc/bash_completion ] && source /usr/local/etc/bash_completion
 ```
 
-### Manual installation
+## Bash (manual installation)
+
 Copy the completion script locally, and add this to you bashrc:
-```
-[ -f /full/path/to/yadm.bash_completion ] && source /full/path/to/yadm.bash_completion
+
+```bash
+[ -f /path/to/yadm/completion/bash/yadm ] && source /path/to/yadm/completion/bash/yadm
 ```
 
-## Zsh completions
-### Homebrew
-If using `homebrew` to install **yadm**, completions should handled automatically.
+## Zsh (manual installation)
 
-### Manual installation
-Copy the completion script `yadm.zsh_completion` locally, rename it to `_yadm`, and add the containing folder to `$fpath` in `.zshrc`:
-```
-fpath=(/path/to/folder/containing_yadm $fpath)
+Add the `completion/zsh` folder to `$fpath` in `.zshrc`:
+
+```zsh
+fpath=(/path/to/yadm/completion/zsh $fpath)
 autoload -U compinit
 compinit
 ```
 
-### Installation using [zplug](https://github.com/b4b4r07/zplug)
+## Zsh (using [zplug](https://github.com/b4b4r07/zplug))
+
 Load `_yadm` as a plugin in your `.zshrc`:
-```
+
+```zsh
 fpath=("$ZPLUG_HOME/bin" $fpath)
-zplug "TheLocehiliosan/yadm", rename-to:_yadm, use:"completion/yadm.zsh_completion", as:command, defer:2
+zplug "TheLocehiliosan/yadm", use:"completion/zsh/_yadm", as:command, defer:2
 ```
+
+## Fish (manual installation)
+
+Copy the completion script `yadm.fish` to any folder within `$fish_complete_path`. For example, for local installation, you can copy it to `$HOME/.config/fish/completions/` and it will be loaded when `yadm` is invoked.

completion/bash/yadm

@@ -1,10 +1,12 @@
 # test if git completion is missing, but loader exists, attempt to load
-if ! declare -F _git > /dev/null && declare -F _completion_loader > /dev/null; then
-  _completion_loader git
+if ! declare -F _git > /dev/null && ! declare -F __git_wrap__git_main > /dev/null; then
+  if declare -F _completion_loader > /dev/null; then
+    _completion_loader git
+  fi
 fi
 
 # only operate if git completion is present
-if declare -F _git > /dev/null; then
+if declare -F _git > /dev/null || declare -F __git_wrap__git_main > /dev/null; then
 
   _yadm() {
 
@@ -18,7 +20,7 @@ if declare -F _git > /dev/null; then
       antepenultimate=${COMP_WORDS[COMP_CWORD-2]}
     fi
 
-    local GIT_DIR
+    local -x GIT_DIR
     # shellcheck disable=SC2034
     GIT_DIR="$(yadm introspect repo 2>/dev/null)"
 
@@ -60,19 +62,37 @@ if declare -F _git > /dev/null; then
 		  ;;
     esac
 
+    local yadm_switches=( $(yadm introspect switches 2>/dev/null) )
+
     # this condition is so files are completed properly for --yadm-xxx options
-    if [[ ! "$penultimate" =~ ^- ]]; then
+    if [[ " ${yadm_switches[*]} " != *" $penultimate "* ]]; then
       # TODO: somehow solve the problem with [--yadm-xxx option] being
       #       incompatible with what git expects, namely [--arg=option]
-      _git
+      if declare -F _git > /dev/null; then
+        _git
+      else
+        __git_wrap__git_main
+      fi
     fi
     if [[ "$current" =~ ^- ]]; then
       local matching
-      matching=$(compgen -W "$(yadm introspect switches 2>/dev/null)" -- "$current")
+      matching=$(compgen -W "${yadm_switches[*]}" -- "$current")
       __gitcompappend "$matching"
     fi
 
-    if [ "$COMP_CWORD" == 1 ] || [[ "$antepenultimate" =~ ^- ]] ; then
+    # Find the index of where the sub-command argument should go.
+    local command_idx
+    for (( command_idx=1 ; command_idx < ${#COMP_WORDS[@]} ; command_idx++ )); do
+      local command_idx_arg="${COMP_WORDS[$command_idx]}"
+      if [[ " ${yadm_switches[*]} " = *" $command_idx_arg "* ]]; then
+        let command_idx++
+      elif [[ "$command_idx_arg" = -* ]]; then
+        :
+      else
+        break
+      fi
+    done
+    if [[ "$COMP_CWORD" = "$command_idx" ]]; then
       local matching
       matching=$(compgen -W "$(yadm introspect commands 2>/dev/null)" -- "$current")
       __gitcompappend "$matching"

completion/fish/yadm.fish

@@ -0,0 +1,77 @@
+#!/usr/bin/fish
+
+function __fish_yadm_universial_optspecs
+    string join \n 'a-yadm-dir=' 'b-yadm-repo=' 'c-yadm-config=' \
+                    'd-yadm-encrypt=' 'e-yadm-archive=' 'f-yadm-bootstrap='
+end
+
+function __fish_yadm_needs_command
+    # Figure out if the current invocation already has a command.
+    set -l cmd (commandline -opc)
+    set -e cmd[1]
+    argparse -s (__fish_yadm_universial_optspecs) -- $cmd 2>/dev/null
+    or return 0
+    if set -q argv[1]
+        echo $argv[1]
+        return 1
+    end
+    return 0
+end
+
+function __fish_yadm_using_command
+    set -l cmd (__fish_yadm_needs_command)
+    test -z "$cmd"
+    and return 1
+    contains -- $cmd $argv
+    and return 0
+end
+
+# yadm's specific autocomplete
+complete -x -c yadm -n '__fish_yadm_needs_command'       -a 'clone'      -d 'Clone an existing repository'
+complete -F -c yadm -n '__fish_yadm_using_command clone' -s w            -d 'work-tree to use (default: $HOME)'
+complete -f -c yadm -n '__fish_yadm_using_command clone' -s b            -d 'branch to clone'
+complete -x -c yadm -n '__fish_yadm_using_command clone' -s f            -d 'force to overwrite'
+complete -x -c yadm -n '__fish_yadm_using_command clone' -l bootstrap    -d 'force bootstrap to run'
+complete -x -c yadm -n '__fish_yadm_using_command clone' -l no-bootstrap -d 'prevent bootstrap from beingrun'
+
+complete -x -c yadm -n '__fish_yadm_needs_command' -a 'alt'       -d 'Create links for alternates'
+complete -x -c yadm -n '__fish_yadm_needs_command' -a 'bootstrap' -d 'Execute $HOME/.config/yadm/bootstrap'
+complete -x -c yadm -n '__fish_yadm_needs_command' -a 'perms'     -d 'Fix perms for private files'
+complete -x -c yadm -n '__fish_yadm_needs_command' -a 'enter'     -d 'Run sub-shell with GIT variables set'
+complete    -c yadm -n '__fish_yadm_needs_command' -a 'git-crypt' -d 'Run git-crypt commands for the yadm repo'
+complete -x -c yadm -n '__fish_yadm_needs_command' -a 'help'      -d 'Print a summary of yadm commands'
+complete -x -c yadm -n '__fish_yadm_needs_command' -a 'upgrade'   -d 'Upgrade to version 2 of yadm directory structure'
+complete -x -c yadm -n '__fish_yadm_needs_command' -a 'version'   -d 'Print the version of yadm'
+
+complete -x -c yadm -n '__fish_yadm_needs_command' -a 'init' -d 'Initialize an empty repository'
+complete -x -c yadm -n '__fish_yadm_using_command init' -s f -d 'force to overwrite'
+complete -F -c yadm -n '__fish_yadm_using_command init' -s w -d 'set work-tree (default: $HOME)'
+
+complete -x -c yadm -n '__fish_yadm_needs_command' -a 'list' -d 'List tracked files at current directory'
+complete -x -c yadm -n '__fish_yadm_using_command list' -s a -d 'list all managed files instead'
+
+complete -x -c yadm -n '__fish_yadm_needs_command' -a 'encrypt' -d 'Encrypt files'
+complete -x -c yadm -n '__fish_yadm_needs_command' -a 'decrypt' -d 'Decrypt files'
+complete -x -c yadm -n '__fish_yadm_using_command decrypt' -s l -d 'list the files stored without extracting'
+
+complete -x -c yadm -n '__fish_yadm_needs_command' -a 'introspect' -d 'Report internal yadm data'
+complete -x -c yadm -n '__fish_yadm_using_command introspect' -a (printf -- '%s\n' 'commands configs repo switches') -d 'category'
+
+complete -x -c yadm -n '__fish_yadm_needs_command' -a 'gitconfig' -d 'Pass options to the git config command'
+complete -x -c yadm -n '__fish_yadm_needs_command' -a 'config'    -d 'Configure a setting'
+for name in (yadm introspect configs)
+    complete -x -c yadm -n '__fish_yadm_using_command config' -a '$name' -d 'yadm config'
+end
+
+# yadm universial options
+complete --force-files -c yadm -s Y -l yadm-dir       -d 'Override location of yadm directory'
+complete --force-files -c yadm      -l yadm-repo      -d 'Override location of yadm repository'
+complete --force-files -c yadm      -l yadm-config    -d 'Override location of yadm configuration file'
+complete --force-files -c yadm      -l yadm-encrypt   -d 'Override location of yadm encryption configuration'
+complete --force-files -c yadm      -l yadm-archive   -d 'Override location of yadm encrypted files archive'
+complete --force-files -c yadm      -l yadm-bootstrap -d 'Override location of yadm bootstrap program'
+
+# wraps git's autocomplete
+set -l GIT_DIR (yadm introspect repo)
+# setup the correct git-dir by appending it to git's argunment
+complete -c yadm -w "git --git-dir=$GIT_DIR"

completion/yadm.zsh_completion

@@ -1,46 +0,0 @@
-#compdef yadm
-_yadm(){
-    local -a _1st_arguments
-     _1st_arguments=(
-        'help:Display yadm command help'
-        'init:Initialize an empty repository'
-        'config:Configure a setting'
-        'list:List tracked files'
-        'alt:Create links for alternates'
-        'bootstrap:Execute $HOME/.config/yadm/bootstrap'
-        'encrypt:Encrypt files'
-        'decrypt:Decrypt files'
-        'perms:Fix perms for private files'
-        'add:git add'
-        'push:git push'
-        'pull:git pull'
-        'diff:git diff'
-        'checkout:git checkout'
-        'co:git co'
-        'commit:git commit'
-        'ci:git ci'
-        'status:git status'
-        'st:git st'
-        'reset:git reset'
-        'log:git log'
-    )
-
-    local context state line expl
-    local -A opt_args
-
-    _arguments '*:: :->subcmds' && return 0
-
-    if (( CURRENT == 1 )); then
-        _describe -t commands "yadm commands" _1st_arguments -V1
-        return
-    fi
-
-    case "$words[1]" in
-        *)
-            _arguments ':filenames:_files'
-        ;;
-    esac
-
-}
-
-_yadm "$@"

completion/zsh/_yadm

@@ -0,0 +1,180 @@
+#compdef yadm
+
+# This completion tries to fallback to git's completion for git commands.
+
+zstyle -T ':completion:*:yadm:argument-1:descriptions:' format && \
+    zstyle ':completion:*:yadm:argument-1:descriptions' format '%d:'
+zstyle -T ':completion:*:yadm:*:yadm' group-name && \
+    zstyle ':completion:*:yadm:*:yadm' group-name ''
+
+function _yadm-add(){
+  local -a yadm_options yadm_path
+  yadm_path="$(yadm rev-parse --show-toplevel)"
+  yadm_options=($(yadm status --porcelain=v1 |
+    awk -v yadm_path=${yadm_path} '{printf "%s/%s:%s\n",  yadm_path, $2, $1}' ))
+
+  _describe 'command' yadm_options
+  _files
+}
+
+function _yadm-checkout(){
+    _yadm-add
+}
+
+_yadm-alt() {
+    return 0
+}
+
+_yadm-bootstrap() {
+    return 0
+}
+
+_yadm-clone() {
+    _arguments \
+        '(--bootstrap --no-bootstrap)--bootstrap[force bootstrap, without prompt]' \
+        '(--bootstrap --no-bootstrap)--no-bootstrap[prevent bootstrap, without prompt]' \
+        '-f[force overwrite of existing repository]' \
+        '-w[yadm work tree path]: :_files -/'
+
+    local curcontext="${curcontext%:*:*}:git:"
+
+    words=("git" "${words[@]}") CURRENT=$((CURRENT + 1)) service=git _git
+}
+
+_yadm-config() {
+    # TODO: complete config names
+}
+
+_yadm-decrypt() {
+    _arguments \
+        '-l[list files]'
+}
+
+_yadm-encrypt() {
+    return 0
+}
+
+_yadm-enter() {
+    _arguments \
+        ':command: _command_names -e' \
+        '*::arguments: _normal'
+}
+
+_yadm-git-crypt() {
+    # TODO: complete git-crypt options
+}
+
+_yadm-help() {
+    return 0
+}
+
+_yadm-init() {
+    _arguments \
+        '-f[force overwrite of existing repository]' \
+        '-w[work tree path]: :_files -/'
+}
+
+_yadm-list() {
+    _arguments \
+        '-a[list all tracked files]'
+}
+
+_yadm-perms() {
+    return 0
+}
+
+_yadm-transcrypt() {
+    integer _ret=1
+    _call_function _ret _transcrypt
+    return _ret
+}
+
+_yadm-upgrade() {
+    _arguments \
+        '-f[force deinit of submodules]' \
+        ': '
+}
+
+_yadm-version() {
+    return 0
+}
+
+_yadm_commands() {
+    local -a commands=(
+        alt:'create links for alternates'
+        bootstrap:'execute bootstrap'
+        clone:'clone an existing yadm repository'
+        config:'configure an yadm setting'
+        decrypt:'decrypt files'
+        encrypt:'encrypt files'
+        enter:'run sub-shell with GIT variables set'
+        git-crypt:'run git-crypt commands for the yadm repository'
+        gitconfig:'run the git config command'
+        help:'display yadm help information'
+        init:'initialize an empty yadm repository'
+        list:'list files tracked by yadm'
+        perms:'fix perms for private files'
+        transcrypt:'run transcrypt commands for the yadm repository'
+        upgrade:'upgrade legacy yadm paths'
+        version:'show yadm version'
+    )
+
+    local oldcontext="$curcontext"
+    local curcontext="${curcontext%:*:*}:git:"
+
+    words=("git" "${words[-1]}") CURRENT=2 service=git _git
+
+    curcontext="$oldcontext"
+    _describe -t yadm "yadm commands" commands
+
+    return 0
+}
+
+_yadm() {
+    local curcontext=$curcontext state state_descr line
+    declare -A opt_args
+
+    _arguments -C \
+      '(-Y --yadm-dir)'{-Y,--yadm-dir}'[override the standard yadm directory]: :_files -/' \
+      '--yadm-data[override the standard yadm data directory]: :_files -/' \
+      '--yadm-repo[override the standard repo path]: :_files -/' \
+      '--yadm-config[override the standard config path]: :_files -/' \
+      '--yadm-encrypt[override the standard encrypt path]: :_files -/' \
+      '--yadm-archive[override the standard archive path]: :_files -/' \
+      '--yadm-bootstrap[override the standard bootstrap path]: :_files' \
+      '--help[display yadm help information]' \
+      '--version[show yadm version]' \
+      '(-): :->command' \
+      '(-)*:: :->option-or-argument' && return
+
+    local -a repo_args
+    (( $+opt_args[--yadm-repo] )) && repo_args+=(--yadm-repo "$opt_args[--yadm-repo]")
+    (( $+opt_args[--yadm-data] )) && repo_args+=(--yadm-data "$opt_args[--yadm-data]")
+    local -x GIT_DIR="$(_call_program gitdir yadm "${repo_args[@]}" introspect repo)"
+    [[ -z "$GIT_DIR" ]] && return 1
+
+    integer _ret=1
+    case $state in
+        (command)
+            _yadm_commands && _ret=0
+            ;;
+        (option-or-argument)
+            curcontext=${curcontext%:*:*}:yadm-${words[1]}:
+            if ! _call_function _ret _yadm-${words[1]}; then
+
+                # Translate gitconfig to use the regular completion for config
+                [[ ${words[1]} = "gitconfig" ]] && words[1]=config
+
+                words=("git" "${(@)words}")
+                CURRENT=$(( CURRENT + 1 ))
+
+                curcontext=${curcontext%:*:*}:git:
+                service=git _git && _ret=0
+            fi
+            ;;
+    esac
+
+    return _ret
+}
+
+(( $+functions[_git] )) && _yadm

contrib/bootstrap/bootstrap-in-dir

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+# Save this file as ~/.config/yadm/bootstrap and make it executable. It will
+# execute all executable files (excluding templates and editor backups) in the
+# ~/.config/yadm/bootstrap.d directory when run.
+
+set -eu
+
+# Directory to look for bootstrap executables in
+BOOTSTRAP_D="${BASH_SOURCE[0]}.d"
+
+if [[ ! -d "$BOOTSTRAP_D" ]]; then
+    echo "Error: bootstrap directory '$BOOTSTRAP_D' not found" >&2
+    exit 1
+fi
+
+find -L "$BOOTSTRAP_D" -type f | sort | while IFS= read -r bootstrap; do
+    if [[ -x "$bootstrap" && ! "$bootstrap" =~ "##" && ! "$bootstrap" =~ "~$" ]]; then
+        if ! "$bootstrap"; then
+            echo "Error: bootstrap '$bootstrap' failed" >&2
+            exit 1
+        fi
+    fi
+done

contrib/commands/README.md

@@ -0,0 +1,9 @@
+## Contributed Commands
+
+Although these commands are available as part of the official
+**yadm** source tree, they have a somewhat different status. The intention is to
+keep interesting and potentially useful commands here, building a library of
+examples that might help others.
+
+I recommend *careful review* of any code from here before using it. No
+guarantees of code quality is assumed.

contrib/commands/yadm-untracked

@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+
+# To run: `yadm-untracked <config-file>`
+#
+# If you wish to create a YADM alias to run this as, for example `yadm untracked`
+# then the following command will add the alias:
+#     `yadm gitconfig alias.untracked '!<PATH>/yadm-untracked'`
+
+# Possible script improvements:
+# - Reduce the amount of configuration; I have not figured out a way to
+#   get rid of the non-recursive and ignore. The recursive list could be
+#   built from the directories that are present in `yadm list`
+
+# Configuration... The script looks at the following 3 arrays:
+#
+# yadm_tracked_recursively
+#     The directories and files in this list are searched recursively to build
+#     a list of files that you expect are tracked with `yadm`. Items in this
+#     list are relative to the root of your YADM repo (which is $HOME for most).
+
+# yadm_tracked_nonrecursively
+#     Same as above but don't search recursively
+#
+# ignore_files_and_dirs
+#     A list of directories and files that will not be reported as untracked if
+#     found in the above two searches.
+#
+# Example configuration file (uncomment it to use):
+# yadm_tracked_recursively=(
+#     bin .config .vim
+# )
+#
+# yadm_tracked_nonrecursively=(
+#     ~
+# )
+#
+# ignore_files_and_dirs=(
+#     .CFUserTextEncoding .DS_Store .config/gh
+#     .vim/autoload/plug.vim
+# )
+
+if [[ $# -ne 1 ]]; then
+    echo 'Usage: yadm-untracked <config-file>'
+    exit 1
+fi
+
+yadm_tracked_recursively=()
+yadm_tracked_nonrecursively=()
+ignore_files_and_dirs=()
+
+source $1
+
+root=`yadm enter echo '$GIT_WORK_TREE'`
+
+cd $root
+
+find_list=$(mktemp -t find_list)
+find ${yadm_tracked_recursively[*]} -type f >$find_list
+find ${yadm_tracked_nonrecursively[*]} -maxdepth 1 -type f |
+  awk "{sub(\"^\./\", \"\"); sub(\"^$root/\", \"\"); print }" >>$find_list
+sort -o $find_list $find_list
+
+yadm_list=$(mktemp -t yadm_list)
+yadm list >$yadm_list
+find ${ignore_files_and_dirs[*]} -type f >>$yadm_list
+sort -o $yadm_list $yadm_list
+
+# Show the files not in `yadm list`
+comm -23 $find_list $yadm_list
+
+rm -f $find_list $yadm_list

contrib/hooks/encrypt_with_checksums/README.md

@@ -0,0 +1,9 @@
+## Track checksums of encrypted files
+
+Contributed by Martin Zuther
+
+Hook         | Description
+----         | -----------
+post_encrypt | Collects the checksums of encrypted files, and stores them in .config/yadm/files.checksums
+post_list    | Prints the names of encrypted files
+post_status  | Reports untracked changes within encrypted files

contrib/hooks/encrypt_with_checksums/post_encrypt

@@ -0,0 +1,96 @@
+#!/usr/bin/env bash
+
+# yadm - Yet Another Dotfiles Manager
+# Copyright (C) 2015-2021 Tim Byrne and Martin Zuther
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+YADM_CHECKSUMS="$YADM_HOOK_DIR/files.checksums"
+WARNING_MESSAGE="No checksums were created"
+
+# unpack exported array; filenames including a newline character (\n)
+# are NOT supported
+OLD_IFS="$IFS"
+IFS=$'\n'
+YADM_ENCRYPT_INCLUDE_FILES=( $YADM_ENCRYPT_INCLUDE_FILES )
+IFS="$OLD_IFS"
+
+
+function get_checksum_command {
+    # check if "shasum" exists and supports the algorithm (which is
+    # tested by sending an empty string to "shasum")
+    if command -v "shasum" > /dev/null && printf "" | shasum --algorithm "256" &> /dev/null; then
+        printf "shasum --algorithm 256"
+    # check if "sha256sum" exists
+    elif command -v "sha256sum" > /dev/null; then
+        printf "sha256sum"
+    # check if "gsha256sum" exists
+    elif command -v "gsha256sum" > /dev/null; then
+        printf "gsha256sum"
+    else
+        # display warning in bright yellow
+        printf "\033[1;33m" >&2
+        printf "\nWARNING: \"shasum\", \"sha256sum\" and \"gsha256sum\" not found.   %s\n" "$WARNING_MESSAGE." >&2
+
+        # reset output color
+        printf "\033[0m" >&2
+
+        # signal error
+        return 1
+    fi
+}
+
+
+# get checksum command
+CHECKSUM_COMMAND=$(get_checksum_command)
+
+# no command found
+if (($?)); then
+    # return original exit status of yadm command
+    exit "$YADM_HOOK_EXIT"
+fi
+
+# empty (or create) checksum file
+true > "$YADM_CHECKSUMS"
+
+# calculate checksums for encrypted files
+for included in "${YADM_ENCRYPT_INCLUDE_FILES[@]}"; do
+    # highlight any errors in red
+    printf "\033[0;31m"
+
+    # calculate checksums
+    $CHECKSUM_COMMAND "$included" >> "$YADM_CHECKSUMS"
+    ERROR_CODE=$?
+
+    # reset output color
+    printf "\033[0m"
+
+    # handle errors
+    if (($ERROR_CODE)); then
+        # display warning in bright yellow
+        printf "\033[1;33m" >&2
+        printf "\nWARNING: an error occurred.  Please inspect the checksum file.\n" >&2
+
+        # reset output color
+        printf "\033[0m" >&2
+
+        # exit and signal error
+        exit $ERROR_CODE
+    fi
+done
+
+# announce success and return original exit status of yadm command
+printf "Wrote SHA-256 checksums: %s\n" "$YADM_CHECKSUMS"
+exit "$YADM_HOOK_EXIT"

contrib/hooks/encrypt_with_checksums/post_list

@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+
+# yadm - Yet Another Dotfiles Manager
+# Copyright (C) 2015-2021 Tim Byrne and Martin Zuther
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+YADM_CHECKSUMS="$YADM_HOOK_DIR/files.checksums"
+
+
+# is current directory on yadm's work path?
+# (adapted from https://unix.stackexchange.com/a/6438/122163)
+if [ "${PWD##$YADM_HOOK_WORK}" != "$PWD" ]; then
+    ON_WORK_PATH=1
+else
+    ON_WORK_PATH=0
+fi
+
+
+# list all files or only those in the subdirectories below?
+OPTION_LIST_ALL=0
+for argument in "${YADM_HOOK_FULL_COMMAND[@]}"; do
+    # mimick git ls-files by displaying all files when not on work
+    # path
+    if [ "$argument" = "-a" ] || [ $ON_WORK_PATH -eq 0 ]; then
+        OPTION_LIST_ALL=1
+        break
+    fi
+done
+
+
+# if there is no checksum file, exit with original status of yadm
+# command
+if [ ! -f "$YADM_CHECKSUMS" ]; then
+    exit "$YADM_HOOK_EXIT"
+fi
+
+# list encrypted files
+while IFS= read -r filename; do
+    # remove checksums from file names
+    filename="${filename##[a-zA-Z0-9]* }"
+
+    # list only files in the subdirectories below (i.e. files
+    # whose relative path doesn't begin with "../")
+    if [ $OPTION_LIST_ALL -eq 0 ]; then
+        REL_PATH=$(relative_path "$PWD" "$YADM_HOOK_WORK/$filename")
+
+        if [ "$REL_PATH" = "${REL_PATH##../}" ]; then
+            printf "%s\n" "$REL_PATH"
+        fi
+    # list all files
+    else
+        printf "%s\n" "$filename"
+    fi
+done < "$YADM_CHECKSUMS"
+
+# return original exit status of yadm command
+exit "$YADM_HOOK_EXIT"

contrib/hooks/encrypt_with_checksums/post_status

@@ -0,0 +1,100 @@
+#!/usr/bin/env bash
+
+# yadm - Yet Another Dotfiles Manager
+# Copyright (C) 2015-2021 Tim Byrne and Martin Zuther
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+YADM_CHECKSUMS="$YADM_HOOK_DIR/files.checksums"
+WARNING_MESSAGE="Checksums were not verified"
+
+# unpack exported array; filenames including a newline character (\n)
+# are NOT supported
+OLD_IFS="$IFS"
+IFS=$'\n'
+YADM_ENCRYPT_INCLUDE_FILES=( $YADM_ENCRYPT_INCLUDE_FILES )
+IFS="$OLD_IFS"
+
+
+function get_checksum_command {
+    # check if "shasum" exists and supports the algorithm (which is
+    # tested by sending an empty string to "shasum")
+    if command -v "shasum" > /dev/null && printf "" | shasum --algorithm "256" &> /dev/null; then
+        printf "shasum --algorithm 256"
+    # check if "sha256sum" exists
+    elif command -v "sha256sum" > /dev/null; then
+        printf "sha256sum"
+    # check if "gsha256sum" exists
+    elif command -v "gsha256sum" > /dev/null; then
+        printf "gsha256sum"
+    else
+        # display warning in bright yellow
+        printf "\033[1;33m" >&2
+        printf "\nWARNING: \"shasum\", \"sha256sum\" and \"gsha256sum\" not found.   %s\n" "$WARNING_MESSAGE." >&2
+
+        # reset output color
+        printf "\033[0m" >&2
+
+        # signal error
+        return 1
+    fi
+}
+
+
+# if there is no checksum file, exit with original status of yadm
+# command
+if [ ! -f "$YADM_CHECKSUMS" ]; then
+    exit "$YADM_HOOK_EXIT"
+fi
+
+# get checksum command
+CHECKSUM_COMMAND=$(get_checksum_command)
+
+# no command found
+if (($?)); then
+    # return original exit status of yadm command
+    exit "$YADM_HOOK_EXIT"
+fi
+
+# check encrypted files for differences and capture output and error
+# messages
+YADM_CHECKSUM_OUTPUT=$($CHECKSUM_COMMAND --check "$YADM_CHECKSUMS" 2>&1)
+ERROR_CODE=$?
+
+# handle mismatched checksums and errors
+if (($ERROR_CODE)); then
+    printf "\nSome SHA-256 sums do not match (or an error occurred):\n\n"
+
+    # display differing files and errors (highlighted in red)
+    printf "\033[0;31m"
+
+    while IFS= read -r line; do
+        # beautify output and get rid of unnecessary lines
+        line="${line%%*: [Oo][Kk]}"
+        line="${line%%: [Ff][Aa][Ii][Ll][Ee][Dd]}"
+        line="${line##*WARNING:*did NOT match}"
+
+        if [ -n "$line" ]; then
+            printf "%s\n" "$line"
+        fi
+    done <<< "$YADM_CHECKSUM_OUTPUT"
+
+    # reset output color
+    printf "\033[0m"
+
+    # display advice for differing files and signal error
+    printf "\nConsider running either \"yadm encrypt\" or \"yadm decrypt\".\n"
+    exit $ERROR_CODE
+fi

contrib/hooks/parsing_full_command_example/README.md

@@ -0,0 +1,7 @@
+## Example of parsing `$YADM_HOOK_FULL_COMMAND`
+
+Contributed by Tim Byrne
+
+Hook    | Description
+----    | -----------
+pre_log | Provides an example of parsing `$YADM_HOOK_FULL_COMMAND` in Bash

contrib/hooks/parsing_full_command_example/pre_log

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+# yadm exposes all parameters of the command which triggers a hook. Those
+# parameters are exported as the environment variable YADM_HOOK_FULL_COMMAND.
+# Any spaces, tabs, or backslashes in those parameters are escaped with a
+# backslash. The function `parse_full_command()` is a demonstration of parsing
+# those values which may be escaped.
+
+function parse_full_command() {
+  local delim=$'\x1e' # ASCII Record Separator
+  local space=$'\x1f' # ASCII Unit Separator
+  local tab=$'\t'     # ASCII TAB
+  local cmd
+  cmd="$YADM_HOOK_FULL_COMMAND"
+  cmd="${cmd//\\ /$space}"      # swap escaped spaces for `1f`
+  cmd="${cmd//\\\\/\\}"         # fix escaped backslashes
+  cmd="${cmd//\\$tab/$tab}"     # fix escaped tabs
+  cmd="${cmd// /$delim}"        # convert space delimiters to `1c`
+  cmd="${cmd//$space/ }"        # convert `1f` back to spaces
+  # parse data into an array
+  IFS=$delim read -r -a full_cmd <<< "$cmd"
+}
+parse_full_command
+for param in "${full_cmd[@]}"; do
+  echo "Parameter: '$param'"
+done

docker-compose.yml

@@ -1,7 +0,0 @@
----
-version: '3'
-services:
-  testbed:
-    volumes:
-      - .:/yadm:ro
-    image: yadm/testbed:2019-09-25

pylintrc

@@ -8,7 +8,7 @@ max-attributes=8
 max-statements=65
 
 [SIMILARITIES]
-min-similarity-lines=6
+min-similarity-lines=8
 
 [MESSAGES CONTROL]
 disable=redefined-outer-name

test/Dockerfile

@@ -0,0 +1,73 @@
+FROM ubuntu:18.04
+MAINTAINER Tim Byrne <sultan@locehilios.com>
+
+# Shellcheck and esh versions
+ARG SC_VER=0.8.0
+ARG ESH_VER=0.3.1
+
+# Install prerequisites and configure UTF-8 locale
+RUN \
+  echo "en_US.UTF-8 UTF-8" > /etc/locale.gen \
+  && apt-get update \
+  && DEBIAN_FRONTEND=noninteractive \
+    apt-get install -y --no-install-recommends \
+    expect \
+    git \
+    gnupg \
+    locales \
+    lsb-release \
+    make \
+    man \
+    python3-pip \
+    vim-tiny \
+    xz-utils \
+  && rm -rf /var/lib/apt/lists/* \
+  && update-locale LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8'
+
+ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8'
+
+# Convenience settings for the testbed's root account
+RUN echo 'set -o vi' >> /root/.bashrc
+
+# Create a flag to identify when running inside the yadm testbed
+RUN touch /.yadmtestbed
+
+# Install shellcheck
+ADD https://github.com/koalaman/shellcheck/releases/download/v$SC_VER/shellcheck-v$SC_VER.linux.x86_64.tar.xz /opt
+RUN cd /opt \
+  && tar xf shellcheck-v$SC_VER.linux.x86_64.tar.xz \
+  && rm -f shellcheck-v$SC_VER.linux.x86_64.tar.xz \
+  && ln -s /opt/shellcheck-v$SC_VER/shellcheck /usr/local/bin
+
+# Upgrade pip3 and install requirements
+COPY test/requirements.txt /tmp/requirements.txt
+RUN python3 -m pip install --upgrade pip setuptools \
+  && python3 -m pip install --upgrade -r /tmp/requirements.txt \
+  && rm -f /tmp/requirements
+
+# Install esh
+ADD https://raw.githubusercontent.com/jirutka/esh/v$ESH_VER/esh /usr/local/bin
+RUN chmod +x /usr/local/bin/esh
+
+# Create workdir and dummy Makefile to be used if no /yadm volume is mounted
+RUN mkdir /yadm \
+  && echo "test:" > /yadm/Makefile \
+  && echo "\t@echo 'The yadm project must be mounted at /yadm'" >> /yadm/Makefile \
+  && echo "\t@echo 'Try using a docker parameter like -v \"\$\$PWD:/yadm:ro\"'" >> /yadm/Makefile \
+  && echo "\t@false" >> /yadm/Makefile
+
+# Include released versions of yadm to test upgrades
+ADD https://raw.githubusercontent.com/TheLocehiliosan/yadm/1.12.0/yadm /usr/local/bin/yadm-1.12.0
+ADD https://raw.githubusercontent.com/TheLocehiliosan/yadm/2.5.0/yadm /usr/local/bin/yadm-2.5.0
+RUN chmod +x /usr/local/bin/yadm-*
+
+# Configure git to make it easier to test yadm manually
+RUN git config --system user.email "test@yadm.io" \
+  && git config --system user.name "Yadm Test"
+
+# /yadm will be the work directory for all tests
+# docker commands should mount the local yadm project as /yadm
+WORKDIR /yadm
+
+# By default, run all tests defined
+CMD make test

test/conftest.py

@@ -8,31 +8,42 @@ import os
 import platform
 import pwd
 from subprocess import Popen, PIPE
+import py
 import pytest
 
 
+def pytest_addoption(parser):
+    """Add options to pytest"""
+    parser.addoption(
+        "--force-linters",
+        action="store_true",
+        default=False,
+        help="Run linters regardless of installed versions",
+    )
+
+
 @pytest.fixture(scope='session')
 def shellcheck_version():
     """Version of shellcheck supported"""
-    return '0.4.6'
+    return '0.8.0'
 
 
 @pytest.fixture(scope='session')
 def pylint_version():
     """Version of pylint supported"""
-    return '2.4.1'
+    return '2.6.0'
 
 
 @pytest.fixture(scope='session')
 def flake8_version():
     """Version of flake8 supported"""
-    return '3.7.8'
+    return '3.8.4'
 
 
 @pytest.fixture(scope='session')
 def yamllint_version():
     """Version of yamllint supported"""
-    return '1.17.0'
+    return '1.25.0'
 
 
 @pytest.fixture(scope='session')
@@ -57,12 +68,29 @@ def tst_distro(runner):
     return distro
 
 
+@pytest.fixture(scope='session')
+def tst_distro_family(runner):
+    """Test session's distro_family"""
+    family = ''
+    with contextlib.suppress(Exception):
+        run = runner(command=[
+            'grep', '-oP', r'ID_LIKE=\K.+', '/etc/os-release'], report=False)
+        family = run.out.strip()
+    return family
+
+
 @pytest.fixture(scope='session')
 def tst_sys():
     """Test session's uname value"""
     return platform.system()
 
 
+@pytest.fixture(scope='session')
+def tst_arch():
+    """Test session's uname value"""
+    return platform.machine()
+
+
 @pytest.fixture(scope='session')
 def supported_commands():
     """List of supported commands
@@ -78,12 +106,14 @@ def supported_commands():
         'decrypt',
         'encrypt',
         'enter',
+        'git-crypt',
         'gitconfig',
         'help',
         'init',
         'introspect',
         'list',
         'perms',
+        'transcrypt',
         'upgrade',
         'version',
         ]
@@ -96,6 +126,7 @@ def supported_configs():
     This list should be updated every time yadm learns a new config.
     """
     return [
+        'local.arch',
         'local.class',
         'local.hostname',
         'local.os',
@@ -105,10 +136,14 @@ def supported_configs():
         'yadm.auto-exclude',
         'yadm.auto-perms',
         'yadm.auto-private-dirs',
+        'yadm.cipher',
         'yadm.git-program',
         'yadm.gpg-perms',
         'yadm.gpg-program',
         'yadm.gpg-recipient',
+        'yadm.openssl-ciphername',
+        'yadm.openssl-old',
+        'yadm.openssl-program',
         'yadm.ssh-perms',
         ]
 
@@ -123,6 +158,7 @@ def supported_switches():
         '--yadm-archive',
         '--yadm-bootstrap',
         '--yadm-config',
+        '--yadm-data',
         '--yadm-dir',
         '--yadm-encrypt',
         '--yadm-repo',
@@ -162,6 +198,10 @@ class Runner():
             self.command = ' '.join([str(cmd) for cmd in command])
         else:
             self.command = command
+        if env is None:
+            env = {}
+        merged_env = os.environ.copy()
+        merged_env.update(env)
         self.inp = inp
         self.wrap(expect)
         process = Popen(
@@ -171,7 +211,7 @@ class Runner():
             stderr=PIPE,
             shell=shell,
             cwd=cwd,
-            env=env,
+            env=merged_env,
         )
         input_bytes = self.inp
         if self.inp:
@@ -262,13 +302,17 @@ def yadm():
 @pytest.fixture()
 def paths(tmpdir, yadm):
     """Function scoped test paths"""
+
     dir_root = tmpdir.mkdir('root')
-    dir_work = dir_root.mkdir('work')
-    dir_yadm = dir_root.mkdir('yadm')
-    dir_repo = dir_yadm.mkdir('repo.git')
-    dir_hooks = dir_yadm.mkdir('hooks')
     dir_remote = dir_root.mkdir('remote')
-    file_archive = dir_yadm.join('files.gpg')
+    dir_work = dir_root.mkdir('work')
+    dir_xdg_data = dir_root.mkdir('xdg_data')
+    dir_xdg_home = dir_root.mkdir('xdg_home')
+    dir_data = dir_xdg_data.mkdir('yadm')
+    dir_yadm = dir_xdg_home.mkdir('yadm')
+    dir_hooks = dir_yadm.mkdir('hooks')
+    dir_repo = dir_data.mkdir('repo.git')
+    file_archive = dir_data.join('archive')
     file_bootstrap = dir_yadm.join('bootstrap')
     file_config = dir_yadm.join('config')
     file_encrypt = dir_yadm.join('encrypt')
@@ -276,24 +320,32 @@ def paths(tmpdir, yadm):
         'Paths', [
             'pgm',
             'root',
-            'work',
-            'yadm',
-            'repo',
-            'hooks',
             'remote',
+            'work',
+            'xdg_data',
+            'xdg_home',
+            'data',
+            'yadm',
+            'hooks',
+            'repo',
             'archive',
             'bootstrap',
             'config',
             'encrypt',
             ])
+    os.environ['XDG_CONFIG_HOME'] = str(dir_xdg_home)
+    os.environ['XDG_DATA_HOME'] = str(dir_xdg_data)
     return paths(
         yadm,
         dir_root,
-        dir_work,
-        dir_yadm,
-        dir_repo,
-        dir_hooks,
         dir_remote,
+        dir_work,
+        dir_xdg_data,
+        dir_xdg_home,
+        dir_data,
+        dir_yadm,
+        dir_hooks,
+        dir_repo,
         file_archive,
         file_bootstrap,
         file_config,
@@ -302,11 +354,11 @@ def paths(tmpdir, yadm):
 
 
 @pytest.fixture()
-def yadm_y(paths):
+def yadm_cmd(paths):
     """Generate custom command_list function"""
     def command_list(*args):
         """Produce params for running yadm with -Y"""
-        return [paths.pgm, '-Y', str(paths.yadm)] + list(args)
+        return [paths.pgm] + list(args)
     return command_list
 
 
@@ -445,15 +497,6 @@ def ds1_dset(tst_sys):
     dset = DataSet()
     dset.add_file('t1')
     dset.add_file('d1/t2')
-    dset.add_file(f'test_alt##S')
-    dset.add_file(f'test_alt##S.H')
-    dset.add_file(f'test_alt##S.H.U')
-    dset.add_file(f'test_alt##C.S.H.U')
-    dset.add_file(f'test alt/test alt##S')
-    dset.add_file(f'test alt/test alt##S.H')
-    dset.add_file(f'test alt/test alt##S.H.U')
-    dset.add_file(f'test alt/test alt##C.S.H.U')
-    dset.add_file(f'test_alt_copy##{tst_sys}')
     dset.add_file(f'test_alt_copy##os.{tst_sys}')
     dset.add_file('u1', tracked=False)
     dset.add_file('d2/u2', tracked=False)
@@ -553,3 +596,32 @@ def ds1(ds1_work_copy, paths, ds1_dset):
     dscopy = copy.deepcopy(ds1_dset)
     dscopy.relative_to(copy.deepcopy(paths.work))
     return dscopy
+
+
+@pytest.fixture(scope='session')
+def gnupg(tmpdir_factory, runner):
+    """Location of GNUPGHOME"""
+
+    def register_gpg_password(password):
+        """Publish a new GPG mock password"""
+        py.path.local('/tmp/mock-password').write(password)
+
+    home = tmpdir_factory.mktemp('gnupghome')
+    home.chmod(0o700)
+    conf = home.join('gpg.conf')
+    conf.write('no-secmem-warning\n')
+    conf.chmod(0o600)
+    agentconf = home.join('gpg-agent.conf')
+    agentconf.write(
+        f'pinentry-program {os.path.abspath("test/pinentry-mock")}\n'
+        'max-cache-ttl 0\n'
+    )
+    agentconf.chmod(0o600)
+    data = collections.namedtuple('GNUPG', ['home', 'pw'])
+    env = os.environ.copy()
+    env['GNUPGHOME'] = home
+
+    # this pre-populates std files in the GNUPGHOME
+    runner(['gpg', '-k'], env=env)
+
+    return data(home, register_gpg_password)

test/pinentry-mock

@@ -0,0 +1,12 @@
+#!/bin/bash
+# This program is a custom mock pinentry program
+# It always uses whatever password is found in the /tmp directory
+password="$(cat /tmp/mock-password 2>/dev/null)"
+echo "OK Pleased to meet you"
+while read -r line; do
+  if [[ $line =~ GETPIN ]]; then
+    echo -n "D "
+    echo "$password"
+  fi
+  echo "OK";
+done

test/requirements.txt

@@ -0,0 +1,6 @@
+envtpl
+flake8==3.8.4
+j2cli
+pylint==2.6.0
+pytest==6.2.1
+yamllint==1.25.0

test/test_alt.py

@@ -22,12 +22,12 @@ def test_alt_source(
         tracked, encrypt, exclude,
         yadm_alt):
     """Test yadm alt operates on all expected sources of alternates"""
-    yadm_dir = setup_standard_yadm_dir(paths)
+    yadm_dir, yadm_data = setup_standard_yadm_dir(paths)
 
     utils.create_alt_files(
         paths, '##default', tracked=tracked, encrypt=encrypt, exclude=exclude,
         yadm_alt=yadm_alt, yadm_dir=yadm_dir)
-    run = runner([paths.pgm, '-Y', yadm_dir, 'alt'])
+    run = runner([paths.pgm, '-Y', yadm_dir, '--yadm-data', yadm_data, 'alt'])
     assert run.success
     assert run.err == ''
     linked = utils.parse_alt_output(run.out)
@@ -40,7 +40,7 @@ def test_alt_source(
         link_file = paths.work.join(link_path)
         if tracked or (encrypt and not exclude):
             assert link_file.islink()
-            target = py.path.local(link_file.readlink())
+            target = py.path.local(os.path.realpath(link_file))
             if target.isfile():
                 assert link_file.read() == source_file_content
                 assert str(source_file) in linked
@@ -53,35 +53,68 @@ def test_alt_source(
             assert str(source_file) not in linked
 
 
+@pytest.mark.usefixtures('ds1_copy')
+@pytest.mark.parametrize('yadm_alt', [True, False], ids=['alt', 'worktree'])
+def test_relative_link(runner, paths, yadm_alt):
+    """Confirm links created are relative"""
+    yadm_dir, yadm_data = setup_standard_yadm_dir(paths)
+
+    utils.create_alt_files(
+        paths, '##default', tracked=True, encrypt=False, exclude=False,
+        yadm_alt=yadm_alt, yadm_dir=yadm_dir)
+    run = runner([paths.pgm, '-Y', yadm_dir, '--yadm-data', yadm_data, 'alt'])
+    assert run.success
+    assert run.err == ''
+
+    basepath = yadm_dir.join('alt') if yadm_alt else paths.work
+
+    for link_path in TEST_PATHS:
+        source_file_content = link_path + '##default'
+        source_file = basepath.join(source_file_content)
+        link_file = paths.work.join(link_path)
+        link = link_file.readlink()
+        relpath = os.path.relpath(
+            source_file, start=os.path.dirname(link_file))
+        assert link == relpath
+
+
 @pytest.mark.usefixtures('ds1_copy')
 @pytest.mark.parametrize('suffix', [
     '##default',
+    '##default,e.txt', '##default,extension.txt',
+    '##a.$tst_arch', '##arch.$tst_arch',
     '##o.$tst_sys', '##os.$tst_sys',
     '##d.$tst_distro', '##distro.$tst_distro',
+    '##f.$tst_distro_family', '##distro_family.$tst_distro_family',
     '##c.$tst_class', '##class.$tst_class',
     '##h.$tst_host', '##hostname.$tst_host',
     '##u.$tst_user', '##user.$tst_user',
     ])
 def test_alt_conditions(
         runner, paths,
-        tst_sys, tst_distro, tst_host, tst_user, suffix):
+        tst_arch, tst_sys, tst_distro, tst_distro_family, tst_host, tst_user,
+        suffix):
     """Test conditions supported by yadm alt"""
-    yadm_dir = setup_standard_yadm_dir(paths)
+    yadm_dir, yadm_data = setup_standard_yadm_dir(paths)
 
     # set the class
     tst_class = 'testclass'
-    utils.set_local(paths, 'class', tst_class)
+    utils.set_local(paths, 'class', tst_class + ".before")
+    utils.set_local(paths, 'class', tst_class, add=True)
+    utils.set_local(paths, 'class', tst_class + ".after", add=True)
 
     suffix = string.Template(suffix).substitute(
+        tst_arch=tst_arch,
         tst_sys=tst_sys,
         tst_distro=tst_distro,
+        tst_distro_family=tst_distro_family,
         tst_class=tst_class,
         tst_host=tst_host,
         tst_user=tst_user,
     )
 
     utils.create_alt_files(paths, suffix)
-    run = runner([paths.pgm, '-Y', yadm_dir, 'alt'])
+    run = runner([paths.pgm, '-Y', yadm_dir, '--yadm-data', yadm_data, 'alt'])
     assert run.success
     assert run.err == ''
     linked = utils.parse_alt_output(run.out)
@@ -89,7 +122,7 @@ def test_alt_conditions(
     for link_path in TEST_PATHS:
         source_file = link_path + suffix
         assert paths.work.join(link_path).islink()
-        target = py.path.local(paths.work.join(link_path).readlink())
+        target = py.path.local(os.path.realpath(paths.work.join(link_path)))
         if target.isfile():
             assert paths.work.join(link_path).read() == source_file
             assert str(paths.work.join(source_file)) in linked
@@ -101,18 +134,18 @@ def test_alt_conditions(
 
 @pytest.mark.usefixtures('ds1_copy')
 @pytest.mark.parametrize(
-    'kind', ['default', '', None, 'envtpl', 'j2cli', 'j2'])
+    'kind', ['default', '', None, 'envtpl', 'j2cli', 'j2', 'esh'])
 @pytest.mark.parametrize('label', ['t', 'template', 'yadm', ])
 def test_alt_templates(
         runner, paths, kind, label):
     """Test templates supported by yadm alt"""
-    yadm_dir = setup_standard_yadm_dir(paths)
+    yadm_dir, yadm_data = setup_standard_yadm_dir(paths)
 
     suffix = f'##{label}.{kind}'
     if kind is None:
         suffix = f'##{label}'
     utils.create_alt_files(paths, suffix)
-    run = runner([paths.pgm, '-Y', yadm_dir, 'alt'])
+    run = runner([paths.pgm, '-Y', yadm_dir, '--yadm-data', yadm_data, 'alt'])
     assert run.success
     assert run.err == ''
     created = utils.parse_alt_output(run.out, linked=False)
@@ -127,15 +160,15 @@ def test_alt_templates(
 
 @pytest.mark.usefixtures('ds1_copy')
 @pytest.mark.parametrize('autoalt', [None, 'true', 'false'])
-def test_auto_alt(runner, yadm_y, paths, autoalt):
+def test_auto_alt(runner, yadm_cmd, paths, autoalt):
     """Test auto alt"""
 
     # set the value of auto-alt
     if autoalt:
-        os.system(' '.join(yadm_y('config', 'yadm.auto-alt', autoalt)))
+        os.system(' '.join(yadm_cmd('config', 'yadm.auto-alt', autoalt)))
 
     utils.create_alt_files(paths, '##default')
-    run = runner(yadm_y('status'))
+    run = runner(yadm_cmd('status'))
     assert run.success
     assert run.err == ''
     linked = utils.parse_alt_output(run.out)
@@ -146,7 +179,8 @@ def test_auto_alt(runner, yadm_y, paths, autoalt):
             assert not paths.work.join(link_path).exists()
         else:
             assert paths.work.join(link_path).islink()
-            target = py.path.local(paths.work.join(link_path).readlink())
+            target = py.path.local(
+                os.path.realpath(paths.work.join(link_path)))
             if target.isfile():
                 assert paths.work.join(link_path).read() == source_file
                 # no linking output when run via auto-alt
@@ -159,7 +193,7 @@ def test_auto_alt(runner, yadm_y, paths, autoalt):
 
 
 @pytest.mark.usefixtures('ds1_copy')
-def test_stale_link_removal(runner, yadm_y, paths):
+def test_stale_link_removal(runner, yadm_cmd, paths):
     """Stale links to alternative files are removed
 
     This test ensures that when an already linked alternative becomes invalid
@@ -174,7 +208,7 @@ def test_stale_link_removal(runner, yadm_y, paths):
     utils.create_alt_files(paths, f'##class.{tst_class}')
 
     # run alt to trigger linking
-    run = runner(yadm_y('alt'))
+    run = runner(yadm_cmd('alt'))
     assert run.success
     assert run.err == ''
     linked = utils.parse_alt_output(run.out)
@@ -183,7 +217,7 @@ def test_stale_link_removal(runner, yadm_y, paths):
     for stale_path in TEST_PATHS:
         source_file = stale_path + '##class.' + tst_class
         assert paths.work.join(stale_path).islink()
-        target = py.path.local(paths.work.join(stale_path).readlink())
+        target = py.path.local(os.path.realpath(paths.work.join(stale_path)))
         if target.isfile():
             assert paths.work.join(stale_path).read() == source_file
             assert str(paths.work.join(source_file)) in linked
@@ -196,7 +230,7 @@ def test_stale_link_removal(runner, yadm_y, paths):
     utils.set_local(paths, 'class', 'changedclass')
 
     # run alt to trigger linking
-    run = runner(yadm_y('alt'))
+    run = runner(yadm_cmd('alt'))
     assert run.success
     assert run.err == ''
     linked = utils.parse_alt_output(run.out)
@@ -209,7 +243,7 @@ def test_stale_link_removal(runner, yadm_y, paths):
 
 
 @pytest.mark.usefixtures('ds1_repo_copy')
-def test_template_overwrite_symlink(runner, yadm_y, paths, tst_sys):
+def test_template_overwrite_symlink(runner, yadm_cmd, paths, tst_sys):
     """Remove symlinks before processing a template
 
     If a symlink is in the way of the output of a template, the target of the
@@ -226,7 +260,7 @@ def test_template_overwrite_symlink(runner, yadm_y, paths, tst_sys):
     template = paths.work.join('test_link##template.default')
     template.write('test-data')
 
-    run = runner(yadm_y('add', target, template))
+    run = runner(yadm_cmd('add', target, template))
     assert run.success
     assert run.err == ''
     assert run.out == ''
@@ -239,12 +273,13 @@ def test_template_overwrite_symlink(runner, yadm_y, paths, tst_sys):
 @pytest.mark.parametrize('style', ['symlink', 'template'])
 def test_ensure_alt_path(runner, paths, style):
     """Test that directories are created before making alternates"""
-    yadm_dir = setup_standard_yadm_dir(paths)
+    yadm_dir, yadm_data = setup_standard_yadm_dir(paths)
     suffix = 'default' if style == 'symlink' else 'template'
     filename = 'a/b/c/file'
     source = yadm_dir.join(f'alt/{filename}##{suffix}')
     source.write('test-data', ensure=True)
-    run = runner([paths.pgm, '-Y', yadm_dir, 'add', source])
+    run = runner([
+        paths.pgm, '-Y', yadm_dir, '--yadm-data', yadm_data, 'add', source])
     assert run.success
     assert run.err == ''
     assert run.out == ''
@@ -254,6 +289,7 @@ def test_ensure_alt_path(runner, paths, style):
 def setup_standard_yadm_dir(paths):
     """Configure a yadm home within the work tree"""
     std_yadm_dir = paths.work.mkdir('.config').mkdir('yadm')
-    std_yadm_dir.join('repo.git').mksymlinkto(paths.repo, absolute=1)
+    std_yadm_data = paths.work.mkdir('.local').mkdir('share').mkdir('yadm')
+    std_yadm_data.join('repo.git').mksymlinkto(paths.repo, absolute=1)
     std_yadm_dir.join('encrypt').mksymlinkto(paths.encrypt, absolute=1)
-    return std_yadm_dir
+    return std_yadm_dir, std_yadm_data

test/test_alt_copy.py

@@ -4,12 +4,6 @@ import os
 import pytest
 
 
-@pytest.mark.parametrize(
-    'cygwin',
-    [pytest.param(True, marks=pytest.mark.deprecated), False],
-    ids=['cygwin', 'no-cygwin'])
-@pytest.mark.parametrize(
-    'compatibility', [True, False], ids=['compat', 'no-compat'])
 @pytest.mark.parametrize(
     'setting, expect_link, pre_existing', [
         (None, True, None),
@@ -27,20 +21,14 @@ import pytest
     ])
 @pytest.mark.usefixtures('ds1_copy')
 def test_alt_copy(
-        runner, yadm_y, paths, tst_sys,
-        setting, expect_link, pre_existing,
-        compatibility, cygwin):
+        runner, yadm_cmd, paths, tst_sys,
+        setting, expect_link, pre_existing):
     """Test yadm.alt-copy"""
 
-    option = 'yadm.cygwin-copy' if cygwin else 'yadm.alt-copy'
-
     if setting is not None:
-        os.system(' '.join(yadm_y('config', option, str(setting))))
+        os.system(' '.join(yadm_cmd('config', 'yadm.alt-copy', str(setting))))
 
-    if compatibility:
-        expected_content = f'test_alt_copy##{tst_sys}'
-    else:
-        expected_content = f'test_alt_copy##os.{tst_sys}'
+    expected_content = f'test_alt_copy##os.{tst_sys}'
 
     alt_path = paths.work.join('test_alt_copy')
     if pre_existing == 'symlink':
@@ -48,11 +36,7 @@ def test_alt_copy(
     elif pre_existing == 'file':
         alt_path.write('wrong content')
 
-    env = os.environ.copy()
-    if compatibility:
-        env['YADM_COMPATIBILITY'] = '1'
-
-    run = runner(yadm_y('alt'), env=env)
+    run = runner(yadm_cmd('alt'))
     assert run.success
     assert run.err == ''
     assert 'Linking' in run.out

test/test_assert_private_dirs.py

@@ -8,7 +8,8 @@ pytestmark = pytest.mark.usefixtures('ds1_copy')
 PRIVATE_DIRS = ['.gnupg', '.ssh']
 
 
-def test_pdirs_missing(runner, yadm_y, paths):
+@pytest.mark.parametrize('home', [True, False], ids=['home', 'not-home'])
+def test_pdirs_missing(runner, yadm_cmd, paths, home):
     """Private dirs (private dirs missing)
 
     When a git command is run
@@ -23,8 +24,12 @@ def test_pdirs_missing(runner, yadm_y, paths):
             path.remove()
         assert not path.exists()
 
+    env = {'DEBUG': 'yes'}
+    if home:
+        env['HOME'] = paths.work
+
     # run status
-    run = runner(command=yadm_y('status'), env={'DEBUG': 'yes'})
+    run = runner(command=yadm_cmd('status'), env=env)
     assert run.success
     assert run.err == ''
     assert 'On branch master' in run.out
@@ -33,16 +38,22 @@ def test_pdirs_missing(runner, yadm_y, paths):
     # and are protected
     for pdir in PRIVATE_DIRS:
         path = paths.work.join(pdir)
-        assert path.exists()
-        assert oct(path.stat().mode).endswith('00'), 'Directory is not secured'
+        if home:
+            assert path.exists()
+            assert oct(path.stat().mode).endswith('00'), ('Directory is '
+                                                          'not secured')
+        else:
+            assert not path.exists()
 
     # confirm directories are created before command is run:
-    assert re.search(
-        r'Creating.+\.gnupg.+Creating.+\.ssh.+Running git command git status',
-        run.out, re.DOTALL), 'directories created before command is run'
+    if home:
+        assert re.search(
+            (r'Creating.+\.(gnupg|ssh).+Creating.+\.(gnupg|ssh).+'
+             r'Running git command git status'),
+            run.out, re.DOTALL), 'directories created before command is run'
 
 
-def test_pdirs_missing_apd_false(runner, yadm_y, paths):
+def test_pdirs_missing_apd_false(runner, yadm_cmd, paths):
     """Private dirs (private dirs missing / yadm.auto-private-dirs=false)
 
     When a git command is run
@@ -59,11 +70,11 @@ def test_pdirs_missing_apd_false(runner, yadm_y, paths):
         assert not path.exists()
 
     # set configuration
-    os.system(' '.join(yadm_y(
+    os.system(' '.join(yadm_cmd(
         'config', '--bool', 'yadm.auto-private-dirs', 'false')))
 
     # run status
-    run = runner(command=yadm_y('status'))
+    run = runner(command=yadm_cmd('status'))
     assert run.success
     assert run.err == ''
     assert 'On branch master' in run.out
@@ -73,7 +84,7 @@ def test_pdirs_missing_apd_false(runner, yadm_y, paths):
         assert not paths.work.join(pdir).exists()
 
 
-def test_pdirs_exist_apd_false(runner, yadm_y, paths):
+def test_pdirs_exist_apd_false(runner, yadm_cmd, paths):
     """Private dirs (private dirs exist / yadm.auto-perms=false)
 
     When a git command is run
@@ -91,11 +102,11 @@ def test_pdirs_exist_apd_false(runner, yadm_y, paths):
         assert oct(path.stat().mode).endswith('77'), 'Directory is secure.'
 
     # set configuration
-    os.system(' '.join(yadm_y(
+    os.system(' '.join(yadm_cmd(
         'config', '--bool', 'yadm.auto-perms', 'false')))
 
     # run status
-    run = runner(command=yadm_y('status'))
+    run = runner(command=yadm_cmd('status'))
     assert run.success
     assert run.err == ''
     assert 'On branch master' in run.out

test/test_bootstrap.py

@@ -14,7 +14,7 @@ import pytest
         'executable',
     ])
 def test_bootstrap(
-        runner, yadm_y, paths, exists, executable, code, expect):
+        runner, yadm_cmd, paths, exists, executable, code, expect):
     """Test bootstrap command"""
     if exists:
         paths.bootstrap.write('')
@@ -25,7 +25,11 @@ def test_bootstrap(
             f'exit {code}\n'
         )
         paths.bootstrap.chmod(0o775)
-    run = runner(command=yadm_y('bootstrap'))
+    run = runner(command=yadm_cmd('bootstrap'))
     assert run.code == code
-    assert run.err == ''
-    assert expect in run.out
+    if exists and executable:
+        assert run.err == ''
+        assert expect in run.out
+    else:
+        assert expect in run.err
+        assert run.out == ''

test/test_clean.py

@@ -1,11 +1,11 @@
 """Test clean"""
 
 
-def test_clean_command(runner, yadm_y):
+def test_clean_command(runner, yadm_cmd):
     """Run with clean command"""
-    run = runner(command=yadm_y('clean'))
+    run = runner(command=yadm_cmd('clean'))
     # do nothing, this is a dangerous Git command when managing dot files
     # report the command as disabled and exit as a failure
     assert run.failure
-    assert run.err == ''
-    assert 'disabled' in run.out
+    assert run.out == ''
+    assert 'disabled' in run.err

test/test_clone.py

@@ -24,7 +24,7 @@ BOOTSTRAP_MSG = 'Bootstrap successful'
         'conflicts',
     ])
 def test_clone(
-        runner, paths, yadm_y, repo_config, ds1,
+        runner, paths, yadm_cmd, repo_config, ds1,
         good_remote, repo_exists, force, conflicts):
     """Test basic clone operation"""
 
@@ -53,28 +53,30 @@ def test_clone(
     if force:
         args += ['-f']
     args += [remote_url]
-    run = runner(command=yadm_y(*args))
+    run = runner(command=yadm_cmd(*args))
 
     if not good_remote:
         # clone should fail
         assert run.failure
-        assert run.err != ''
-        assert 'Unable to fetch origin' in run.out
+        assert run.out == ''
+        assert 'Unable to clone the repository' in run.err
         assert not paths.repo.exists()
     elif repo_exists and not force:
         # can't overwrite data
         assert run.failure
-        assert run.err == ''
-        assert 'Git repo already exists' in run.out
+        assert run.out == ''
+        assert 'Git repo already exists' in run.err
     else:
         # clone should succeed, and repo should be configured properly
         assert successful_clone(run, paths, repo_config)
 
+        # these clones should have master as HEAD
+        verify_head(paths, 'master')
+
         # ensure conflicts are handled properly
         if conflicts:
             assert 'NOTE' in run.out
-            assert 'Merging origin/master failed' in run.out
-            assert 'Conflicts preserved' in run.out
+            assert 'Local files with content that differs' in run.out
 
         # confirm correct Git origin
         run = runner(
@@ -86,25 +88,19 @@ def test_clone(
 
         # ensure conflicts are really preserved
         if conflicts:
-            # test to see if the work tree is actually "clean"
+            # test that the conflicts are preserved in the work tree
             run = runner(
-                command=yadm_y('status', '-uno', '--porcelain'),
+                command=yadm_cmd('status', '-uno', '--porcelain'),
                 cwd=paths.work)
             assert run.success
             assert run.err == ''
-            assert run.out == '', 'worktree has unexpected changes'
+            assert str(ds1.tracked[0].path) in run.out
 
-            # test to see if the conflicts are stashed
-            run = runner(command=yadm_y('stash', 'list'), cwd=paths.work)
+            # verify content of the conflicts
+            run = runner(command=yadm_cmd('diff'), cwd=paths.work)
             assert run.success
             assert run.err == ''
-            assert 'Conflicts preserved' in run.out, 'conflicts not stashed'
-
-            # verify content of the stashed conflicts
-            run = runner(command=yadm_y('stash', 'show', '-p'), cwd=paths.work)
-            assert run.success
-            assert run.err == ''
-            assert '\n+conflict' in run.out, 'conflicts not stashed'
+            assert '\n+conflict' in run.out, 'conflict overwritten'
 
     # another force-related assertion
     if old_repo:
@@ -130,7 +126,7 @@ def test_clone(
         'existing, answer y',
     ])
 def test_clone_bootstrap(
-        runner, paths, yadm_y, repo_config, bs_exists, bs_param, answer):
+        runner, paths, yadm_cmd, repo_config, bs_exists, bs_param, answer):
     """Test bootstrap clone features"""
 
     # establish a bootstrap
@@ -144,7 +140,7 @@ def test_clone_bootstrap(
     expect = []
     if answer:
         expect.append(('Would you like to execute it now', answer))
-    run = runner(command=yadm_y(*args), expect=expect)
+    run = runner(command=yadm_cmd(*args), expect=expect)
 
     if answer:
         assert 'Would you like to execute it now' in run.out
@@ -161,6 +157,7 @@ def test_clone_bootstrap(
         assert BOOTSTRAP_MSG not in run.out
 
     assert successful_clone(run, paths, repo_config, expected_code)
+    verify_head(paths, 'master')
 
     if not bs_exists:
         assert BOOTSTRAP_MSG not in run.out
@@ -197,7 +194,7 @@ def create_bootstrap(paths, exists):
         'missing gnupg, tracked',
     ])
 def test_clone_perms(
-        runner, yadm_y, paths, repo_config,
+        runner, yadm_cmd, paths, repo_config,
         private_type, in_repo, in_work):
     """Test clone permission-related functions"""
 
@@ -222,10 +219,14 @@ def test_clone_perms(
         paths.work.remove()
         paths.work.mkdir()
 
+    env = {'HOME': paths.work}
     run = runner(
-        yadm_y('clone', '-d', '-w', paths.work, f'file://{paths.remote}'))
+        yadm_cmd('clone', '-d', '-w', paths.work, f'file://{paths.remote}'),
+        env=env
+    )
 
     assert successful_clone(run, paths, repo_config)
+    verify_head(paths, 'master')
     if in_work:
         # private directories which already exist, should be left as they are,
         # which in this test is "insecure".
@@ -233,20 +234,20 @@ def test_clone_perms(
             f'initial private dir perms drwxrwxrwx.+.{private_type}',
             run.out)
         assert re.search(
-            f'pre-merge private dir perms drwxrwxrwx.+.{private_type}',
+            f'pre-checkout private dir perms drwxrwxrwx.+.{private_type}',
             run.out)
         assert re.search(
-            f'post-merge private dir perms drwxrwxrwx.+.{private_type}',
+            f'post-checkout private dir perms drwxrwxrwx.+.{private_type}',
             run.out)
     else:
         # private directories which are created, should be done prior to
-        # merging, and with secure permissions.
+        # checkout, and with secure permissions.
         assert 'initial private dir perms' not in run.out
         assert re.search(
-            f'pre-merge private dir perms drwx------.+.{private_type}',
+            f'pre-checkout private dir perms drwx------.+.{private_type}',
             run.out)
         assert re.search(
-            f'post-merge private dir perms drwx------.+.{private_type}',
+            f'post-checkout private dir perms drwx------.+.{private_type}',
             run.out)
 
     # standard perms still apply afterwards unless disabled with auto.perms
@@ -256,8 +257,9 @@ def test_clone_perms(
 
 
 @pytest.mark.usefixtures('remote')
-@pytest.mark.parametrize('branch', ['master', 'valid', 'invalid'])
-def test_alternate_branch(runner, paths, yadm_y, repo_config, branch):
+@pytest.mark.parametrize(
+    'branch', ['master', 'default', 'valid', 'invalid'])
+def test_alternate_branch(runner, paths, yadm_cmd, repo_config, branch):
     """Test cloning a branch other than master"""
 
     # add a "valid" branch to the remote
@@ -265,6 +267,12 @@ def test_alternate_branch(runner, paths, yadm_y, repo_config, branch):
     os.system(
         f'GIT_DIR="{paths.remote}" git commit '
         f'--allow-empty -m "This branch is valid"')
+    if branch != 'default':
+        # When branch == 'default', the "default" branch of the remote repo
+        # will remain "valid" to validate identification the correct default
+        # branch by inspecting the repo. Otherwise it will be set back to
+        # "master"
+        os.system(f'GIT_DIR="{paths.remote}" git checkout master')
 
     # clear out the work path
     paths.work.remove()
@@ -274,15 +282,15 @@ def test_alternate_branch(runner, paths, yadm_y, repo_config, branch):
 
     # run the clone command
     args = ['clone', '-w', paths.work]
-    if branch != 'master':
+    if branch not in ['master', 'default']:
         args += ['-b', branch]
     args += [remote_url]
-    run = runner(command=yadm_y(*args))
+    run = runner(command=yadm_cmd(*args))
 
     if branch == 'invalid':
         assert run.failure
-        assert 'ERROR: Clone failed' in run.out
-        assert f"'origin/{branch}' does not exist in {remote_url}" in run.out
+        assert 'ERROR: Unable to clone the repository' in run.err
+        assert f"Remote branch {branch} not found in upstream" in run.err
     else:
         assert successful_clone(run, paths, repo_config)
 
@@ -293,17 +301,18 @@ def test_alternate_branch(runner, paths, yadm_y, repo_config, branch):
         assert run.success
         assert run.err == ''
         assert f'origin\t{remote_url}' in run.out
-        run = runner(command=yadm_y('show'))
-        if branch == 'valid':
-            assert 'This branch is valid' in run.out
-        else:
+        run = runner(command=yadm_cmd('show'))
+        if branch == 'master':
             assert 'Initial commit' in run.out
+            verify_head(paths, 'master')
+        else:
+            assert 'This branch is valid' in run.out
+            verify_head(paths, 'valid')
 
 
 def successful_clone(run, paths, repo_config, expected_code=0):
     """Assert clone is successful"""
     assert run.code == expected_code
-    assert 'Initialized' in run.out
     assert oct(paths.repo.stat().mode).endswith('00'), 'Repo is not secured'
     assert repo_config('core.bare') == 'false'
     assert repo_config('status.showUntrackedFiles') == 'no'
@@ -320,3 +329,17 @@ def remote(paths, ds1_repo_copy):
     # cannot be applied to another fixture.
     paths.remote.remove()
     paths.repo.move(paths.remote)
+
+
+def test_no_repo(runner, yadm_cmd, ):
+    """Test cloning without specifying a repo"""
+    run = runner(command=yadm_cmd('clone', '-f'))
+    assert run.failure
+    assert run.out == ''
+    assert 'ERROR: Unable to clone the repository' in run.err
+    assert 'repository \'repo.git\' does not exist' in run.err
+
+
+def verify_head(paths, branch):
+    """Assert the local repo has the correct head branch"""
+    assert paths.repo.join('HEAD').read() == f'ref: refs/heads/{branch}\n'

test/test_compat_alt.py

@@ -1,450 +0,0 @@
-"""Test alt"""
-
-import os
-import string
-import py
-import pytest
-import utils
-
-# These tests are for the alternate processing in YADM_COMPATIBILITY=1 mode
-pytestmark = pytest.mark.deprecated
-
-# These test IDs are broken. During the writing of these tests, problems have
-# been discovered in the way yadm orders matching files.
-BROKEN_TEST_IDS = [
-    'test_wild[tracked-##C.S.H.U-C-S%-H%-U]',
-    'test_wild[tracked-##C.S.H.U-C-S-H%-U]',
-    'test_wild[encrypted-##C.S.H.U-C-S%-H%-U]',
-    'test_wild[encrypted-##C.S.H.U-C-S-H%-U]',
-    ]
-
-PRECEDENCE = [
-    '##',
-    '##$tst_sys',
-    '##$tst_sys.$tst_host',
-    '##$tst_sys.$tst_host.$tst_user',
-    '##$tst_class',
-    '##$tst_class.$tst_sys',
-    '##$tst_class.$tst_sys.$tst_host',
-    '##$tst_class.$tst_sys.$tst_host.$tst_user',
-    ]
-
-WILD_TEMPLATES = [
-    '##$tst_class',
-    '##$tst_class.$tst_sys',
-    '##$tst_class.$tst_sys.$tst_host',
-    '##$tst_class.$tst_sys.$tst_host.$tst_user',
-    ]
-
-TEST_PATHS = [utils.ALT_FILE1, utils.ALT_FILE2, utils.ALT_DIR]
-
-WILD_TESTED = set()
-
-
-@pytest.mark.parametrize('precedence_index', range(len(PRECEDENCE)))
-@pytest.mark.parametrize(
-    'tracked, encrypt, exclude', [
-        (False, False, False),
-        (True, False, False),
-        (False, True, False),
-        (False, True, True),
-    ], ids=[
-        'untracked',
-        'tracked',
-        'encrypted',
-        'excluded',
-    ])
-@pytest.mark.usefixtures('ds1_copy')
-def test_alt(runner, yadm_y, paths,
-             tst_sys, tst_host, tst_user,
-             tracked, encrypt, exclude,
-             precedence_index):
-    """Test alternate linking
-
-    This test is done by iterating for the number of templates in PRECEDENCE.
-    With each iteration, another file is left off the list. So with each
-    iteration, the template with the "highest precedence" is left out. The file
-    using the highest precedence should be the one linked.
-    """
-
-    # set the class
-    tst_class = 'testclass'
-    utils.set_local(paths, 'class', tst_class)
-
-    # process the templates in PRECEDENCE
-    precedence = list()
-    for template in PRECEDENCE:
-        precedence.append(
-            string.Template(template).substitute(
-                tst_class=tst_class,
-                tst_host=tst_host,
-                tst_sys=tst_sys,
-                tst_user=tst_user,
-            )
-        )
-
-    # create files using a subset of files
-    for suffix in precedence[0:precedence_index+1]:
-        utils.create_alt_files(paths, suffix, tracked=tracked,
-                               encrypt=encrypt, exclude=exclude)
-
-    # run alt to trigger linking
-    env = os.environ.copy()
-    env['YADM_COMPATIBILITY'] = '1'
-    run = runner(yadm_y('alt'), env=env)
-    assert run.success
-    assert run.err == ''
-    linked = utils.parse_alt_output(run.out)
-
-    # assert the proper linking has occurred
-    for file_path in TEST_PATHS:
-        source_file = file_path + precedence[precedence_index]
-        if tracked or (encrypt and not exclude):
-            assert paths.work.join(file_path).islink()
-            target = py.path.local(paths.work.join(file_path).readlink())
-            if target.isfile():
-                assert paths.work.join(file_path).read() == source_file
-                assert str(paths.work.join(source_file)) in linked
-            else:
-                assert paths.work.join(file_path).join(
-                    utils.CONTAINED).read() == source_file
-                assert str(paths.work.join(source_file)) in linked
-        else:
-            assert not paths.work.join(file_path).exists()
-            assert str(paths.work.join(source_file)) not in linked
-
-
-def short_template(template):
-    """Translate template into something short for test IDs"""
-    return string.Template(template).substitute(
-        tst_class='C',
-        tst_host='H',
-        tst_sys='S',
-        tst_user='U',
-    )
-
-
-@pytest.mark.parametrize('wild_user', [True, False], ids=['U%', 'U'])
-@pytest.mark.parametrize('wild_host', [True, False], ids=['H%', 'H'])
-@pytest.mark.parametrize('wild_sys', [True, False], ids=['S%', 'S'])
-@pytest.mark.parametrize('wild_class', [True, False], ids=['C%', 'C'])
-@pytest.mark.parametrize('template', WILD_TEMPLATES, ids=short_template)
-@pytest.mark.parametrize(
-    'tracked, encrypt', [
-        (True, False),
-        (False, True),
-    ], ids=[
-        'tracked',
-        'encrypted',
-    ])
-@pytest.mark.usefixtures('ds1_copy')
-def test_wild(request, runner, yadm_y, paths,
-              tst_sys, tst_host, tst_user,
-              tracked, encrypt,
-              wild_class, wild_host, wild_sys, wild_user,
-              template):
-    """Test wild linking
-
-    These tests are done by creating permutations of the possible files using
-    WILD_TEMPLATES. Each case is then tested (while skipping the already tested
-    permutations for efficiency).
-    """
-
-    if request.node.name in BROKEN_TEST_IDS:
-        pytest.xfail(
-            'This test is known to be broken. '
-            'This bug only affects deprecated features.')
-
-    tst_class = 'testclass'
-
-    # determine the "wild" version of the suffix
-    str_class = '%' if wild_class else tst_class
-    str_host = '%' if wild_host else tst_host
-    str_sys = '%' if wild_sys else tst_sys
-    str_user = '%' if wild_user else tst_user
-    wild_suffix = string.Template(template).substitute(
-        tst_class=str_class,
-        tst_host=str_host,
-        tst_sys=str_sys,
-        tst_user=str_user,
-    )
-
-    # determine the "standard" version of the suffix
-    std_suffix = string.Template(template).substitute(
-        tst_class=tst_class,
-        tst_host=tst_host,
-        tst_sys=tst_sys,
-        tst_user=tst_user,
-    )
-
-    # skip over duplicate tests (this seems to be the simplest way to cover the
-    # permutations of tests, while skipping duplicates.)
-    test_key = f'{tracked}{encrypt}{wild_suffix}{std_suffix}'
-    if test_key in WILD_TESTED:
-        return
-    WILD_TESTED.add(test_key)
-
-    # set the class
-    utils.set_local(paths, 'class', tst_class)
-
-    # create files using the wild suffix
-    utils.create_alt_files(paths, wild_suffix, tracked=tracked,
-                           encrypt=encrypt, exclude=False)
-
-    # run alt to trigger linking
-    env = os.environ.copy()
-    env['YADM_COMPATIBILITY'] = '1'
-    run = runner(yadm_y('alt'), env=env)
-    assert run.success
-    assert run.err == ''
-    linked = utils.parse_alt_output(run.out)
-
-    # assert the proper linking has occurred
-    for file_path in TEST_PATHS:
-        source_file = file_path + wild_suffix
-        assert paths.work.join(file_path).islink()
-        target = py.path.local(paths.work.join(file_path).readlink())
-        if target.isfile():
-            assert paths.work.join(file_path).read() == source_file
-            assert str(paths.work.join(source_file)) in linked
-        else:
-            assert paths.work.join(file_path).join(
-                utils.CONTAINED).read() == source_file
-            assert str(paths.work.join(source_file)) in linked
-
-    # create files using the standard suffix
-    utils.create_alt_files(paths, std_suffix, tracked=tracked,
-                           encrypt=encrypt, exclude=False)
-
-    # run alt to trigger linking
-    env = os.environ.copy()
-    env['YADM_COMPATIBILITY'] = '1'
-    run = runner(yadm_y('alt'), env=env)
-    assert run.success
-    assert run.err == ''
-    linked = utils.parse_alt_output(run.out)
-
-    # assert the proper linking has occurred
-    for file_path in TEST_PATHS:
-        source_file = file_path + std_suffix
-        assert paths.work.join(file_path).islink()
-        target = py.path.local(paths.work.join(file_path).readlink())
-        if target.isfile():
-            assert paths.work.join(file_path).read() == source_file
-            assert str(paths.work.join(source_file)) in linked
-        else:
-            assert paths.work.join(file_path).join(
-                utils.CONTAINED).read() == source_file
-            assert str(paths.work.join(source_file)) in linked
-
-
-@pytest.mark.usefixtures('ds1_copy')
-def test_local_override(runner, yadm_y, paths,
-                        tst_sys, tst_host, tst_user):
-    """Test local overrides"""
-
-    # define local overrides
-    utils.set_local(paths, 'class', 'or-class')
-    utils.set_local(paths, 'hostname', 'or-hostname')
-    utils.set_local(paths, 'os', 'or-os')
-    utils.set_local(paths, 'user', 'or-user')
-
-    # create files, the first would normally be the most specific version
-    # however, the second is the overridden version which should be preferred.
-    utils.create_alt_files(
-        paths, f'##or-class.{tst_sys}.{tst_host}.{tst_user}')
-    utils.create_alt_files(
-        paths, '##or-class.or-os.or-hostname.or-user')
-
-    # run alt to trigger linking
-    env = os.environ.copy()
-    env['YADM_COMPATIBILITY'] = '1'
-    run = runner(yadm_y('alt'), env=env)
-    assert run.success
-    assert run.err == ''
-    linked = utils.parse_alt_output(run.out)
-
-    # assert the proper linking has occurred
-    for file_path in TEST_PATHS:
-        source_file = file_path + '##or-class.or-os.or-hostname.or-user'
-        assert paths.work.join(file_path).islink()
-        target = py.path.local(paths.work.join(file_path).readlink())
-        if target.isfile():
-            assert paths.work.join(file_path).read() == source_file
-            assert str(paths.work.join(source_file)) in linked
-        else:
-            assert paths.work.join(file_path).join(
-                utils.CONTAINED).read() == source_file
-            assert str(paths.work.join(source_file)) in linked
-
-
-@pytest.mark.parametrize('suffix', ['AAA', 'ZZZ', 'aaa', 'zzz'])
-@pytest.mark.usefixtures('ds1_copy')
-def test_class_case(runner, yadm_y, paths, tst_sys, suffix):
-    """Test range of class cases"""
-
-    # set the class
-    utils.set_local(paths, 'class', suffix)
-
-    # create files
-    endings = [suffix]
-    if tst_sys == 'Linux':
-        # Only create all of these side-by-side on Linux, which is
-        # unquestionably case-sensitive. This would break tests on
-        # case-insensitive systems.
-        endings = ['AAA', 'ZZZ', 'aaa', 'zzz']
-    for ending in endings:
-        utils.create_alt_files(paths, f'##{ending}')
-
-    # run alt to trigger linking
-    env = os.environ.copy()
-    env['YADM_COMPATIBILITY'] = '1'
-    run = runner(yadm_y('alt'), env=env)
-    assert run.success
-    assert run.err == ''
-    linked = utils.parse_alt_output(run.out)
-
-    # assert the proper linking has occurred
-    for file_path in TEST_PATHS:
-        source_file = file_path + f'##{suffix}'
-        assert paths.work.join(file_path).islink()
-        target = py.path.local(paths.work.join(file_path).readlink())
-        if target.isfile():
-            assert paths.work.join(file_path).read() == source_file
-            assert str(paths.work.join(source_file)) in linked
-        else:
-            assert paths.work.join(file_path).join(
-                utils.CONTAINED).read() == source_file
-            assert str(paths.work.join(source_file)) in linked
-
-
-@pytest.mark.parametrize('autoalt', [None, 'true', 'false'])
-@pytest.mark.usefixtures('ds1_copy')
-def test_auto_alt(runner, yadm_y, paths, autoalt):
-    """Test setting auto-alt"""
-
-    # set the value of auto-alt
-    if autoalt:
-        os.system(' '.join(yadm_y('config', 'yadm.auto-alt', autoalt)))
-
-    # create file
-    suffix = '##'
-    utils.create_alt_files(paths, suffix)
-
-    # run status to possibly trigger linking
-    env = os.environ.copy()
-    env['YADM_COMPATIBILITY'] = '1'
-    run = runner(yadm_y('status'), env=env)
-    assert run.success
-    assert run.err == ''
-    linked = utils.parse_alt_output(run.out)
-
-    # assert the proper linking has occurred
-    for file_path in TEST_PATHS:
-        source_file = file_path + suffix
-        if autoalt == 'false':
-            assert not paths.work.join(file_path).exists()
-        else:
-            assert paths.work.join(file_path).islink()
-            target = py.path.local(paths.work.join(file_path).readlink())
-            if target.isfile():
-                assert paths.work.join(file_path).read() == source_file
-                # no linking output when run via auto-alt
-                assert str(paths.work.join(source_file)) not in linked
-            else:
-                assert paths.work.join(file_path).join(
-                    utils.CONTAINED).read() == source_file
-                # no linking output when run via auto-alt
-                assert str(paths.work.join(source_file)) not in linked
-
-
-@pytest.mark.parametrize('delimiter', ['.', '_'])
-@pytest.mark.usefixtures('ds1_copy')
-def test_delimiter(runner, yadm_y, paths,
-                   tst_sys, tst_host, tst_user, delimiter):
-    """Test delimiters used"""
-
-    suffix = '##' + delimiter.join([tst_sys, tst_host, tst_user])
-
-    # create file
-    utils.create_alt_files(paths, suffix)
-
-    # run alt to trigger linking
-    env = os.environ.copy()
-    env['YADM_COMPATIBILITY'] = '1'
-    run = runner(yadm_y('alt'), env=env)
-    assert run.success
-    assert run.err == ''
-    linked = utils.parse_alt_output(run.out)
-
-    # assert the proper linking has occurred
-    # only a delimiter of '.' is valid
-    for file_path in TEST_PATHS:
-        source_file = file_path + suffix
-        if delimiter == '.':
-            assert paths.work.join(file_path).islink()
-            target = py.path.local(paths.work.join(file_path).readlink())
-            if target.isfile():
-                assert paths.work.join(file_path).read() == source_file
-                assert str(paths.work.join(source_file)) in linked
-            else:
-                assert paths.work.join(file_path).join(
-                    utils.CONTAINED).read() == source_file
-                assert str(paths.work.join(source_file)) in linked
-        else:
-            assert not paths.work.join(file_path).exists()
-            assert str(paths.work.join(source_file)) not in linked
-
-
-@pytest.mark.usefixtures('ds1_copy')
-def test_invalid_links_removed(runner, yadm_y, paths):
-    """Links to invalid alternative files are removed
-
-    This test ensures that when an already linked alternative becomes invalid
-    due to a change in class, the alternate link is removed.
-    """
-
-    # set the class
-    tst_class = 'testclass'
-    utils.set_local(paths, 'class', tst_class)
-
-    # create files which match the test class
-    utils.create_alt_files(paths, f'##{tst_class}')
-
-    # run alt to trigger linking
-    env = os.environ.copy()
-    env['YADM_COMPATIBILITY'] = '1'
-    run = runner(yadm_y('alt'), env=env)
-    assert run.success
-    assert run.err == ''
-    linked = utils.parse_alt_output(run.out)
-
-    # assert the proper linking has occurred
-    for file_path in TEST_PATHS:
-        source_file = file_path + '##' + tst_class
-        assert paths.work.join(file_path).islink()
-        target = py.path.local(paths.work.join(file_path).readlink())
-        if target.isfile():
-            assert paths.work.join(file_path).read() == source_file
-            assert str(paths.work.join(source_file)) in linked
-        else:
-            assert paths.work.join(file_path).join(
-                utils.CONTAINED).read() == source_file
-            assert str(paths.work.join(source_file)) in linked
-
-    # change the class so there are no valid alternates
-    utils.set_local(paths, 'class', 'changedclass')
-
-    # run alt to trigger linking
-    env = os.environ.copy()
-    env['YADM_COMPATIBILITY'] = '1'
-    run = runner(yadm_y('alt'), env=env)
-    assert run.success
-    assert run.err == ''
-    linked = utils.parse_alt_output(run.out)
-
-    # assert the linking is removed
-    for file_path in TEST_PATHS:
-        source_file = file_path + '##' + tst_class
-        assert not paths.work.join(file_path).exists()
-        assert str(paths.work.join(source_file)) not in linked

test/test_compat_jinja.py

@@ -1,198 +0,0 @@
-"""Test jinja"""
-
-import os
-import pytest
-import utils
-
-# These tests are for the template processing in YADM_COMPATIBILITY=1 mode
-pytestmark = pytest.mark.deprecated
-
-
-@pytest.fixture(scope='module')
-def envtpl_present(runner):
-    """Is envtpl present and working?"""
-    try:
-        run = runner(command=['envtpl', '-h'])
-        if run.success:
-            return True
-    except OSError:
-        pass
-    return False
-
-
-@pytest.mark.usefixtures('ds1_copy')
-def test_local_override(runner, yadm_y, paths,
-                        tst_distro, envtpl_present):
-    """Test local overrides"""
-    if not envtpl_present:
-        pytest.skip('Unable to test without envtpl.')
-
-    # define local overrides
-    utils.set_local(paths, 'class', 'or-class')
-    utils.set_local(paths, 'hostname', 'or-hostname')
-    utils.set_local(paths, 'os', 'or-os')
-    utils.set_local(paths, 'user', 'or-user')
-
-    template = (
-        'j2-{{ YADM_CLASS }}-'
-        '{{ YADM_OS }}-{{ YADM_HOSTNAME }}-'
-        '{{ YADM_USER }}-{{ YADM_DISTRO }}'
-        '-{%- '
-        f"include '{utils.INCLUDE_FILE}'"
-        ' -%}'
-    )
-    expected = (
-        f'j2-or-class-or-os-or-hostname-or-user-{tst_distro}'
-        f'-{utils.INCLUDE_CONTENT}'
-    )
-
-    utils.create_alt_files(paths, '##yadm.j2', content=template,
-                           includefile=True)
-
-    # os.system(f'find {paths.work}' + ' -name *j2 -ls -exec cat \'{}\' ";"')
-    # os.system(f'find {paths.work}')
-    # run alt to trigger linking
-    env = os.environ.copy()
-    env['YADM_COMPATIBILITY'] = '1'
-    run = runner(yadm_y('alt'), env=env)
-    assert run.success
-    assert run.err == ''
-    created = utils.parse_alt_output(run.out, linked=False)
-
-    # assert the proper creation has occurred
-    for file_path in (utils.ALT_FILE1, utils.ALT_FILE2):
-        source_file = file_path + '##yadm.j2'
-        assert paths.work.join(file_path).isfile()
-        lines = paths.work.join(file_path).readlines(cr=False)
-        assert lines[0] == source_file
-        assert lines[1] == expected
-        assert str(paths.work.join(source_file)) in created
-
-
-@pytest.mark.parametrize('autoalt', [None, 'true', 'false'])
-@pytest.mark.usefixtures('ds1_copy')
-def test_auto_alt(runner, yadm_y, paths, autoalt, tst_sys,
-                  envtpl_present):
-    """Test setting auto-alt"""
-
-    if not envtpl_present:
-        pytest.skip('Unable to test without envtpl.')
-
-    # set the value of auto-alt
-    if autoalt:
-        os.system(' '.join(yadm_y('config', 'yadm.auto-alt', autoalt)))
-
-    # create file
-    jinja_suffix = '##yadm.j2'
-    utils.create_alt_files(paths, jinja_suffix, content='{{ YADM_OS }}')
-
-    # run status to possibly trigger linking
-    env = os.environ.copy()
-    env['YADM_COMPATIBILITY'] = '1'
-    run = runner(yadm_y('status'), env=env)
-    assert run.success
-    assert run.err == ''
-    created = utils.parse_alt_output(run.out, linked=False)
-
-    # assert the proper creation has occurred
-    for file_path in (utils.ALT_FILE1, utils.ALT_FILE2):
-        source_file = file_path + jinja_suffix
-        if autoalt == 'false':
-            assert not paths.work.join(file_path).exists()
-        else:
-            assert paths.work.join(file_path).isfile()
-            lines = paths.work.join(file_path).readlines(cr=False)
-            assert lines[0] == source_file
-            assert lines[1] == tst_sys
-            # no created output when run via auto-alt
-            assert str(paths.work.join(source_file)) not in created
-
-
-@pytest.mark.usefixtures('ds1_copy')
-def test_jinja_envtpl_missing(runner, paths):
-    """Test operation when envtpl is missing"""
-
-    script = f"""
-        YADM_TEST=1 source {paths.pgm}
-        process_global_args -Y "{paths.yadm}"
-        set_operating_system
-        configure_paths
-        YADM_COMPATIBILITY=1
-        ENVTPL_PROGRAM='envtpl_missing' main alt
-    """
-
-    utils.create_alt_files(paths, '##yadm.j2')
-
-    run = runner(command=['bash'], inp=script)
-    assert run.success
-    assert run.err == ''
-    assert f'envtpl not available, not creating' in run.out
-
-
-@pytest.mark.parametrize(
-    'tracked, encrypt, exclude', [
-        (False, False, False),
-        (True, False, False),
-        (False, True, False),
-        (False, True, True),
-    ], ids=[
-        'untracked',
-        'tracked',
-        'encrypted',
-        'excluded',
-    ])
-@pytest.mark.usefixtures('ds1_copy')
-def test_jinja(runner, yadm_y, paths,
-               tst_sys, tst_host, tst_user, tst_distro,
-               tracked, encrypt, exclude,
-               envtpl_present):
-    """Test jinja processing"""
-
-    if not envtpl_present:
-        pytest.skip('Unable to test without envtpl.')
-
-    jinja_suffix = '##yadm.j2'
-
-    # set the class
-    tst_class = 'testclass'
-    utils.set_local(paths, 'class', tst_class)
-
-    template = (
-        'j2-{{ YADM_CLASS }}-'
-        '{{ YADM_OS }}-{{ YADM_HOSTNAME }}-'
-        '{{ YADM_USER }}-{{ YADM_DISTRO }}'
-        '-{%- '
-        f"include '{utils.INCLUDE_FILE}'"
-        ' -%}'
-    )
-    expected = (
-        f'j2-{tst_class}-'
-        f'{tst_sys}-{tst_host}-'
-        f'{tst_user}-{tst_distro}'
-        f'-{utils.INCLUDE_CONTENT}'
-    )
-
-    utils.create_alt_files(paths, jinja_suffix, content=template,
-                           tracked=tracked, encrypt=encrypt, exclude=exclude,
-                           includefile=True)
-
-    # run alt to trigger linking
-    env = os.environ.copy()
-    env['YADM_COMPATIBILITY'] = '1'
-    run = runner(yadm_y('alt'), env=env)
-    assert run.success
-    assert run.err == ''
-    created = utils.parse_alt_output(run.out, linked=False)
-
-    # assert the proper creation has occurred
-    for file_path in (utils.ALT_FILE1, utils.ALT_FILE2):
-        source_file = file_path + jinja_suffix
-        if tracked or (encrypt and not exclude):
-            assert paths.work.join(file_path).isfile()
-            lines = paths.work.join(file_path).readlines(cr=False)
-            assert lines[0] == source_file
-            assert lines[1] == expected
-            assert str(paths.work.join(source_file)) in created
-        else:
-            assert not paths.work.join(file_path).exists()
-            assert str(paths.work.join(source_file)) not in created

test/test_config.py

@@ -10,7 +10,7 @@ TEST_VALUE = 'testvalue'
 TEST_FILE = f'[{TEST_SECTION}]\n\t{TEST_ATTRIBUTE} = {TEST_VALUE}'
 
 
-def test_config_no_params(runner, yadm_y, supported_configs):
+def test_config_no_params(runner, yadm_cmd, supported_configs):
     """No parameters
 
     Display instructions
@@ -18,7 +18,7 @@ def test_config_no_params(runner, yadm_y, supported_configs):
     Exit with 0
     """
 
-    run = runner(yadm_y('config'))
+    run = runner(yadm_cmd('config'))
 
     assert run.success
     assert run.err == ''
@@ -27,21 +27,21 @@ def test_config_no_params(runner, yadm_y, supported_configs):
         assert config in run.out
 
 
-def test_config_read_missing(runner, yadm_y):
+def test_config_read_missing(runner, yadm_cmd):
     """Read missing attribute
 
     Display an empty value
     Exit with 0
     """
 
-    run = runner(yadm_y('config', TEST_KEY))
+    run = runner(yadm_cmd('config', TEST_KEY))
 
     assert run.success
     assert run.err == ''
     assert run.out == ''
 
 
-def test_config_write(runner, yadm_y, paths):
+def test_config_write(runner, yadm_cmd, paths):
     """Write attribute
 
     Display no output
@@ -49,7 +49,7 @@ def test_config_write(runner, yadm_y, paths):
     Exit with 0
     """
 
-    run = runner(yadm_y('config', TEST_KEY, TEST_VALUE))
+    run = runner(yadm_cmd('config', TEST_KEY, TEST_VALUE))
 
     assert run.success
     assert run.err == ''
@@ -57,7 +57,7 @@ def test_config_write(runner, yadm_y, paths):
     assert paths.config.read().strip() == TEST_FILE
 
 
-def test_config_read(runner, yadm_y, paths):
+def test_config_read(runner, yadm_cmd, paths):
     """Read attribute
 
     Display value
@@ -65,14 +65,14 @@ def test_config_read(runner, yadm_y, paths):
     """
 
     paths.config.write(TEST_FILE)
-    run = runner(yadm_y('config', TEST_KEY))
+    run = runner(yadm_cmd('config', TEST_KEY))
 
     assert run.success
     assert run.err == ''
     assert run.out.strip() == TEST_VALUE
 
 
-def test_config_update(runner, yadm_y, paths):
+def test_config_update(runner, yadm_cmd, paths):
     """Update attribute
 
     Display no output
@@ -82,7 +82,7 @@ def test_config_update(runner, yadm_y, paths):
 
     paths.config.write(TEST_FILE)
 
-    run = runner(yadm_y('config', TEST_KEY, TEST_VALUE + 'extra'))
+    run = runner(yadm_cmd('config', TEST_KEY, TEST_VALUE + 'extra'))
 
     assert run.success
     assert run.err == ''
@@ -92,7 +92,7 @@ def test_config_update(runner, yadm_y, paths):
 
 
 @pytest.mark.usefixtures('ds1_repo_copy')
-def test_config_local_read(runner, yadm_y, paths, supported_local_configs):
+def test_config_local_read(runner, yadm_cmd, paths, supported_local_configs):
     """Read local attribute
 
     Display value from the repo config
@@ -107,14 +107,14 @@ def test_config_local_read(runner, yadm_y, paths, supported_local_configs):
 
     # run yadm config
     for config in supported_local_configs:
-        run = runner(yadm_y('config', config))
+        run = runner(yadm_cmd('config', config))
         assert run.success
         assert run.err == ''
         assert run.out.strip() == f'value_of_{config}'
 
 
 @pytest.mark.usefixtures('ds1_repo_copy')
-def test_config_local_write(runner, yadm_y, paths, supported_local_configs):
+def test_config_local_write(runner, yadm_cmd, paths, supported_local_configs):
     """Write local attribute
 
     Display no output
@@ -124,7 +124,7 @@ def test_config_local_write(runner, yadm_y, paths, supported_local_configs):
 
     # run yadm config
     for config in supported_local_configs:
-        run = runner(yadm_y('config', config, f'value_of_{config}'))
+        run = runner(yadm_cmd('config', config, f'value_of_{config}'))
         assert run.success
         assert run.err == ''
         assert run.out == ''
@@ -137,3 +137,27 @@ def test_config_local_write(runner, yadm_y, paths, supported_local_configs):
         assert run.success
         assert run.err == ''
         assert run.out.strip() == f'value_of_{config}'
+
+
+def test_config_without_parent_directory(runner, yadm_cmd, paths):
+    """Write and read attribute to/from config file with non-existent parent dir
+
+    Update configuration file
+    Display value
+    Exit with 0
+    """
+
+    config_file = paths.root + '/folder/does/not/exist/config'
+
+    run = runner(
+        yadm_cmd('--yadm-config', config_file, 'config', TEST_KEY, TEST_VALUE))
+
+    assert run.success
+    assert run.err == ''
+    assert run.out == ''
+
+    run = runner(yadm_cmd('--yadm-config', config_file, 'config', TEST_KEY))
+
+    assert run.success
+    assert run.err == ''
+    assert run.out.strip() == TEST_VALUE

test/test_encryption.py

@@ -2,6 +2,7 @@
 
 import os
 import pipes
+import time
 import pytest
 
 KEY_FILE = 'test/test_key'
@@ -13,30 +14,54 @@ PASSPHRASE = 'ExamplePassword'
 pytestmark = pytest.mark.usefixtures('config_git')
 
 
-def add_asymmetric_key():
+def add_asymmetric_key(runner, gnupg):
     """Add asymmetric key"""
-    os.system(f'gpg --import {pipes.quote(KEY_FILE)}')
-    os.system(f'gpg --import-ownertrust < {pipes.quote(KEY_TRUST)}')
+    env = os.environ.copy()
+    env['GNUPGHOME'] = gnupg.home
+    runner(
+        ['gpg', '--import', pipes.quote(KEY_FILE)],
+        env=env,
+        shell=True,
+    )
+    runner(
+        ['gpg', '--import-ownertrust', '<', pipes.quote(KEY_TRUST)],
+        env=env,
+        shell=True,
+    )
 
 
-def remove_asymmetric_key():
+def remove_asymmetric_key(runner, gnupg):
     """Remove asymmetric key"""
-    os.system(
-        f'gpg --batch --yes '
-        f'--delete-secret-keys {pipes.quote(KEY_FINGERPRINT)}')
-    os.system(f'gpg --batch --yes --delete-key {pipes.quote(KEY_FINGERPRINT)}')
+    env = os.environ.copy()
+    env['GNUPGHOME'] = gnupg.home
+    runner(
+        [
+            'gpg', '--batch', '--yes',
+            '--delete-secret-keys', pipes.quote(KEY_FINGERPRINT)
+        ],
+        env=env,
+        shell=True,
+    )
+    runner(
+        [
+            'gpg', '--batch', '--yes',
+            '--delete-key', pipes.quote(KEY_FINGERPRINT)
+        ],
+        env=env,
+        shell=True,
+    )
 
 
 @pytest.fixture
-def asymmetric_key():
+def asymmetric_key(runner, gnupg):
     """Fixture for asymmetric key, removed in teardown"""
-    add_asymmetric_key()
+    add_asymmetric_key(runner, gnupg)
     yield KEY_NAME
-    remove_asymmetric_key()
+    remove_asymmetric_key(runner, gnupg)
 
 
 @pytest.fixture
-def encrypt_targets(yadm_y, paths):
+def encrypt_targets(yadm_cmd, paths):
     """Fixture for setting up data to encrypt
 
     This fixture:
@@ -53,7 +78,7 @@ def encrypt_targets(yadm_y, paths):
     """
 
     # init empty yadm repo
-    os.system(' '.join(yadm_y('init', '-w', str(paths.work), '-f')))
+    os.system(' '.join(yadm_cmd('init', '-w', str(paths.work), '-f')))
 
     expected = []
 
@@ -95,7 +120,7 @@ def encrypt_targets(yadm_y, paths):
 
 
 @pytest.fixture(scope='session')
-def decrypt_targets(tmpdir_factory, runner):
+def decrypt_targets(tmpdir_factory, runner, gnupg):
     """Fixture for setting data to decrypt
 
     This fixture:
@@ -117,17 +142,21 @@ def decrypt_targets(tmpdir_factory, runner):
     tmpdir.join('subdir/decrypt3').write('subdir/decrypt3')
     expected.append('subdir/decrypt3')
 
+    gnupg.pw(PASSPHRASE)
+    env = os.environ.copy()
+    env['GNUPGHOME'] = gnupg.home
     run = runner(
         ['tar', 'cvf', '-'] +
         expected +
         ['|', 'gpg', '--batch', '--yes', '-c'] +
-        ['--passphrase', pipes.quote(PASSPHRASE)] +
         ['--output', pipes.quote(str(symmetric))],
         cwd=tmpdir,
+        env=env,
         shell=True)
     assert run.success
 
-    add_asymmetric_key()
+    gnupg.pw('')
+    add_asymmetric_key(runner, gnupg)
     run = runner(
         ['tar', 'cvf', '-'] +
         expected +
@@ -135,9 +164,10 @@ def decrypt_targets(tmpdir_factory, runner):
         ['-r', pipes.quote(KEY_NAME)] +
         ['--output', pipes.quote(str(asymmetric))],
         cwd=tmpdir,
+        env=env,
         shell=True)
     assert run.success
-    remove_asymmetric_key()
+    remove_asymmetric_key(runner, gnupg)
 
     return {
         'asymmetric': asymmetric,
@@ -147,8 +177,8 @@ def decrypt_targets(tmpdir_factory, runner):
 
 
 @pytest.mark.parametrize(
-    'mismatched_phrase', [False, True],
-    ids=['matching_phrase', 'mismatched_phrase'])
+    'bad_phrase', [False, True],
+    ids=['good_phrase', 'bad_phrase'])
 @pytest.mark.parametrize(
     'missing_encrypt', [False, True],
     ids=['encrypt_exists', 'encrypt_missing'])
@@ -156,42 +186,43 @@ def decrypt_targets(tmpdir_factory, runner):
     'overwrite', [False, True],
     ids=['clean', 'overwrite'])
 def test_symmetric_encrypt(
-        runner, yadm_y, paths, encrypt_targets,
-        overwrite, missing_encrypt, mismatched_phrase):
+        runner, yadm_cmd, paths, encrypt_targets,
+        gnupg, bad_phrase, overwrite, missing_encrypt):
     """Test symmetric encryption"""
 
     if missing_encrypt:
         paths.encrypt.remove()
 
-    matched_phrase = PASSPHRASE
-    if mismatched_phrase:
-        matched_phrase = 'mismatched'
+    if bad_phrase:
+        gnupg.pw('')
+    else:
+        gnupg.pw(PASSPHRASE)
 
     if overwrite:
         paths.archive.write('existing archive')
 
-    run = runner(yadm_y('encrypt'), expect=[
-        ('passphrase:', PASSPHRASE),
-        ('passphrase:', matched_phrase),
-        ])
+    env = os.environ.copy()
+    env['GNUPGHOME'] = gnupg.home
+    run = runner(yadm_cmd('encrypt'), env=env)
 
-    if missing_encrypt or mismatched_phrase:
+    if missing_encrypt or bad_phrase:
         assert run.failure
     else:
         assert run.success
         assert run.err == ''
 
     if missing_encrypt:
-        assert 'does not exist' in run.out
-    elif mismatched_phrase:
-        assert 'invalid passphrase' in run.out
+        assert 'does not exist' in run.err
+    elif bad_phrase:
+        assert 'Invalid passphrase' in run.err
     else:
-        assert encrypted_data_valid(runner, paths.archive, encrypt_targets)
+        assert encrypted_data_valid(
+            runner, gnupg, paths.archive, encrypt_targets)
 
 
 @pytest.mark.parametrize(
-    'wrong_phrase', [False, True],
-    ids=['correct_phrase', 'wrong_phrase'])
+    'bad_phrase', [False, True],
+    ids=['good_phrase', 'bad_phrase'])
 @pytest.mark.parametrize(
     'archive_exists', [True, False],
     ids=['archive_exists', 'archive_missing'])
@@ -199,16 +230,18 @@ def test_symmetric_encrypt(
     'dolist', [False, True],
     ids=['decrypt', 'list'])
 def test_symmetric_decrypt(
-        runner, yadm_y, paths, decrypt_targets,
-        dolist, archive_exists, wrong_phrase):
+        runner, yadm_cmd, paths, decrypt_targets, gnupg,
+        dolist, archive_exists, bad_phrase):
     """Test decryption"""
 
     # init empty yadm repo
-    os.system(' '.join(yadm_y('init', '-w', str(paths.work), '-f')))
+    os.system(' '.join(yadm_cmd('init', '-w', str(paths.work), '-f')))
 
-    phrase = PASSPHRASE
-    if wrong_phrase:
-        phrase = 'wrong-phrase'
+    if bad_phrase:
+        gnupg.pw('')
+        time.sleep(1)  # allow gpg-agent cache to expire
+    else:
+        gnupg.pw(PASSPHRASE)
 
     if archive_exists:
         decrypt_targets['symmetric'].copy(paths.archive)
@@ -216,15 +249,18 @@ def test_symmetric_decrypt(
     # to test overwriting
     paths.work.join('decrypt1').write('pre-existing file')
 
+    env = os.environ.copy()
+    env['GNUPGHOME'] = gnupg.home
+
     args = []
 
     if dolist:
         args.append('-l')
-    run = runner(yadm_y('decrypt') + args, expect=[('passphrase:', phrase)])
+    run = runner(yadm_cmd('decrypt') + args, env=env)
 
-    if archive_exists and not wrong_phrase:
+    if archive_exists and not bad_phrase:
         assert run.success
-        assert run.err == ''
+        assert 'encrypted with 1 passphrase' in run.err
         if dolist:
             for filename in decrypt_targets['expected']:
                 if filename != 'decrypt1':  # this one should exist
@@ -248,37 +284,59 @@ def test_symmetric_decrypt(
     'overwrite', [False, True],
     ids=['clean', 'overwrite'])
 def test_asymmetric_encrypt(
-        runner, yadm_y, paths, encrypt_targets,
+        runner, yadm_cmd, paths, encrypt_targets, gnupg,
         overwrite, key_exists, ask):
     """Test asymmetric encryption"""
 
     # specify encryption recipient
     if ask:
-        os.system(' '.join(yadm_y('config', 'yadm.gpg-recipient', 'ASK')))
+        os.system(' '.join(yadm_cmd('config', 'yadm.gpg-recipient', 'ASK')))
         expect = [('Enter the user ID', KEY_NAME), ('Enter the user ID', '')]
     else:
-        os.system(' '.join(yadm_y('config', 'yadm.gpg-recipient', KEY_NAME)))
+        os.system(' '.join(yadm_cmd('config', 'yadm.gpg-recipient', KEY_NAME)))
         expect = []
 
     if overwrite:
         paths.archive.write('existing archive')
 
     if not key_exists:
-        remove_asymmetric_key()
+        remove_asymmetric_key(runner, gnupg)
 
-    run = runner(yadm_y('encrypt'), expect=expect)
+    env = os.environ.copy()
+    env['GNUPGHOME'] = gnupg.home
+
+    run = runner(yadm_cmd('encrypt'), env=env, expect=expect)
 
     if key_exists:
         assert run.success
-        assert encrypted_data_valid(runner, paths.archive, encrypt_targets)
+        assert encrypted_data_valid(
+            runner, gnupg, paths.archive, encrypt_targets)
     else:
         assert run.failure
-        assert 'Unable to write' in run.out
+        assert 'Unable to write' in run.out if expect else run.err
 
     if ask:
         assert 'Enter the user ID' in run.out
 
 
+@pytest.mark.usefixtures('asymmetric_key')
+@pytest.mark.usefixtures('encrypt_targets')
+def test_multi_key(runner, yadm_cmd, gnupg):
+    """Test multiple recipients"""
+
+    # specify two encryption recipient
+    os.system(' '.join(yadm_cmd(
+        'config', 'yadm.gpg-recipient', f'"second-key {KEY_NAME}"')))
+
+    env = os.environ.copy()
+    env['GNUPGHOME'] = gnupg.home
+
+    run = runner(yadm_cmd('encrypt'), env=env)
+
+    assert run.failure
+    assert 'second-key: skipped: No public key' in run.err
+
+
 @pytest.mark.usefixtures('asymmetric_key')
 @pytest.mark.parametrize(
     'key_exists', [True, False],
@@ -287,12 +345,12 @@ def test_asymmetric_encrypt(
     'dolist', [False, True],
     ids=['decrypt', 'list'])
 def test_asymmetric_decrypt(
-        runner, yadm_y, paths, decrypt_targets,
+        runner, yadm_cmd, paths, decrypt_targets, gnupg,
         dolist, key_exists):
     """Test decryption"""
 
     # init empty yadm repo
-    os.system(' '.join(yadm_y('init', '-w', str(paths.work), '-f')))
+    os.system(' '.join(yadm_cmd('init', '-w', str(paths.work), '-f')))
 
     decrypt_targets['asymmetric'].copy(paths.archive)
 
@@ -300,13 +358,15 @@ def test_asymmetric_decrypt(
     paths.work.join('decrypt1').write('pre-existing file')
 
     if not key_exists:
-        remove_asymmetric_key()
+        remove_asymmetric_key(runner, gnupg)
 
     args = []
 
     if dolist:
         args.append('-l')
-    run = runner(yadm_y('decrypt') + args)
+    env = os.environ.copy()
+    env['GNUPGHOME'] = gnupg.home
+    run = runner(yadm_cmd('decrypt') + args, env=env)
 
     if key_exists:
         assert run.success
@@ -320,14 +380,15 @@ def test_asymmetric_decrypt(
                 assert paths.work.join(filename).read() == filename
     else:
         assert run.failure
-        assert 'Unable to extract encrypted files' in run.out
+        assert 'Unable to extract encrypted files' in run.err
 
 
 @pytest.mark.parametrize(
     'untracked',
     [False, 'y', 'n'],
     ids=['tracked', 'untracked_answer_y', 'untracked_answer_n'])
-def test_offer_to_add(runner, yadm_y, paths, encrypt_targets, untracked):
+def test_offer_to_add(
+        runner, yadm_cmd, paths, encrypt_targets, gnupg, untracked):
     """Test offer to add encrypted archive
 
     All the other encryption tests use an archive outside of the work tree.
@@ -336,28 +397,32 @@ def test_offer_to_add(runner, yadm_y, paths, encrypt_targets, untracked):
     """
 
     worktree_archive = paths.work.join('worktree-archive.tar.gpg')
-    expect = [
-        ('passphrase:', PASSPHRASE),
-        ('passphrase:', PASSPHRASE),
-        ]
+
+    expect = []
+
+    gnupg.pw(PASSPHRASE)
+    env = os.environ.copy()
+    env['GNUPGHOME'] = gnupg.home
 
     if untracked:
         expect.append(('add it now', untracked))
     else:
         worktree_archive.write('exists')
-        os.system(' '.join(yadm_y('add', str(worktree_archive))))
+        os.system(' '.join(yadm_cmd('add', str(worktree_archive))))
 
     run = runner(
-        yadm_y('encrypt', '--yadm-archive', str(worktree_archive)),
+        yadm_cmd('encrypt', '--yadm-archive', str(worktree_archive)),
+        env=env,
         expect=expect
         )
 
     assert run.success
     assert run.err == ''
-    assert encrypted_data_valid(runner, worktree_archive, encrypt_targets)
+    assert encrypted_data_valid(
+        runner, gnupg, worktree_archive, encrypt_targets)
 
     run = runner(
-        yadm_y('status', '--porcelain', '-uall', str(worktree_archive)))
+        yadm_cmd('status', '--porcelain', '-uall', str(worktree_archive)))
     assert run.success
     assert run.err == ''
 
@@ -372,22 +437,20 @@ def test_offer_to_add(runner, yadm_y, paths, encrypt_targets, untracked):
         assert f'AM {worktree_archive.basename}' in run.out
 
 
-def test_encrypt_added_to_exclude(runner, yadm_y, paths):
+@pytest.mark.usefixtures('ds1_copy')
+def test_encrypt_added_to_exclude(runner, yadm_cmd, paths, gnupg):
     """Confirm that .config/yadm/encrypt is added to exclude"""
 
-    expect = [
-        ('passphrase:', PASSPHRASE),
-        ('passphrase:', PASSPHRASE),
-        ]
+    gnupg.pw(PASSPHRASE)
+    env = os.environ.copy()
+    env['GNUPGHOME'] = gnupg.home
 
     exclude_file = paths.repo.join('info/exclude')
     paths.encrypt.write('test-encrypt-data\n')
+    paths.work.join('test-encrypt-data').write('')
     exclude_file.write('original-data', ensure=True)
 
-    run = runner(
-        yadm_y('encrypt'),
-        expect=expect,
-        )
+    run = runner(yadm_cmd('encrypt'), env=env)
 
     assert 'test-encrypt-data' in paths.repo.join('info/exclude').read()
     assert 'original-data' in paths.repo.join('info/exclude').read()
@@ -395,14 +458,16 @@ def test_encrypt_added_to_exclude(runner, yadm_y, paths):
     assert run.err == ''
 
 
-def encrypted_data_valid(runner, encrypted, expected):
+def encrypted_data_valid(runner, gnupg, encrypted, expected):
     """Verify encrypted data matches expectations"""
+    gnupg.pw(PASSPHRASE)
+    env = os.environ.copy()
+    env['GNUPGHOME'] = gnupg.home
     run = runner([
         'gpg',
-        '--passphrase', pipes.quote(PASSPHRASE),
         '-d', pipes.quote(str(encrypted)),
         '2>/dev/null',
-        '|', 'tar', 't'], shell=True, report=False)
+        '|', 'tar', 't'], env=env, shell=True, report=False)
     file_count = 0
     for filename in run.out.splitlines():
         if filename.endswith('/'):

test/test_enter.py

@@ -17,7 +17,7 @@ import pytest
         'shell-noexec',
     ])
 @pytest.mark.usefixtures('ds1_copy')
-def test_enter(runner, yadm_y, paths, shell, success):
+def test_enter(runner, yadm_cmd, paths, shell, success):
     """Enter tests"""
     env = os.environ.copy()
     if shell == 'delete':
@@ -33,17 +33,18 @@ def test_enter(runner, yadm_y, paths, shell, success):
     else:
         env['SHELL'] = shell
 
-    run = runner(command=yadm_y('enter'), env=env)
+    run = runner(command=yadm_cmd('enter'), env=env)
     assert run.success == success
-    assert run.err == ''
     prompt = f'yadm shell ({paths.repo})'
     if success:
         assert run.out.startswith('Entering yadm repo')
         assert run.out.rstrip().endswith('Leaving yadm repo')
-    if not success:
-        assert 'does not refer to an executable' in run.out
+        assert run.err == ''
+    else:
+        assert 'does not refer to an executable' in run.err
     if 'env' in shell:
         assert f'GIT_DIR={paths.repo}' in run.out
+        assert f'GIT_WORK_TREE={paths.work}' in run.out
         assert f'PROMPT={prompt}' in run.out
         assert f'PS1={prompt}' in run.out
 
@@ -58,20 +59,52 @@ def test_enter(runner, yadm_y, paths, shell, success):
         'csh',
         'zsh',
     ])
+@pytest.mark.parametrize(
+    'cmd',
+    [False, 'cmd', 'cmd-bad-exit'],
+    ids=['no-cmd', 'cmd', 'cmd-bad-exit'])
+@pytest.mark.parametrize(
+    'term', ['', 'dumb'],
+    ids=['term-empty', 'term-dumb'])
 @pytest.mark.usefixtures('ds1_copy')
-def test_enter_shell_ops(runner, yadm_y, paths, shell, opts, path):
+def test_enter_shell_ops(runner, yadm_cmd, paths, shell,
+                         opts, path, cmd, term):
     """Enter tests for specific shell options"""
 
+    change_exit = '\nfalse' if cmd == 'cmd-bad-exit' else ''
+
     # Create custom shell to detect options passed
     custom_shell = paths.root.join(shell)
-    custom_shell.write('#!/bin/sh\necho OPTS=$*\necho PROMPT=$PROMPT')
+    custom_shell.write(
+        f'#!/bin/sh\necho OPTS=$*\necho PROMPT=$PROMPT{change_exit}'
+    )
     custom_shell.chmod(0o775)
 
+    test_cmd = ['test1', 'test2', 'test3']
+
+    enter_cmd = ['enter']
+    if cmd:
+        enter_cmd += test_cmd
+
     env = os.environ.copy()
+    env['TERM'] = term
     env['SHELL'] = custom_shell
 
-    run = runner(command=yadm_y('enter'), env=env)
-    assert run.success
+    if shell == 'zsh' and term == 'dumb':
+        opts += ' --no-zle'
+
+    run = runner(command=yadm_cmd(*enter_cmd), env=env)
+    if cmd == 'cmd-bad-exit':
+        assert run.failure
+    else:
+        assert run.success
     assert run.err == ''
     assert f'OPTS={opts}' in run.out
     assert f'PROMPT=yadm shell ({paths.repo}) {path} >' in run.out
+    if cmd:
+        assert '-c ' + ' '.join(test_cmd) in run.out
+        assert 'Entering yadm repo' not in run.out
+        assert 'Leaving yadm repo' not in run.out
+    else:
+        assert 'Entering yadm repo' in run.out
+        assert 'Leaving yadm repo' in run.out

test/test_ext_crypt.py

@@ -0,0 +1,48 @@
+"""Test external encryption commands"""
+
+import pytest
+
+
+@pytest.mark.parametrize(
+    'crypt',
+    [False, 'installed', 'installed-but-failed'],
+    ids=['not-installed', 'installed', 'installed-but-failed']
+)
+@pytest.mark.parametrize(
+        'cmd,var', [
+            ['git_crypt', 'GIT_CRYPT_PROGRAM'],
+            ['transcrypt', 'TRANSCRYPT_PROGRAM'],
+        ],
+        ids=['git-crypt', 'transcrypt'])
+def test_ext_encryption(runner, yadm, paths, tmpdir, crypt, cmd, var):
+    """External encryption tests"""
+
+    paths.repo.ensure(dir=True)
+    bindir = tmpdir.mkdir('bin')
+    pgm = bindir.join('test-ext-crypt')
+
+    if crypt:
+        pgm.write('#!/bin/sh\necho ext-crypt ran\n')
+        pgm.chmod(0o775)
+    if crypt == 'installed-but-failed':
+        pgm.write('false\n', mode='a')
+
+    script = f"""
+        YADM_TEST=1 source {yadm}
+        YADM_REPO={paths.repo}
+        {var}="{pgm}"
+        {cmd} "param1"
+    """
+
+    run = runner(command=['bash'], inp=script)
+
+    if crypt:
+        if crypt == 'installed-but-failed':
+            assert run.failure
+        else:
+            assert run.success
+        assert run.out.strip() == 'ext-crypt ran'
+        assert run.err == ''
+    else:
+        assert run.failure
+        assert f"command '{pgm}' cannot be located" in run.err

test/test_git.py

@@ -5,7 +5,7 @@ import pytest
 
 
 @pytest.mark.usefixtures('ds1_copy')
-def test_git(runner, yadm_y, paths):
+def test_git(runner, yadm_cmd, paths):
     """Test series of passthrough git commands
 
     Passthru unknown commands to Git
@@ -17,14 +17,14 @@ def test_git(runner, yadm_y, paths):
     """
 
     # passthru unknown commands to Git
-    run = runner(command=yadm_y('bogus'))
+    run = runner(command=yadm_cmd('bogus'))
     assert run.failure
     assert "git: 'bogus' is not a git command." in run.err
     assert "See 'git --help'" in run.err
     assert run.out == ''
 
     # git command 'add' - badfile
-    run = runner(command=yadm_y('add', '-v', 'does_not_exist'))
+    run = runner(command=yadm_cmd('add', '-v', 'does_not_exist'))
     assert run.code == 128
     assert "pathspec 'does_not_exist' did not match any files" in run.err
     assert run.out == ''
@@ -32,19 +32,19 @@ def test_git(runner, yadm_y, paths):
     # git command 'add'
     newfile = paths.work.join('test_git')
     newfile.write('test_git')
-    run = runner(command=yadm_y('add', '-v', str(newfile)))
+    run = runner(command=yadm_cmd('add', '-v', str(newfile)))
     assert run.success
     assert run.err == ''
     assert "add 'test_git'" in run.out
 
     # git command 'status'
-    run = runner(command=yadm_y('status'))
+    run = runner(command=yadm_cmd('status'))
     assert run.success
     assert run.err == ''
     assert re.search(r'new file:\s+test_git', run.out)
 
     # git command 'commit'
-    run = runner(command=yadm_y('commit', '-m', 'Add test_git'))
+    run = runner(command=yadm_cmd('commit', '-m', 'Add test_git'))
     assert run.success
     assert run.err == ''
     assert '1 file changed' in run.out
@@ -52,7 +52,7 @@ def test_git(runner, yadm_y, paths):
     assert re.search(r'create mode .+ test_git', run.out)
 
     # git command 'log'
-    run = runner(command=yadm_y('log', '--oneline'))
+    run = runner(command=yadm_cmd('log', '--oneline'))
     assert run.success
     assert run.err == ''
     assert 'Add test_git' in run.out

test/test_help.py

@@ -1,17 +1,19 @@
 """Test help"""
+import pytest
 
 
-def test_missing_command(runner, yadm_y):
+def test_missing_command(runner, yadm_cmd):
     """Run without any command"""
-    run = runner(command=yadm_y())
+    run = runner(command=yadm_cmd())
     assert run.failure
     assert run.err == ''
     assert run.out.startswith('Usage: yadm')
 
 
-def test_help_command(runner, yadm_y):
+@pytest.mark.parametrize('cmd', ['--help', 'help'])
+def test_help_command(runner, yadm_cmd, cmd):
     """Run with help command"""
-    run = runner(command=yadm_y('help'))
+    run = runner(command=yadm_cmd(cmd))
     assert run.failure
     assert run.err == ''
     assert run.out.startswith('Usage: yadm')

test/test_hooks.py

@@ -21,8 +21,9 @@ import pytest
         'pre-post-success',
         'pre-post-fail',
     ])
+@pytest.mark.parametrize('cmd', ['--version', 'version'])
 def test_hooks(
-        runner, yadm_y, paths,
+        runner, yadm_cmd, paths, cmd,
         pre, pre_code, post, post_code):
     """Test pre/post hook"""
 
@@ -33,7 +34,7 @@ def test_hooks(
         create_hook(paths, 'post_version', post_code)
 
     # run yadm
-    run = runner(yadm_y('version'))
+    run = runner(yadm_cmd(cmd))
     # when a pre hook fails, yadm should exit with the hook's code
     assert run.code == pre_code
     assert run.err == ''
@@ -53,7 +54,7 @@ def test_hooks(
 
 # repo fixture is needed to test the population of YADM_HOOK_WORK
 @pytest.mark.usefixtures('ds1_repo_copy')
-def test_hook_env(runner, yadm_y, paths):
+def test_hook_env(runner, yadm_cmd, paths):
     """Test hook environment"""
 
     # test will be done with a non existent "git" passthru command
@@ -62,10 +63,10 @@ def test_hook_env(runner, yadm_y, paths):
 
     # write the hook
     hook = paths.hooks.join(f'post_{cmd}')
-    hook.write('#!/bin/sh\nenv\n')
+    hook.write('#!/bin/bash\nenv\ndeclare\n')
     hook.chmod(0o755)
 
-    run = runner(yadm_y(cmd, 'extra_args'))
+    run = runner(yadm_cmd(cmd, 'extra_args'))
 
     # expect passthru to fail
     assert run.failure
@@ -74,9 +75,89 @@ def test_hook_env(runner, yadm_y, paths):
     # verify hook environment
     assert 'YADM_HOOK_EXIT=1\n' in run.out
     assert f'YADM_HOOK_COMMAND={cmd}\n' in run.out
+    assert f'YADM_HOOK_DIR={paths.yadm}\n' in run.out
     assert f'YADM_HOOK_FULL_COMMAND={cmd} extra_args\n' in run.out
     assert f'YADM_HOOK_REPO={paths.repo}\n' in run.out
     assert f'YADM_HOOK_WORK={paths.work}\n' in run.out
+    assert 'YADM_ENCRYPT_INCLUDE_FILES=\n' in run.out
+
+    # verify the hook environment contains certain exported functions
+    for func in [
+            'builtin_dirname',
+            'relative_path',
+            'unix_path',
+            'mixed_path',
+    ]:
+        assert f'BASH_FUNC_{func}' in run.out
+
+    # verify the hook environment contains the list of encrypted files
+    script = f"""
+        YADM_TEST=1 source {paths.pgm}
+        YADM_HOOKS="{paths.hooks}"
+        HOOK_COMMAND="{cmd}"
+        ENCRYPT_INCLUDE_FILES=(a b c)
+        invoke_hook "post"
+    """
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.err == ''
+    assert 'YADM_ENCRYPT_INCLUDE_FILES=a\nb\nc\n' in run.out
+
+
+def test_escaped(runner, yadm_cmd, paths):
+    """Test escaped values in YADM_HOOK_FULL_COMMAND"""
+
+    # test will be done with a non existent "git" passthru command
+    # which should exit with a failing code
+    cmd = 'passthrucmd'
+
+    # write the hook
+    hook = paths.hooks.join(f'post_{cmd}')
+    hook.write('#!/bin/bash\nenv\n')
+    hook.chmod(0o755)
+
+    run = runner(yadm_cmd(cmd, 'a b', 'c\td', 'e\\f'))
+
+    # expect passthru to fail
+    assert run.failure
+
+    # verify escaped values
+    assert (
+        f'YADM_HOOK_FULL_COMMAND={cmd} '
+        'a\\ b c\\\td e\\\\f\n') in run.out
+
+
+@pytest.mark.parametrize('condition', ['exec', 'no-exec', 'mingw'])
+def test_executable(runner, paths, condition):
+    """Verify hook must be exectuable"""
+    cmd = 'version'
+    hook = paths.hooks.join(f'pre_{cmd}')
+    hook.write('#!/bin/sh\necho HOOK\n')
+    hook.chmod(0o644)
+    if condition == 'exec':
+        hook.chmod(0o755)
+
+    mingw = 'OPERATING_SYSTEM="MINGWx"' if condition == 'mingw' else ''
+    script = f"""
+        YADM_TEST=1 source {paths.pgm}
+        YADM_HOOKS="{paths.hooks}"
+        HOOK_COMMAND="{cmd}"
+        {mingw}
+        invoke_hook "pre"
+    """
+    run = runner(command=['bash'], inp=script)
+
+    if condition != 'mingw':
+        assert run.success
+        assert run.err == ''
+    else:
+        assert run.failure
+        assert 'Permission denied' in run.err
+
+    if condition == 'exec':
+        assert 'HOOK' in run.out
+    elif condition == 'no-exec':
+        assert 'HOOK' not in run.out
 
 
 def create_hook(paths, name, code):

test/test_init.py

@@ -19,7 +19,7 @@ import pytest
     ])
 @pytest.mark.usefixtures('ds1_work_copy')
 def test_init(
-        runner, yadm_y, paths, repo_config, alt_work, repo_present, force):
+        runner, yadm_cmd, paths, repo_config, alt_work, repo_present, force):
     """Test init
 
     Repos should have attribs:
@@ -45,18 +45,22 @@ def test_init(
 
     # command args
     args = ['init']
+    cwd = None
     if alt_work:
-        args.extend(['-w', paths.work])
+        if force:
+            cwd = paths.work.dirname
+            args.extend(['-w', paths.work.basename])
+        else:
+            args.extend(['-w', paths.work])
     if force:
         args.append('-f')
 
     # run init
-    run = runner(yadm_y(*args), env={'HOME': home})
-    assert run.err == ''
+    run = runner(yadm_cmd(*args), env={'HOME': home}, cwd=cwd)
 
     if repo_present and not force:
         assert run.failure
-        assert 'repo already exists' in run.out
+        assert 'repo already exists' in run.err
         assert old_repo.isfile(), 'Missing original repo'
     else:
         assert run.success
@@ -64,6 +68,8 @@ def test_init(
 
         if repo_present:
             assert not old_repo.isfile(), 'Original repo still exists'
+        else:
+            assert run.err == ''
 
         if alt_work:
             assert repo_config('core.worktree') == paths.work

test/test_introspect.py

@@ -13,13 +13,13 @@ import pytest
         'switches',
     ])
 def test_introspect_category(
-        runner, yadm_y, paths, name,
+        runner, yadm_cmd, paths, name,
         supported_commands, supported_configs, supported_switches):
     """Validate introspection category"""
     if name:
-        run = runner(command=yadm_y('introspect', name))
+        run = runner(command=yadm_cmd('introspect', name))
     else:
-        run = runner(command=yadm_y('introspect'))
+        run = runner(command=yadm_cmd('introspect'))
 
     assert run.success
     assert run.err == ''

test/test_list.py

@@ -11,7 +11,7 @@ import pytest
         'subdir',
     ])
 @pytest.mark.usefixtures('ds1_copy')
-def test_list(runner, yadm_y, paths, ds1, location):
+def test_list(runner, yadm_cmd, paths, ds1, location):
     """List tests"""
     if location == 'work':
         run_dir = paths.work
@@ -23,7 +23,7 @@ def test_list(runner, yadm_y, paths, ds1, location):
     with run_dir.as_cwd():
         # test with '-a'
         # should get all tracked files, relative to the work path
-        run = runner(command=yadm_y('list', '-a'))
+        run = runner(command=yadm_cmd('list', '-a'))
         assert run.success
         assert run.err == ''
         returned_files = set(run.out.splitlines())
@@ -33,7 +33,7 @@ def test_list(runner, yadm_y, paths, ds1, location):
         # should get all tracked files, relative to the work path unless in a
         # subdir, then those should be a limited set of files, relative to the
         # subdir
-        run = runner(command=yadm_y('list'))
+        run = runner(command=yadm_cmd('list'))
         assert run.success
         assert run.err == ''
         returned_files = set(run.out.splitlines())

test/test_perms.py

@@ -6,12 +6,13 @@ import pytest
 
 @pytest.mark.parametrize('autoperms', ['notest', 'unset', 'true', 'false'])
 @pytest.mark.usefixtures('ds1_copy')
-def test_perms(runner, yadm_y, paths, ds1, autoperms):
+def test_perms(runner, yadm_cmd, paths, ds1, autoperms):
     """Test perms"""
     # set the value of auto-perms
     if autoperms != 'notest':
         if autoperms != 'unset':
-            os.system(' '.join(yadm_y('config', 'yadm.auto-perms', autoperms)))
+            os.system(' '.join(
+                yadm_cmd('config', 'yadm.auto-perms', autoperms)))
 
     # privatepaths will hold all paths that should become secured
     privatepaths = [paths.work.join('.ssh'), paths.work.join('.gnupg')]
@@ -38,7 +39,7 @@ def test_perms(runner, yadm_y, paths, ds1, autoperms):
     cmd = 'perms'
     if autoperms != 'notest':
         cmd = 'status'
-    run = runner(yadm_y(cmd))
+    run = runner(yadm_cmd(cmd), env={'HOME': paths.work})
     assert run.success
     assert run.err == ''
     if cmd == 'perms':
@@ -62,15 +63,15 @@ def test_perms(runner, yadm_y, paths, ds1, autoperms):
 @pytest.mark.parametrize('sshperms', [None, 'true', 'false'])
 @pytest.mark.parametrize('gpgperms', [None, 'true', 'false'])
 @pytest.mark.usefixtures('ds1_copy')
-def test_perms_control(runner, yadm_y, paths, ds1, sshperms, gpgperms):
+def test_perms_control(runner, yadm_cmd, paths, ds1, sshperms, gpgperms):
     """Test fine control of perms"""
     # set the value of ssh-perms
     if sshperms:
-        os.system(' '.join(yadm_y('config', 'yadm.ssh-perms', sshperms)))
+        os.system(' '.join(yadm_cmd('config', 'yadm.ssh-perms', sshperms)))
 
     # set the value of gpg-perms
     if gpgperms:
-        os.system(' '.join(yadm_y('config', 'yadm.gpg-perms', gpgperms)))
+        os.system(' '.join(yadm_cmd('config', 'yadm.gpg-perms', gpgperms)))
 
     # privatepaths will hold all paths that should become secured
     privatepaths = [paths.work.join('.ssh'), paths.work.join('.gnupg')]
@@ -81,7 +82,7 @@ def test_perms_control(runner, yadm_y, paths, ds1, sshperms, gpgperms):
         assert not oct(private.stat().mode).endswith('00'), (
             'Path started secured')
 
-    run = runner(yadm_y('perms'))
+    run = runner(yadm_cmd('perms'), env={'HOME': paths.work})
     assert run.success
     assert run.err == ''
     assert run.out == ''

test/test_syntax.py

@@ -10,20 +10,22 @@ def test_yadm_syntax(runner, yadm):
     assert run.success
 
 
-def test_shellcheck(runner, yadm, shellcheck_version):
+def test_shellcheck(pytestconfig, runner, yadm, shellcheck_version):
     """Passes shellcheck"""
-    run = runner(command=['shellcheck', '-V'], report=False)
-    if f'version: {shellcheck_version}' not in run.out:
-        pytest.skip('Unsupported shellcheck version')
+    if not pytestconfig.getoption("--force-linters"):
+        run = runner(command=['shellcheck', '-V'], report=False)
+        if f'version: {shellcheck_version}' not in run.out:
+            pytest.skip('Unsupported shellcheck version')
     run = runner(command=['shellcheck', '-s', 'bash', yadm])
     assert run.success
 
 
-def test_pylint(runner, pylint_version):
+def test_pylint(pytestconfig, runner, pylint_version):
     """Passes pylint"""
-    run = runner(command=['pylint', '--version'], report=False)
-    if f'pylint {pylint_version}' not in run.out:
-        pytest.skip('Unsupported pylint version')
+    if not pytestconfig.getoption("--force-linters"):
+        run = runner(command=['pylint', '--version'], report=False)
+        if f'pylint {pylint_version}' not in run.out:
+            pytest.skip('Unsupported pylint version')
     pyfiles = list()
     for tfile in os.listdir('test'):
         if tfile.endswith('.py'):
@@ -32,21 +34,32 @@ def test_pylint(runner, pylint_version):
     assert run.success
 
 
-def test_flake8(runner, flake8_version):
+def test_flake8(pytestconfig, runner, flake8_version):
     """Passes flake8"""
-    run = runner(command=['flake8', '--version'], report=False)
-    if not run.out.startswith(flake8_version):
-        pytest.skip('Unsupported flake8 version')
+    if not pytestconfig.getoption("--force-linters"):
+        run = runner(command=['flake8', '--version'], report=False)
+        if not run.out.startswith(flake8_version):
+            pytest.skip('Unsupported flake8 version')
     run = runner(command=['flake8', 'test'])
     assert run.success
 
 
-def test_yamllint(runner, yamllint_version):
+def test_yamllint(pytestconfig, runner, yamllint_version):
     """Passes yamllint"""
-    run = runner(command=['yamllint', '--version'], report=False)
-    if not run.out.strip().endswith(yamllint_version):
-        pytest.skip('Unsupported yamllint version')
+    if not pytestconfig.getoption("--force-linters"):
+        run = runner(command=['yamllint', '--version'], report=False)
+        if not run.out.strip().endswith(yamllint_version):
+            pytest.skip('Unsupported yamllint version')
     run = runner(
         command=['yamllint', '-s', '$(find . -name \\*.yml)'],
         shell=True)
     assert run.success
+
+
+def test_man(runner):
+    """Check for warnings from man"""
+    run = runner(
+        command=['man', '--warnings', './yadm.1'])
+    assert run.success
+    assert run.err == ''
+    assert 'yadm - Yet Another Dotfiles Manager' in run.out

test/test_unit_configure_paths.py

@@ -2,19 +2,21 @@
 
 import pytest
 
-ARCHIVE = 'files.gpg'
+ARCHIVE = 'archive'
 BOOTSTRAP = 'bootstrap'
 CONFIG = 'config'
 ENCRYPT = 'encrypt'
 HOME = '/testhome'
 REPO = 'repo.git'
 YDIR = '.config/yadm'
+YDATA = '.local/share/yadm'
 
 
 @pytest.mark.parametrize(
     'override, expect', [
         (None, {}),
-        ('-Y', {}),
+        ('-Y', {'yadm': 'YADM_DIR'}),
+        ('--yadm-data', {'data': 'YADM_DATA'}),
         ('--yadm-repo', {'repo': 'YADM_REPO', 'git': 'GIT_DIR'}),
         ('--yadm-config', {'config': 'YADM_CONFIG'}),
         ('--yadm-encrypt', {'encrypt': 'YADM_ENCRYPT'}),
@@ -23,59 +25,70 @@ YDIR = '.config/yadm'
     ], ids=[
         'default',
         'override yadm dir',
+        'override yadm data',
         'override repo',
         'override config',
         'override encrypt',
         'override archive',
         'override bootstrap',
     ])
-def test_config(runner, paths, override, expect):
+@pytest.mark.parametrize(
+    'path', ['.', './override', 'override', '.override', '/override'], ids=[
+        'cwd', './relative', 'relative', 'hidden relative', 'absolute'
+    ])
+def test_config(runner, paths, override, expect, path):
     """Test configure_paths"""
-    opath = 'override'
-    matches = match_map()
-    args = []
+    if path.startswith('/'):
+        expected_path = path
+    else:
+        expected_path = str(paths.root.join(path))
+
+    args = [override, path] if override else []
+
     if override == '-Y':
-        matches = match_map('/' + opath)
+        matches = match_map(expected_path)
+    elif override == '--yadm-data':
+        matches = match_map(None, expected_path)
+    else:
+        matches = match_map()
 
-    if override:
-        args = [override, '/' + opath]
-        for ekey in expect.keys():
-            matches[ekey] = f'{expect[ekey]}="/{opath}"'
-        run_test(
-            runner, paths,
-            [override, opath],
-            ['must specify a fully qualified'], 1)
+    for ekey in expect.keys():
+        matches[ekey] = f'{expect[ekey]}="{expected_path}"'
 
-    run_test(runner, paths, args, matches.values(), 0)
+    run_test(runner, paths, args, matches.values(), cwd=str(paths.root))
 
 
-def match_map(yadm_dir=None):
+def match_map(yadm_dir=None, yadm_data=None):
     """Create a dictionary of matches, relative to yadm_dir"""
     if not yadm_dir:
         yadm_dir = '/'.join([HOME, YDIR])
+    if not yadm_data:
+        yadm_data = '/'.join([HOME, YDATA])
     return {
         'yadm': f'YADM_DIR="{yadm_dir}"',
-        'repo': f'YADM_REPO="{yadm_dir}/{REPO}"',
+        'repo': f'YADM_REPO="{yadm_data}/{REPO}"',
         'config': f'YADM_CONFIG="{yadm_dir}/{CONFIG}"',
         'encrypt': f'YADM_ENCRYPT="{yadm_dir}/{ENCRYPT}"',
-        'archive': f'YADM_ARCHIVE="{yadm_dir}/{ARCHIVE}"',
+        'archive': f'YADM_ARCHIVE="{yadm_data}/{ARCHIVE}"',
         'bootstrap': f'YADM_BOOTSTRAP="{yadm_dir}/{BOOTSTRAP}"',
-        'git': f'GIT_DIR="{yadm_dir}/{REPO}"',
+        'git': f'GIT_DIR="{yadm_data}/{REPO}"',
         }
 
 
-def run_test(runner, paths, args, expected_matches, expected_code=0):
+def run_test(runner, paths, args, expected_matches, cwd=None):
     """Run proces global args, and run configure_paths"""
     argstring = ' '.join(['"'+a+'"' for a in args])
     script = f"""
         YADM_TEST=1 HOME="{HOME}" source {paths.pgm}
         process_global_args {argstring}
-        HOME="{HOME}" set_yadm_dir
+        XDG_CONFIG_HOME=
+        XDG_DATA_HOME=
+        HOME="{HOME}" set_yadm_dirs
         configure_paths
         declare -p | grep -E '(YADM|GIT)_'
     """
-    run = runner(command=['bash'], inp=script)
-    assert run.code == expected_code
+    run = runner(command=['bash'], inp=script, cwd=cwd)
+    assert run.success
     assert run.err == ''
     for match in expected_matches:
         assert match in run.out

test/test_unit_copy_perms.py

@@ -0,0 +1,53 @@
+"""Unit tests: copy_perms"""
+import os
+import pytest
+
+OCTAL = '7654'
+NON_OCTAL = '9876'
+
+
+@pytest.mark.parametrize(
+    'stat_broken', [True, False], ids=['normal', 'stat broken'])
+def test_copy_perms(runner, yadm, tmpdir, stat_broken):
+    """Test function copy_perms"""
+    src_mode = 0o754
+    dst_mode = 0o644
+    source = tmpdir.join('source')
+    source.write('test', ensure=True)
+    source.chmod(src_mode)
+
+    dest = tmpdir.join('dest')
+    dest.write('test', ensure=True)
+    dest.chmod(dst_mode)
+
+    override_stat = ''
+    if stat_broken:
+        override_stat = 'function stat() { echo broken; }'
+    script = f"""
+        YADM_TEST=1 source {yadm}
+        {override_stat}
+        copy_perms "{source}" "{dest}"
+    """
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.err == ''
+    assert run.out == ''
+    expected = dst_mode if stat_broken else src_mode
+    assert oct(os.stat(dest).st_mode)[-3:] == oct(expected)[-3:]
+
+
+@pytest.mark.parametrize(
+    'stat_output', [OCTAL, NON_OCTAL], ids=['octal', 'non-octal'])
+def test_get_mode(runner, yadm, stat_output):
+    """Test function get_mode"""
+    script = f"""
+        YADM_TEST=1 source {yadm}
+        function stat() {{ echo {stat_output}; }}
+        mode=$(get_mode abc)
+        echo "MODE:$mode"
+    """
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.err == ''
+    expected = OCTAL if stat_output == OCTAL else ""
+    assert f'MODE:{expected}\n' in run.out

test/test_unit_encryption.py

@@ -0,0 +1,135 @@
+"""Unit tests: encryption functions"""
+
+import pytest
+
+
+@pytest.mark.parametrize('condition', ['default', 'override'])
+def test_get_cipher(runner, paths, condition):
+    """Test _get_cipher()"""
+
+    if condition == 'override':
+        paths.config.write('[yadm]\n\tcipher = override-cipher')
+
+    script = f"""
+        YADM_TEST=1 source {paths.pgm}
+        YADM_DIR="{paths.yadm}"
+        set_yadm_dirs
+        configure_paths
+        _get_cipher test-archive
+        echo "output_archive:$output_archive"
+        echo "yadm_cipher:$yadm_cipher"
+    """
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.err == ''
+    assert 'output_archive:test-archive' in run.out
+    if condition == 'override':
+        assert 'yadm_cipher:override-cipher' in run.out
+    else:
+        assert 'yadm_cipher:gpg' in run.out
+
+
+@pytest.mark.parametrize('cipher', ['gpg', 'openssl', 'bad'])
+@pytest.mark.parametrize('mode', ['_encrypt_to', '_decrypt_from'])
+def test_encrypt_decrypt(runner, paths, cipher, mode):
+    """Test _encrypt_to() & _decrypt_from"""
+
+    script = f"""
+        YADM_TEST=1 source {paths.pgm}
+        YADM_DIR="{paths.yadm}"
+        set_yadm_dirs
+        configure_paths
+        function mock_openssl() {{ echo openssl $*; }}
+        function mock_gpg() {{ echo gpg $*; }}
+        function _get_cipher() {{
+            output_archive="$1"
+            yadm_cipher="{cipher}"
+        }}
+        OPENSSL_PROGRAM=mock_openssl
+        GPG_PROGRAM=mock_gpg
+        {mode} {paths.archive}
+    """
+    run = runner(command=['bash'], inp=script)
+
+    if cipher != 'bad':
+        assert run.success
+        assert run.out.startswith(cipher)
+        assert str(paths.archive) in run.out
+        assert run.err == ''
+    else:
+        assert run.failure
+        assert 'Unknown cipher' in run.err
+
+
+@pytest.mark.parametrize('condition', ['default', 'override'])
+def test_get_openssl_ciphername(runner, paths, condition):
+    """Test _get_openssl_ciphername()"""
+
+    if condition == 'override':
+        paths.config.write('[yadm]\n\topenssl-ciphername = override-cipher')
+
+    script = f"""
+        YADM_TEST=1 source {paths.pgm}
+        YADM_DIR="{paths.yadm}"
+        set_yadm_dirs
+        configure_paths
+        result=$(_get_openssl_ciphername)
+        echo "result:$result"
+    """
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.err == ''
+    if condition == 'override':
+        assert run.out.strip() == 'result:override-cipher'
+    else:
+        assert run.out.strip() == 'result:aes-256-cbc'
+
+
+@pytest.mark.parametrize('condition', ['old', 'not-old'])
+def test_set_openssl_options(runner, paths, condition):
+    """Test _set_openssl_options()"""
+
+    if condition == 'old':
+        paths.config.write('[yadm]\n\topenssl-old = true')
+
+    script = f"""
+        YADM_TEST=1 source {paths.pgm}
+        YADM_DIR="{paths.yadm}"
+        set_yadm_dirs
+        configure_paths
+        function _get_openssl_ciphername() {{ echo "testcipher"; }}
+        _set_openssl_options
+        echo "result:${{OPENSSL_OPTS[@]}}"
+    """
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.err == ''
+    if condition == 'old':
+        assert '-testcipher -salt -md md5' in run.out
+    else:
+        assert '-testcipher -salt -pbkdf2 -iter 100000 -md sha512' in run.out
+
+
+@pytest.mark.parametrize('recipient', ['ASK', 'present', ''])
+def test_set_gpg_options(runner, paths, recipient):
+    """Test _set_gpg_options()"""
+
+    paths.config.write(f'[yadm]\n\tgpg-recipient = {recipient}')
+
+    script = f"""
+        YADM_TEST=1 source {paths.pgm}
+        YADM_DIR="{paths.yadm}"
+        set_yadm_dirs
+        configure_paths
+        _set_gpg_options
+        echo "result:${{GPG_OPTS[@]}}"
+    """
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.err == ''
+    if recipient == 'ASK':
+        assert run.out.strip() == 'result:--no-default-recipient -e'
+    elif recipient != '':
+        assert run.out.strip() == f'result:-e -r {recipient}'
+    else:
+        assert run.out.strip() == 'result:-c'

test/test_unit_is_valid_branch_name.py

@@ -1,40 +0,0 @@
-"""Unit tests: is_valid_branch_name"""
-import pytest
-
-# Git branches do not allow:
-#  * path component that begins with "."
-#  * double dot
-#  * "~", "^", ":", "\", space
-#  * end with a "/"
-#  * end with ".lock"
-
-
-@pytest.mark.parametrize(
-    'branch, expected', [
-        ('master', 'valid'),
-        ('path/branch', 'valid'),
-        ('path/.branch', 'invalid'),
-        ('path..branch', 'invalid'),
-        ('path~branch', 'invalid'),
-        ('path^branch', 'invalid'),
-        ('path:branch', 'invalid'),
-        ('path\\branch', 'invalid'),
-        ('path branch', 'invalid'),
-        ('path/branch/', 'invalid'),
-        ('branch.lock', 'invalid'),
-    ])
-def test_is_valid_branch_name(runner, yadm, branch, expected):
-    """Test function is_valid_branch_name()"""
-
-    script = f"""
-        YADM_TEST=1 source {yadm}
-        if is_valid_branch_name "{branch}"; then
-            echo valid
-        else
-            echo invalid
-        fi
-    """
-    run = runner(command=['bash'], inp=script)
-    assert run.success
-    assert run.err == ''
-    assert run.out.strip() == expected

test/test_unit_issue_legacy_path_warning.py

@@ -6,34 +6,36 @@ import pytest
     'legacy_path', [
         None,
         'repo.git',
-        'config',
-        'encrypt',
         'files.gpg',
-        'bootstrap',
-        'hooks/pre_command',
-        'hooks/post_command',
         ],
     )
+@pytest.mark.parametrize(
+    'override', [True, False], ids=['override', 'no-override'])
 @pytest.mark.parametrize(
     'upgrade', [True, False], ids=['upgrade', 'no-upgrade'])
-def test_legacy_warning(tmpdir, runner, yadm, upgrade, legacy_path):
+def test_legacy_warning(tmpdir, runner, yadm, upgrade, override, legacy_path):
     """Use issue_legacy_path_warning"""
     home = tmpdir.mkdir('home')
 
     if legacy_path:
-        home.mkdir(f'.yadm').ensure(legacy_path)
+        home.ensure(f'.config/yadm/{str(legacy_path)}')
 
+    override = 'YADM_OVERRIDE_REPO=override' if override else ''
     main_args = 'MAIN_ARGS=("upgrade")' if upgrade else ''
     script = f"""
+        XDG_CONFIG_HOME=
+        XDG_DATA_HOME=
         HOME={home}
         YADM_TEST=1 source {yadm}
         {main_args}
+        {override}
+        set_yadm_dirs
         issue_legacy_path_warning
     """
     run = runner(command=['bash'], inp=script)
     assert run.success
-    assert run.err == ''
-    if legacy_path and not upgrade:
-        assert 'Legacy configuration paths have been detected' in run.out
+    assert run.out == ''
+    if legacy_path and (not upgrade) and (not override):
+        assert 'Legacy paths have been detected' in run.err
     else:
-        assert run.out.rstrip() == ''
+        assert 'Legacy paths have been detected' not in run.err

test/test_unit_parse_encrypt.py

@@ -46,12 +46,8 @@ def test_empty(runner, paths, encrypt):
     assert 'EIF_COUNT:0' in run.out, 'EIF should be empty'
 
 
-@pytest.mark.usefixtures('ds1_repo_copy')
-def test_file_parse_encrypt(runner, paths):
-    """Test parse_encrypt
-
-    Test an array of supported features of the encrypt configuration.
-    """
+def create_test_encrypt_data(paths):
+    """Generate test data for testing encrypt"""
 
     edata = ''
     expected = set()
@@ -126,6 +122,30 @@ def test_file_parse_encrypt(runner, paths):
     expected.add('ex ex/file4')
     expected.add('ex ex/file6.text')
 
+    # double star
+    edata += 'doublestar/**/file*\n'
+    edata += '!**/file3\n'
+    paths.work.join('doublestar/a/b/file1').write('', ensure=True)
+    paths.work.join('doublestar/c/d/file2').write('', ensure=True)
+    paths.work.join('doublestar/e/f/file3').write('', ensure=True)
+    paths.work.join('doublestar/g/h/nomatch').write('', ensure=True)
+    expected.add('doublestar/a/b/file1')
+    expected.add('doublestar/c/d/file2')
+    # doublestar/e/f/file3 is excluded
+
+    return edata, expected
+
+
+@pytest.mark.usefixtures('ds1_repo_copy')
+def test_file_parse_encrypt(runner, paths):
+    """Test parse_encrypt
+
+    Test an array of supported features of the encrypt configuration.
+    """
+
+    # generate test data & expectations
+    edata, expected = create_test_encrypt_data(paths)
+
     # write encrypt file
     print(f'ENCRYPT:\n---\n{edata}---\n')
     paths.encrypt.write(edata)
@@ -166,6 +186,8 @@ def run_parse_encrypt(
         export YADM_ENCRYPT
         GIT_DIR={paths.repo}
         export GIT_DIR
+        YADM_WORK={paths.work}
+        export YADM_WORK
         {parse_cmd}
         export ENCRYPT_INCLUDE_FILES
         export PARSE_ENCRYPT_SHORT

test/test_unit_private_dirs.py

@@ -0,0 +1,34 @@
+"""Unit tests: private_dirs"""
+import pytest
+
+
+@pytest.mark.parametrize(
+    'gnupghome',
+    [True, False],
+    ids=['gnupghome-set', 'gnupghome-unset'],
+)
+@pytest.mark.parametrize('param', ['all', 'gnupg'])
+def test_relative_path(runner, paths, gnupghome, param):
+    """Test translate_to_relative"""
+
+    alt_gnupghome = 'alt/gnupghome'
+    env_gnupghome = paths.work.join(alt_gnupghome)
+
+    script = f"""
+        YADM_TEST=1 source {paths.pgm}
+        YADM_WORK={paths.work}
+        private_dirs {param}
+    """
+
+    env = {}
+    if gnupghome:
+        env['GNUPGHOME'] = env_gnupghome
+
+    expected = alt_gnupghome if gnupghome else '.gnupg'
+    if param == 'all':
+        expected = f'.ssh {expected}'
+
+    run = runner(command=['bash'], inp=script, env=env)
+    assert run.success
+    assert run.err == ''
+    assert run.out.strip() == expected

test/test_unit_query_distro.py

@@ -1,26 +1,32 @@
 """Unit tests: query_distro"""
+import pytest
 
 
-def test_lsb_release_present(runner, yadm, tst_distro):
+@pytest.mark.parametrize(
+    'condition', ['lsb_release', 'os-release', 'os-release-quotes', 'missing'])
+def test_query_distro(runner, yadm, tst_distro, tmp_path, condition):
     """Match lsb_release -si when present"""
+    test_release = 'testrelease'
+    lsb_release = ''
+    os_release = tmp_path.joinpath('os-release')
+    if 'os-release' in condition:
+        quotes = '"' if 'quotes' in condition else ''
+        os_release.write_text(
+            f"testing\nID={quotes}{test_release}{quotes}\nrelease")
+    if condition != 'lsb_release':
+        lsb_release = 'LSB_RELEASE_PROGRAM="missing_lsb_release"'
     script = f"""
         YADM_TEST=1 source {yadm}
+        {lsb_release}
+        OS_RELEASE="{os_release}"
         query_distro
     """
     run = runner(command=['bash'], inp=script)
     assert run.success
     assert run.err == ''
-    assert run.out.rstrip() == tst_distro
-
-
-def test_lsb_release_missing(runner, yadm):
-    """Empty value when missing"""
-    script = f"""
-        YADM_TEST=1 source {yadm}
-        LSB_RELEASE_PROGRAM="missing_lsb_release"
-        query_distro
-    """
-    run = runner(command=['bash'], inp=script)
-    assert run.success
-    assert run.err == ''
-    assert run.out.rstrip() == ''
+    if condition == 'lsb_release':
+        assert run.out.rstrip() == tst_distro
+    elif 'os-release' in condition:
+        assert run.out.rstrip() == test_release
+    else:
+        assert run.out.rstrip() == ''

test/test_unit_query_distro_family.py

@@ -0,0 +1,26 @@
+"""Unit tests: query_distro_family"""
+import pytest
+
+
+@pytest.mark.parametrize(
+    'condition', ['os-release', 'os-release-quotes', 'missing'])
+def test_query_distro_family(runner, yadm, tmp_path, condition):
+    """Match ID_LIKE when present"""
+    test_family = 'testfamily'
+    os_release = tmp_path.joinpath('os-release')
+    if 'os-release' in condition:
+        quotes = '"' if 'quotes' in condition else ''
+        os_release.write_text(
+            f"testing\nID_LIKE={quotes}{test_family}{quotes}\nfamily")
+    script = f"""
+        YADM_TEST=1 source {yadm}
+        OS_RELEASE="{os_release}"
+        query_distro_family
+    """
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.err == ''
+    if 'os-release' in condition:
+        assert run.out.rstrip() == test_family
+    else:
+        assert run.out.rstrip() == ''

test/test_unit_record_score.py

@@ -8,16 +8,16 @@ INIT_VARS = """
     local_host=testhost
     local_user=testuser
     alt_scores=()
-    alt_filenames=()
     alt_targets=()
+    alt_sources=()
     alt_template_cmds=()
 """
 
 REPORT_RESULTS = """
     echo "SIZE:${#alt_scores[@]}"
     echo "SCORES:${alt_scores[@]}"
-    echo "FILENAMES:${alt_filenames[@]}"
     echo "TARGETS:${alt_targets[@]}"
+    echo "SOURCES:${alt_sources[@]}"
 """
 
 
@@ -27,7 +27,7 @@ def test_dont_record_zeros(runner, yadm):
     script = f"""
         YADM_TEST=1 source {yadm}
         {INIT_VARS}
-        record_score "0" "testfile" "testtarget"
+        record_score "0" "testtgt" "testsrc"
         {REPORT_RESULTS}
     """
     run = runner(command=['bash'], inp=script)
@@ -35,8 +35,8 @@ def test_dont_record_zeros(runner, yadm):
     assert run.err == ''
     assert 'SIZE:0\n' in run.out
     assert 'SCORES:\n' in run.out
-    assert 'FILENAMES:\n' in run.out
     assert 'TARGETS:\n' in run.out
+    assert 'SOURCES:\n' in run.out
 
 
 def test_new_scores(runner, yadm):
@@ -45,9 +45,9 @@ def test_new_scores(runner, yadm):
     script = f"""
         YADM_TEST=1 source {yadm}
         {INIT_VARS}
-        record_score "1" "file_one"   "targ_one"
-        record_score "2" "file_two"   "targ_two"
-        record_score "4" "file_three" "targ_three"
+        record_score "1" "tgt_one"   "src_one"
+        record_score "2" "tgt_two"   "src_two"
+        record_score "4" "tgt_three" "src_three"
         {REPORT_RESULTS}
     """
     run = runner(command=['bash'], inp=script)
@@ -55,8 +55,8 @@ def test_new_scores(runner, yadm):
     assert run.err == ''
     assert 'SIZE:3\n' in run.out
     assert 'SCORES:1 2 4\n' in run.out
-    assert 'FILENAMES:file_one file_two file_three\n' in run.out
-    assert 'TARGETS:targ_one targ_two targ_three\n' in run.out
+    assert 'TARGETS:tgt_one tgt_two tgt_three\n' in run.out
+    assert 'SOURCES:src_one src_two src_three\n' in run.out
 
 
 @pytest.mark.parametrize('difference', ['lower', 'equal', 'higher'])
@@ -64,7 +64,7 @@ def test_existing_scores(runner, yadm, difference):
     """Test existing scores"""
 
     expected_score = '2'
-    expected_target = 'existing_target'
+    expected_src = 'existing_src'
     if difference == 'lower':
         score = '1'
     elif difference == 'equal':
@@ -72,15 +72,15 @@ def test_existing_scores(runner, yadm, difference):
     else:
         score = '4'
         expected_score = '4'
-        expected_target = 'new_target'
+        expected_src = 'new_src'
 
     script = f"""
         YADM_TEST=1 source {yadm}
         {INIT_VARS}
         alt_scores=(2)
-        alt_filenames=("testfile")
-        alt_targets=("existing_target")
-        record_score "{score}" "testfile" "new_target"
+        alt_targets=("testtgt")
+        alt_sources=("existing_src")
+        record_score "{score}" "testtgt" "new_src"
         {REPORT_RESULTS}
     """
     run = runner(command=['bash'], inp=script)
@@ -88,21 +88,21 @@ def test_existing_scores(runner, yadm, difference):
     assert run.err == ''
     assert 'SIZE:1\n' in run.out
     assert f'SCORES:{expected_score}\n' in run.out
-    assert 'FILENAMES:testfile\n' in run.out
-    assert f'TARGETS:{expected_target}\n' in run.out
+    assert 'TARGETS:testtgt\n' in run.out
+    assert f'SOURCES:{expected_src}\n' in run.out
 
 
 def test_existing_template(runner, yadm):
-    """Record nothing if a template command is registered for this file"""
+    """Record nothing if a template command is registered for this target"""
 
     script = f"""
         YADM_TEST=1 source {yadm}
         {INIT_VARS}
         alt_scores=(1)
-        alt_filenames=("testfile")
-        alt_targets=()
+        alt_targets=("testtgt")
+        alt_sources=()
         alt_template_cmds=("existing_template")
-        record_score "2" "testfile" "new_target"
+        record_score "2" "testtgt" "new_src"
         {REPORT_RESULTS}
     """
     run = runner(command=['bash'], inp=script)
@@ -110,5 +110,32 @@ def test_existing_template(runner, yadm):
     assert run.err == ''
     assert 'SIZE:1\n' in run.out
     assert 'SCORES:1\n' in run.out
-    assert 'FILENAMES:testfile\n' in run.out
-    assert 'TARGETS:\n' in run.out
+    assert 'TARGETS:testtgt\n' in run.out
+    assert 'SOURCES:\n' in run.out
+
+
+def test_config_first(runner, yadm):
+    """Verify YADM_CONFIG is always processed first"""
+
+    config = 'yadm_config_file'
+    script = f"""
+        YADM_TEST=1 source {yadm}
+        {INIT_VARS}
+        YADM_CONFIG={config}
+        record_score "1" "tgt_before" "src_before"
+        record_template "tgt_tmp" "cmd_tmp" "src_tmp"
+        record_score "2" "{config}"   "src_config"
+        record_score "3" "tgt_after"  "src_after"
+        {REPORT_RESULTS}
+        echo "CMD_VALUE:${{alt_template_cmds[@]}}"
+        echo "CMD_INDEX:${{!alt_template_cmds[@]}}"
+    """
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.err == ''
+    assert 'SIZE:3\n' in run.out
+    assert 'SCORES:2 1 3\n' in run.out
+    assert f'TARGETS:{config} tgt_before tgt_tmp tgt_after\n' in run.out
+    assert 'SOURCES:src_config src_before src_tmp src_after\n' in run.out
+    assert 'CMD_VALUE:cmd_tmp\n' in run.out
+    assert 'CMD_INDEX:2\n' in run.out

test/test_unit_record_template.py

@@ -1,16 +1,16 @@
 """Unit tests: record_template"""
 
 INIT_VARS = """
-    alt_filenames=()
-    alt_template_cmds=()
     alt_targets=()
+    alt_template_cmds=()
+    alt_sources=()
 """
 
 REPORT_RESULTS = """
-    echo "SIZE:${#alt_filenames[@]}"
-    echo "FILENAMES:${alt_filenames[@]}"
+    echo "SIZE:${#alt_targets[@]}"
+    echo "TARGETS:${alt_targets[@]}"
     echo "CMDS:${alt_template_cmds[@]}"
-    echo "TARGS:${alt_targets[@]}"
+    echo "SOURCES:${alt_sources[@]}"
 """
 
 
@@ -20,18 +20,18 @@ def test_new_template(runner, yadm):
     script = f"""
         YADM_TEST=1 source {yadm}
         {INIT_VARS}
-        record_template "file_one"   "cmd_one"   "targ_one"
-        record_template "file_two"   "cmd_two"   "targ_two"
-        record_template "file_three" "cmd_three" "targ_three"
+        record_template "tgt_one"   "cmd_one"   "src_one"
+        record_template "tgt_two"   "cmd_two"   "src_two"
+        record_template "tgt_three" "cmd_three" "src_three"
         {REPORT_RESULTS}
     """
     run = runner(command=['bash'], inp=script)
     assert run.success
     assert run.err == ''
     assert 'SIZE:3\n' in run.out
-    assert 'FILENAMES:file_one file_two file_three\n' in run.out
+    assert 'TARGETS:tgt_one tgt_two tgt_three\n' in run.out
     assert 'CMDS:cmd_one cmd_two cmd_three\n' in run.out
-    assert 'TARGS:targ_one targ_two targ_three\n' in run.out
+    assert 'SOURCES:src_one src_two src_three\n' in run.out
 
 
 def test_existing_template(runner, yadm):
@@ -40,16 +40,16 @@ def test_existing_template(runner, yadm):
     script = f"""
         YADM_TEST=1 source {yadm}
         {INIT_VARS}
-        alt_filenames=("testfile")
+        alt_targets=("testtgt")
         alt_template_cmds=("existing_cmd")
-        alt_targets=("existing_targ")
-        record_template "testfile" "new_cmd" "new_targ"
+        alt_sources=("existing_src")
+        record_template "testtgt" "new_cmd" "new_src"
         {REPORT_RESULTS}
     """
     run = runner(command=['bash'], inp=script)
     assert run.success
     assert run.err == ''
     assert 'SIZE:1\n' in run.out
-    assert 'FILENAMES:testfile\n' in run.out
+    assert 'TARGETS:testtgt\n' in run.out
     assert 'CMDS:new_cmd\n' in run.out
-    assert 'TARGS:new_targ\n' in run.out
+    assert 'SOURCES:new_src\n' in run.out

test/test_unit_relative_path.py

@@ -0,0 +1,31 @@
+"""Unit tests: relative_path"""
+import pytest
+
+
+@pytest.mark.parametrize(
+    'base,full_path,expected',
+    [
+        ("/A/B/C", "/A", "../.."),
+        ("/A/B/C", "/A/B", ".."),
+        ("/A/B/C", "/A/B/C", ""),
+        ("/A/B/C", "/A/B/C/D", "D"),
+        ("/A/B/C", "/A/B/C/D/E", "D/E"),
+        ("/A/B/C", "/A/B/D", "../D"),
+        ("/A/B/C", "/A/B/D/E", "../D/E"),
+        ("/A/B/C", "/A/D", "../../D"),
+        ("/A/B/C", "/A/D/E", "../../D/E"),
+        ("/A/B/C", "/D/E/F", "../../../D/E/F"),
+    ],
+)
+def test_relative_path(runner, paths, base, full_path, expected):
+    """Test translate_to_relative"""
+
+    script = f"""
+        YADM_TEST=1 source {paths.pgm}
+        relative_path "{base}" "{full_path}"
+    """
+
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.err == ''
+    assert run.out.strip() == expected

test/test_unit_report_invalid_alts.py

@@ -0,0 +1,30 @@
+"""Unit tests: report_invalid_alts"""
+import pytest
+
+
+@pytest.mark.parametrize('valid', [True, False], ids=['valid', 'no_valid'])
+@pytest.mark.parametrize('previous', [True, False], ids=['prev', 'no_prev'])
+def test_report_invalid_alts(runner, yadm, valid, previous):
+    """Use report_invalid_alts"""
+
+    lwi = ''
+    alts = 'INVALID_ALT=()'
+    if previous:
+        lwi = 'LEGACY_WARNING_ISSUED=1'
+    if not valid:
+        alts = 'INVALID_ALT=("file##invalid")'
+
+    script = f"""
+        YADM_TEST=1 source {yadm}
+        {lwi}
+        {alts}
+        report_invalid_alts
+    """
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.out == ''
+    if not valid and not previous:
+        assert 'WARNING' in run.err
+        assert 'file##invalid' in run.err
+    else:
+        assert run.err == ''

test/test_unit_score_file.py

@@ -6,25 +6,33 @@ CONDITION = {
         'labels': ['default'],
         'modifier': 0,
         },
+    'arch': {
+        'labels': ['a', 'arch'],
+        'modifier': 1,
+        },
     'system': {
         'labels': ['o', 'os'],
-        'modifier': 1,
+        'modifier': 2,
         },
     'distro': {
         'labels': ['d', 'distro'],
-        'modifier': 2,
+        'modifier': 4,
+        },
+    'distro_family': {
+        'labels': ['f', 'distro_family'],
+        'modifier': 8,
         },
     'class': {
         'labels': ['c', 'class'],
-        'modifier': 4,
+        'modifier': 16,
         },
     'hostname': {
         'labels': ['h', 'hostname'],
-        'modifier': 8,
+        'modifier': 32,
         },
     'user': {
         'labels': ['u', 'user'],
-        'modifier': 16,
+        'modifier': 64,
         },
     }
 TEMPLATE_LABELS = ['t', 'template', 'yadm']
@@ -44,6 +52,12 @@ def calculate_score(filename):
             label, value = condition.split('.', 1)
         if label in CONDITION['default']['labels']:
             score += 1000
+        elif label in CONDITION['arch']['labels']:
+            if value == 'testarch':
+                score += 1000 + CONDITION['arch']['modifier']
+            else:
+                score = 0
+                break
         elif label in CONDITION['system']['labels']:
             if value == 'testsystem':
                 score += 1000 + CONDITION['system']['modifier']
@@ -82,6 +96,8 @@ def calculate_score(filename):
 
 @pytest.mark.parametrize(
     'default', ['default', None], ids=['default', 'no-default'])
+@pytest.mark.parametrize(
+    'arch', ['arch', None], ids=['arch', 'no-arch'])
 @pytest.mark.parametrize(
     'system', ['system', None], ids=['system', 'no-system'])
 @pytest.mark.parametrize(
@@ -93,10 +109,11 @@ def calculate_score(filename):
 @pytest.mark.parametrize(
     'user', ['user', None], ids=['user', 'no-user'])
 def test_score_values(
-        runner, yadm, default, system, distro, cla, host, user):
+        runner, yadm, default, arch, system, distro, cla, host, user):
     """Test score results"""
     # pylint: disable=too-many-branches
     local_class = 'testclass'
+    local_arch = 'testarch'
     local_system = 'testsystem'
     local_distro = 'testdistro'
     local_host = 'testhost'
@@ -111,6 +128,18 @@ def test_score_values(
                     newfile += ','
                 newfile += label
                 filenames[newfile] = calculate_score(newfile)
+    if arch:
+        for filename in list(filenames):
+            for match in [True, False]:
+                for label in CONDITION[arch]['labels']:
+                    newfile = filename
+                    if not newfile.endswith('##'):
+                        newfile += ','
+                    newfile += '.'.join([
+                        label,
+                        local_arch if match else 'badarch'
+                        ])
+                    filenames[newfile] = calculate_score(newfile)
     if system:
         for filename in list(filenames):
             for match in [True, False]:
@@ -176,6 +205,8 @@ def test_score_values(
         YADM_TEST=1 source {yadm}
         score=0
         local_class={local_class}
+        local_classes=({local_class})
+        local_arch={local_arch}
         local_system={local_system}
         local_distro={local_distro}
         local_host={local_host}
@@ -196,9 +227,32 @@ def test_score_values(
     assert run.out == expected
 
 
+@pytest.mark.parametrize('ext', [None, 'e', 'extension'])
+def test_extensions(runner, yadm, ext):
+    """Verify extensions do not effect scores"""
+    local_user = 'testuser'
+    filename = f'filename##u.{local_user}'
+    if ext:
+        filename += f',{ext}.xyz'
+    expected = ''
+    script = f"""
+        YADM_TEST=1 source {yadm}
+        score=0
+        local_user={local_user}
+        score_file "{filename}"
+        echo "$score"
+    """
+    expected = f'{1000 + CONDITION["user"]["modifier"]}\n'
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.err == ''
+    assert run.out == expected
+
+
 def test_score_values_templates(runner, yadm):
     """Test score results"""
     local_class = 'testclass'
+    local_arch = 'arch'
     local_system = 'testsystem'
     local_distro = 'testdistro'
     local_host = 'testhost'
@@ -217,6 +271,7 @@ def test_score_values_templates(runner, yadm):
         YADM_TEST=1 source {yadm}
         score=0
         local_class={local_class}
+        local_arch={local_arch}
         local_system={local_system}
         local_distro={local_distro}
         local_host={local_host}
@@ -260,3 +315,37 @@ def test_template_recording(runner, yadm, cmd_generated):
     assert run.success
     assert run.err == ''
     assert run.out.rstrip() == expected
+
+
+def test_underscores_in_distro_and_family(runner, yadm):
+    """Test replacing spaces in distro / distro_family with underscores"""
+    local_distro = 'test distro'
+    local_distro_family = 'test family'
+    filenames = {
+            'filename##distro.test distro': 1004,
+            'filename##distro.test-distro': 0,
+            'filename##distro.test_distro': 1004,
+            'filename##distro_family.test family': 1008,
+            'filename##distro_family.test-family': 0,
+            'filename##distro_family.test_family': 1008,
+            }
+
+    script = f"""
+        YADM_TEST=1 source {yadm}
+        score=0
+        local_distro="{local_distro}"
+        local_distro_family="{local_distro_family}"
+    """
+    expected = ''
+    for filename in filenames:
+        script += f"""
+            score_file "{filename}"
+            echo "{filename}"
+            echo "$score"
+        """
+        expected += filename + '\n'
+        expected += str(filenames[filename]) + '\n'
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.err == ''
+    assert run.out == expected

test/test_unit_set_local_alt_values.py

@@ -7,6 +7,7 @@ import utils
     'override', [
         False,
         'class',
+        'arch',
         'os',
         'hostname',
         'user',
@@ -14,6 +15,7 @@ import utils
     ids=[
         'no-override',
         'override-class',
+        'override-arch',
         'override-os',
         'override-hostname',
         'override-user',
@@ -21,20 +23,24 @@ import utils
     )
 @pytest.mark.usefixtures('ds1_copy')
 def test_set_local_alt_values(
-        runner, yadm, paths, tst_sys, tst_host, tst_user, override):
+        runner, yadm, paths, tst_arch, tst_sys, tst_host, tst_user, override):
     """Use issue_legacy_path_warning"""
     script = f"""
         YADM_TEST=1 source {yadm} &&
         set_operating_system &&
-        YADM_DIR={paths.yadm} configure_paths &&
+        YADM_DIR={paths.yadm} YADM_DATA={paths.data} configure_paths &&
         set_local_alt_values
         echo "class='$local_class'"
+        echo "arch='$local_arch'"
         echo "os='$local_system'"
         echo "host='$local_host'"
         echo "user='$local_user'"
     """
 
-    if override:
+    if override == 'class':
+        utils.set_local(paths, override, 'first')
+        utils.set_local(paths, override, 'override', add=True)
+    elif override:
         utils.set_local(paths, override, 'override')
 
     run = runner(command=['bash'], inp=script)
@@ -46,32 +52,41 @@ def test_set_local_alt_values(
     else:
         assert "class=''" in run.out
 
+    if override == 'arch':
+        assert "arch='override'" in run.out
+    else:
+        assert f"arch='{tst_arch}'" in run.out
+
     if override == 'os':
         assert "os='override'" in run.out
     else:
         assert f"os='{tst_sys}'" in run.out
 
     if override == 'hostname':
-        assert f"host='override'" in run.out
+        assert "host='override'" in run.out
     else:
         assert f"host='{tst_host}'" in run.out
 
     if override == 'user':
-        assert f"user='override'" in run.out
+        assert "user='override'" in run.out
     else:
         assert f"user='{tst_user}'" in run.out
 
 
-def test_distro(runner, yadm):
-    """Assert that local_distro is set"""
+def test_distro_and_family(runner, yadm):
+    """Assert that local_distro/local_distro_family are set"""
 
     script = f"""
         YADM_TEST=1 source {yadm}
+        function config() {{ echo "$1"; }}
         function query_distro() {{ echo "testdistro"; }}
+        function query_distro_family() {{ echo "testfamily"; }}
         set_local_alt_values
         echo "distro='$local_distro'"
+        echo "distro_family='$local_distro_family'"
     """
     run = runner(command=['bash'], inp=script)
     assert run.success
     assert run.err == ''
-    assert run.out.strip() == "distro='testdistro'"
+    assert "distro='testdistro'" in run.out
+    assert "distro_family='testfamily'" in run.out

test/test_unit_set_os.py

@@ -6,12 +6,14 @@ import pytest
 @pytest.mark.parametrize(
     'proc_value, expected_os', [
         ('missing', 'uname'),
-        ('has Microsoft inside', 'WSL'),
+        ('has microsoft inside', 'WSL'),  # case insensitive
+        ('has Microsoft inside', 'WSL'),  # case insensitive
         ('another value', 'uname'),
     ], ids=[
         '/proc/version missing',
-        '/proc/version includes MS',
-        '/proc/version excludes MS',
+        '/proc/version includes ms',
+        '/proc/version excludes Ms',
+        'another value',
     ])
 def test_set_operating_system(
         runner, paths, tst_sys, proc_value, expected_os):

test/test_unit_set_yadm_dir.py

@@ -1,35 +1,48 @@
-"""Unit tests: set_yadm_dir"""
+"""Unit tests: set_yadm_dirs"""
 import pytest
 
 
 @pytest.mark.parametrize(
-    'condition',
-    ['basic', 'override', 'xdg_config_home', 'legacy'],
+    'condition', [
+        'basic',
+        'override',
+        'override_data',
+        'xdg_config_home',
+        'xdg_data_home'
+        ],
     )
-def test_set_yadm_dir(runner, yadm, condition):
-    """Test set_yadm_dir"""
+def test_set_yadm_dirs(runner, yadm, condition):
+    """Test set_yadm_dirs"""
     setup = ''
     if condition == 'override':
         setup = 'YADM_DIR=/override'
+    elif condition == 'override_data':
+        setup = 'YADM_DATA=/override'
     elif condition == 'xdg_config_home':
         setup = 'XDG_CONFIG_HOME=/xdg'
-    elif condition == 'legacy':
-        setup = 'YADM_COMPATIBILITY=1'
+    elif condition == 'xdg_data_home':
+        setup = 'XDG_DATA_HOME=/xdg'
     script = f"""
         HOME=/testhome
         YADM_TEST=1 source {yadm}
+        XDG_CONFIG_HOME=
+        XDG_DATA_HOME=
         {setup}
-        set_yadm_dir
-        echo "$YADM_DIR"
+        set_yadm_dirs
+        echo "YADM_DIR=$YADM_DIR"
+        echo "YADM_DATA=$YADM_DATA"
     """
     run = runner(command=['bash'], inp=script)
     assert run.success
     assert run.err == ''
     if condition == 'basic':
-        assert run.out.rstrip() == '/testhome/.config/yadm'
+        assert 'YADM_DIR=/testhome/.config/yadm' in run.out
+        assert 'YADM_DATA=/testhome/.local/share/yadm' in run.out
     elif condition == 'override':
-        assert run.out.rstrip() == '/override'
+        assert 'YADM_DIR=/override' in run.out
+    elif condition == 'override_data':
+        assert 'YADM_DATA=/override' in run.out
     elif condition == 'xdg_config_home':
-        assert run.out.rstrip() == '/xdg/yadm'
-    elif condition == 'legacy':
-        assert run.out.rstrip() == '/testhome/.yadm'
+        assert 'YADM_DIR=/xdg/yadm' in run.out
+    elif condition == 'xdg_data_home':
+        assert 'YADM_DATA=/xdg/yadm' in run.out

test/test_unit_template_default.py

@@ -1,18 +1,27 @@
 """Unit tests: template_default"""
+import os
+
+FILE_MODE = 0o754
 
 # these values are also testing the handling of bizarre characters
 LOCAL_CLASS = "default_Test+@-!^Class"
+LOCAL_CLASS2 = "default_Test+@-|^2nd_Class withSpace"
+LOCAL_ARCH = "default_Test+@-!^Arch"
 LOCAL_SYSTEM = "default_Test+@-!^System"
 LOCAL_HOST = "default_Test+@-!^Host"
 LOCAL_USER = "default_Test+@-!^User"
 LOCAL_DISTRO = "default_Test+@-!^Distro"
+LOCAL_DISTRO_FAMILY = "default_Test+@-!^Family"
 TEMPLATE = f'''
 start of template
-default class  = >{{{{yadm.class}}}}<
-default os     = >{{{{yadm.os}}}}<
-default host   = >{{{{yadm.hostname}}}}<
-default user   = >{{{{yadm.user}}}}<
-default distro = >{{{{yadm.distro}}}}<
+default class         = >{{{{yadm.class}}}}<
+default arch          = >{{{{yadm.arch}}}}<
+default os            = >{{{{yadm.os}}}}<
+default host          = >{{{{yadm.hostname}}}}<
+default user          = >{{{{yadm.user}}}}<
+default distro        = >{{{{yadm.distro}}}}<
+default distro_family = >{{{{yadm.distro_family}}}}<
+classes = >{{{{yadm.classes}}}}<
 {{% if yadm.class == "else1" %}}
 wrong else 1
 {{% else %}}
@@ -27,9 +36,21 @@ Multiple lines
 {{% else %}}
 Should not be included...
 {{% endif %}}
+{{% if yadm.class == "{LOCAL_CLASS2}" %}}
+Included section for second class
+{{% endif %}}
 {{% if yadm.class == "wrongclass2" %}}
 wrong class 2
 {{% endif %}}
+{{% if yadm.arch == "wrongarch1" %}}
+wrong arch 1
+{{% endif %}}
+{{% if yadm.arch == "{LOCAL_ARCH}" %}}
+Included section for arch = {{{{yadm.arch}}}} ({{{{yadm.arch}}}} repeated)
+{{% endif %}}
+{{% if yadm.arch == "wrongarch2" %}}
+wrong arch 2
+{{% endif %}}
 {{% if yadm.os == "wrongos1" %}}
 wrong os 1
 {{% endif %}}
@@ -66,46 +87,106 @@ Included section for distro = {{{{yadm.distro}}}} ({{{{yadm.distro}}}} again)
 {{% if yadm.distro == "wrongdistro2" %}}
 wrong distro 2
 {{% endif %}}
+{{% if yadm.distro_family == "wrongfamily1" %}}
+wrong family 1
+{{% endif %}}
+{{% if yadm.distro_family == "{LOCAL_DISTRO_FAMILY}" %}}
+Included section for distro_family = \
+{{{{yadm.distro_family}}}} ({{{{yadm.distro_family}}}} again)
+{{% endif %}}
+{{% if yadm.distro_family == "wrongfamily2" %}}
+wrong family 2
+{{% endif %}}
 end of template
 '''
 EXPECTED = f'''
 start of template
-default class  = >{LOCAL_CLASS}<
-default os     = >{LOCAL_SYSTEM}<
-default host   = >{LOCAL_HOST}<
-default user   = >{LOCAL_USER}<
-default distro = >{LOCAL_DISTRO}<
+default class         = >{LOCAL_CLASS}<
+default arch          = >{LOCAL_ARCH}<
+default os            = >{LOCAL_SYSTEM}<
+default host          = >{LOCAL_HOST}<
+default user          = >{LOCAL_USER}<
+default distro        = >{LOCAL_DISTRO}<
+default distro_family = >{LOCAL_DISTRO_FAMILY}<
+classes = >{LOCAL_CLASS2}
+{LOCAL_CLASS}<
 Included section from else
 Included section for class = {LOCAL_CLASS} ({LOCAL_CLASS} repeated)
 Multiple lines
+Included section for second class
+Included section for arch = {LOCAL_ARCH} ({LOCAL_ARCH} repeated)
 Included section for os = {LOCAL_SYSTEM} ({LOCAL_SYSTEM} repeated)
 Included section for host = {LOCAL_HOST} ({LOCAL_HOST} again)
 Included section for user = {LOCAL_USER} ({LOCAL_USER} repeated)
 Included section for distro = {LOCAL_DISTRO} ({LOCAL_DISTRO} again)
+Included section for distro_family = \
+{LOCAL_DISTRO_FAMILY} ({LOCAL_DISTRO_FAMILY} again)
 end of template
 '''
 
+INCLUDE_BASIC = 'basic\n'
+INCLUDE_VARIABLES = '''\
+included <{{ yadm.class }}> file
+
+empty line above
+'''
+INCLUDE_NESTED = 'no newline at the end'
+
+TEMPLATE_INCLUDE = '''\
+The first line
+{% include empty %}
+An empty file removes the line above
+{%include basic%}
+{% include "./variables.{{ yadm.os }}"  %}
+{% include dir/nested %}
+Include basic again:
+{% include basic %}
+'''
+EXPECTED_INCLUDE = f'''\
+The first line
+An empty file removes the line above
+basic
+included <{LOCAL_CLASS}> file
+
+empty line above
+no newline at the end
+Include basic again:
+basic
+'''
+
 
 def test_template_default(runner, yadm, tmpdir):
     """Test template_default"""
 
     input_file = tmpdir.join('input')
     input_file.write(TEMPLATE, ensure=True)
+    input_file.chmod(FILE_MODE)
     output_file = tmpdir.join('output')
 
+    # ensure overwrite works when file exists as read-only (there is some
+    # special processing when this is encountered because some environments do
+    # not properly overwrite read-only files)
+    output_file.write('existing')
+    output_file.chmod(0o400)
+
     script = f"""
         YADM_TEST=1 source {yadm}
+        set_awk
         local_class="{LOCAL_CLASS}"
+        local_classes=("{LOCAL_CLASS2}" "{LOCAL_CLASS}")
+        local_arch="{LOCAL_ARCH}"
         local_system="{LOCAL_SYSTEM}"
         local_host="{LOCAL_HOST}"
         local_user="{LOCAL_USER}"
         local_distro="{LOCAL_DISTRO}"
+        local_distro_family="{LOCAL_DISTRO_FAMILY}"
         template_default "{input_file}" "{output_file}"
     """
     run = runner(command=['bash'], inp=script)
     assert run.success
     assert run.err == ''
     assert output_file.read() == EXPECTED
+    assert os.stat(output_file).st_mode == os.stat(input_file).st_mode
 
 
 def test_source(runner, yadm, tmpdir):
@@ -113,13 +194,69 @@ def test_source(runner, yadm, tmpdir):
 
     input_file = tmpdir.join('input')
     input_file.write('{{yadm.source}}', ensure=True)
+    input_file.chmod(FILE_MODE)
     output_file = tmpdir.join('output')
 
     script = f"""
         YADM_TEST=1 source {yadm}
+        set_awk
         template_default "{input_file}" "{output_file}"
     """
     run = runner(command=['bash'], inp=script)
     assert run.success
     assert run.err == ''
     assert output_file.read().strip() == str(input_file)
+    assert os.stat(output_file).st_mode == os.stat(input_file).st_mode
+
+
+def test_include(runner, yadm, tmpdir):
+    """Test include"""
+
+    empty_file = tmpdir.join('empty')
+    empty_file.write('', ensure=True)
+
+    basic_file = tmpdir.join('basic')
+    basic_file.write(INCLUDE_BASIC)
+
+    variables_file = tmpdir.join(f'variables.{LOCAL_SYSTEM}')
+    variables_file.write(INCLUDE_VARIABLES)
+
+    nested_file = tmpdir.join('dir').join('nested')
+    nested_file.write(INCLUDE_NESTED, ensure=True)
+
+    input_file = tmpdir.join('input')
+    input_file.write(TEMPLATE_INCLUDE)
+    input_file.chmod(FILE_MODE)
+    output_file = tmpdir.join('output')
+
+    script = f"""
+        YADM_TEST=1 source {yadm}
+        set_awk
+        local_class="{LOCAL_CLASS}"
+        local_system="{LOCAL_SYSTEM}"
+        template_default "{input_file}" "{output_file}"
+    """
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.err == ''
+    assert output_file.read() == EXPECTED_INCLUDE
+    assert os.stat(output_file).st_mode == os.stat(input_file).st_mode
+
+
+def test_env(runner, yadm, tmpdir):
+    """Test env"""
+
+    input_file = tmpdir.join('input')
+    input_file.write('{{env.PWD}}', ensure=True)
+    input_file.chmod(FILE_MODE)
+    output_file = tmpdir.join('output')
+
+    script = f"""
+        YADM_TEST=1 source {yadm}
+        set_awk
+        template_default "{input_file}" "{output_file}"
+    """
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.err == ''
+    assert output_file.read().strip() == os.environ['PWD']

test/test_unit_template_esh.py

@@ -0,0 +1,166 @@
+"""Unit tests: template_esh"""
+import os
+
+FILE_MODE = 0o754
+
+LOCAL_CLASS = "esh_Test+@-!^Class"
+LOCAL_CLASS2 = "esh_Test+@-|^2nd_Class withSpace"
+LOCAL_ARCH = "esh_Test+@-!^Arch"
+LOCAL_SYSTEM = "esh_Test+@-!^System"
+LOCAL_HOST = "esh_Test+@-!^Host"
+LOCAL_USER = "esh_Test+@-!^User"
+LOCAL_DISTRO = "esh_Test+@-!^Distro"
+LOCAL_DISTRO_FAMILY = "esh_Test+@-!^Family"
+TEMPLATE = f'''
+start of template
+esh class         = ><%=$YADM_CLASS%><
+esh arch          = ><%=$YADM_ARCH%><
+esh os            = ><%=$YADM_OS%><
+esh host          = ><%=$YADM_HOSTNAME%><
+esh user          = ><%=$YADM_USER%><
+esh distro        = ><%=$YADM_DISTRO%><
+esh distro_family = ><%=$YADM_DISTRO_FAMILY%><
+esh classes = ><%=$YADM_CLASSES%><
+<% if [ "$YADM_CLASS" = "wrongclass1" ]; then -%>
+wrong class 1
+<% fi -%>
+<% if [ "$YADM_CLASS" = "{LOCAL_CLASS}" ]; then -%>
+Included esh section for class = <%=$YADM_CLASS%> (<%=$YADM_CLASS%> repeated)
+<% fi -%>
+<% if [ "$YADM_CLASS" = "wrongclass2" ]; then -%>
+wrong class 2
+<% fi -%>
+<% echo "$YADM_CLASSES" | while IFS='' read cls; do
+   if [ "$cls" = "{LOCAL_CLASS2}" ]; then -%>
+Included esh section for second class
+<% fi; done -%>
+<% if [ "$YADM_ARCH" = "wrongarch1" ]; then -%>
+wrong arch 1
+<% fi -%>
+<% if [ "$YADM_ARCH" = "{LOCAL_ARCH}" ]; then -%>
+Included esh section for arch = <%=$YADM_ARCH%> (<%=$YADM_ARCH%> repeated)
+<% fi -%>
+<% if [ "$YADM_ARCH" = "wrongarch2" ]; then -%>
+wrong arch 2
+<% fi -%>
+<% if [ "$YADM_OS" = "wrongos1" ]; then -%>
+wrong os 1
+<% fi -%>
+<% if [ "$YADM_OS" = "{LOCAL_SYSTEM}" ]; then -%>
+Included esh section for os = <%=$YADM_OS%> (<%=$YADM_OS%> repeated)
+<% fi -%>
+<% if [ "$YADM_OS" = "wrongos2" ]; then -%>
+wrong os 2
+<% fi -%>
+<% if [ "$YADM_HOSTNAME" = "wronghost1" ]; then -%>
+wrong host 1
+<% fi -%>
+<% if [ "$YADM_HOSTNAME" = "{LOCAL_HOST}" ]; then -%>
+Included esh section for host = <%=$YADM_HOSTNAME%> (<%=$YADM_HOSTNAME%> again)
+<% fi -%>
+<% if [ "$YADM_HOSTNAME" = "wronghost2" ]; then -%>
+wrong host 2
+<% fi -%>
+<% if [ "$YADM_USER" = "wronguser1" ]; then -%>
+wrong user 1
+<% fi -%>
+<% if [ "$YADM_USER" = "{LOCAL_USER}" ]; then -%>
+Included esh section for user = <%=$YADM_USER%> (<%=$YADM_USER%> repeated)
+<% fi -%>
+<% if [ "$YADM_USER" = "wronguser2" ]; then -%>
+wrong user 2
+<% fi -%>
+<% if [ "$YADM_DISTRO" = "wrongdistro1" ]; then -%>
+wrong distro 1
+<% fi -%>
+<% if [ "$YADM_DISTRO" = "{LOCAL_DISTRO}" ]; then -%>
+Included esh section for distro = <%=$YADM_DISTRO%> (<%=$YADM_DISTRO%> again)
+<% fi -%>
+<% if [ "$YADM_DISTRO" = "wrongdistro2" ]; then -%>
+wrong distro 2
+<% fi -%>
+<% if [ "$YADM_DISTRO_FAMILY" = "wrongfamily1" ]; then -%>
+wrong family 1
+<% fi -%>
+<% if [ "$YADM_DISTRO_FAMILY" = "{LOCAL_DISTRO_FAMILY}" ]; then -%>
+Included esh section for distro_family = \
+<%=$YADM_DISTRO_FAMILY%> (<%=$YADM_DISTRO_FAMILY%> again)
+<% fi -%>
+<% if [ "$YADM_DISTRO" = "wrongfamily2" ]; then -%>
+wrong family 2
+<% fi -%>
+end of template
+'''
+EXPECTED = f'''
+start of template
+esh class         = >{LOCAL_CLASS}<
+esh arch          = >{LOCAL_ARCH}<
+esh os            = >{LOCAL_SYSTEM}<
+esh host          = >{LOCAL_HOST}<
+esh user          = >{LOCAL_USER}<
+esh distro        = >{LOCAL_DISTRO}<
+esh distro_family = >{LOCAL_DISTRO_FAMILY}<
+esh classes = >{LOCAL_CLASS2} {LOCAL_CLASS}<
+Included esh section for class = {LOCAL_CLASS} ({LOCAL_CLASS} repeated)
+Included esh section for second class
+Included esh section for arch = {LOCAL_ARCH} ({LOCAL_ARCH} repeated)
+Included esh section for os = {LOCAL_SYSTEM} ({LOCAL_SYSTEM} repeated)
+Included esh section for host = {LOCAL_HOST} ({LOCAL_HOST} again)
+Included esh section for user = {LOCAL_USER} ({LOCAL_USER} repeated)
+Included esh section for distro = {LOCAL_DISTRO} ({LOCAL_DISTRO} again)
+Included esh section for distro_family = \
+{LOCAL_DISTRO_FAMILY} ({LOCAL_DISTRO_FAMILY} again)
+end of template
+'''
+
+
+def test_template_esh(runner, yadm, tmpdir):
+    """Test processing by esh"""
+
+    input_file = tmpdir.join('input')
+    input_file.write(TEMPLATE, ensure=True)
+    input_file.chmod(FILE_MODE)
+    output_file = tmpdir.join('output')
+
+    # ensure overwrite works when file exists as read-only (there is some
+    # special processing when this is encountered because some environments do
+    # not properly overwrite read-only files)
+    output_file.write('existing')
+    output_file.chmod(0o400)
+
+    script = f"""
+        YADM_TEST=1 source {yadm}
+        local_class="{LOCAL_CLASS}"
+        local_classes=("{LOCAL_CLASS2}" "{LOCAL_CLASS}")
+        local_arch="{LOCAL_ARCH}"
+        local_system="{LOCAL_SYSTEM}"
+        local_host="{LOCAL_HOST}"
+        local_user="{LOCAL_USER}"
+        local_distro="{LOCAL_DISTRO}"
+        local_distro_family="{LOCAL_DISTRO_FAMILY}"
+        template_esh "{input_file}" "{output_file}"
+    """
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.err == ''
+    assert output_file.read().strip() == str(EXPECTED).strip()
+    assert os.stat(output_file).st_mode == os.stat(input_file).st_mode
+
+
+def test_source(runner, yadm, tmpdir):
+    """Test YADM_SOURCE"""
+
+    input_file = tmpdir.join('input')
+    input_file.write('<%= $YADM_SOURCE %>', ensure=True)
+    input_file.chmod(FILE_MODE)
+    output_file = tmpdir.join('output')
+
+    script = f"""
+        YADM_TEST=1 source {yadm}
+        template_esh "{input_file}" "{output_file}"
+    """
+    run = runner(command=['bash'], inp=script)
+    assert run.success
+    assert run.err == ''
+    assert output_file.read().strip() == str(input_file)
+    assert os.stat(output_file).st_mode == os.stat(input_file).st_mode

test/test_unit_template_j2.py

@@ -1,32 +1,54 @@
 """Unit tests: template_j2cli & template_envtpl"""
+import os
 import pytest
 
+FILE_MODE = 0o754
+
 LOCAL_CLASS = "j2_Test+@-!^Class"
+LOCAL_CLASS2 = "j2_Test+@-|^2nd_Class withSpace"
+LOCAL_ARCH = "j2_Test+@-!^Arch"
 LOCAL_SYSTEM = "j2_Test+@-!^System"
 LOCAL_HOST = "j2_Test+@-!^Host"
 LOCAL_USER = "j2_Test+@-!^User"
 LOCAL_DISTRO = "j2_Test+@-!^Distro"
+LOCAL_DISTRO_FAMILY = "j2_Test+@-!^Family"
 TEMPLATE = f'''
 start of template
-j2 class  = >{{{{YADM_CLASS}}}}<
-j2 os     = >{{{{YADM_OS}}}}<
-j2 host   = >{{{{YADM_HOSTNAME}}}}<
-j2 user   = >{{{{YADM_USER}}}}<
-j2 distro = >{{{{YADM_DISTRO}}}}<
+j2 class         = >{{{{YADM_CLASS}}}}<
+j2 arch          = >{{{{YADM_ARCH}}}}<
+j2 os            = >{{{{YADM_OS}}}}<
+j2 host          = >{{{{YADM_HOSTNAME}}}}<
+j2 user          = >{{{{YADM_USER}}}}<
+j2 distro        = >{{{{YADM_DISTRO}}}}<
+j2 distro_family = >{{{{YADM_DISTRO_FAMILY}}}}<
+j2 classes = >{{{{YADM_CLASSES}}}}<
 {{%- if YADM_CLASS == "wrongclass1" %}}
 wrong class 1
 {{%- endif %}}
 {{%- if YADM_CLASS == "{LOCAL_CLASS}" %}}
-Included section for class = {{{{YADM_CLASS}}}} ({{{{YADM_CLASS}}}} repeated)
+Included j2 section for class = \
+{{{{YADM_CLASS}}}} ({{{{YADM_CLASS}}}} repeated)
 {{%- endif %}}
 {{%- if YADM_CLASS == "wrongclass2" %}}
 wrong class 2
 {{%- endif %}}
+{{%- if "{LOCAL_CLASS2}" in YADM_CLASSES.split("\\n") %}}
+Included j2 section for second class
+{{%- endif %}}
+{{%- if YADM_ARCH == "wrongarch1" %}}
+wrong arch 1
+{{%- endif %}}
+{{%- if YADM_ARCH == "{LOCAL_ARCH}" %}}
+Included j2 section for arch = {{{{YADM_ARCH}}}} ({{{{YADM_ARCH}}}} repeated)
+{{%- endif %}}
+{{%- if YADM_ARCH == "wrongarch2" %}}
+wrong arch 2
+{{%- endif %}}
 {{%- if YADM_OS == "wrongos1" %}}
 wrong os 1
 {{%- endif %}}
 {{%- if YADM_OS == "{LOCAL_SYSTEM}" %}}
-Included section for os = {{{{YADM_OS}}}} ({{{{YADM_OS}}}} repeated)
+Included j2 section for os = {{{{YADM_OS}}}} ({{{{YADM_OS}}}} repeated)
 {{%- endif %}}
 {{%- if YADM_OS == "wrongos2" %}}
 wrong os 2
@@ -35,7 +57,8 @@ wrong os 2
 wrong host 1
 {{%- endif %}}
 {{%- if YADM_HOSTNAME == "{LOCAL_HOST}" %}}
-Included section for host = {{{{YADM_HOSTNAME}}}} ({{{{YADM_HOSTNAME}}}} again)
+Included j2 section for host = \
+{{{{YADM_HOSTNAME}}}} ({{{{YADM_HOSTNAME}}}} again)
 {{%- endif %}}
 {{%- if YADM_HOSTNAME == "wronghost2" %}}
 wrong host 2
@@ -44,7 +67,7 @@ wrong host 2
 wrong user 1
 {{%- endif %}}
 {{%- if YADM_USER == "{LOCAL_USER}" %}}
-Included section for user = {{{{YADM_USER}}}} ({{{{YADM_USER}}}} repeated)
+Included j2 section for user = {{{{YADM_USER}}}} ({{{{YADM_USER}}}} repeated)
 {{%- endif %}}
 {{%- if YADM_USER == "wronguser2" %}}
 wrong user 2
@@ -53,25 +76,44 @@ wrong user 2
 wrong distro 1
 {{%- endif %}}
 {{%- if YADM_DISTRO == "{LOCAL_DISTRO}" %}}
-Included section for distro = {{{{YADM_DISTRO}}}} ({{{{YADM_DISTRO}}}} again)
+Included j2 section for distro = \
+{{{{YADM_DISTRO}}}} ({{{{YADM_DISTRO}}}} again)
 {{%- endif %}}
 {{%- if YADM_DISTRO == "wrongdistro2" %}}
 wrong distro 2
 {{%- endif %}}
+{{%- if YADM_DISTRO_FAMILY == "wrongfamily1" %}}
+wrong family 1
+{{%- endif %}}
+{{%- if YADM_DISTRO_FAMILY == "{LOCAL_DISTRO_FAMILY}" %}}
+Included j2 section for distro_family = \
+{{{{YADM_DISTRO_FAMILY}}}} ({{{{YADM_DISTRO_FAMILY}}}} again)
+{{%- endif %}}
+{{%- if YADM_DISTRO_FAMILY == "wrongfamily2" %}}
+wrong family 2
+{{%- endif %}}
 end of template
 '''
 EXPECTED = f'''
 start of template
-j2 class  = >{LOCAL_CLASS}<
-j2 os     = >{LOCAL_SYSTEM}<
-j2 host   = >{LOCAL_HOST}<
-j2 user   = >{LOCAL_USER}<
-j2 distro = >{LOCAL_DISTRO}<
-Included section for class = {LOCAL_CLASS} ({LOCAL_CLASS} repeated)
-Included section for os = {LOCAL_SYSTEM} ({LOCAL_SYSTEM} repeated)
-Included section for host = {LOCAL_HOST} ({LOCAL_HOST} again)
-Included section for user = {LOCAL_USER} ({LOCAL_USER} repeated)
-Included section for distro = {LOCAL_DISTRO} ({LOCAL_DISTRO} again)
+j2 class         = >{LOCAL_CLASS}<
+j2 arch          = >{LOCAL_ARCH}<
+j2 os            = >{LOCAL_SYSTEM}<
+j2 host          = >{LOCAL_HOST}<
+j2 user          = >{LOCAL_USER}<
+j2 distro        = >{LOCAL_DISTRO}<
+j2 distro_family = >{LOCAL_DISTRO_FAMILY}<
+j2 classes = >{LOCAL_CLASS2}
+{LOCAL_CLASS}<
+Included j2 section for class = {LOCAL_CLASS} ({LOCAL_CLASS} repeated)
+Included j2 section for second class
+Included j2 section for arch = {LOCAL_ARCH} ({LOCAL_ARCH} repeated)
+Included j2 section for os = {LOCAL_SYSTEM} ({LOCAL_SYSTEM} repeated)
+Included j2 section for host = {LOCAL_HOST} ({LOCAL_HOST} again)
+Included j2 section for user = {LOCAL_USER} ({LOCAL_USER} repeated)
+Included j2 section for distro = {LOCAL_DISTRO} ({LOCAL_DISTRO} again)
+Included j2 section for distro_family = \
+{LOCAL_DISTRO_FAMILY} ({LOCAL_DISTRO_FAMILY} again)
 end of template
 '''
 
@@ -82,21 +124,32 @@ def test_template_j2(runner, yadm, tmpdir, processor):
 
     input_file = tmpdir.join('input')
     input_file.write(TEMPLATE, ensure=True)
+    input_file.chmod(FILE_MODE)
     output_file = tmpdir.join('output')
 
+    # ensure overwrite works when file exists as read-only (there is some
+    # special processing when this is encountered because some environments do
+    # not properly overwrite read-only files)
+    output_file.write('existing')
+    output_file.chmod(0o400)
+
     script = f"""
         YADM_TEST=1 source {yadm}
         local_class="{LOCAL_CLASS}"
+        local_classes=("{LOCAL_CLASS2}" "{LOCAL_CLASS}")
+        local_arch="{LOCAL_ARCH}"
         local_system="{LOCAL_SYSTEM}"
         local_host="{LOCAL_HOST}"
         local_user="{LOCAL_USER}"
         local_distro="{LOCAL_DISTRO}"
+        local_distro_family="{LOCAL_DISTRO_FAMILY}"
         template_{processor} "{input_file}" "{output_file}"
     """
     run = runner(command=['bash'], inp=script)
     assert run.success
     assert run.err == ''
     assert output_file.read() == EXPECTED
+    assert os.stat(output_file).st_mode == os.stat(input_file).st_mode
 
 
 @pytest.mark.parametrize('processor', ('j2cli', 'envtpl'))
@@ -105,6 +158,7 @@ def test_source(runner, yadm, tmpdir, processor):
 
     input_file = tmpdir.join('input')
     input_file.write('{{YADM_SOURCE}}', ensure=True)
+    input_file.chmod(FILE_MODE)
     output_file = tmpdir.join('output')
 
     script = f"""
@@ -115,3 +169,4 @@ def test_source(runner, yadm, tmpdir, processor):
     assert run.success
     assert run.err == ''
     assert output_file.read().strip() == str(input_file)
+    assert os.stat(output_file).st_mode == os.stat(input_file).st_mode

test/test_unit_upgrade.py

@@ -1,53 +1,39 @@
 """Unit tests: upgrade"""
 import pytest
 
-LEGACY_PATHS = [
-    'config',
-    'encrypt',
-    'files.gpg',
-    'bootstrap',
-    'hooks/pre_command',
-    'hooks/post_command',
-]
 
-# used:
-# YADM_COMPATIBILITY
-# YADM_DIR
-# YADM_LEGACY_DIR
-# GIT_PROGRAM
-@pytest.mark.parametrize('condition', ['compat', 'equal', 'existing_repo'])
+@pytest.mark.parametrize('condition', ['override', 'equal', 'existing_repo'])
 def test_upgrade_errors(tmpdir, runner, yadm, condition):
     """Test upgrade() error conditions"""
 
-    compatibility = 'YADM_COMPATIBILITY=1' if condition == 'compat' else ''
-
     home = tmpdir.mkdir('home')
     yadm_dir = home.join('.config/yadm')
-    legacy_dir = home.join('.yadm')
+    yadm_data = home.join('.local/share/yadm')
+    override = ''
+    if condition == 'override':
+        override = 'override'
     if condition == 'equal':
-        legacy_dir = yadm_dir
+        yadm_data = yadm_dir
     if condition == 'existing_repo':
         yadm_dir.ensure_dir('repo.git')
-        legacy_dir.ensure_dir('repo.git')
+        yadm_data.ensure_dir('repo.git')
 
     script = f"""
         YADM_TEST=1 source {yadm}
-        {compatibility}
         YADM_DIR="{yadm_dir}"
-        YADM_REPO="{yadm_dir}/repo.git"
-        YADM_LEGACY_DIR="{legacy_dir}"
+        YADM_DATA="{yadm_data}"
+        YADM_REPO="{yadm_data}/repo.git"
+        YADM_LEGACY_ARCHIVE="files.gpg"
+        YADM_OVERRIDE_REPO="{override}"
         upgrade
     """
     run = runner(command=['bash'], inp=script)
     assert run.failure
-    assert run.err == ''
-    assert 'Unable to upgrade' in run.out
-    if condition == 'compat':
-        assert 'YADM_COMPATIBILITY' in run.out
-    if condition == 'equal':
-        assert 'has been resolved as' in run.out
-    if condition == 'existing_repo':
-        assert 'already exists' in run.out
+    assert 'Unable to upgrade' in run.err
+    if condition in ['override', 'equal']:
+        assert 'Paths have been overridden' in run.err
+    elif condition == 'existing_repo':
+        assert 'already exists' in run.err
 
 
 @pytest.mark.parametrize(
@@ -59,22 +45,29 @@ def test_upgrade(tmpdir, runner, yadm, condition):
     mock for git. echo will return true, simulating a positive result from "git
     ls-files". Also echo will report the parameters for "git mv".
     """
+    legacy_paths = ('config', 'encrypt', 'bootstrap', 'hooks/pre_cmd')
     home = tmpdir.mkdir('home')
     yadm_dir = home.join('.config/yadm')
-    legacy_dir = home.join('.yadm')
+    yadm_data = home.join('.local/share/yadm')
+    yadm_legacy = home.join('.yadm')
 
     if condition != 'no-paths':
-        legacy_dir.join('repo.git/config').write('test-repo', ensure=True)
-        for lpath in LEGACY_PATHS:
-            legacy_dir.join(lpath).write(lpath, ensure=True)
+        yadm_dir.join('repo.git/config').write('test-repo', ensure=True)
+        yadm_dir.join('files.gpg').write('files.gpg', ensure=True)
+        for path in legacy_paths:
+            yadm_legacy.join(path).write(path, ensure=True)
 
     mock_git = ""
-    if condition in ['tracked', 'submodules']:
+    if condition != 'no-paths':
         mock_git = f'''
             function git() {{
                 echo "$@"
-                if [[ "$*" == *.gitmodules* ]]; then
-                    return { '0' if condition == 'submodules' else '1' }
+                if [[ "$*" = *"submodule status" ]]; then
+                    { 'echo " 1234567 mymodule (1.0)"'
+                        if condition == 'submodules' else ':' }
+                fi
+                if [[ "$*" = *ls-files* ]]; then
+                    return { 1 if condition == 'untracked' else 0 }
                 fi
                 return 0
             }}
@@ -82,9 +75,11 @@ def test_upgrade(tmpdir, runner, yadm, condition):
 
     script = f"""
         YADM_TEST=1 source {yadm}
+        YADM_LEGACY_DIR="{yadm_legacy}"
         YADM_DIR="{yadm_dir}"
-        YADM_REPO="{yadm_dir}/repo.git"
-        YADM_LEGACY_DIR="{legacy_dir}"
+        YADM_DATA="{yadm_data}"
+        YADM_REPO="{yadm_data}/repo.git"
+        YADM_ARCHIVE="{yadm_data}/archive"
         GIT_PROGRAM="git"
         {mock_git}
         function cd {{ echo "$@";}}
@@ -96,25 +91,32 @@ def test_upgrade(tmpdir, runner, yadm, condition):
     if condition == 'no-paths':
         assert 'Upgrade is not necessary' in run.out
     else:
-        for lpath in LEGACY_PATHS + ['repo.git']:
+        for (lpath, npath) in [
+                ('repo.git', 'repo.git'), ('files.gpg', 'archive')]:
             expected = (
-                f'Moving {legacy_dir.join(lpath)} '
-                f'to {yadm_dir.join(lpath)}')
+                f'Moving {yadm_dir.join(lpath)} '
+                f'to {yadm_data.join(npath)}')
+            assert expected in run.out
+        for path in legacy_paths:
+            expected = (
+                f'Moving {yadm_legacy.join(path)} '
+                f'to {yadm_dir.join(path)}')
             assert expected in run.out
         if condition == 'untracked':
-            assert 'test-repo' in yadm_dir.join('repo.git/config').read()
-            for lpath in LEGACY_PATHS:
-                assert lpath in yadm_dir.join(lpath).read()
+            assert 'test-repo' in yadm_data.join('repo.git/config').read()
+            assert 'files.gpg' in yadm_data.join('archive').read()
+            for path in legacy_paths:
+                assert path in yadm_dir.join(path).read()
         elif condition in ['tracked', 'submodules']:
-            for lpath in LEGACY_PATHS:
-                expected = (
-                    f'mv {legacy_dir.join(lpath)} '
-                    f'{yadm_dir.join(lpath)}')
-                assert expected in run.out
+            expected = (
+                f'mv {yadm_dir.join("files.gpg")} '
+                f'{yadm_data.join("archive")}')
+            assert expected in run.out
             assert 'files tracked by yadm have been renamed' in run.out
             if condition == 'submodules':
-                assert 'submodule deinit -f .' in run.out
-                assert 'submodule update --init --recursive' in run.out
+                assert 'submodule deinit -- mymodule' in run.out
+                assert 'submodule update --init --recursive -- mymodule' \
+                    in run.out
             else:
-                assert 'submodule deinit -f .' not in run.out
+                assert 'submodule deinit' not in run.out
                 assert 'submodule update --init --recursive' not in run.out

test/test_unit_x_program.py

@@ -16,24 +16,24 @@ import pytest
     ])
 @pytest.mark.parametrize('program', ['git', 'gpg'])
 def test_x_program(
-        runner, yadm_y, paths, program, executable, success, value, match):
+        runner, yadm_cmd, paths, program, executable, success, value, match):
     """Set yadm.X-program, and test result of require_X"""
 
     # set configuration
     if executable:
-        os.system(' '.join(yadm_y(
+        os.system(' '.join(yadm_cmd(
             'config', f'yadm.{program}-program', executable)))
 
     # test require_[git,gpg]
     script = f"""
         YADM_TEST=1 source {paths.pgm}
-        YADM_CONFIG="{paths.config}"
+        YADM_OVERRIDE_CONFIG="{paths.config}"
+        configure_paths
         require_{program}
         echo ${program.upper()}_PROGRAM
     """
     run = runner(command=['bash'], inp=script)
     assert run.success == success
-    assert run.err == ''
 
     # [GIT,GPG]_PROGRAM set correctly
     if value == 'program':
@@ -43,4 +43,6 @@ def test_x_program(
 
     # error reported about bad config
     if match:
-        assert match in run.out
+        assert match in run.err
+    else:
+        assert run.err == ''

test/test_upgrade.py

@@ -0,0 +1,129 @@
+"""Test upgrade"""
+
+import os
+import pytest
+
+
+@pytest.mark.parametrize(
+    'versions', [
+        ('1.12.0', '2.5.0'),
+        ('1.12.0',),
+        ('2.5.0',),
+    ], ids=[
+        '1.12.0 -> 2.5.0 -> latest',
+        '1.12.0 -> latest',
+        '2.5.0 -> latest',
+    ])
+@pytest.mark.parametrize(
+    'submodule', [False, True],
+    ids=['no submodule', 'with submodules'])
+def test_upgrade(tmpdir, runner, versions, submodule):
+    """Upgrade tests"""
+    # pylint: disable=too-many-statements
+    home = tmpdir.mkdir('HOME')
+    env = {'HOME': str(home)}
+
+    if submodule:
+        ext_repo = tmpdir.mkdir('ext_repo')
+        ext_repo.join('afile').write('some data')
+
+        for cmd in (('init',), ('add', 'afile'), ('commit', '-m', 'test')):
+            run = runner(['git', '-C', str(ext_repo), *cmd])
+            assert run.success
+
+    os.environ.pop('XDG_CONFIG_HOME', None)
+    os.environ.pop('XDG_DATA_HOME', None)
+
+    def run_version(version, *args, check_stderr=True):
+        yadm = 'yadm-%s' % version if version else '/yadm/yadm'
+        run = runner([yadm, *args], shell=True, cwd=str(home), env=env)
+        assert run.success
+        if check_stderr:
+            assert run.err == ''
+        return run
+
+    # Initialize the repo with the first version
+    first = versions[0]
+    run_version(first, 'init')
+
+    home.join('file').write('some data')
+    run_version(first, 'add', 'file')
+    run_version(first, 'commit', '-m', '"First commit"')
+
+    if submodule:
+        # When upgrading via 2.5.0 we can't have a submodule that's been added
+        # after being cloned as 2.5.0 fails the upgrade in that case.
+        can_upgrade_cloned_submodule = '2.5.0' not in versions[1:]
+        if can_upgrade_cloned_submodule:
+            # Check out a repo and then add it as a submodule
+            run = runner(['git', '-C', str(home), 'clone', str(ext_repo), 'b'])
+            assert run.success
+            run_version(first, 'submodule', 'add', str(ext_repo), 'b')
+
+        # Add submodule without first checking it out
+        run_version(first, 'submodule', 'add', str(ext_repo), 'a',
+                    check_stderr=False)
+        run_version(first, 'submodule', 'add', str(ext_repo), 'c',
+                    check_stderr=False)
+
+        run_version(first, 'commit', '-m', '"Add submodules"')
+
+    for path in ('.yadm', '.config/yadm'):
+        yadm_dir = home.join(path)
+        if yadm_dir.exists():
+            break
+
+    yadm_dir.join('bootstrap').write('init stuff')
+    run_version(first, 'add', yadm_dir.join('bootstrap'))
+    run_version(first, 'commit', '-m', 'bootstrap')
+
+    yadm_dir.join('encrypt').write('secret')
+
+    hooks_dir = yadm_dir.mkdir('hooks')
+    hooks_dir.join('pre_status').write('status')
+    hooks_dir.join('post_commit').write('commit')
+
+    run_version(first, 'config', 'local.class', 'test')
+    run_version(first, 'config', 'foo.bar', 'true')
+
+    # Run upgrade with intermediate versions and latest
+    latest = None
+    for version in versions[1:] + (latest,):
+        run = run_version(version, 'upgrade', check_stderr=not submodule)
+        if submodule:
+            lines = run.err.splitlines()
+            if can_upgrade_cloned_submodule:
+                assert 'Migrating git directory of' in lines[0]
+                assert str(home.join('b/.git')) in lines[1]
+                assert str(yadm_dir.join('repo.git/modules/b')) in lines[2]
+                del lines[:3]
+            for line in lines:
+                assert line.startswith('Submodule')
+                assert 'registered for path' in line
+
+    # Verify result for the final upgrade
+    run_version(latest, 'status')
+
+    run = run_version(latest, 'show', 'HEAD:file')
+    assert run.out == 'some data'
+
+    if submodule:
+        if can_upgrade_cloned_submodule:
+            assert home.join('b/afile').read() == 'some data'
+        assert home.join('a/afile').read() == 'some data'
+        assert home.join('c/afile').read() == 'some data'
+
+    yadm_dir = home.join('.config/yadm')
+
+    assert yadm_dir.join('bootstrap').read() == 'init stuff'
+    assert yadm_dir.join('encrypt').read() == 'secret'
+
+    hooks_dir = yadm_dir.join('hooks')
+    assert hooks_dir.join('pre_status').read() == 'status'
+    assert hooks_dir.join('post_commit').read() == 'commit'
+
+    run = run_version(latest, 'config', 'local.class')
+    assert run.out.rstrip() == 'test'
+
+    run = run_version(latest, 'config', 'foo.bar')
+    assert run.out.rstrip() == 'true'

test/test_version.py

@@ -26,10 +26,13 @@ def test_semantic_version(expected_version):
         'does not conform to MAJOR.MINOR.PATCH')
 
 
+@pytest.mark.parametrize('cmd', ['--version', 'version'])
 def test_reported_version(
-        runner, yadm_y, expected_version):
-    """Report correct version"""
-    run = runner(command=yadm_y('version'))
+        runner, yadm_cmd, cmd, expected_version):
+    """Report correct version and bash/git versions"""
+    run = runner(command=yadm_cmd(cmd))
     assert run.success
     assert run.err == ''
-    assert run.out == f'yadm {expected_version}\n'
+    assert 'bash version' in run.out
+    assert 'git version' in run.out
+    assert run.out.endswith(f'\nyadm version {expected_version}\n')

test/utils.py

@@ -21,11 +21,12 @@ INCLUDE_DIRS = ['', 'test alt']
 INCLUDE_CONTENT = '8780846c02e34c930d0afd127906668f'
 
 
-def set_local(paths, variable, value):
+def set_local(paths, variable, value, add=False):
     """Set local override"""
+    add = "--add" if add else ""
     os.system(
         f'GIT_DIR={str(paths.repo)} '
-        f'git config --local "local.{variable}" "{value}"'
+        f'git config --local {add} "local.{variable}" "{value}"'
     )
 
 

yadm.1

@@ -1,5 +1,5 @@
-." vim: set spell so=8:
-.TH yadm 1 "12 November 2019" "2.0.1"
+.\" vim: set spell so=8:
+.TH yadm 1 "23 January 2023" "3.2.2"
 
 .SH NAME
 
@@ -47,8 +47,6 @@ list
 
 .BR yadm " encrypt
 
-.BR yadm " enter
-
 .BR yadm " decrypt
 .RB [ -l ]
 
@@ -56,7 +54,14 @@ list
 
 .BR yadm " perms
 
+.BR yadm " enter [ command ]
+
+.BR yadm " git-crypt [ options ]
+
+.BR yadm " transcrypt [ options ]
+
 .BR yadm " upgrade
+.RB [ -f ]
 
 .BR yadm " introspect
 .I category
@@ -111,37 +116,15 @@ if it exists.
 .TP
 .BI clone " url
 Clone a remote repository for tracking dotfiles.
-After the contents of the remote repository have been fetched, a "merge" of
-.I origin/master
-is attempted.
+After the contents of the remote repository have been fetched, a "check out" of
+the remote HEAD branch is attempted.
 If there are conflicting files already present in the
 .IR work-tree ,
-this merge will fail and instead a "reset" of
-.I origin/master
-will be done, followed by a "stash". This "stash" operation will preserve the
-original data.
-
-You can review the stashed conflicts by running the command
-
-.RS
-.RS
-yadm stash show -p
-.RE
-
-from within your
-.I $HOME
-directory. If you want to restore the stashed data, you can run
-
-.RS
-yadm stash apply
-.RE
-or
-.RS
-yadm stash pop
-.RE
+the local version will be left unmodified and you'll have to review and resolve
+the difference.
 
 The repository is stored in
-.IR $HOME/.config/yadm/repo.git .
+.IR $HOME/.local/share/yadm/repo.git .
 By default,
 .I $HOME
 will be used as the
@@ -150,8 +133,7 @@ but this can be overridden with the
 .BR -w " option.
 yadm can be forced to overwrite an existing repository by providing the
 .BR -f " option.
-If you want to use a branch other than
-.IR origin/master ,
+If you want to use a branch other than the remote HEAD branch
 you can specify it using the
 .BR -b " option.
 By default yadm will ask the user if the bootstrap program should be run (if it
@@ -159,18 +141,17 @@ exists). The options
 .BR --bootstrap " or " --no-bootstrap
 will either force the bootstrap to be run, or prevent it from being run,
 without prompting the user.
-.RE
 .TP
 .B config
 This command manages configurations for yadm.
-This command works exactly they way
+This command works exactly the way
 .BR git-config (1)
 does.
 See the CONFIGURATION section for more details.
 .TP
 .B decrypt
 Decrypt all files stored in
-.IR $HOME/.config/yadm/files.gpg .
+.IR $HOME/.local/share/yadm/archive .
 Files decrypted will be relative to the configured
 .IR work-tree " (usually
 .IR $HOME ).
@@ -187,16 +168,37 @@ See the ENCRYPTION section for more details.
 Run a sub-shell with all Git variables set. Exit the sub-shell the same way you
 leave your normal shell (usually with the "exit" command). This sub-shell can
 be used to easily interact with your yadm repository using "git" commands. This
-could be useful if you are using a tool which uses Git directly. For example,
+could be useful if you are using a tool which uses Git directly, such as tig,
+vim-fugitive, git-cola, etc.
+
+Optionally, you can provide a command after "enter", and instead of invoking
+your shell, that command will be run with all of the Git variables exposed to
+the command's environment.
+
 Emacs Tramp and Magit can manage files by using this configuration:
+
 .RS
     (add-to-list 'tramp-methods
          '("yadm"
            (tramp-login-program "yadm")
            (tramp-login-args (("enter")))
+           (tramp-login-env (("SHELL") ("/bin/sh")))
            (tramp-remote-shell "/bin/sh")
            (tramp-remote-shell-args ("-c"))))
 .RE
+
+.RS
+With this config, use (magit-status "/yadm::").
+.RE
+.TP
+.BI git-crypt " options
+If git-crypt is installed, this command allows you to pass options directly to
+git-crypt, with the environment configured to use the yadm repository.
+
+git-crypt enables transparent encryption and decryption of files in a git repository.
+You can read
+https://github.com/AGWA/git-crypt
+for details.
 .TP
 .B gitconfig
 Pass options to the
@@ -224,7 +226,7 @@ Print a summary of yadm commands.
 .B init
 Initialize a new, empty repository for tracking dotfiles.
 The repository is stored in
-.IR $HOME/.config/yadm/repo.git .
+.IR $HOME/.local/share/yadm/repo.git .
 By default,
 .I $HOME
 will be used as the
@@ -257,50 +259,43 @@ configuration
 .I yadm.auto-perms
 to "false".
 .TP
+.BI transcrypt " options
+If transcrypt is installed, this command allows you to pass options directly to
+transcrypt, with the environment configured to use the yadm repository.
+
+transcrypt enables transparent encryption and decryption of files in a git repository.
+You can read
+https://github.com/elasticdog/transcrypt
+for details.
+.TP
 .B upgrade
-Version 2 of yadm uses a different directory for storing your configurations.
-When you start to use version 2 for the first time, you may see warnings about
+Version 3 of yadm uses a different directory for storing data.
+When you start to use version 3 for the first time, you may see warnings about
 moving your data to this new directory.
 The easiest way to accomplish this is by running "yadm upgrade".
 This command will start by moving your yadm repo to the new path.
-Next it will move any configuration data to the new path.
-If the configurations are tracked within your yadm repo, this command will
-"stage" the renaming of those files in the repo's index.
-Upgrading will also re-initialize all submodules you have added (otherwise they
-will be broken when the repo moves).
+Next it will move any archive data.
+If the archive is tracked within your yadm repo, this command will
+"stage" the renaming of that file in the repo's index.
+
+Upgrading will attempt to de-initialize and re-initialize your submodules. If
+your submodules cannot be de-initialized, the upgrade will fail. The most
+common reason submodules will fail to de-initialize is because they have local
+modifications. If you are willing to lose the local modifications to those
+submodules, you can use the
+.B -f
+option with the "upgrade" command to force the de-initialization.
+
 After running "yadm upgrade", you should run "yadm status" to review changes
 which have been staged, and commit them to your repository.
 
 You can read
-https://yadm.io/docs/upgrade_from_1
+https://yadm.io/docs/upgrade_from_2
 for more information.
 .TP
 .B version
 Print the version of yadm.
 
-.SH COMPATIBILITY
-
-Beginning with version 2.0.0, yadm introduced a couple major changes which may
-require you to adjust your configurations.
-See the
-.B upgrade
-command for help making those adjustments.
-
-First, yadm now uses the "XDG Base Directory Specification" to find its
-configurations. You can read
-https://yadm.io/docs/upgrade_from_1
-for more information.
-
-Second, the naming conventions for alternate files have been changed.
-You can read https://yadm.io/docs/alternates for more information.
-
-If you want to retain the old functionality, you can set an environment variable,
-.IR YADM_COMPATIBILITY=1 .
-Doing so will automatically use the old yadm directory, and process alternates
-the same as the pre-2.0.0 version. This compatibility mode is deprecated, and
-will be removed in future versions. This mode exists solely for transitioning
-to the new paths and naming of alternates.
-
 .SH OPTIONS
 
 yadm supports a set of universal options that alter the paths it uses. The
@@ -315,10 +310,14 @@ alias yadm='yadm --yadm-repo /alternate/path/to/repo'
 .RE
 
 The following is the full list of universal options.
-Each option should be followed by a fully qualified path.
+Each option should be followed by a path.
 .TP
 .B -Y,--yadm-dir
 Override the yadm directory.
+yadm stores its configurations relative to this directory.
+.TP
+.B --yadm-data
+Override the yadm data directory.
 yadm stores its data relative to this directory.
 .TP
 .B --yadm-repo
@@ -359,12 +358,6 @@ The following is the full list of supported configurations:
 If set to "true", alternate files will be copies instead of symbolic links.
 This might be desirable, because some systems may not properly support
 symlinks.
-
-NOTE: The deprecated
-.I yadm.cygwin-copy
-option used by older versions of yadm has been replaced by
-.IR yadm.alt-copy .
-The old option will be removed in the next version of yadm.
 .TP
 .B yadm.auto-alt
 Disable the automatic linking described in the section ALTERNATES. If disabled,
@@ -386,6 +379,11 @@ This feature is enabled by default.
 .B yadm.auto-private-dirs
 Disable the automatic creating of private directories described in the section PERMISSIONS.
 .TP
+.B yadm.cipher
+Configure which encryption system is used by the encrypt/decrypt commands.
+Valid options are "gpg" and "openssl". The default is "gpg".
+Detailed information can be found in the section ENCRYPTION.
+.TP
 .B yadm.git-program
 Specify an alternate program to use instead of "git".
 By default, the first "git" found in $PATH is used.
@@ -403,18 +401,33 @@ By default, the first "gpg" found in $PATH is used.
 Asymmetrically encrypt files with a gpg public/private key pair.
 Provide a "key ID" to specify which public key to encrypt with.
 The key must exist in your public keyrings.
+Multiple recipients can be specified (separated by space).
 If left blank or not provided, symmetric encryption is used instead.
 If set to "ASK", gpg will interactively ask for recipients.
 See the ENCRYPTION section for more details.
 This feature is disabled by default.
 .TP
+.B yadm.openssl-ciphername
+Specify which cipher should be used by openssl.
+"aes-256-cbc" is used by default.
+.TP
+.B yadm.openssl-old
+Newer versions of openssl support the pbkdf2 key derivation function. This is
+used by default. If this configuration is set to "true", openssl operations
+will use options compatible with older versions of openssl. If you change this
+option, you will need to recreate your encrypted archive.
+.TP
+.B yadm.openssl-program
+Specify an alternate program to use instead of "openssl".
+By default, the first "openssl" found in $PATH is used.
+.TP
 .B yadm.ssh-perms
 Disable the permission changes to
 .IR $HOME/.ssh/* .
 This feature is enabled by default.
 
 .RE
-The following four "local" configurations are not stored in the
+The following five "local" configurations are not stored in the
 .IR $HOME/.config/yadm/config,
 they are stored in the local repository.
 
@@ -422,6 +435,14 @@ they are stored in the local repository.
 .B local.class
 Specify a class for the purpose of symlinking alternate files.
 By default, no class will be matched.
+The local host can be assigned multiple classes using command:
+
+.RS
+yadm config --add local.class <additional-class>
+.RE
+.TP
+.B local.arch
+Override the architecture for the purpose of symlinking alternate files.
 .TP
 .B local.hostname
 Override the hostname for the purpose of symlinking alternate files.
@@ -465,15 +486,11 @@ Valid if the value matches the current user.
 Current user is calculated by running
 .BR "id -u -n" .
 .TP
-.BR distro , " d
-Valid if the value matches the distro.
-Distro is calculated by running
-.BR "lsb_release -si" .
-.TP
-.BR os , " o
-Valid if the value matches the OS.
-OS is calculated by running
-.BR "uname -s" .
+.BR hostname , " h
+Valid if the value matches the short hostname.
+Hostname is calculated by running
+.BR "uname -n" ,
+and trimming off any domain.
 .TP
 .BR class , " c
 Valid if the value matches the
@@ -484,16 +501,41 @@ Class must be manually set using
 See the CONFIGURATION section for more details about setting
 .BR local.class .
 .TP
-.BR hostname , " h
-Valid if the value matches the short hostname.
-Hostname is calculated by running
-.BR "hostname" ,
-and trimming off any domain.
+.BR distro , " d
+Valid if the value matches the distro.
+Distro is calculated by running
+.B "lsb_release -si"
+or by inspecting the ID from
+.BR "/etc/os-release" .
+.TP
+.BR distro_family , " f
+Valid if the value matches the distro family.
+Distro family is calculated by inspecting the ID_LIKE line from
+.BR "/etc/os-release" .
+.TP
+.BR os , " o
+Valid if the value matches the OS.
+OS is calculated by running
+.BR "uname -s" .
+.TP
+.BR arch , " a
+Valid if the value matches the architecture.
+Architecture is calculated by running
+.BR "uname -m" .
 .TP
 .B default
 Valid when no other alternate is valid.
+.TP
+.BR extension , " e
+A special "condition" that doesn't affect the selection process. Its purpose is
+instead to allow the alternate file to end with a certain extension to
+e.g. make editors highlight the content properly.
 .LP
 
+.BR NOTE :
+The OS for "Windows Subsystem for Linux" is reported as "WSL", even
+though uname identifies as "Linux".
+
 You may use any number of conditions, in any order.
 An alternate will only be used if ALL conditions are valid.
 For all files managed by yadm's repository or listed in
@@ -553,7 +595,7 @@ If no "##default" version exists and no files have valid conditions, then no
 link will be created.
 
 Links are also created for directories named this way, as long as they have at
-least one yadm managed file within them.
+least one yadm managed file within them (at the top level).
 
 yadm will automatically create these links by default. This can be disabled
 using the
@@ -572,8 +614,9 @@ command. The following sets the class to be "Work".
 
   yadm config local.class Work
 
-Similarly, the values of os, hostname, and user can be manually overridden
-using the configuration options
+Similarly, the values of architecture, os, hostname, and user can be manually
+overridden using the configuration options
+.BR local.arch ,
 .BR local.os ,
 .BR local.hostname ,
 and
@@ -595,6 +638,15 @@ upon
 which is available on most *nix systems. To use this processor,
 specify the value of "default" or just leave the value off (e.g. "##template").
 .TP
+.B ESH
+ESH is a template processor written in POSIX compliant shell. It allows
+executing shell commands within templates. This can be used to reference your
+own configurations within templates, for example:
+
+  <% yadm config mysection.myconfig %>
+
+To use the ESH template processor, specify the value of "esh"
+.TP
 .B j2cli
 To use the j2cli Jinja template processor, specify the value of "j2"  or
 "j2cli".
@@ -612,24 +664,37 @@ to create or overwrite files.
 
 During processing, the following variables are available in the template:
 
- Default         Jinja           Description
- -------------   -------------   --------------------------
- yadm.class      YADM_CLASS      Locally defined yadm class
- yadm.distro     YADM_DISTRO     lsb_release -si
- yadm.hostname   YADM_HOSTNAME   hostname (without domain)
- yadm.os         YADM_OS         uname -s
- yadm.user       YADM_USER       id -u -n
- yadm.source     YADM_SOURCE     Template filename
+ Default              Jinja or ESH         Description
+ -------------        -------------        ----------------------------
+ yadm.arch            YADM_ARCH            uname -m
+ yadm.class           YADM_CLASS           Last locally defined class
+ yadm.classes         YADM_CLASSES         All classes
+ yadm.distro          YADM_DISTRO          lsb_release -si
+ yadm.distro_family   YADM_DISTRO_FAMILY   ID_LIKE from /etc/os-release
+ yadm.hostname        YADM_HOSTNAME        uname -n (without domain)
+ yadm.os              YADM_OS              uname -s
+ yadm.source          YADM_SOURCE          Template filename
+ yadm.user            YADM_USER            id -u -n
+ env.VAR                                   Environment variable VAR
+
+.BR NOTE :
+The OS for "Windows Subsystem for Linux" is reported as "WSL", even
+though uname identifies as "Linux".
+
+.BR NOTE :
+If lsb_release is not available, DISTRO will be the ID specified in
+/etc/os-release.
 
 Examples:
 
 .I whatever##template
 with the following content
 
-  {% if yadm.user == 'harvey' %}
+  {% if yadm.user == "harvey" %}
   config={{yadm.class}}-{{yadm.os}}
   {% else %}
   config=dev-whatever
+  {% include "whatever.extra" %}
   {% endif %}
 
 would output a file named
@@ -638,9 +703,12 @@ with the following content if the user is "harvey":
 
   config=work-Linux
 
-and the following otherwise:
+and the following otherwise (if
+.I whatever.extra
+contains admin=false):
 
   config=dev-whatever
+  admin=false
 
 An equivalent Jinja template named
 .I whatever##template.j2
@@ -650,8 +718,20 @@ would look like:
   config={{YADM_CLASS}}-{{YADM_OS}}
   {% else -%}
   config=dev-whatever
+  {% include 'whatever.extra' %}
   {% endif -%}
 
+An equivalent ESH templated named
+.I whatever##template.esh
+would look like:
+
+  <% if [ "$YADM_USER" = "harvey" ]; then -%>
+  config=<%= $YADM_CLASS %>-<%= $YADM_OS %>
+  <% else -%>
+  config=dev-whatever
+  <%+ whatever.extra %>
+  <% fi -%>
+
 .SH ENCRYPTION
 
 It can be useful to manage confidential files, like SSH or GPG keys, across
@@ -659,9 +739,15 @@ multiple systems. However, doing so would put plain text data into a Git
 repository, which often resides on a public system. yadm can make it easy to
 encrypt and decrypt a set of files so the encrypted version can be maintained
 in the Git repository.
-This feature will only work if the
+This feature will only work if a supported tool is available.
+Both
 .BR gpg (1)
-command is available.
+and
+.BR openssl (1)
+are supported.
+gpg is used by default, but openssl can be configured with the
+.I yadm.cipher
+configuration.
 
 To use this feature, a list of patterns must be created and saved as
 .IR $HOME/.config/yadm/encrypt .
@@ -675,17 +761,19 @@ For example:
     .gnupg/*.gpg
 .RE
 
-Standard filename expansions (*, ?, [) are supported. Other shell expansions
-like brace and tilde are not supported. Spaces in paths are supported, and
-should not be quoted. If a directory is specified, its contents will be
-included, but not recursively. Paths beginning with a "!" will be excluded.
+Standard filename expansions (*, ?, [) are supported.
+If you have Bash version 4, you may use "**" to match all subdirectories.
+Other shell expansions like brace and tilde are not supported.
+Spaces in paths are supported, and should not be quoted.
+If a directory is specified, its contents will be included, but not recursively.
+Paths beginning with a "!" will be excluded.
 
 The
 .B yadm encrypt
 command will find all files matching the patterns, and prompt for a password. Once a
 password has confirmed, the matching files will be encrypted and saved as
-.IR $HOME/.config/yadm/files.gpg .
-The patterns and files.gpg should be added to the yadm repository so they are
+.IR $HOME/.local/share/yadm/archive .
+The "encrypt" and "archive" files should be added to the yadm repository so they are
 available across multiple systems.
 
 To decrypt these files later, or on another system run
@@ -715,6 +803,22 @@ This can be disabled using the
 .I yadm.auto-exclude
 configuration.
 
+.B Using transcrypt or git-crypt
+
+A completely separate option for encrypting data is to install and use
+transcrypt or git-crypt.
+Once installed, you can use these tools by running
+.B "yadm transcrypt"
+or
+.BR "yadm git-crypt" .
+These tools enables transparent encryption and decryption of files in a git
+repository. See the following web sites for more information:
+
+- https://github.com/elasticdog/transcrypt
+
+- https://github.com/AGWA/git-crypt
+.LP
+
 .SH PERMISSIONS
 
 When files are checked out of a Git repository, their initial permissions are
@@ -722,7 +826,7 @@ dependent upon the user's umask. Because of this, yadm will automatically
 update the permissions of some file paths. The "group" and "others" permissions
 will be removed from the following files:
 
-.RI - " $HOME/.config/yadm/files.gpg
+.RI - " $HOME/.local/share/yadm/archive
 
 - All files matching patterns in
 .I $HOME/.config/yadm/encrypt
@@ -798,7 +902,8 @@ The command which triggered the hook
 The exit status of the yadm command
 .TP
 .B YADM_HOOK_FULL_COMMAND
-The yadm command with all command line arguments
+The yadm command with all command line arguments (parameters are space
+delimited, and any space, tab or backslash will be escaped with a backslash)
 .TP
 .B YADM_HOOK_REPO
 The path to the yadm repository
@@ -817,12 +922,25 @@ is defined as a fully qualified path, this directory will be
 Otherwise it will be
 .IR "$HOME/.config/yadm" .
 
+Similarly, yadm's data files are relative to the "yadm data directory".
+yadm uses the "XDG Base Directory Specification" to determine this directory.
+If the environment variable
+.B $XDG_DATA_HOME
+is defined as a fully qualified path, this directory will be
+.IR "$XDG_DATA_HOME/yadm" .
+Otherwise it will be
+.IR "$HOME/.local/share/yadm" .
+
 The following are the default paths yadm uses for its own data.
 Most of these paths can be altered using universal options.
 See the OPTIONS section for details.
 .TP
 .I $HOME/.config/yadm
-The yadm directory. By default, all data yadm stores is relative to this
+The yadm directory. By default, all configs yadm stores is relative to this
+directory.
+.TP
+.I $HOME/.local/share/yadm
+The yadm data directory. By default, all data yadm stores is relative to this
 directory.
 .TP
 .I $YADM_DIR/config
@@ -833,13 +951,13 @@ This is a directory to keep "alternate files" without having them side-by-side
 with the resulting symlink or processed template. Alternate files placed in
 this directory will be created relative to $HOME instead.
 .TP
-.I $YADM_DIR/repo.git
+.I $YADM_DATA/repo.git
 Git repository used by yadm.
 .TP
 .I $YADM_DIR/encrypt
 List of globs used for encrypt/decrypt
 .TP
-.I $YADM_DIR/files.gpg
+.I $YADM_DATA/archive
 All files encrypted with
 .B yadm encrypt
 are stored in this file.
@@ -864,7 +982,7 @@ Initial push of master to origin
 .B echo ".ssh/*.key" >> $HOME/.config/yadm/encrypt
 Add a new pattern to the list of encrypted files
 .TP
-.B yadm encrypt ; yadm add ~/.config/yadm/files.gpg ; yadm commit
+.B yadm encrypt ; yadm add ~/.local/share/yadm/archive ; yadm commit
 Commit a new set of encrypted files
 
 .SH REPORTING BUGS
@@ -881,5 +999,8 @@ Tim Byrne <sultan@locehilios.com>
 
 .BR git (1),
 .BR gpg (1)
+.BR openssl (1)
+.BR transcrypt (1)
+.BR git-crypt (1)
 
 https://yadm.io/

yadm.md

@@ -24,117 +24,129 @@
 
        yadm encrypt
 
-       yadm enter
-
        yadm decrypt [-l]
 
        yadm alt
 
        yadm perms
 
-       yadm upgrade
+       yadm enter [ command ]
+
+       yadm git-crypt [ options ]
+
+       yadm transcrypt [ options ]
+
+       yadm upgrade [-f]
 
        yadm introspect category
 
 
 ## DESCRIPTION
-       yadm is a tool for managing a collection of files across multiple  com-
-       puters,  using  a  shared Git repository.  In addition, yadm provides a
-       feature to select alternate versions of files for  particular  systems.
-       Lastly,  yadm  supplies the ability to manage a subset of secure files,
+       yadm  is a tool for managing a collection of files across multiple com‐
+       puters, using a shared Git repository.  In addition,  yadm  provides  a
+       feature  to  select alternate versions of files for particular systems.
+       Lastly, yadm supplies the ability to manage a subset of  secure  files,
        which are encrypted before they are included in the repository.
 
 
 ## COMMANDS
        git-command or git-alias
-              Any command not internally handled by yadm is passed through  to
-              git(1).   Git commands or aliases are invoked with the yadm man-
+              Any  command not internally handled by yadm is passed through to
+              git(1).  Git commands or aliases are invoked with the yadm  man‐
               aged repository.  The working directory for Git commands will be
               the configured work-tree (usually $HOME).
 
-              Dotfiles  are  managed by using standard git commands; add, com-
+              Dotfiles are managed by using standard git commands;  add,  com‐
               mit, push, pull, etc.
 
-              The config command is not passed directly through.  Instead  use
+              The  config command is not passed directly through.  Instead use
               the gitconfig command (see below).
 
-       alt    Create  symbolic  links  and  process  templates for any managed
-              files matching the naming rules described in the ALTERNATES  and
-              TEMPLATES  sections.  It is usually unnecessary to run this com-
-              mand, as yadm automatically  processes  alternates  by  default.
-              This  automatic behavior can be disabled by setting the configu-
+       alt    Create symbolic links and  process  templates  for  any  managed
+              files  matching the naming rules described in the ALTERNATES and
+              TEMPLATES sections. It is usually unnecessary to run  this  com‐
+              mand,  as  yadm  automatically  processes alternates by default.
+              This automatic behavior can be disabled by setting the  configu‐
               ration yadm.auto-alt to "false".
 
        bootstrap
               Execute $HOME/.config/yadm/bootstrap if it exists.
 
        clone url
-              Clone a remote repository for tracking dotfiles.  After the con-
-              tents  of  the remote repository have been fetched, a "merge" of
-              origin/master is attempted.   If  there  are  conflicting  files
-              already  present  in  the  work-tree,  this  merge will fail and
-              instead a "reset" of origin/master will be done, followed  by  a
-              "stash". This "stash" operation will preserve the original data.
+              Clone a remote repository for tracking dotfiles.  After the con‐
+              tents of the remote repository have been fetched, a "check  out"
+              of  the remote HEAD branch is attempted.  If there are conflict‐
+              ing files already present in the work-tree,  the  local  version
+              will  be  left  unmodified and you'll have to review and resolve
+              the difference.
 
-              You can review the stashed conflicts by running the command
+              The repository is  stored  in  $HOME/.local/share/yadm/repo.git.
+              By default, $HOME will be used as the work-tree, but this can be
+              overridden with the -w option.  yadm can be forced to  overwrite
+              an  existing repository by providing the -f option.  If you want
+              to use a branch other than the remote HEAD branch you can  spec‐
+              ify  it  using the -b option.  By default yadm will ask the user
+              if the bootstrap program should be run (if it exists).  The  op‐
+              tions  --bootstrap or --no-bootstrap will either force the boot‐
+              strap to be run, or prevent it from being run, without prompting
+              the user.
 
-                     yadm stash show -p
-
-              from within your $HOME directory. If you  want  to  restore  the
-              stashed data, you can run
-
-                     yadm stash apply
-              or
-                     yadm stash pop
-
-              The  repository  is  stored  in $HOME/.config/yadm/repo.git.  By
-              default, $HOME will be used as the work-tree, but  this  can  be
-              overridden  with the -w option.  yadm can be forced to overwrite
-              an existing repository by providing the -f option.  If you  want
-              to  use  a  branch  other than origin/master, you can specify it
-              using the -b option.  By default yadm will ask the user  if  the
-              bootstrap  program  should  be  run  (if it exists). The options
-              --bootstrap or --no-bootstrap will either force the bootstrap to
-              be  run,  or  prevent  it  from being run, without prompting the
-              user.
-
-       config This command manages  configurations  for  yadm.   This  command
-              works  exactly  they way git-config(1) does.  See the CONFIGURA-
-              TION section for more details.
+       config This  command  manages  configurations  for  yadm.  This command
+              works exactly the way git-config(1) does.  See the CONFIGURATION
+              section for more details.
 
        decrypt
-              Decrypt all files stored in $HOME/.config/yadm/files.gpg.  Files
-              decrypted  will be relative to the configured work-tree (usually
-              $HOME).  Using the -l option will list the files stored  without
-              extracting them.
+              Decrypt  all  files  stored  in $HOME/.local/share/yadm/archive.
+              Files decrypted will be relative  to  the  configured  work-tree
+              (usually $HOME).  Using the -l option will list the files stored
+              without extracting them.
 
        encrypt
-              Encrypt  all  files  matching  the patterns found in $HOME/.con-
-              fig/yadm/encrypt.  See the ENCRYPTION section for more  details.
+              Encrypt all files matching the  patterns  found  in  $HOME/.con‐
+              fig/yadm/encrypt.  See the ENCRYPTION section for more details.
 
        enter  Run  a  sub-shell with all Git variables set. Exit the sub-shell
               the same way you leave  your  normal  shell  (usually  with  the
               "exit"  command).  This sub-shell can be used to easily interact
               with your yadm repository using "git" commands.  This  could  be
-              useful  if  you  are  using  a tool which uses Git directly. For
-              example, Emacs Tramp and Magit can manage files  by  using  this
-              configuration:
+              useful  if you are using a tool which uses Git directly, such as
+              tig, vim-fugitive, git-cola, etc.
+
+              Optionally, you can provide a command after "enter", and instead
+              of invoking your shell, that command will be run with all of the
+              Git variables exposed to the command's environment.
+
+              Emacs Tramp and Magit can manage files by using this  configura‐
+              tion:
+
                   (add-to-list 'tramp-methods
                        '("yadm"
                          (tramp-login-program "yadm")
                          (tramp-login-args (("enter")))
+                         (tramp-login-env (("SHELL") ("/bin/sh")))
                          (tramp-remote-shell "/bin/sh")
                          (tramp-remote-shell-args ("-c"))))
 
+              With this config, use (magit-status "/yadm::").
+
+       git-crypt options
+              If  git-crypt  is installed, this command allows you to pass op‐
+              tions directly to git-crypt, with the environment configured  to
+              use the yadm repository.
+
+              git-crypt enables transparent encryption and decryption of files
+              in a git repository.  You can read  https://github.com/AGWA/git-
+              crypt for details.
+
        gitconfig
               Pass  options to the git config command. Since yadm already uses
-              the config command to manage its own configurations,  this  com-
-              mand is provided as a way to change configurations of the repos-
+              the config command to manage its own configurations,  this  com‐
+              mand is provided as a way to change configurations of the repos‐
               itory managed by yadm.  One useful case might  be  to  configure
               the  repository so untracked files are shown in status commands.
               yadm initially configures its repository so that untracked files
               are  not  shown.   If  you wish use the default Git behavior (to
-              show untracked files and directories), you can remove this  con-
+              show untracked files and directories), you can remove this  con‐
               figuration.
 
                      yadm gitconfig --unset status.showUntrackedFiles
@@ -142,7 +154,7 @@
        help   Print a summary of yadm commands.
 
        init   Initialize  a  new, empty repository for tracking dotfiles.  The
-              repository  is  stored   in   $HOME/.config/yadm/repo.git.    By
+              repository is stored  in  $HOME/.local/share/yadm/repo.git.   By
               default,  $HOME  will  be used as the work-tree, but this can be
               overridden with the -w option.  yadm can be forced to  overwrite
               an existing repository by providing the -f option.
@@ -157,69 +169,69 @@
               support command line completion.
 
        perms  Update  permissions as described in the PERMISSIONS section.  It
-              is usually unnecessary to run this command,  as  yadm  automati-
+              is usually unnecessary to run this command,  as  yadm  automati‐
               cally  processes permissions by default. This automatic behavior
               can be disabled by setting the configuration yadm.auto-perms  to
               "false".
 
-       upgrade
-              Version  2  of  yadm uses a different directory for storing your
-              configurations.  When you start to use version 2 for  the  first
-              time,  you  may  see warnings about moving your data to this new
-              directory.  The easiest way to accomplish  this  is  by  running
-              "yadm  upgrade".   This  command  will start by moving your yadm
-              repo to the new path.  Next it will move any configuration  data
-              to  the new path.  If the configurations are tracked within your
-              yadm repo, this command will "stage" the renaming of those files
-              in the repo's index.  Upgrading will also re-initialize all sub-
-              modules you have added (otherwise they will be broken  when  the
-              repo moves).  After running "yadm upgrade", you should run "yadm
-              status" to review changes which have  been  staged,  and  commit
-              them to your repository.
+       transcrypt options
+              If  transcrypt is installed, this command allows you to pass op‐
+              tions directly to transcrypt, with the environment configured to
+              use the yadm repository.
 
-              You can read https://yadm.io/docs/upgrade_from_1 for more infor-
+              transcrypt  enables  transparent  encryption  and  decryption of
+              files    in    a    git    repository.      You     can     read
+              https://github.com/elasticdog/transcrypt for details.
+
+       upgrade
+              Version  3  of yadm uses a different directory for storing data.
+              When you start to use version 3 for the first time, you may  see
+              warnings about moving your data to this new directory.  The eas‐
+              iest way to accomplish this is by running "yadm upgrade".   This
+              command  will  start  by  moving your yadm repo to the new path.
+              Next it will move any archive data.  If the archive  is  tracked
+              within your yadm repo, this command will "stage" the renaming of
+              that file in the repo's index.
+
+              Upgrading will attempt to de-initialize and  re-initialize  your
+              submodules. If your submodules cannot be de-initialized, the up‐
+              grade will fail. The most common reason submodules will fail  to
+              de-initialize  is  because they have local modifications. If you
+              are willing to lose the local modifications to those submodules,
+              you  can  use  the -f option with the "upgrade" command to force
+              the de-initialization.
+
+              After running "yadm upgrade", you should run  "yadm  status"  to
+              review  changes  which have been staged, and commit them to your
+              repository.
+
+              You can read https://yadm.io/docs/upgrade_from_2 for more infor‐
               mation.
 
        version
               Print the version of yadm.
 
 
-## COMPATIBILITY
-       Beginning with version 2.0.0, yadm introduced a  couple  major  changes
-       which  may  require you to adjust your configurations.  See the upgrade
-       command for help making those adjustments.
-
-       First, yadm now uses the "XDG Base Directory Specification" to find its
-       configurations.  You  can  read https://yadm.io/docs/upgrade_from_1 for
-       more information.
-
-       Second, the naming conventions for alternate files have  been  changed.
-       You can read https://yadm.io/docs/alternates for more information.
-
-       If you want to retain the old functionality, you can set an environment
-       variable, YADM_COMPATIBILITY=1.  Doing so will  automatically  use  the
-       old  yadm  directory,  and process alternates the same as the pre-2.0.0
-       version. This compatibility mode is deprecated, and will be removed  in
-       future  versions.  This mode exists solely for transitioning to the new
-       paths and naming of alternates.
-
-
 ## OPTIONS
-       yadm supports a set of universal options that alter the paths it  uses.
-       The  default paths are documented in the FILES section. Any path speci-
-       fied by these options must be fully qualified. If you  always  want  to
-       override  one  or  more  of  these paths, it may be useful to create an
-       alias for the yadm command.  For example, the following alias could  be
+       yadm  supports a set of universal options that alter the paths it uses.
+       The default paths are documented in the FILES section. Any path  speci‐
+       fied  by  these  options must be fully qualified. If you always want to
+       override one or more of these paths, it may  be  useful  to  create  an
+       alias  for the yadm command.  For example, the following alias could be
        used to override the repository directory.
 
               alias yadm='yadm --yadm-repo /alternate/path/to/repo'
 
-       The  following  is  the  full  list  of universal options.  Each option
-       should be followed by a fully qualified path.
+       The following is the full  list  of  universal  options.   Each  option
+       should be followed by a path.
 
        -Y,--yadm-dir
-              Override the yadm directory.  yadm stores its data  relative  to
-              this directory.
+              Override  the  yadm  directory.   yadm stores its configurations
+              relative to this directory.
+
+       --yadm-data
+              Override the yadm data directory.  yadm stores its data relative
+              to this directory.
 
        --yadm-repo
               Override the location of the yadm repository.
@@ -249,65 +261,90 @@
        The following is the full list of supported configurations:
 
        yadm.alt-copy
-              If set to "true", alternate files will be copies instead of sym-
+              If set to "true", alternate files will be copies instead of sym‐
               bolic links.  This might be desirable, because some systems  may
               not properly support symlinks.
 
-              NOTE:  The deprecated yadm.cygwin-copy option used by older ver-
-              sions of yadm has  been  replaced  by  yadm.alt-copy.   The  old
-              option will be removed in the next version of yadm.
-
        yadm.auto-alt
-              Disable  the  automatic  linking described in the section ALTER-
+              Disable  the  automatic  linking described in the section ALTER‐
               NATES. If disabled, you may still run  "yadm  alt"  manually  to
-              create  the alternate links. This feature is enabled by default.
+              create the alternate links. This feature is enabled by default.
 
        yadm.auto-exclude
-              Disable  the  automatic  exclusion  of   patterns   defined   in
+              Disable   the   automatic   exclusion  of  patterns  defined  in
               $HOME/.config/yadm/encrypt.  This feature is enabled by default.
 
        yadm.auto-perms
-              Disable the automatic permission changes described in  the  sec-
+              Disable the automatic permission changes described in  the  sec‐
               tion  PERMISSIONS.   If  disabled,  you may still run yadm perms
-              manually to update permissions.   This  feature  is  enabled  by
-              default.
+              manually to update permissions.  This feature is enabled by  de‐
+              fault.
 
        yadm.auto-private-dirs
               Disable  the automatic creating of private directories described
               in the section PERMISSIONS.
 
+       yadm.cipher
+              Configure which encryption system is used by the encrypt/decrypt
+              commands.  Valid options are "gpg" and "openssl". The default is
+              "gpg".  Detailed information can be found in the section ENCRYP‐
+              TION.
+
        yadm.git-program
-              Specify an alternate  program  to  use  instead  of  "git".   By
-              default, the first "git" found in $PATH is used.
+              Specify  an  alternate  program to use instead of "git".  By de‐
+              fault, the first "git" found in $PATH is used.
 
        yadm.gpg-perms
-              Disable  the permission changes to $HOME/.gnupg/*.  This feature
+              Disable the permission changes to $HOME/.gnupg/*.  This  feature
               is enabled by default.
 
        yadm.gpg-program
-              Specify an alternate  program  to  use  instead  of  "gpg".   By
-              default, the first "gpg" found in $PATH is used.
+              Specify  an  alternate  program to use instead of "gpg".  By de‐
+              fault, the first "gpg" found in $PATH is used.
 
        yadm.gpg-recipient
               Asymmetrically encrypt files with a gpg public/private key pair.
-              Provide a "key ID" to specify which public key to encrypt  with.
-              The  key  must  exist in your public keyrings.  If left blank or
-              not provided, symmetric encryption is used instead.  If  set  to
-              "ASK",  gpg  will  interactively  ask  for  recipients.  See the
-              ENCRYPTION section for more details.  This feature  is  disabled
-              by default.
+              Provide  a "key ID" to specify which public key to encrypt with.
+              The key must exist in your public keyrings.  Multiple recipients
+              can  be  specified  (separated  by space).  If left blank or not
+              provided, symmetric encryption  is  used  instead.   If  set  to
+              "ASK",  gpg  will interactively ask for recipients.  See the EN‐
+              CRYPTION section for more details.  This feature is disabled  by
+              default.
+
+       yadm.openssl-ciphername
+              Specify  which  cipher should be used by openssl.  "aes-256-cbc"
+              is used by default.
+
+       yadm.openssl-old
+              Newer versions of openssl  support  the  pbkdf2  key  derivation
+              function.  This is used by default. If this configuration is set
+              to "true", openssl operations will use options  compatible  with
+              older  versions  of openssl. If you change this option, you will
+              need to recreate your encrypted archive.
+
+       yadm.openssl-program
+              Specify an alternate program to use instead  of  "openssl".   By
+              default, the first "openssl" found in $PATH is used.
 
        yadm.ssh-perms
               Disable the permission changes to $HOME/.ssh/*.  This feature is
               enabled by default.
 
-       The following  four  "local"  configurations  are  not  stored  in  the
+       The following  five  "local"  configurations  are  not  stored  in  the
        $HOME/.config/yadm/config, they are stored in the local repository.
 
 
        local.class
               Specify  a  class for the purpose of symlinking alternate files.
-              By default, no class will be matched.
+              By default, no class will be matched.  The local host can be as‐
+              signed multiple classes using command:
+
+              yadm config --add local.class <additional-class>
+
+       local.arch
+              Override  the  architecture for the purpose of symlinking alter‐
+              nate files.
 
        local.hostname
               Override the hostname for the purpose  of  symlinking  alternate
@@ -325,55 +362,72 @@
        to have an automated way of choosing an alternate version of a file for
        a different operating system, host, user, etc.
 
-       yadm will automatically create a symbolic link to the appropriate  ver-
-       sion  of  a  file, when a valid suffix is appended to the filename. The
-       suffix contains the conditions that must be met for  that  file  to  be
+       yadm  will automatically create a symbolic link to the appropriate ver‐
+       sion of a file, when a valid suffix is appended to  the  filename.  The
+       suffix  contains  the  conditions  that must be met for that file to be
        used.
 
-       The suffix begins with "##", followed by any number of conditions sepa-
+       The suffix begins with "##", followed by any number of conditions sepa‐
        rated by commas.
 
          ##<condition>[,<condition>,...]
 
-       Each condition is an attribute/value pair, separated by a period.  Some
-       conditions  do  not require a "value", and in that case, the period and
-       value can be omitted. Most attributes can be abbreviated  as  a  single
+       Each  condition is an attribute/value pair, separated by a period. Some
+       conditions do not require a "value", and in that case, the  period  and
+       value  can  be  omitted. Most attributes can be abbreviated as a single
        letter.
 
          <attribute>[.<value>]
 
-       These are the supported attributes, in the order of the weighted prece-
+       These are the supported attributes, in the order of the weighted prece‐
        dence:
 
 
        template, t
-              Valid when the value matches  a  supported  template  processor.
+              Valid  when  the  value  matches a supported template processor.
               See the TEMPLATES section for more details.
 
        user, u
-              Valid  if  the  value matches the current user.  Current user is
+              Valid if the value matches the current user.   Current  user  is
               calculated by running id -u -n.
 
-       distro, d
-              Valid if the value matches the distro.  Distro is calculated  by
-              running lsb_release -si.
-
-       os, o  Valid  if the value matches the OS.  OS is calculated by running
-              uname -s.
+       hostname, h
+              Valid if the value matches the short hostname.  Hostname is cal‐
+              culated by running uname -n, and trimming off any domain.
 
        class, c
               Valid if the value matches the local.class configuration.  Class
               must be manually set using yadm config local.class <class>.  See
-              the  CONFIGURATION  section  for  more  details  about   setting
-              local.class.
+              the CONFIGURATION section for more  details  about  setting  lo‐
+              cal.class.
 
-       hostname, h
-              Valid if the value matches the short hostname.  Hostname is cal-
-              culated by running hostname, and trimming off any domain.
+       distro, d
+              Valid  if the value matches the distro.  Distro is calculated by
+              running lsb_release -si or by inspecting the ID from /etc/os-re‐
+              lease.
+
+       distro_family, f
+              Valid  if the value matches the distro family.  Distro family is
+              calculated by inspecting the ID_LIKE line from /etc/os-release.
+
+       os, o  Valid if the value matches the OS.  OS is calculated by  running
+              uname -s.
+
+       arch, a
+              Valid  if  the  value matches the architecture.  Architecture is
+              calculated by running uname -m.
 
        default
               Valid when no other alternate is valid.
 
+       extension, e
+              A special "condition" that doesn't affect the selection process.
+              Its purpose is instead to allow the alternate file to end with a
+              certain extension to e.g. make  editors  highlight  the  content
+              properly.
+
+       NOTE:  The  OS  for "Windows Subsystem for Linux" is reported as "WSL",
+       even though uname identifies as "Linux".
 
        You may use any number of conditions, in any order.  An alternate  will
        only  be  used  if  ALL conditions are valid.  For all files managed by
@@ -384,15 +438,15 @@
        The "most appropriate" version is determined by calculating a score for
        each  version  of  a  file. A template is always scored higher than any
        symlink condition. The number of conditions is the next largest  factor
-       in  scoring.   Files  with  more conditions will always be favored. Any
-       invalid condition will disqualify that file completely.
+       in scoring.  Files with more conditions will always be favored. Any in‐
+       valid condition will disqualify that file completely.
 
        If you don't care to have all versions of alternates stored in the same
        directory  as  the  generated  symlink,  you  can  place  them  in  the
        $HOME/.config/yadm/alt directory. The generated  symlink  or  processed
        template will be created using the same relative path.
 
-       Alternate  linking may best be demonstrated by example. Assume the fol-
+       Alternate  linking may best be demonstrated by example. Assume the fol‐
        lowing files are managed by yadm's repository:
 
          - $HOME/path/example.txt##default
@@ -407,7 +461,7 @@
        If running on a Macbook named "host2", yadm will create a symbolic link
        which looks like this:
 
-       $HOME/path/example.txt    ->    $HOME/path/example.txt##os.Darwin,host-
+       $HOME/path/example.txt    ->    $HOME/path/example.txt##os.Darwin,host‐
        name.host2
 
        However, on another Mackbook named "host3", yadm will create a symbolic
@@ -434,9 +488,9 @@
        then no link will be created.
 
        Links  are also created for directories named this way, as long as they
-       have at least one yadm managed file within them.
+       have at least one yadm managed file within them (at the top level).
 
-       yadm will automatically create these links by default. This can be dis-
+       yadm will automatically create these links by default. This can be dis‐
        abled  using  the yadm.auto-alt configuration.  Even if disabled, links
        can be manually created by running yadm alt.
 
@@ -448,9 +502,9 @@
 
          yadm config local.class Work
 
-       Similarly, the values of os, hostname, and user can be  manually  over-
-       ridden  using  the  configuration options local.os, local.hostname, and
-       local.user.
+       Similarly, the values of architecture, os, hostname, and  user  can  be
+       manually  overridden  using  the  configuration options local.arch, lo‐
+       cal.os, local.hostname, and local.user.
 
 
 ## TEMPLATES
@@ -462,64 +516,95 @@
 
        default
               This is yadm's built-in template processor.  This  processor  is
-              very basic, with a Jinja-like syntax. The advantage of this pro-
+              very basic, with a Jinja-like syntax. The advantage of this pro‐
               cessor is that it only depends upon awk, which is  available  on
               most  *nix  systems. To use this processor, specify the value of
               "default" or just leave the value off (e.g. "##template").
 
-       j2cli  To use the j2cli Jinja template processor, specify the value  of
+       ESH    ESH is a template processor written in POSIX compliant shell. It
+              allows  executing  shell  commands within templates. This can be
+              used to reference your own configurations within templates,  for
+              example:
+
+                <% yadm config mysection.myconfig %>
+
+              To use the ESH template processor, specify the value of "esh"
+
+       j2cli  To  use the j2cli Jinja template processor, specify the value of
               "j2"  or "j2cli".
 
        envtpl To use the envtpl Jinja template processor, specify the value of
               "j2" or "envtpl".
 
+       NOTE: Specifying "j2" as the processor will attempt to use j2cli or en‐
+       vtpl, whichever is available.
 
-       NOTE: Specifying "j2" as the processor will attempt  to  use  j2cli  or
-       envtpl, whichever is available.
-
-       If  the  template  processor  specified is available, templates will be
+       If the template processor specified is  available,  templates  will  be
        processed to create or overwrite files.
 
-       During processing, the following variables are available  in  the  tem-
+       During  processing,  the  following variables are available in the tem‐
        plate:
 
-        Default         Jinja           Description
-        -------------   -------------   --------------------------
-        yadm.class      YADM_CLASS      Locally defined yadm class
-        yadm.distro     YADM_DISTRO     lsb_release -si
-        yadm.hostname   YADM_HOSTNAME   hostname (without domain)
-        yadm.os         YADM_OS         uname -s
-        yadm.user       YADM_USER       id -u -n
-        yadm.source     YADM_SOURCE     Template filename
+        Default              Jinja or ESH         Description
+        -------------        -------------        ----------------------------
+        yadm.arch            YADM_ARCH            uname -m
+        yadm.class           YADM_CLASS           Last locally defined class
+        yadm.classes         YADM_CLASSES         All classes
+        yadm.distro          YADM_DISTRO          lsb_release -si
+        yadm.distro_family   YADM_DISTRO_FAMILY   ID_LIKE from /etc/os-release
+        yadm.hostname        YADM_HOSTNAME        uname -n (without domain)
+        yadm.os              YADM_OS              uname -s
+        yadm.source          YADM_SOURCE          Template filename
+        yadm.user            YADM_USER            id -u -n
+        env.VAR                                   Environment variable VAR
+
+       NOTE: The OS for "Windows Subsystem for Linux" is  reported  as  "WSL",
+       even though uname identifies as "Linux".
+
+       NOTE:  If lsb_release is not available, DISTRO will be the ID specified
+       in /etc/os-release.
 
        Examples:
 
        whatever##template with the following content
 
-         {% if yadm.user == 'harvey' %}
+         {% if yadm.user == "harvey" %}
          config={{yadm.class}}-{{yadm.os}}
          {% else %}
          config=dev-whatever
+         {% include "whatever.extra" %}
          {% endif %}
 
-       would  output  a  file named whatever with the following content if the
+       would output a file named whatever with the following  content  if  the
        user is "harvey":
 
          config=work-Linux
 
-       and the following otherwise:
+       and the following otherwise (if whatever.extra contains admin=false):
 
          config=dev-whatever
+         admin=false
 
-       An equivalent Jinja template  named  whatever##template.j2  would  look
+       An  equivalent  Jinja  template  named whatever##template.j2 would look
        like:
 
          {% if YADM_USER == 'harvey' -%}
          config={{YADM_CLASS}}-{{YADM_OS}}
          {% else -%}
          config=dev-whatever
+         {% include 'whatever.extra' %}
          {% endif -%}
 
+       An equivalent ESH templated  named  whatever##template.esh  would  look
+       like:
+
+         <% if [ "$YADM_USER" = "harvey" ]; then -%>
+         config=<%= $YADM_CLASS %>-<%= $YADM_OS %>
+         <% else -%>
+         config=dev-whatever
+         <%+ whatever.extra %>
+         <% fi -%>
+
 
 ## ENCRYPTION
        It  can  be  useful to manage confidential files, like SSH or GPG keys,
@@ -527,7 +612,9 @@
        into a Git repository, which often resides on a public system. yadm can
        make it easy to encrypt and decrypt a set of  files  so  the  encrypted
        version  can  be  maintained  in the Git repository.  This feature will
-       only work if the gpg(1) command is available.
+       only work if a supported tool is available.  Both gpg(1) and openssl(1)
+       are  supported.   gpg is used by default, but openssl can be configured
+       with the yadm.cipher configuration.
 
        To use this feature, a list of patterns must be created  and  saved  as
        $HOME/.config/yadm/encrypt.   This  list of patterns should be relative
@@ -536,41 +623,53 @@
                   .ssh/*.key
                   .gnupg/*.gpg
 
-       Standard filename expansions (*,  ?,  [)  are  supported.  Other  shell
-       expansions  like brace and tilde are not supported. Spaces in paths are
-       supported, and should not be quoted. If a directory is  specified,  its
-       contents  will be included, but not recursively. Paths beginning with a
+       Standard filename expansions (*, ?, [) are supported.  If you have Bash
+       version  4,  you may use "**" to match all subdirectories.  Other shell
+       expansions like brace and tilde are not supported.  Spaces in paths are
+       supported,  and should not be quoted.  If a directory is specified, its
+       contents will be included, but not recursively.  Paths beginning with a
        "!" will be excluded.
 
        The yadm encrypt command will find all files matching the patterns, and
-       prompt  for  a  password.  Once  a password has confirmed, the matching
-       files will be encrypted and saved as $HOME/.config/yadm/files.gpg.  The
-       patterns  and  files.gpg should be added to the yadm repository so they
-       are available across multiple systems.
+       prompt for a password. Once a  password  has  confirmed,  the  matching
+       files  will  be encrypted and saved as $HOME/.local/share/yadm/archive.
+       The "encrypt" and "archive" files should be added to the  yadm  reposi‐
+       tory so they are available across multiple systems.
 
        To decrypt these files later, or on another system run yadm decrypt and
-       provide  the  correct password.  After files are decrypted, permissions
+       provide the correct password.  After files are  decrypted,  permissions
        are automatically updated as described in the PERMISSIONS section.
 
-       Symmetric encryption is used by default, but asymmetric encryption  may
+       Symmetric  encryption is used by default, but asymmetric encryption may
        be enabled using the yadm.gpg-recipient configuration.
 
-       NOTE:  It is recommended that you use a private repository when keeping
+       NOTE: It is recommended that you use a private repository when  keeping
        confidential files, even though they are encrypted.
 
        Patterns found in $HOME/.config/yadm/encrypt are automatically added to
-       the  repository's  info/exclude  file  every  time yadm encrypt is run.
-       This is to prevent accidentally committing sensitive data to the repos-
+       the repository's info/exclude file every  time  yadm  encrypt  is  run.
+       This is to prevent accidentally committing sensitive data to the repos‐
        itory.  This can be disabled using the yadm.auto-exclude configuration.
 
+       Using transcrypt or git-crypt
+
+       A completely separate option for encrypting data is to install and  use
+       transcrypt  or  git-crypt.   Once installed, you can use these tools by
+       running yadm transcrypt or yadm git-crypt.  These tools enables  trans‐
+       parent  encryption and decryption of files in a git repository. See the
+       following web sites for more information:
+
+       - https://github.com/elasticdog/transcrypt
+
+       - https://github.com/AGWA/git-crypt
 
 ## PERMISSIONS
-       When files are checked out of a Git repository, their  initial  permis-
+       When files are checked out of a Git repository, their  initial  permis‐
        sions  are  dependent upon the user's umask. Because of this, yadm will
        automatically update the permissions of some file  paths.  The  "group"
        and "others" permissions will be removed from the following files:
 
-       - $HOME/.config/yadm/files.gpg
+       - $HOME/.local/share/yadm/archive
 
        - All files matching patterns in $HOME/.config/yadm/encrypt
 
@@ -578,11 +677,11 @@
 
        - The GPG directory and files, .gnupg/*
 
-       yadm will automatically update permissions by default. This can be dis-
-       abled using the yadm.auto-perms configuration. Even if  disabled,  per-
-       missions  can  be  manually  updated  by  running yadm perms.  The .ssh
-       directory processing can be disabled using the yadm.ssh-perms  configu-
-       ration.  The  .gnupg  directory  processing  can  be disabled using the
+       yadm will automatically update permissions by default. This can be dis‐
+       abled using the yadm.auto-perms configuration. Even if  disabled,  per‐
+       missions  can  be manually updated by running yadm perms.  The .ssh di‐
+       rectory processing can be disabled using the yadm.ssh-perms  configura‐
+       tion.  The  .gnupg  directory  processing  can  be  disabled  using the
        yadm.gpg-perms configuration.
 
        When cloning a repo which includes data in a .ssh or .gnupg  directory,
@@ -592,18 +691,18 @@
 
        When running a Git command and .ssh or .gnupg directories do not exist,
        yadm will create those directories with mask 0700 prior to running  the
-       Git command. This can be disabled using the yadm.auto-private-dirs con-
+       Git command. This can be disabled using the yadm.auto-private-dirs con‐
        figuration.
 
 
 ## HOOKS
-       For every command yadm supports, a  program  can  be  provided  to  run
-       before  or  after  that command. These are referred to as "hooks". yadm
+       For every command yadm supports, a program can be provided to  run  be‐
+       fore  or  after  that  command.  These are referred to as "hooks". yadm
        looks for hooks in the directory $HOME/.config/yadm/hooks.   Each  hook
        is named using a prefix of pre_ or post_, followed by the command which
-       should trigger the hook. For example, to create a  hook  which  is  run
-       after  every  yadm  pull command, create a hook named post_pull.  Hooks
-       must have the executable file permission set.
+       should trigger the hook. For example, to create a hook which is run af‐
+       ter every yadm pull command, create a hook named post_pull.  Hooks must
+       have the executable file permission set.
 
        If a pre_ hook is defined, and the hook terminates with a non-zero exit
        status,  yadm  will  refuse  to run the yadm command. For example, if a
@@ -621,7 +720,9 @@
               The exit status of the yadm command
 
        YADM_HOOK_FULL_COMMAND
-              The yadm command with all command line arguments
+              The yadm command with all command line arguments (parameters are
+              space delimited, and any space, tab or backslash will be escaped
+              with a backslash)
 
        YADM_HOOK_REPO
               The path to the yadm repository
@@ -632,35 +733,45 @@
 
 ## FILES
        All  of  yadm's  configurations  are  relative to the "yadm directory".
-       yadm uses the "XDG Base  Directory  Specification"  to  determine  this
-       directory.   If the environment variable $XDG_CONFIG_HOME is defined as
-       a fully qualified path, this directory will  be  $XDG_CONFIG_HOME/yadm.
+       yadm uses the "XDG Base Directory Specification" to determine this  di‐
+       rectory.   If the environment variable $XDG_CONFIG_HOME is defined as a
+       fully qualified path, this  directory  will  be  $XDG_CONFIG_HOME/yadm.
        Otherwise it will be $HOME/.config/yadm.
 
+       Similarly, yadm's data files are relative to the "yadm data directory".
+       yadm uses the "XDG Base Directory Specification" to determine this  di‐
+       rectory.   If  the  environment variable $XDG_DATA_HOME is defined as a
+       fully qualified path, this directory will be $XDG_DATA_HOME/yadm.  Oth‐
+       erwise it will be $HOME/.local/share/yadm.
+
        The  following  are the default paths yadm uses for its own data.  Most
        of these paths can be altered using universal options.  See the OPTIONS
        section for details.
 
        $HOME/.config/yadm
-              The yadm directory. By default, all data yadm stores is relative
-              to this directory.
+              The yadm directory. By default, all configs yadm stores is rela‐
+              tive to this directory.
+
+       $HOME/.local/share/yadm
+              The yadm data directory. By default, all  data  yadm  stores  is
+              relative to this directory.
 
        $YADM_DIR/config
               Configuration file for yadm.
 
        $YADM_DIR/alt
-              This is a directory to keep  "alternate  files"  without  having
-              them  side-by-side  with the resulting symlink or processed tem-
-              plate. Alternate files placed in this directory will be  created
+              This  is  a  directory  to keep "alternate files" without having
+              them side-by-side with the resulting symlink or  processed  tem‐
+              plate.  Alternate files placed in this directory will be created
               relative to $HOME instead.
 
-       $YADM_DIR/repo.git
+       $YADM_DATA/repo.git
               Git repository used by yadm.
 
        $YADM_DIR/encrypt
               List of globs used for encrypt/decrypt
 
-       $YADM_DIR/files.gpg
+       $YADM_DATA/archive
               All files encrypted with yadm encrypt are stored in this file.
 
 
@@ -680,7 +791,7 @@
        echo .ssh/*.key >> $HOME/.config/yadm/encrypt
               Add a new pattern to the list of encrypted files
 
-       yadm encrypt ; yadm add ~/.config/yadm/files.gpg ; yadm commit
+       yadm encrypt ; yadm add ~/.local/share/yadm/archive ; yadm commit
               Commit a new set of encrypted files
 
 
@@ -695,7 +806,7 @@
 
 
 ## SEE ALSO
-       git(1), gpg(1)
+       git(1), gpg(1) openssl(1) transcrypt(1) git-crypt(1)
 
        https://yadm.io/
 

yadm.spec

@@ -1,18 +1,13 @@
 %{!?_pkgdocdir: %global _pkgdocdir %{_docdir}/%{name}-%{version}}
 Name: yadm
 Summary: Yet Another Dotfiles Manager
-Version: 2.0.1
+Version: 3.2.2
 Group: Development/Tools
 Release: 1%{?dist}
 URL: https://yadm.io
 License: GPL-3.0-only
 Requires: bash
 Requires: git
-%if 0%{?fedora} || 0%{?rhel_version} || 0%{?centos_version} >= 700
-Requires: /usr/bin/hostname
-%else
-Requires: /bin/hostname
-%endif
 
 Source: %{name}.tar.gz
 BuildRoot: %{_tmppath}/%{name}-%{version}-build
@@ -34,7 +29,7 @@ encrypted before they are included in the repository.
 
 # this is done to allow paths other than yadm-x.x.x (for example, when building
 # from branches instead of release tags)
-cd *yadm-*
+test -f yadm || cd *yadm-*
 
 %{__mkdir} -p %{buildroot}%{_bindir}
 %{__cp}  yadm %{buildroot}%{_bindir}