lxsetup

lxsetup is a CLI to help you run all the steps of a new Linux machine setup, based on a config file lxsetup.json, some programs lists and a few custom scripts. It helps to see the progress of installation, run only broken or new steps, and run tests related to each step.

DISCLAIMER: This project is in Documentation Driven Development and Test Driven Development, so the CLI itself is not implemented yet.... First I define the JSON format, secondly I write the documentation, then I write feature and unit tests, and finally I can implement the CLI.

Project status

• 01.09.2022: The JSON format is almost ready and the documentation too (it would need some feedbacks actually to make adjustments)... I just started to write the first tests and the basics things to implement in this CLI. (It's the first time I build a CLI, in Bash, with automated testing in Bash... so I'm learning along the way and refactoring things).
• 05.09.2022: I gave up on developing in Bash. I'm not very used to Bash and it seems to be a huge mess to read and manage tons of values from a JSON file... So, I switch to PHP (even if it will add a dependency to use the tool). I've made good progress with Start.php and Helper.php and lxsetup. I'm happy with the switch, I'm so more productive. I'm experiencing developing it in vanilla PHP in OOP, it's fun.

Table of content

• lxsetup
  • Core concepts
    • Files structure
    • Files content examples
    • How to create your setup repository
  • How to use lxsetup ?
    • Install the CLI
    • Setup the machine the first time
    • Update the machine
    • Run all tests
    • Edit the setup
    • Run a specific step
    • Display the help
    • Display the changelog
  • General rules
  • lxsetup.json file syntax
    • Writing scripts
    • Writing steps
    • Writing tests
    • Writing extra tests
    • Writing requirements
    • Writing conditions
  • How to change lxsetup itself ?
    • Setup the project
    • Run the tests
  • Conclusion
    • License

Core concepts

lxsetup need to access a setup repository (in this document we call it in a very original way setup). This repos contains the whole logic of the installation of your computer.

Files structure

Example of a simple structure of setup repository:

setup
├── apps
│   ├── firefox-dev.sh
│   ├── php.sh
│   └── virtualbox.sh
├── lists
│   ├── flatpak.list
│   └── snap.list
├── lxsetup.json
└── system
    ├── clone-repos.sh
    ├── drivers.sh
    └── import-config.sh

• apps folder contains custom scripts to install applications (think about apps that can't be installed in a single command like dnf install ffmpeg).
• lists folder contains lists of programs IDs (files extensions must be .list)
• lxsetup.json: the configuration file of this setup. It contains metadata, all the steps and tests for our setup.
• system folder contains custom scripts you wrote to setup things on your system (like drivers, files, dotfiles, ...). Same idea as apps but separated for easier managment.

Files content examples

App script example: apps/virtualbox.sh

#!/bin/bash
## Description: install VirtualBox 6.1 from the Oracle RPM repository

## Virtualbox
sudo rpm --import https://www.virtualbox.org/download/oracle_vbox.asc
sudo dnf config-manager --add-repo https://download.virtualbox.org/virtualbox/rpm/fedora/virtualbox.repo
sudo dnf update -y
sudo dnf install -y kernel-devel-$(uname -r) kernel-headers
sudo dnf install -y VirtualBox-6.1

Package managers programs list example: list/flatpak.list

# List of programs installed with Flatpak
## Multimedia
org.gimp.GIMP
com.obsproject.Studio
org.darktable.Darktable

## Productivity
org.freefilesync.FreeFileSync
org.videolan.VLC

System script example: system/git-config.sh

#!/bin/bash
# Configure global Git configuration with my default identity
git config --global init.defaultBranch main
git config --global user.name "Samuel Roland"
git config --global user.email samuelroland@noreply.codeberg.org

The configuration file: lxsetup.json (read comments to understand values)

{
	//Setup info
	"name": "Fedora 36 setup",
	"author": "Samuel Roland",
	//Steps to run in the given order
	"steps": {

		//Step of type "command" to run a single command (see "command")
		"add-flathub": {
			"type": "command",
			"description": "Adding the Flathub repository to Flatpak",
			"command": "sudo flatpak remote-add flathub https://flathub.org/repo/flathub.flatpakrepo",
			
			//We write a test of this operation 
			// so we can know if successful or not
			// (the default "assert" value is "success"
			// -> it asserts that the command has a success exit code)
			"tests": [{
				"description": "Make sure Flatpak remotes repos contains Flathub",
				"given": "flatpak remotes | grep flathub"
			}]
		},

		//Another step of type "package-manager" to install all programs
		// from a specific package manager
		"flatpak-packages": {
			"type": "package-manager",
			"description": "Flatpak packages from Flathub",
			"prefix": "flatpak install -y flathub",
			"list": "flatpak",

			//This is the command to generate the list of installed packages and 
			// to know if all packages are installed. It's like a dynamic list of tests.
			"installed": "flatpak list"
		},

		//Step of type "script" to run a script 
		// (here a custom script in the "apps" folder)
		"virtualbox": {
			"type": "script",
			"file": "virtualbox",
			"description": "Install Virtualbox 6.1",
			"tests": [{
				"description": "Check Virtualbox is installed",
				"given": "virtualbox -h"
			}]
		}
	},

	//Some extra tests
	"extra-tests": {
		"check-firefox-installed": {
			"description": "Check default Firefox is already installed",
			"given": "firefox -v"
		}
	}
}

How to create your setup repository

1. Create a repository to store the elements of your setup
2. Create a file lxsetup.json with this base:

   {
   	"name": "tbd",
   	"author": "tbd",
   	"steps": {
   		"step id": {
   			"type": "script",
   			"file": "<script file name>",
   			"description": "Step description",
   			"tests": [{
   				"description": "Test description",
   				"given": "test command"
   			}]
   		}
   	},
   	"extra-tests": {
   		"test id": {
   			"description": "Test description",
   			"given": "test command"
   		}
   	}
   }
   

3. Create an apps folder and import all your custom scripts to install your apps
4. Create a system folder and import all your custom scripts that change your system
5. Create a lists folder and create 1 file per package manager (with the lists of IDs of each package, like in the example)
6. In lxsetup.json
   1. Define name, author
   2. Define all the steps that need to be executed to install your computer
   3. Define extra tests if needed
7. You're ready to start using lxsetup

How to use lxsetup ?

In your terminal, the current folder must be your setup repos.

Install the CLI

TODO: this step is not defined for the moment because I don't know how I will distribute/publish the CLI. Any idea or best practice? please let me know <3

Setup the machine the first time

The first time you use lxsetup on a machine, you can use this command in the setup folder. It will execute all the steps without looking at what's already setup or not.

lxsetup start

The start commands execute these actions:

1. Make sure lxsetup.lock doesn't exists, if exists display a message and exit.
2. Read lxsetup.json in the current folder and make sure the file is valid
3. Run all the tests of each step to know which steps should be runned.
4. Print the 3 categories of steps (already valid, to run, or ignored steps)
5. Ask for confirmation
6. When confirmed, all steps to run are launched, and the progress and errors are displayed.
7. Run tests of the and display final statistics and state of your computer.

Update the machine

Later on, we will definitly change our setup with new programs, scripts, changes or even removed steps. This option let you easily update a machine with the last changes.

lxsetup update

The update commands execute these actions:

1. Make sure lxsetup.lock exists, if not, display a message and exit.
2. Pull commits in this repository
3. Run tests to know the state of steps
4. Based on lxsetup.lock, define what steps or packages have been added or removed.
5. Display all steps to run, separated in 3 categories: remove steps, added steps, and not valid steps (but already runned in the past)
6. Ask for confirmation
7. When confirmed, all steps to run are launched
8. Run tests of the and display final statistics and state of your computer.

Run all tests

Run all tests in step and extra tests and show the result.

lxsetup test

Edit the setup

Along the time your setup changes, you need to maintain your setup repositories. To quickly make small changes, you can easily access your files and edit them in your terminal.

lxsetup edit

The edit commands execute these actions:

1. Locate the setup folder
2. Fetch commits and warn the user if some remotes commits should be pulled first.
3. Display the list of files for lxsetup in your setup repos (excluding files outside of apps, lists, system, lxsetup.json, and lxsetup.lock). Display some help on how this menu works (how to create a new file, or delete one).
4. While the option is not q:
   1. Wait the user to enter an option (the index of a displayed file, n to create, d to delete, or q to quit)
   2. Open the file to edit, or create a new file with given name or delete the file.
5. Ask if the user wants to validate (commit and push), if yes, ask for commit message, commit the modifications on the edited files (only?) and push the commit.

Run a specific step

Run a specific step with a given id. If <step-id> is not provided, the list of steps with an index will be displayed to let you choose the step.

lxsetup run <step-id>

Display the help

Display the help details with more info than simply lxsetup (with no option)

lxsetup help

Display the changelog

Display the release notes for each version of LxSetup in the descending order.

lxsetup changelog

General rules

With everyting we automate, scripts, steps and tests should support to run multiple times without any problem.

lxsetup.json file syntax

You can look at an real world example of the complete file on the setup repository of Samuel Roland at codeberg.org/samuelroland/setup

Key	Req.	Example	Description	Goal
setup.name	YES	Fedora 36 setup	A name given to the setup.	Display
setup.author	NO	Samuel Roland	The author of the setup.	Display
setup.description	NO	Setup for my 2 Fedora computers.	A description given to the setup.	Display
setup.licence	NO	AGPL-3-or-later	A SPDX (todo:link) identifier of the licence given to this setup.	Display
setup.version	YES	v1.5	A version number for your setup, with the format of your choice.	Display
os	NO		The OS that are compatible with this setup	To not run the setup if not the good OS or version.
os.strict	YES	true	If strict, the setup will be blocked. Else a warning will be displayed.
os.get-command	YES	cat /etc/fedora-release	Command to get OS name and version
os.expected	YES	Fedora release 36 (Thirty Six)	The expected result of compatible-os.get-command.
steps	YES	See docs below...	All the steps in the order in which they will be runned.	Describe the list of steps in a specific order.
extra-tests	NO	See docs below...	Extra tests not related to any steps. They are runned at the end.

Writing scripts

Your setup probably contains multiple custom scripts to install custom applications or programs not available in package managers repositories.

Writing steps

steps is an JSON object containing a list of step indexed by a key. This key is unique and is used to identify the steps in setup.lock. All steps have a type and a description. The order probably matters ! You can't install programs with snap before you have snap installed !

Step types:

1. command: a single command independant (not related to other commands)
   1. The required field command is the command to run
      Example:

      "add-flathub": {
      	"type": "command",
      	"description": "Adding the Flathub repository for Flatpak",
      	"command": "sudo flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo",
      	"tests": [{
      		"description": "Make sure Flatpak remotes repos contains Flathub",
      		"given": "flatpak remotes | grep flathub"
      	}]
      }
      

2. script: a script file to run
   1. The required field file is the file name without the .sh Example:

      "firefox-dev": {
      	"type": "script",
      	"file": "firefox-dev",
      	"description": "Install Firefox Developer edition",
      	"tests": [{
      		"description": "Check Firefox Developer is installed",
      		"given": "/home/$(whoami)/.local/opt/firefox/firefox -v"
      	}]
      }
      

3. package-manager: a bunch of programs in a list installed with the same package manager
   1. The field list is the list file name without the .list Example:

      "flatpak-packages": {
      	"type": "package-manager",
      	"list": "flatpak",
      	"description": "Install my Flatpak packages",
      	"installed": "flatpak list"
      }
      

Writing tests

The tests are a way to know the status of steps. If all tests of a given step are passing, the step is considered valid and doesn't need to be executed. Test objects are written under the array tests (even for one test). They can be stored under the array extra-tests too, if they are not related to any step. The tests commands should only do read operations and must not touch the system. The purpose is to be able to run tests are often as we want to check the state of the setup.

Here is a simple test making sure the firefox -v command has a success exit code (if true, firefox CLI exists so we can assume Firefox is installed).

{
	"description": "Check Firefox is installed",
	"given": "firefox -v"
}

The tests always have a description. The field assert defines which type of assertion we are doing (default assert is success). This impact the necessity and content of the other fields.

Test assertions list:

1. success: make sure a command is successful (the exit code is 0). This is the default assertion, you don't need to specify it.
   1. The field given is the command to test
      Example:

      {
      	"description": "Check Virtualbox is installed",
      	"given": "virtualbox -h"
      }
      

2. Assertions for system services. The required key services is an array of service names
   1. active: make sure a systemd service is active Example:

      {
      	"description": "Check that Nginx and php-fpm services are active",
      	"services": [
      		"nginx",
      		"php-fpm"
      	],
      	"assert": "active"
      },
      

   2. enabled: make sure a systemd service is enabled Example:

      {
      	"description": "Check Nginx service is enabled",
      	"services": [ "nginx" ],
      	"assert": "enabled"
      }
      

   3. masked: make sure a systemd service is masked Example:

      {
      	"description": "Check systemd-networkd-wait-online service is masked",
      	"services": [ "systemd-networkd-wait-online" ],
      	"assert": "masked"
      }
      

   TIP: It's possible to group assertions of services to avoid creating 2 steps for the same service.

   {
   	"description": "Check Nginx service is active and enabled",
   	"services": [ "nginx" ],
   	"assert": ["enabled", "active"]
   }
   

Tips for writing success tests: success tests are very convenient to test a lot of things because of exit codes. Here are a few examples that can help you to write your own tests:

• Need to check a command output contains a given word ? Just write something like mycommand | grep myword and as grep fails if no lines found, this is enough to test it.
• You want to make sure a file exists ? use test: test -e myfile.txt.
• You want to test the current GPU driver is nvidia: lspci -v | grep 'Kernel driver in use: nvidia'.

Writing extra tests

Extra tests are just normal tests, but they have a key to be identifiables. The extra-tests key is a JSON object with a list of test objects. Tests have a key because they need to be idenfiable (to write requirements, and in the lxsetup.lock).

Example

"extra-tests": {
	"check-firefox-installed": {
		"description": "Check default Firefox is already installed (by Fedora)",
		"given": "firefox -v"
	},
	"ssh-codeberg": {
		"description": "Check SSH authentication works for Codeberg",
		"given": "ssh -T git@codeberg.org"
	}
}

Writing requirements

Inside a step, you can add the optional require key with an array of requirements. If one of the requirements are not met, the step will not be executed. Requirements can be custom commands or an extra test.

Requirements types can be:

1. script: the value in given is a script to run that need to be success
2. test: the name of an extra test that need to pass

An example with the code folder creation and cloning of repos. This step needs to have the ssh-setup runned and the ssh authentication working.

"code-folder-setup": {
	"type": "script",
	"file": "code-setup",
	"description": "Setup the ~/code folder with all necessary repository",
	"require": [
		{
			"type": "script",
			"given": "ssh-setup"
		},
		{
			"type": "test",
			"given": "ssh-codeberg"
		},
		{
			"type": "test",
			"given": "ssh-github"
		}
	],
	"tests": [{
		"description": "Code folder exists and contains at least 10 elements",
		"given": "test -e ~/code && test $(ls ~/code | wc -l) -gt 10"
	}]
}

Writing conditions

Conditions are a way to disable some steps based on the result of a command. For ex. here we don't want to install Nvidia drivers if there is no Nvidia GPU.

"nvidia-drivers": {
	"type": "script",
	"description": "Installing Nvidia drivers for the GPU",
	"file": "system/nvidia-drivers",
	"conditions": [
		{
			"description": "Only if the computer has a GPU (not integrated and from Nvidia)",
			"command": "lspci | grep VGA | grep NVIDIA"
		}
	],
	"tests": [
		{
			"description": "GPU driver is set to nvidia",
			"given": "lspci -v | grep 'Kernel driver in use: nvidia'"
		}
	]
}

How to change lxsetup itself ?

Sam, you talked a lot about what the setup repos should contains, but what if I want to modify lxsetup in the first place?

Good question... LxSetup is developed in vanilla PHP in OOP using a bunch of Composer packages.

Here is the class diagram of the CLI. src/lxsetup is the entry point and router running options from LxSetup class (start(), update() are methods to run these options).

classDiagram
	direction TB
	Step o-- Test
	LxSetup o-- Step
	LxSetup -- Start
	LxSetup -- Update
	LxSetup -- Changelog
	class LxSetup {
		+name
		+author
		+os
		+version
		-List~Step~ steps
		-List~Test~ extraTests
		+__construct() void
		+start() void
		+update() void
		+changelog() void
		+showIntro() void
		+showHelp() void
		+getSteps() List~Step~
		-validateConfigFile()
		-osCheck()
	}
	class Start {
		-List~Step~ stepsToRun
		+run()
		-showIntroduction() void
		-askConfirmation() bool
	}
	class Update {
		-List~Step~ stepsToRun
		+run()
		-showIntroduction() void
		-askConfirmation() bool
	}
	class Changelog {
		+title
		+versions
		+show() void
		+currentVersion() string
	}
    class Step {
        -slug
		-type
		-description
		-file
		-command
		-list
		-installed
		-conditions
		-require
		-List~Test~ tests
        +run()
        +getStatus() bool
		+getFailedTests() List~Test~
		+getTestsOutput() string
		+getListElements() array
		+conditionsMet() bool
		+describe() string
    }
    class Test {
		-slug
        -description
		-assert
		-given
		-services
		-output
        +run() bool
		+describe() string
		+getOutput()
    }

todo: document it !

Setup the project

git clone https://codeberg.org/samuelroland/lxsetup.git

Run the tests

Tests of LxSetup are written with Pest, the great PHPUnit wrapper.

vendor/bin/pest

Or if you pest installed globally:

pest

Conclusion

License

lxsetup is released under AGPLv3-or-later. Feel free to use, study, share and improve !

Copyright (C) 2022-present Samuel Roland

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program.  If not, see <http://www.gnu.org/licenses/>.