Commit b903aa61 authored by Pierre Smeyers's avatar Pierre Smeyers
Browse files

docs: add Shell Scripting Guidelines

parent b8220b0c
Loading
Loading
Loading
Loading

docs/dev/shell.md

0 → 100644
+171 −0
Original line number Diff line number Diff line
---
author: Pierre Smeyers
description: This page gives guidelines and recommandations for shell scripting developement in to-be-continuous.
---

# Shell Scripting Guidelines

This page gives guidelines and recommandations for shell scripting developement in _to be continuous_.

## Introduction

_to be continuous_ templates bring a lot of magic. All this magic is mainly implemented with shell scripts.
Fine! But which kind of shell are we talking about? Is it Bash? Regular C Shell? Something else?

All TBC template scripts are run in container images inheriting one of the three following base images:

```mermaid
---
config:
  pie:
    textPosition: 0.6
  themeVariables:
    pieOuterStrokeWidth: "3px"
---
pie
    "Alpine (Ash shell)" : 65
    "Debian (Bash shell)" : 30
    "Fedora (Bash shell)" : 5
```

> [!note]
> TBC will never try to be compliant with other base images than Alpine, Debian or Fedora, and should never make strong assumptions 
> on wich base image is used in such or such template (because users might always override the default provided images with their own, and TBC should keep working).

As a conclusion: **_to be continuous_ template scripts must be executable with [Bash](https://www.gnu.org/software/bash/manual/bash.html) and [Ash](https://en.wikipedia.org/wiki/Almquist_shell)**.

Fortunately, [Ash](https://en.wikipedia.org/wiki/Almquist_shell) is a subset of [Bash](https://www.gnu.org/software/bash/manual/bash.html) so all we have to do is to determine which Bash features are supported in Ash, and which aren't. That's the very purpose of this page.

## Bash features support in Ash

| Name                                              | Description / Example                                                                                                                                                                                            | Ash support                                         |
| ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- |
| [file conditional expressions][cond-expr]         | `[[ -a file ]]`, `[[ -f file ]]`, `[[ -d file ]]`, `[[ -x file ]]`...                                                                                                                                            | ✅ supported                                        |
| [string conditional expressions][cond-expr]       | `[[ string ]]`, `[[ -z string ]]`, `[[ -n string ]]`,<br/> `[[ string1 == string2 ]]`, `[[ string1 != string2 ]]`...                                                                                             | ✅ supported                                        |
| [arithmetic conditional expressions][cond-expr]   | `[[ val1 -eq val2 ]]`, `[[ val1 -ne val2 ]]`, `[[ val1 -lt val2 ]]`, `[[ val1 -ge val2 ]]`...                                                                                                                    | ✅ supported                                        |
| [advanced conditional expressions][cond-expr]     | `[[ -o optname ]]` (true if the shell option _optname_ is enabled)<br/>`[[ -v varname ]]` (true if variable _varname_ is set)<br/>`[[ -R varname ]]` (true if variable _varname_ is set and is a name reference) | ❌ not supported                                    |
| [regex operator][regex-op]                        | `[[ text =~ regex ]]` (true if _text_ matches _regex_)                                                                                                                                                           | ✅ supported<br/>⚠️ except variable `$BASH_REMATCH` |
| [unset or null substitution expansion][expansion] | `${parameter:−word}` (expands to _word_ if `$parameter` is unset or empty; else `$parameter`) | ✅ supported                                        |
| [unset substitution expansion][expansion]         | `${parameter−word}` (expands to _word_ if `$parameter` is unset; else `$parameter`)| ✅ supported                                        |
| [set substitution expansion][expansion]           | `${parameter:+word}` (expands to _word_ if `$parameter` is set and not empty; else empty string) | ✅ supported                                        |
| [substring expansion][expansion]                  | `${parameter:offset}`<br/>`${parameter:offset:length}`                                                                                                                                                           | ✅ supported                                        |
| [length expansion][expansion]                     | `${#parameter}` (length of `$parameter`)                                                                                                                                                                         | ✅ supported                                        |
| [remove prefix expansion][expansion]              | `${parameter#pattern}`                                                                                                                                                                                           | ✅ supported                                        |
| [remove long prefix expansion][expansion]         | `${parameter##pattern}`                                                                                                                                                                                          | ✅ supported                                        |
| [remove suffix expansion][expansion]              | `${parameter%pattern}`                                                                                                                                                                                           | ✅ supported                                        |
| [remove long suffix expansion][expansion]         | `${parameter%%pattern}`                                                                                                                                                                                          | ✅ supported                                        |
| [replace first match expansion][expansion]        | `${parameter/pattern/string}`                                                                                                                                                                                    | ✅ supported                                        |
| [replace all expansion][expansion]                | `${parameter//pattern/string}`                                                                                                                                                                                   | ✅ supported                                        |
| [indirect variable expansion][expansion]          | `${!varname}` (expands to variable whose name is given by `$varname`)                                                                                                                                            | ❌ not supported                                    |
| [replace prefix expansion][expansion]             | `${parameter/#pattern/string}` (replace)<br/>`${parameter/#pattern}` (remove)                                                                                                                                    | ❌ not supported                                    |
| [replace suffix expansion][expansion]             | `${parameter/%pattern/string}` (replace)<br/>`${parameter/%pattern}` (remove)                                                                                                                                    | ❌ not supported                                    |
| [uppercase expansion][expansion]                  | `${parameter^}` (uppercase first letter)<br/>`${parameter^^}` (uppercase whole word)                                                                                                                             | ❌ not supported                                    |
| [lowercase expansion][expansion]                  | `${parameter,}` (lowercase first letter)<br/>`${parameter,,}` (lowercase whole word)                                                                                                                             | ❌ not supported                                    |
| [operator expansion][expansion]                   | `${parameter@u}`, `${parameter@U}`, `${parameter@L}`...                                                                                                                                                          | ❌ not supported                                    |
| [`declare` command][declare]                      | `declare -A mydict` (a dictionary)<br/>`declare -i mynumber` (an integer)<br/>...                                                                                                                                | ❌ not supported                                    |
| [`local` comand][local]                           | `local myvar=$1` (declare a local variable within a function)                                                                                                                                                    | ✅ supported                                        |
| [Arithmetic expansion][arithmetic-expansion]      | `$(( total + incr ))`, `$(( ! value ))`, `$(( bits << 2 ))`, `$(( bits \| 2 ))`...                                                                                                                                                                                  | ✅ supported                                        |
| [Arithmetic command][arithmetic-expansion]        | `(( index++ ))` (increment integer-typed variable `index`)<br/>`(( index+=30 ))` (increase integer-typed variable `index`)<br/>...                                                                                                                                                   | ❌ not supported                                    |
| [Arrays][arrays]                                  | One-dimensional indexed array variables.<br/>[See cheatsheet][cheat-arrays]                                                                                                                                      | ❌ not supported                                    |
| [Dictionaries][arrays]                            | Associative array variables.<br/>[See cheatsheet][cheat-dictionaries]                                                                                                                                            | ❌ not supported                                    |
| [Here Documents][heredoc]                         | [See cheatsheet][cheat-heredoc]                                                                                                                                                                                  | ✅ supported                                        |
| [Here Strings][herestring]                        | `tr '[:lower:]' '[:upper:]' <<< "Will be uppercased"`<br/>[See cheatsheet][cheat-herestring]                                                                                                                     | ❌ not supported                                    |
| [Process Substitution][proc-subst]                | `diff <(ls "$dir1") <(ls "$dir2")`<br/>[See cheatsheet][cheat-proc-subst]                                                                                                                                        | ✅ supported                                        |

[cond-expr]: https://www.gnu.org/software/bash/manual/bash.html#Bash-Conditional-Expressions
[regex-op]: https://www.gnu.org/software/bash/manual/bash.html#index-_005b_005b
[expansion]: https://www.gnu.org/software/bash/manual/bash.html#Shell-Parameter-Expansion
[arithmetic]: https://www.gnu.org/software/bash/manual/bash.html#Shell-Arithmetic-1
[arithmetic-expansion]: https://www.gnu.org/software/bash/manual/bash.html#Arithmetic-Expansion
[arrays]: https://www.gnu.org/software/bash/manual/bash.html#Arrays
[cheat-arrays]: https://devhints.io/bash#arrays
[cheat-dictionaries]: https://devhints.io/bash#dictionaries
[declare]: https://www.gnu.org/software/bash/manual/bash.html#index-declare
[local]: https://www.gnu.org/software/bash/manual/bash.html#index-local
[heredoc]: https://www.gnu.org/software/bash/manual/bash.html#Here-Documents
[cheat-heredoc]: https://devhints.io/bash#heredoc
[herestring]: https://www.gnu.org/software/bash/manual/bash.html#Here-Strings
[cheat-herestring]: https://devhints.io/bash#herestring
[proc-subst]: https://www.gnu.org/software/bash/manual/bash.html#Process-Substitution
[cheat-proc-subst]: https://devhints.io/bash#process-substitution

## Other recommendations

### Shell options

All _to be continuous_ templates must use the following [shell options](https://www.gnu.org/software/bash/manual/bash.html#The-Set-Builtin):

- `set -e`: instructs bash to immediately exit if any command has a non-zero exit status
- `set -o pipefail`: prevents errors in a pipeline from being masked. If any command in a pipeline fails, the pipeline breaks and that return code is used as the return code of the whole pipeline.

### Use `grep` with caution

The `grep` command has a behavior that might cause unexpected behaviors: **returns a non-zero return code when no match is found**.

As a result, due to the [recommended shell options](#shell-options) used in TBC, the following script might fail unintentionally:

```bash
titles=$(grep ^# README.md)
# 💣 the above command will fail and exit the script if no match was found
```

As a conclusion: **don't use `grep` to filter items out of a list**.
For this purpose, prefer using `awk` instead.

The above script can be implemented with `awk`:

```bash
titles=$(awk '/^#/' README.md)
# the above command will never fail, possibly returns an empty string if no match was found
```

**The `grep` command might be used if the return code is tested**:

```bash
if ! grep ^# README.md
then
  echo "Your README.md file doesn't contain any title!"
  continue
fi
```

### Pattern matching should never be used for pure string matching

The [pattern matching](https://www.gnu.org/software/bash/manual/html_node/Pattern-Matching.html) implementation on Alpine messes up with filename expansion.

Here is the problem:

```bash
$ docker run --rm -it alpine

# in the container
$ [[ "test" == "te"* ]]; echo $?
0
# 👍 Ok

$ touch te.file
$ [[ "test" == "te"* ]]; echo $?
1
# 💣 due to '"te"*' that messes up with 'te.file'
```

:information_source: this is specific to Alpine Ash, the above behavior is not reproduced in Debian or Fedora Bash.

As a conclusion: **glob patterns should never be used for pure string matching** in _to be continuous_ template scripts.
Instead, prefer using the regex operator.

The above script can be implemented with the [regex operator](https://www.gnu.org/software/bash/manual/bash.html#index-_005b_005b) (`=~`):

```bash
$ docker run --rm -it alpine

# in the container
$ [[ "test" =~ ^"te" ]]; echo $?
0
# 👍 Ok

$ touch te.file
$ [[ "test" =~ ^"te" ]]; echo $?
0
# 👍 Ok: =~ is not messed up with files
```
+1 −0
Original line number Diff line number Diff line
@@ -83,6 +83,7 @@ nav:
  - Contribute:
    - Workflow: dev/workflow.md
    - Guidelines: dev/guidelines.md
    - Shell Scripting: dev/shell.md
    - Architecture: dev/architecture.md
  - Security: 
    - Overview: secu/index.md