# Testing your frameworks
SVRUnit can also be used to **test** your framework.
Let's imagine you are building a PHP framework, that is published as a composer dependency, as well as a PHAR binary.
You would usually start it with one of these 2 commands:
```bash
# composer dependency
php vendor/bin/myapp xyz
# phar file
php myapp.phar xyz
```
SVRUnit helps you with 2 things.
1. You can create your custom "E2E" setups where you start your framework and **verify the expected output** using SVRUnit
2. You can use SVRUnit [placeholders](/configuration/configuration#placeholders) to test both, the **script** as well as the **PHAR file** with the same tests.
Let's get started.
Because SVRUnit allows you to install it with Composer, it is recommended to install it directly as dependency. You can of course also use the PHAR file for this.
```bash
composer require svrunit/svrunit
```
Now create a **XML configuration** file that specifies 2 test suites.
\
Both test suites use the same test directory, but use **different executable placeholders**.\
With this approach, we can then use a placeholder inside the real tests later, so that we either use the script version of our file, or the PHAR binary, when running the tests.
Please keep in mind, this is totally optional...you don't need to use placeholders.
```xml
tests/svrunit/tests
tests/svrunit/tests
```
Last but not least, we need the specification of our tests in the YAML file.
If you use placeholders for your executable, just use **(($EXEC))** instead of the real path to your executable. If not, just use your real path.
```yaml
commands:
- name: "Test validate command"
command: "(($EXEC)) validate"
expected: "All translations are valid"
....
....
```
That's it!
You can now **manually** run your tests, or even use your **CI/CD pipeline** for this.
php vendor/bin/svrunit --configuration=myconfig.xml
####
# Configuration
## Basics
The XML Configuration contains all your test suites for your tests.
A single test suite defines where to run a subset of your tests.\
This means you can configure a test run for a specific Docker image and ENV variables.
The tests can then be used from either **shared test directories** or **specific** ones for your environment. You can also configure **single files** that contain the full tests for your use case.
<svrunit setupTime="12">
<testsuites>
<testsuite name="Docker Image with PHP 8.1"
group="php8"
dockerImage="mycompany/docker-image:latest"
dockerEnv="PHP_VERSION=8.1,XDEBUG_ENABLED=1">
<directory>./shared</directory>
<directory>./php/v8.1</directory>
</testsuite>
<testsuite name="Docker Image as Command Runner with PHP 8.0"
group="php8"
dockerImage="mycompany/docker-image:latest"
dockerEnv="PHP_VERSION=8.0,XDEBUG_ENABLED=1"
dockerCommandRunner="true">
<directory>./shared</directory>
<directory>./php/v8.0</directory>
</testsuite>
<testsuite name="Docker Container with PHP 7.4"
dockerContainer="my_running_php74">
<directory>./shared</directory>
<directory>./php/v7.4</directory>
</testsuite>
<testsuite name="Local Server Check with File">
<directory>./shared</directory>
<file>./my_server_spec.xml</file>
</testsuite>
</testsuites>
</svrunit>
## Setup / Teardown
Every test run has a single setup and teardown phase.
If you run tests in Docker images you might want to assign a setup time in seconds. \
This is the time that SVRUnit will wait before starting your tests.
Make sure you plan enought time to have everything ready before your tests get started.
```markup
.........
```
## Test Runners
Every test suite is executed on a specific environment.\
Depending on what type of environment you chose (Docker, localhost, ...), a different Test Runner is being used to run the tests.
This helps you to reuse your existing tests on different types of environments.
### Docker Image
If you want to create tests for your built Docker images, you can provide the following configuration. In that case SVRUnit will always start a new image before executing your tests. It will use a name with "svrunit\_" prefix. Containers will be automatically removed on teardown.
You can also assign a comma separated lists of special ENV variables that you want to use when starting the image.
```markup
.....
```
### Docker Image Command Runner
You can also run tests by using your Docker image as command runner.\
In this case, SVRUnit will not launch a new independent container from that image and execute all tests. Instead it will use a simple "docker run" command where the commands of the tests are appended to the inline command.
The difference to the basic launch of a container is, that this option will override the built-in CMD part of the Docker image. So you can also verify this kind of usage when testing Docker images.
The configuration is almost the same as above. Just add the key "dockerCommandRunner" with TRUE.
```markup
.....
```
### Docker Container
If you want to start tests in a running Docker Container, provide these configuration values to let SVRUnit know where to run your tests.
```markup
.....
```
### Local Tests
If you just want to test your local server, simply configure your test suite without any additional information. This will start all your tests on the executing system.
```markup
.....
```
## Advanced
### Groups
You can define groups for your Test Suites.
This helps you to build categories of different tests, e.g. all PHP 8.x Docker Images, or all test suites that run "Security" tests.
You can then use these groups for filters when running SVRUnit.
Just define the group you want to run, using the **--group=xy** argument.
```xml
./shared
./php/v8.0
```
### Placeholders
SVRUnit allows you to use placeholders for specific things.\
Here is a list of available placeholders.
#### Executable (EXEC)
Use this to define the executable that should be tested.
This is a great way to test different versions of your application with the same tests. This could be a /bin/script version of your framework, next to the phar binary. Both executables tested with the same SVRUnit tests.
You can use the placeholder **(($EXEC))** within your tests.
```yaml
- name: "Test validate command"
command: "(($EXEC)) validate"
expected: "All translations are valid"
```
Now create 2 test suites with the same tests, but different executables.
```xml
tests/svrunit/tests
tests/svrunit/tests
```
Both of your application versions are now tested with the same test files.
# list:suites
You can use this command to list all available test suites of your provided configuration file.\
This is great to get an overview of some more complex setups.
```bash
php svrunit.phar list:suites --configuration=./svrunit.xml
```
# list:groups
You can use this command to list all available test groups of your provided configuration file.\
This is great to get an overview of some more complex setups.
You can then use these groups as filter when [running your tests](/commands/run-tests).
```bash
php svrunit.phar list:groups --configuration=./svrunit.xml
```
# test
SVRUnit has a separate "**test**" command that you can use to run your tests.\
This is also the default command, so you can also skip that command name.
The command requires that you provide the **XML configuration** file as argument.
```bash
php svrunit.phar test --configuration=./svrunit.xml
```
That's it, your tests should now run.
If you want to get more out of this command, please see this list of available options that you can use.
### Command Options
| Option | Description |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| --configuration | Required argument to specify the XML configuration file. |
| --group | Only run test suites that contain your provided group name. |
| --exclude-group | Exclude one or more groups.
--exclude-group=group1,group2
|
| --debug | Show additonal debug output directly when your tests are executed |
| --stop-on-error | If provided, SVRUnit will immediately fail and stop executing tests, once the first test fails.
Use this to save some time when running your tests.
|
| --report-html | Generates a [HTML report](/reports/html-report) |
| --report-junit | Generates a [JUnit report](/reports/junit-report) |
# Command
Use this assertion type to run any free command and verify its output with your expected value.
This is based on "string contains" assertions.
## Expecting
Use this type of assertion to verify against an expected output of the command.
```yaml
commands:
- name: "NVM can switch Node version"
command: "nvm use 14 && echo SVRUNIT: $(node -v)"
expected: "SVRUNIT: v14."
```
## Not Expecting
Use this type if you do not want your output to show the provided value.
```yaml
commands:
- name: "Xdebug not Corrupted"
command: "php -v"
not_expected: "zend_mm_heap corrupted"
```
## Expecting AND conditions
Use this type if you do not want your output to show the provided value.
```yaml
commands:
- name: "Verify PHP 7.4 and OPcache v7.4"
command: "php -v"
expected_and:
- "PHP 7.4"
- "OPcache v7.4"
```
## Expecting OR conditions
Use this type if you do not want your output to show the provided value.
```yaml
commands
- name: "Verify any of the allowed PHP versions"
command: "php -v"
expected_or:
- "PHP 8.1"
- "PHP 8.0"
- "PHP 7.4"
```
## Setup / Teardown
This test also offers separate command options as setup and teardown routine.\
Commands provided with these options, will be run prior and/or after the actual command of this test.
```
commands:
- name: "..."
command: "...."
expected: "..."
setup: "mkdir -p test_assets"
teardown: "rm -f test_assets"
```
# Directory Exists
This assertion type helps you to verify if a certain directory exists.
## Existing
Provide a TRUE (1) as expected to check that the directory should exist.
```yaml
directory_exists:
- name: "DocRoot exists"
directory: "/var/www/html"
expected: 1
```
## Not Existing
Provide a FALSE (0) as expected to check that the directory does not exist at all.
```yaml
directory_exists:
- name: "DocRoot must not exist"
directory: "/var/www/html/public"
expected: 0
```
# File Exists
This assertion type helps you to verify if a certain directory exists.
## Existing
Provide a TRUE (1) as expected to check that the file should exist.
```yaml
file_exists:
- name: "SSH Private Key exists"
file: "/var/www/.ssh/id_rsa"
expected: 1
```
## Not Existing
Provide a FALSE (0) as expected to check that the file does not exist at all.
```yaml
file_exists:
- name: "SSH Private Key must not exist"
file: "/var/www/.ssh/id_rsa"
expected: 0
```
# File Content
This assertion type helps you to verify if a certain text can be found inside the content of a file.
This is based on a "string contains" check.
## Expecting
Use this type to verify that the expected content does exist in the file.
```yaml
file_content:
- name: "Verify webservice entry in /etc/hosts"
file: "/etc/hosts"
expected: "my-webservice x.x.x.x"
```
## Not Expecting
Use this type to ensure that the content does not exist in the file.
```yaml
file_content:
- name: "Verify webservice entry not in /etc/hosts"
file: "/etc/hosts"
not_expected: "my-webservice x.x.x.x"
```
# File Permissions
This assertion type verifies the permissions of a file. It helps you to quickly check expected permission sets.
## Expecting
This type checks if the file has 775 permissions.
```yaml
file_permission:
- name: "My File has 775"
file: "/var/www/my-file.txt"
expected: "775"
```
## Not Expecting
Use this type to verify that the file does not have 775 permissions.
```yaml
file_permission:
- name: "My File has not 775"
file: "/var/www/my-file.txt"
not_expected: "775"
```
# PHP INI
This assertion helps you to quickly identify if a PHP Ini value has a certain value.
## Expecting
This type checks that the expected value is being used in the PHP ini settings.
```yaml
php_ini:
- name: "Memory Limit is unlimited"
setting: "memory_limit"
value: "-1"
```
## Not Expecting
Use this type to ensure that the value is something else than the provided expected value.
```yaml
php_ini:
- name: "Memory Limit is not unlimited"
setting: "memory_limit"
not_value: "-1"
```
## Expecting OR with Operators
Use this to test against a **list of possible values**. \
You can also use it with 1 condition, if you just want to use the **operators**.
The operator **=** is based on simple **string** values.\
\
The operators **<**, **<=**, **>**, **>=** are obviously only possible for **INT** or **FLOAT** values. \
SVRUnit will automatically extract the values and compare thoses based on a float value.\
\
It also supports a correct **conversion of units** like KB, MB, GB and more out of the box for you.
```yaml
php_ini:
- name: "Verify PHP 7.1 or >= 7.4 is used"
setting: "PHP_VERSION"
value_or:
- "operator": "="
"value": "7.1"
- "operator": ">="
"value": "7.4"
```
## PHP INI for "CLI" vs. "WEB"
As you know, PHP settings can either be configured when used with CLI, or when used for a HTTP web server like Apache.
The "PHP INI" Test module allows you to set a mode to "**cli**" or "**web**".\
This makes sure that you really test what you think you are testing.
The mode "**cli**" will simply check the requested INI setting using a CLI command.
The mode "**web**" will do a little bit more for you.\
It will search for your Apache DocRoot directory. When found, it places a temporary PHP script that will then be fetched with curl on your localhost.\
This makes sure to verify the real used INI setting of your web server.\
Afterwards this temporary PHP script is deleted again.
```yaml
php_ini:
- name: "Memory Limit is >= 512M"
setting: "memory_limit"
mode: "web"
value_or:
- "operator": ">="
"value": "512M"
```
{% hint style="warning" %}
Please keep in mind, the mode "web" is pretty new.\
If you figure out any problems or missing features, just let us know, or do a pull request :) \
Thank you!
{% endhint %}
# PHP Modules
This type of assertions is a quick way to verify if a certain PHP module is installed and active. Just use the lowercase name of the module to verify against.
```yaml
php_module:
- name: "PHP Module installed: intl"
module: "intl"
```
# HTML Report
With SVRUnit it's possible to create a HTML report from a test run. This allows you to inspect your test run and its results directly from your browser.
The report will be generated in the folder `./reports`within your working directory.
To start with the reporter, simply append the option `--report-html` to the text command.
```bash
php svrunit.phar --configuration=./svrunit.xml --report-html
```
�
# JUnit Report
With SVRUnit it's possible to create a JUnit XML report file from the test run.
This will be automatically generated in the folder `./reports`within your working directory.
To start with the reporter, simply append the option `--report-junit` to the text command.
```bash
php svrunit.phar --configuration=./svrunit.xml --report-junit
```
The output XML file looks like this.\
It contains an overview of all test suites and their test cases along with their results.
```markup
Expected: xy
Actual: xyz
```
�