Florian Engelhardt

Posted on Sep 22, 2020 • Edited on Feb 22

Growing the PHP Core—One Test at a Time

#php #testing #contributorswanted

In September 2000, I started my vocational training at an internet agency with two teams: one doing JavaServer Pages, and one was doing PHP. I was assigned to the PHP team, and when presented with the language, I immediately knew that no one will ever use this. I was wrong. Today, my entire career is built on PHP. It’s time to give back to the community by writing tests for the PHP core itself!

Prepare Your machine!

Before you start writing tests for PHP, let’s start with running the tests that already exist. Fetch the PHP source from GitHub and compile it to do so.

$ git clone git@github.com:php/php-src.git
$ cd php-src
$ ./buildconf
$ ./configure --with-zlib
$ make -j `nproc`

I recommend creating a fork upfront because it makes creating a pull request with your test easier, later on.

If you do not have a compiler and build tools already installed on your Linux computer, you should install the development-tools group on Fedora or the build-essential on Debian Linux. The ./configure command may exit with an error condition; this usually occurs when build requirements are not met, so install whatever ./configure is missing and re-run that step. Keep in mind that you need to install the development packages — in my case, the configure script stated it was missing libxml, which was, in fact, installed. What it really missed was the header files, which are in the development package (usually named with a dev or devel suffix). Note that the --with-zlib is mandatory in this case, as you need the zlib extension which is not built by default.

After your build is complete, you can find the PHP binary in ./sapi/cli/php. Go ahead and check what you just created:

$ ./sapi/cli/php –v
PHP 8.4.0-dev (cli) (built: Feb 21 2024 16:29:11) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.4.0-dev, Copyright (c) Zend Technologies

Now that you have a freshly built and running PHP binary, you can finally run the tests included within the GitHub repository. Since PHP 7.4 tests may run in parallel, you can give the number of parallel jobs with the -j argument.

$ make TEST_PHP_ARGS=-j`nproc` test
...
=============================================================
TEST RESULT SUMMARY
-------------------------------------------------------------
Exts skipped    :   46
Exts tested     :   26
-------------------------------------------------------------
Number of tests : 15670             10668
Tests skipped   : 5002 ( 31.9%) --------
Tests warned    :    0 (  0.0%) (  0.0%)
Tests failed    :    0 (  0.0%) (  0.0%)
Expected fail   :   32 (  0.2%) (  0.3%)
Tests passed    : 10636 ( 67.9%) ( 99.7%)
-------------------------------------------------------------
Time taken      :   76 seconds
=============================================================

This looks good and it only took 76 seconds to run 10636 tests — quite fast. Zero warnings and zero failures, hooray! You can see that 5002 tests have been skipped and 32 have been expected to fail. You will learn about why this is in the next part.

What Do Tests Look Like?

Let’s take a look at what such a test case may look like. PHP test files have the file ending .phpt and consist of several sections, three of which are mandatory: the TEST, the FILE and the EXPECT or EXPECTF section.

--TEST--
strlen() function
--FILE--
<?php
var_dump(strlen('Hello World!'));
?>
--EXPECT--
int(12)

The TEST Section

This is a short description of what you are testing. It should be as short as possible, as this is what is printed when running the tests. You can put additional details in the DESCRIPTION section.

The SKIPIF Section

This section is optional and is executed by the PHP binary under test. If the resulting output starts with the word skip, the test case is skipped. If it starts with xfail the test case is run, but it is expected to fail.

The FILE Section

This is the actual test case, PHP code enclosed by PHP tags. This is where you do whatever you want to test. As you have to create output that is then matched against your expectation, you usually find var_dump all over the place.

The EXPECT Section

This section must exactly match the output from the executed code in the FILE section to pass.

The EXPECTF Section

EXPECTF can be used as an alternative to the EXPECT section and allows the usage of substitution tags you may know from the printf functions family:

--EXPECTF--
int(%d)

The XFAIL Section

XFAIL is another optional section. If present, the test case is expected to fail; you don’t need to echo out xfail in the SKIPIF section at all if you have this. It contains a description of why this test case is expected to fail. This feature is mainly used if the test is already finished, but the implementation isn’t yet, or for upstream bugs. It usually contains a link to where a discussion about this topic can be found.

The CLEAN Section

This exists so you can clean up after yourself. An example might be temporary files you created during the test. Keep in mind, this section's code is executed independently from the FILES section, so you have no access to variables declared over there. Also this section is run regardless of the outcome of the test.

What else?

You may find more in depth details on the file format at the PHP Quality Assurance Team Web Page.

What Can You Test?

Now that you know what a PHP test looks like it is time to find something to test. Head over to codecov.io.

When I started looking for something to test, I found that the zlib_get_coding_type() function in ext/zlib/zlib.c was not covered at all (this was back in the PHP 7.1 branch and on the now retired gcov.php.net).

The next step was to check out what this function was supposed to do in the PHP Documentation. What I saw in the documentation, but also in the code itself, was that this function returns the string gzip or deflate or the Boolean value false. The linked zlib.output_compression directive documentation gave one additional bit of information: the zlib output compression feature reacts on the HTTP Accept-Encoding header sent with the client HTTP request.

For the test, this means there are four possible cases to check for:

the absence of the Accept-Encoding header
the Accept-Encoding being gzip
the Accept-Encoding being deflate
the Accept-Encoding being anything else

The last case is treated the same as the first case: The function is expected to return the Boolean value false.

Time To Write That Test!

Let's start with the first test in the file test/zlib_get_coding_type.phpt for the case that there is a Accept-Encoding header set to gzip. One question that might arise now is: How do I add an HTTP-Request in the tests? The answer is: you don't, but we know that HTTP-Request headers will be exposed via the $_SERVER super global, so we can inject the fake HTTP header there.

--TEST--
zlib_get_coding_type()
--EXTENSIONS--
zlib
--FILE--
<?php
$_SERVER["HTTP_ACCEPT_ENCODING"] = "gzip";
ini_set('zlib.output_compression', 'Off');
var_dump(zlib_get_coding_type());
ini_set('zlib.output_compression', 'On');
var_dump(zlib_get_coding_type());
?>
--EXPECT--
bool(false)
string(4) "gzip"

You can run this single test via:

$ make test TESTS=test/zlib_get_coding_type.phpt

Sadly, this gives you a failed test. You can easily see why this is the case, by checking the test directory contents where you find your .phpt test file and some other files with various file endings. One that is particularly interesting, in this case, is the zlib_get_coding_type.log file:

---- EXPECTED OUTPUT
bool(false)
string(4) "gzip"
---- ACTUAL OUTPUT
bool(false)
Warning: ini_set(): Cannot change zlib.output_compression - headers already sent in test/zlib_get_coding_type.php on line 4
bool(false)
---- FAILED

And this is your first time learning about PHP internals. You cannot change the output compression setting after your script created any form of output. For this to work, there seems to be a handler that is called when we change the zlib.output_compression directive. Try searching for "zlib.output_compression" in the ext/zlib/zlib.c source code file. You find it in the call to the STD_PHP_INI_BOOLEAN macro along with the pointer to the function OnUpdate_zlib_output_compression in which you can spot the warning you received. To work around this warning, you can change the FILE section:

<?php
$_SERVER["HTTP_ACCEPT_ENCODING"] = "gzip";
ini_set('zlib.output_compression', 'Off');
$off = zlib_get_coding_type();
ini_set('zlib.output_compression', 'On');
$on = zlib_get_coding_type();
var_dump($off, $on);
?>

Rerunning the test case results in yet another failed test. But this time you know where to look and you will find this:

---- EXPECTED OUTPUT
bool(false)
string(4) "gzip"
---- ACTUAL OUTPUT
bool(false)
bool(false)
---- FAILED

The warning is gone, but test failed again. That's progress!

Looking into the zlib_get_coding_type.log file, you notice that the second call to the zlib_get_coding_type() function returned false and not the expected string gzip. It seems like PHP is not reacting to the HTTP header that you clearly set just few lines before. Did we spot a bug in PHP?

The source code tells you: Open the ext/zlib/zlib.c source code file and search for the variable compression_coding as this is the variable that is evaluated in the function you are testing. You should find some matches, but there is one at the beginning of the file in the C function php_zlib_output_encoding which looks like the only place where something is assigned to the variable in question.

Analysing the source code, not understanding exactly what is going on, and finally asking for help on the PHP Community Chat reveals the following: All of your userspace variables and PHP user-accessible autoglobals ($_GET, $_SERVER, ...) are copy-on-write. Altering those from your script creates a copy for you to work on in userspace, effectively making the HTTP request headers immutable to the userspace. The PHP core continues to use the original, unaltered version.

Now that you know you can not alter or set the HTTP header from within your script to influence PHP’s behavior, this might look like a sad end for your tests code coverage.

Wait, there is more! I did not tell you about another section in the PHPT file format that might come in handy now: The ENV section. This section is used to give environment variables to your script and are passed to the PHP process that runs your test code. This means you can go on and that you have to create a test file per test case. Let’s create a new test file for the gzip case and name it zlib_get_coding_type_gzip.phpt.

--TEST--
zlib_get_coding_type() is gzip
--EXTENSIONS--
zlib
--ENV--
HTTP_ACCEPT_ENCODING=gzip
--FILE--
<?php
ini_set('zlib.output_compression', 'Off');
$off = zlib_get_coding_type();
ini_set('zlib.output_compression', 'On');
$gzip = zlib_get_coding_type();
var_dump($off);
var_dump($gzip);
?>
--EXPECT--
bool(false)
string(4) "gzip"

Run the test and see it fails once more. You might have expected this by now. 😝

Let’s see what happened by looking into the zlib_get_coding_type_gzip.log file.

---- EXPECTED OUTPUT
bool(false)
string(4) "gzip"
---- ACTUAL OUTPUT
�1�0
    ���B�P�Txꆘ8`5ؑ������s�7a�ze��B+���ϑ�,����^���~�^�J1����ʶ`�\0�v@cm��}�ap�X}��'4�ͩqG�^w
---- FAILED

Wow, this looks like some binary garbage, and yeah, this does not match the expected output.

Look at your test case. You told PHP that you accept gzip encoding and you turned on output compression. PHP is basically just doing what we told it to do. The binary garbage you see here is gzip encoded data. This means you succeeded in activating gzip output compression, but forgot to turn it off before dumping out the two variables. Now, all that’s left is to add another call to deactivate output compression again so the final test looks similar to this:

--TEST--
zlib_get_coding_type() is gzip
--EXTENSIONS--
zlib
--ENV--
HTTP_ACCEPT_ENCODING=gzip
--FILE--
<?php
ini_set('zlib.output_compression', 'Off');
$off = zlib_get_coding_type();
ini_set('zlib.output_compression', 'On');
$gzip = zlib_get_coding_type();
ini_set('zlib.output_compression', 'Off');
var_dump($off);
var_dump($gzip);
?>
--EXPECT--
bool(false)
string(4) "gzip"

Run this test again, and it finally passes! 🎊🥳🎉

This leaves you with writing one test case for the Accept-Encoding header being set to deflate and another test case for the Accept-Encoding header being set to an invalid string (basically something that is not gzip or deflate). But this is just diligence from this point on and if you like to cheat you can find the tests in the ext/zlib/tests/ directory named zlib_get_coding_type_basic.phpt, zlib_get_coding_type_br.phpt, zlib_get_coding_type_deflate.phpt and zlib_get_coding_type_gzip.phpt.

Collect Some Evidence!

Now that you have tests for all four cases you could hope you covered that function with tests or (a way better option if you ask me), you could continue and create a code coverage report! For this to work you need to install the lcov tool via your Linux package manager (sudo dnf install lcov for Fedora or sudo apt-get install lcov for Debian). As PHP was built with gcov disabled, we need to run configure again, with enabled gcov and recompile.

$ make clean
$ CCACHE_DISABLE=1 ./configure --with-zlib --enable-gcov
$ make -j `nproc`

This compiles the PHP binary with enabled code coverage generation. Lcov will be used to create an HTML code coverage report afterwards. To create your code coverage report, you need to rerun the tests.

$ make test TESTS=test/zlib_get_coding_type.phpt

While running the tests, gcov generates coverage files with a gcda file ending for every source code file. To generate the HTML code coverage report, run the following:

$ make lcov

You may find the resulting HTML code coverage report in lcov_html/index.html.

It looks like we covered all four cases.

What’s in It for You?

With every test you write PHP becomes more stable and reliable, as functions may not change behavior suddenly between releases.

Writing tests for PHP gives you a deeper understanding of how your favorite language works internally. In fact, by reading up to this point you learned that HTTP request headers are immutable to userspace, that there is such a thing as userspace, and that there are handler functions called and check what you are doing when you want to change ini settings at runtime.

Also knowing the PHPT file format gives you other benefits as well: PHPUnit not only supports the PHPT file format and can execute those tests, part of PHPUnit is tested with PHPT test cases. This led me to become a contributor to PHPUnit: Writig tests for PHPUnit in the PHPT file format.

Shout Out!

I would like to thank everyone involved in the PHP TestFest, but especially Ben Ramsey, Sammy Kaye Powers and everyone else involved in the organization of the 2017 edition. This was what made me write tests for PHP and made me not only a PHP but also a PHPUnit contributor. In the long run, this brought me my first ElePHPant 🐘!

Closing notes

I would like to thank my proof readers Kara Ferguson and Pim Elsfhof. It was a pleasure working with you!

"Wooble" is the property of Sammy Kaye Powers and is used with permission.

The German version is available through the PHP Magazin

Interested in seeing the talk to this blog post? Nomad PHP has you covered.

Top comments (6)

Agustin Gomes • Sep 23 '20

I got stuck at libxml, tried to use --with-zlib but it didn't work for me

configure: error: Package requirements (libxml-2.0 >= 2.9.0) were not met:

Package 'libxml-2.0', required by 'virtual:world', not found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables LIBXML_CFLAGS
and LIBXML_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.

Do I need to do something else specific to that?

Apart from it, it's a nice guide, and it has motivated me to try it out to potentially contribute in the future.

Florian Engelhardt • Sep 23 '20 • Edited

Hey Agustin,

this looks like the libxml header files are not installed.

apt-get install libxml2-dev # for Debian based
# or
yum install libxml2-devel # for Fedora based

This should install the needed header files to compile PHP. Additionally I could find this Stack Overflow Thread on the topic.

Let me know if this helped you!

Happy coding
Florian

Tomas Norre Mikkelsen • Sep 22 '20

Isn't is build-essential instead of build-essentials on Debian systems ? at least it is on my Ubuntu.

Florian Engelhardt • Sep 22 '20

Nice catch. I updated the article accordingly, thank you!