[PHP-DEV] Regression Tests From: Thomas Swan (tswan <email protected>)
Date: 08/06/00

At 04:26 AM 8/6/2000, Manuel Lemos wrote:
Hello Stig,

On 06-Aug-00 06:19:07, you wrote:

>> Now, a greater challenge:  many existing bugs lead PHP to crash.  How do
>> you (plan to) handle crashes in your test suite?  Since you use PHP scripts
>> to test features I supposed you have to use something external, like shell
>> scripts or PHP scripts running independently of the actual test scripts, to
>> detect failures that lead to crashes.

>The idea is very simple: a PHP/shell script runs the PHP interpreter
>(with a dedicated php.ini file and/or customization directives on the
>command line) once for each test.  The stdout of the test is then
>compared to the expected output, if they differ, the test fails.  If it
>gets harder than this, I fear people won't bother writing tests.

I know how regression tests work, but if you run from shell, it will be
hard to run them on non-unix platforms.  OTOH hand, if you run from a
single PHP script it terminates when a bug leads PHP to crash.  You have to
rely on a PHP only solution to a master PHP script call individual test
scripts using only the standalone CGI executable.  Besides that there may
exist bugs that only show when PHP is run as a server module, so the scripts
may have to runnable that way.

A possible option lies in using the output buffer and the 'eval' call.

A single master script could iterate among a series of test or case scripts, capturing and comparing output/expected output, examining the output buffer and any other variables at the end of each test.  The only disadvantage would be testing when output_buffering is disabled.  Regardless,  I think this would remove the "is or isn't a server module" issue.

I have thought if the script were calling other scripts via a URL through the web server you can isolate the scripts from interfering with the master script, unless you hit a really nasty bug which crashes the server.

An example would be something like
{pseudocode}

you have regression_test directory
.
..
case1
case2
case3
...


in each case directory
script.php
options.ini
output.txt
input.txt
expected.txt

an option file could contain
method = http|eval
vars = yes|nor

[files]
outputfile = {some file if needed}
inputfile = {some file if needed}
expectedoutputfile = {another file}

[vars]
var1 = {expected value}
. . .

each case could dump its variables to the output file?

----

for each directory in regression test directory {
        read/parse option file (specifying local eval or get through http:// or other method and any state variables)
        evaluate option file to determine if test can be done (exit if no)
        clear output buffer
        eval or get output from script via method in option file
        compare output buffer with expected output
        compare variables (if any) with expected variable values (generate diff?)
        store results in local vars/file to generate report
        output something to browser to keep it alive... {progress report}
        flush output buffer
}

output report when done... success/fails ... etc ...



-- PHP Development Mailing List To unsubscribe, e-mail: php-dev-unsubscribe <email protected> For additional commands, e-mail: php-dev-help <email protected> To contact the list administrators, e-mail: php-list-admin <email protected>