=encoding utf-8

=head1 NAME

Lintian::Tutorial::TestSuite -- Quick intro to running the Lintian testsuite

=head1 SYNOPSIS

This guide will quickly introduce you to running the Lintian test
suite and some tricks.  The Lintian test suite is fairly large and
accordingly it can take a substantial amount of time to run.  To speed
up development, there are various options to limit the tests to run.

If you are looking for a guide on how to write tests, please consult
L<Lintian::Tutorial::WritingTests>.

=head1 DESCRIPTION

The Lintian test suite is an extensive collection of various test
cases.  The test suite is divided into 4 "sub-suites".  The majority
of tests are currently located in the "tests" sub-suite.

To run the full suite in all its glory, simply invoke:

 $ debian/rules runtests

 OR

 $ mkdir -p debian/test-out
 $ t/runtests -k --dump-logs t debian/test-out

While writing a new tag (or check) you probably only want to run a
particular (subset of the) test(s).  See L</Running a subset of the
tests> for the available options.

When run via I<debian/rules>, the test suite respects
"DEB_BUILD_OPTIONS=parallel=N". When using I<t/runtests> directly, use
I<-jN> to choose the number of threads.  Note that "N" denotes the
amount of "worker" threads.  The test runner will generally have 2
threads more than that.  Also each "worker" will run lintian, which
runs multiple unpacking jobs in parallel as well.

In case you are wondering about the 2 extra threads in the test
runner, the first of them is the "coordinator" thread (which will
generally be waiting when the workers are active) and the second one
is the "output" thread (which handles the fancy output).

=head2 Running a subset of the tests

The following options are available:

=over 4

=item Running a single test

To run a single test by its name, use:

 $ debian/rules runtests onlyrun=$name

 OR

 $ t/runtests --dump-logs t debian/test-out $name

=item Running all tests for a check

To run all tests for a given check, use:

 $ debian/rules runtests onlyrun=$check

 OR

 $ t/runtests --dump-logs -k t debian/test-out $check

$check must be the name of a check (it will test for
checks/$check.desc) or "legacy".  This will run all tests that start
with "$check-".

Note: The "changes" sub-suite in the new test suite does not support
this.

=item Running all tests in a sub-suite

To run all tests in a given sub-suite, use:

 $ debian/rules runtests onlyrun=suite:$suites

 OR

 $ t/runtests --dump-logs -k t debian/test-out suite:$suites

$suites is a comma-separated list of names of sub-suites to run.

Note: this cannot be used to influence the order in which the sub-suites
are run.

=item Running all tests designed for a specific tag

To run all tests that have a "Test-For" or a "Test-Against" for a given
tag, use:

 $ debian/rules runtests onlyrun=tag:$tag

 OR

 $ t/runtests --dump-logs -k t debian/test-out tag:$tag

=back

=head2 Running tests under coverage

It is possible to run most of the tests under L<Devel::Cover>.  This is
done by passing I<--coverage> to I<t/runtests>.  Example:

  $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out

Please note that L<Devel::Cover> does not seem to handle multiple
threads too well.  You may see spurious warnings/errors if you run the
tests with 2 or more active worker threads.

B<Caveat> 1: Coverage for collections (i.e. programs in I<collection/>)
does not seem to work at the moment. Therefore, they often end up with
(next to) zero coverage in the generated reports.

B<Caveat> 2: L<Devel::Cover> sometimes changes the output of Lintian
or tools called by Lintian. Obviously, this leads to test
failures. Therefore, you may see weird test failures (or warnings)
when running under coverage.

=head3 Collecting the coverage you want in a reasonable time

Collecting coverage is excruciatingly slow.  This is not helped by the
fact that it becomes unreliable when run under 2 or more threads.

Fortunately, L<Devel::Cover> "appends" to its cover database.  This
allows you to "slowly" build up the coverage database over multiple
runs. Example:

  $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out suite:scripts
  $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out suite:debs
  $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out suite:source
  ...

Or:

  $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out $check
  $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out legacy


=head1 SEE ALSO

L<Lintian::Tutorial::WritingTests>

=cut

