9 Regression Testing SXEmacs

The source directory tests/automated contains SXEmacs’ automated test suite. The usual way of running all the tests is running make check from the top-level source directory.

The test suite is unfinished and it’s still lacking some essential features. It is nevertheless recommended that you run the tests to confirm that SXEmacs behaves correctly.

If you want to run a specific test case, you can do it from the command-line like this:

$ xemacs -batch -l test-harness.elc -f batch-test-emacs TEST-FILE

If something goes wrong, you can run the test suite interactively by loading test-harness.el into a running SXEmacs and typing M-x test-emacs-test-file RET <filename> RET. You will see a log of passed and failed tests, which should allow you to investigate the source of the error and ultimately fix the bug.

Adding a new test file is trivial: just create a new file here and it will be run. There is no need to byte-compile any of the files in this directory—the test-harness will take care of any necessary byte-compilation.

Look at the existing test cases for the examples of coding test cases. It all boils down to your imagination and judicious use of the macros Assert, Check-Error, Check-Error-Message, and Check-Message.

Here’s a simple example checking case-sensitive and case-insensitive comparisons from case-tests.el.

(with-temp-buffer
  (insert "Test Buffer")
  (let ((case-fold-search t))
    (goto-char (point-min))
    (Assert (eq (search-forward "test buffer" nil t) 12))
    (goto-char (point-min))
    (Assert (eq (search-forward "Test buffer" nil t) 12))
    (goto-char (point-min))
    (Assert (eq (search-forward "Test Buffer" nil t) 12))

    (setq case-fold-search nil)
    (goto-char (point-min))
    (Assert (not (search-forward "test buffer" nil t)))
    (goto-char (point-min))
    (Assert (not (search-forward "Test buffer" nil t)))
    (goto-char (point-min))
    (Assert (eq (search-forward "Test Buffer" nil t) 12))))

This example could be inserted in a file in tests/automated, and it would be a complete test, automatically executed when you run make check after building SXEmacs. More complex tests may require substantial temporary scaffolding to create the environment that elicits the bugs, but the top-level Makefile and test-harness.el handle the running and collection of results from the Assert, Check-Error, Check-Error-Message, and Check-Message macros.

In general, you should avoid using functionality from packages in your tests, because you can’t be sure that everyone will have the required package. However, if you’ve got a test that works, by all means add it. Simply wrap the test in an appropriate test, add a notice that the test was skipped, and update the skipped-test-reasons hashtable. Here’s an example from syntax-tests.el:

;; Test forward-comment at buffer boundaries
(with-temp-buffer

  ;; try to use exactly what you need: featurep, boundp, fboundp
  (if (not (fboundp 'c-mode))

      ;; We should provide a standard function for this boilerplate,
      ;; probably called `Skip-Test' -- check for that API with C-h f
      (let* ((reason "c-mode unavailable")
	     (count (gethash reason skipped-test-reasons)))
	(puthash reason (if (null count) 1 (1+ count))
		 skipped-test-reasons)
	(Print-Skip "comment and parse-partial-sexp tests" reason))

    ;; and here's the test code
    (c-mode)
    (insert "// comment\n")
    (forward-comment -2)
    (Assert (eq (point) (point-min)))
    (let ((point (point)))
      (insert "/* comment */")
      (goto-char point)
      (forward-comment 2)
      (Assert (eq (point) (point-max)))
      (parse-partial-sexp point (point-max)))))

Skip-Test is intended for use with features that are normally present in typical configurations. For truly optional features, or tests that apply to one of several alternative implementations (eg, to GTK widgets, but not Athena, Motif, MS Windows, or Carbon), simply silently omit the test.