8.8 Techniques for SXEmacs Developers

To make a purified SXEmacs, do: make puremacs. To make a quantified SXEmacs, do: make quantmacs.

You simply can’t dump Quantified and Purified images (unless using the portable dumper). Purify gets confused when xemacs frees memory in one process that was allocated in a different process on a different machine!. Run it like so:

temacs -batch -l loadup.el run-temacs xemacs-args...

Before you go through the trouble, are you compiling with all debugging and error-checking off? If not, try that first. Be warned that while Quantify is directly responsible for quite a few optimizations which have been made to SXEmacs, doing a run which generates results which can be acted upon is not necessarily a trivial task.

Also, if you’re still willing to do some runs make sure you configure with the ‘--quantify’ flag. That will keep Quantify from starting to record data until after the loadup is completed and will shut off recording right before it shuts down (which generates enough bogus data to throw most results off). It also enables three additional elisp commands: quantify-start-recording-data, quantify-stop-recording-data and quantify-clear-data.

If you want to make SXEmacs faster, target your favorite slow benchmark, run a profiler like Quantify, gprof, or tcov, and figure out where the cycles are going. In many cases you can localize the problem (because a particular new feature or even a single patch elicited it). Don’t hesitate to use brute force techniques like a global counter incremented at strategic places, especially in combination with other performance indications (e.g., degree of buffer fragmentation into extents).

Specific projects:

• Make the garbage collector faster. Figure out how to write an incremental garbage collector.
• Write a compiler that takes bytecode and spits out C code. Unfortunately, you will then need a C compiler and a more fully developed module system.
• Speed up redisplay.
• Speed up syntax highlighting. It was suggested that “maybe moving some of the syntax highlighting capabilities into C would make a difference.” Wrong idea, I think. When processing one 400kB file a particular low-level routine was being called 40 million times simply for one call to newline-and-indent. Syntax highlighting needs to be rewritten to use a reliable, fast parser, then to trust the pre-parsed structure, and only do re-highlighting locally to a text change. Modern machines are fast enough to implement such parsers in Lisp; but no machine will ever be fast enough to deal with quadratic (or worse) algorithms!
• Implement tail recursion in Emacs Lisp (hard!).

Unfortunately, Emacs Lisp is slow, and is going to stay slow. Function calls in elisp are especially expensive. Iterating over a long list is going to be 30 times faster implemented in C than in Elisp.

Heavily used small code fragments need to be fast. The traditional way to implement such code fragments in C is with macros. But macros in C are known to be broken.

Macro arguments that are repeatedly evaluated may suffer from repeated side effects or suboptimal performance.

Variable names used in macros may collide with caller’s variables, causing (at least) unwanted compiler warnings.

In order to solve these problems, and maintain statement semantics, one should use the do { ... } while (0) trick while trying to reference macro arguments exactly once using local variables.

Let’s take a look at this poor macro definition:

#define MARK_OBJECT(obj) \
  if (!marked_p (obj)) mark_object (obj), did_mark = 1

This macro evaluates its argument twice, and also fails if used like this:

  if (flag) MARK_OBJECT (obj); else do_something();

A much better definition is

#define MARK_OBJECT(obj) do { \
  Lisp_Object mo_obj = (obj); \
  if (!marked_p (mo_obj))     \
    {                         \
      mark_object (mo_obj);   \
      did_mark = 1;           \
    }                         \
} while (0)

Notice the elimination of double evaluation by using the local variable with the obscure name. Writing safe and efficient macros requires great care. The one problem with macros that cannot be portably worked around is, since a C block has no value, a macro used as an expression rather than a statement cannot use the techniques just described to avoid multiple evaluation.

In most cases where a macro has function semantics, an inline function is a better implementation technique. Modern compiler optimizers tend to inline functions even if they have no inline keyword, and configure magic ensures that the inline keyword can be safely used as an additional compiler hint. Inline functions used in a single .c files are easy. The function must already be defined to be static. Just add another inline keyword to the definition.

inline static int
heavily_used_small_function (int arg)
{
  ...
}

Inline functions in header files are trickier, because we would like to make the following optimization if the function is not inlined (for example, because we’re compiling for debugging). We would like the function to be defined externally exactly once, and each calling translation unit would create an external reference to the function, instead of including a definition of the inline function in the object code of every translation unit that uses it. This optimization is currently only available for gcc. But you don’t have to worry about the trickiness; just define your inline functions in header files using this pattern:

INLINE_HEADER int
i_used_to_be_a_crufty_macro_but_look_at_me_now (int arg);
INLINE_HEADER int
i_used_to_be_a_crufty_macro_but_look_at_me_now (int arg)
{
  ...
}

The declaration right before the definition is to prevent warnings when compiling with gcc -Wmissing-declarations. I consider issuing this warning for inline functions a gcc bug, but the gcc maintainers disagree.

Every header which contains inline functions, either directly by using INLINE_HEADER or indirectly by using DECLARE_LRECORD must be added to inline.c’s includes to make the optimization described above work. (Optimization note: if all INLINE_HEADER functions are in fact inlined in all translation units, then the linker can just discard inline.o, since it contains only unreferenced code).

To get started debugging SXEmacs, take a look at the gdbinit file in the src directory. See the section in the SXEmacs FAQ on How to Debug an SXEmacs problem with a debugger.

After making source code changes, run make check to ensure that you haven’t introduced any regressions. If you want to make xemacs more reliable, please improve the test suite in tests/automated.

Did you make sure you didn’t introduce any new compiler warnings?

Before submitting a patch, please try compiling at least once with

configure --enable-mule --enable-debug

Here are things to know when you create a new source file:

• All .c files should #include <config.h> first. Almost all .c files should #include "lisp.h" second.
• Generated header files should be included using the ‘#include <...>’ syntax, not the ‘#include "..."’ syntax. The generated headers are:

  config.h sheap-adjust.h paths.h Emacs.ad.h

  The basic rule is that you should assume builds using ‘--srcdir’ and the ‘#include <...>’ syntax needs to be used when the to-be-included generated file is in a potentially different directory at compile time. The non-obvious C rule is that ‘#include "..."’ means to search for the included file in the same directory as the including file, not in the current directory. Normally this is not a problem but when building with ‘--srcdir’, make will search the ‘VPATH’ for you, while the C compiler knows nothing about it.

• Header files should not include ‘<config.h>’ and ‘"lisp.h"’. It is the responsibility of the .c files that use it to do so.

Here is a checklist of things to do when creating a new lisp object type named foo:

1. create foo.h
2. create foo.c
3. add definitions of syms_of_foo, etc. to foo.c
4. add declarations of syms_of_foo, etc. to symsinit.h
5. add calls to syms_of_foo, etc. to emacs.c
6. add definitions of macros like CHECK_FOO and FOOP to foo.h
7. add the new type index to enum lrecord_type
8. add a DEFINE_LRECORD_IMPLEMENTATION call to foo.c
9. add an INIT_LRECORD_IMPLEMENTATION call to syms_of_foo.c