Streaming of Large Documents
Sometimes source documents are too large to hold in memory. Saxon-EE provides a range
of facilities for
processing such documents in streaming mode: that is, processing data as it is read by the XML parser, without
building a complete tree representation of the document in memory.
Some of these facilities implement new features in the draft XSLT 3.0 standard (also
known as XSLT 2.1). Some
are specific to Saxon, and a few facilities are also available in XQuery.
Inevitably there are things that cannot be done in streaming mode - sorting is an
obvious example. Sometimes,
achieving a streaming transformation means rethinking the design of how it works -
for example, splitting it into
multiple phases. So streaming is rarely a case of simply taking your existing code
and setting a simple switch
to request streamed implementation.
There are basically two ways of doing streaming in Saxon:
Burst-mode streaming: with this approach, the transformation of a large file is broken
up into a sequence of transformations of small pieces of the file. Each piece in turn
is read from the input,
turned into a small tree in memory, transformed, and written to the output file.
This approach works well for files that are fairly flat in structure, for example
a log file holding
millions of log records, where the processing of each log record is independent of
the ones that went before.
A variant of this technique uses the new XSLT 3.0
xsl:iterate instruction to iterate
over the records, in place of
xsl:for-each. This allows working data to be maintained as
the records are processed: this makes it possible, for example, to output totals or
averages at the end of the run,
or to make the processing of one record dependent on what came before it in the file.
instruction also allows early exit from the loop, which makes it possible for a transformation
to process data
from the beginning of a large file without actually reading the whole file.
Burst-mode streaming is available in both XSLT and XQuery, but there is no equivalent
in XQuery to the
Streaming templates: this approach follows the traditional XSLT processing pattern of performing
a recursive descent of the input XML hierarchy by matching template rules to the nodes
at each level, but
does so one element at a time, without building the tree in memory.
Every template belongs to a
mode (perhaps the default, unnamed mode), and streaming is a
property of the mode that can be specified using the new
xsl:mode declaration. If the mode
is declared to be streamable, then every template rule within that mode must obey
the rules for
The rules for what is allowed in streamed processing are quite complicated, but the
is that the template rule for a given node can only read the descendants of that node
once, in order.
There are further rules imposed by limitations in the current Saxon implementation:
for example, although
<xsl:for-each-group group-adjacent="xxx"> is theoretically consistent
with a streamed implementation, it is not currently implemented in Saxon.
The streamed template mechanism applies to XSLT only.
Both these facilities are available in Saxon-EE only. Streamed templates also require
XSLT 3.0 to be enabled by setting
the relevant configuration parameters or command line options.