Saxonica.com

Streaming of Large Documents

Saxon 8.5 introduced a new optimization called streaming copy, specifically designed for processing of large documents, where the need to allocate sufficient memory to hold the entire source tree is traditionally a problem.

This currently works only in XSLT, and is supported only in Saxon-SA (though the stylesheet does not need to be schema-aware). It involves no new language constructs, but needs to be enabled by means of the extension attribute saxon:read-once="yes". It is invoked by a stereotypical coding pattern that the optimizer recognizes and treats specially.

A very simple way of using this technique is when making a selective copy of parts of a document. For example, the following code creates an output document containing all the footnote elements from the source document that have the attribute @type='endnote':


<xsl:template name="main">
  <footnotes>
    <xsl:copy-of select="doc('thesis.xml')//footnote[@type='endnote']"
                   saxon:read-once="yes" xmlns:saxon="http://saxon.sf.net/"/>
  </footnotes>
</xsl:template>

Note the restrictions below on the kind of predicate that may be used.

More typically, the copied nodes will be further processed. For example:


<xsl:function name="f:customers">
 <xsl:copy-of select="doc('customers.xml')/*/customer"
                   saxon:read-once="yes" xmlns:saxon="http://saxon.sf.net/"/>
</xsl:function>

<xsl:template name="main">
  <xsl:apply-templates select="f:customers()"/>
</xsl:template>

<xsl:template match="customer">
  <xsl:value-of select="code, name, location" separator="|"/>
  <xsl:text>&#xa;</xsl:text>
</xsl:template>

It's not necessary for such a stylesheet to have a principal source document, the transformation can be invoked instead using the -it main option from the command line, or its equivalent in the Java API.

The important factors here are:

The implementation of this facility typically uses multithreading. One thread (which operates as a push pipeline) is used to read the source document and filter out the nodes selected by the path expression. The nodes are then handed over to the main processing thread, which iterates over the selected nodes using an XPath pull pipeline. Because multithreading is used, this facility is not used when tracing is enabled. It should also be disabled when using a debugger (there is a method in the Configuration object to achieve this.)

In cases where the entire stylesheet can be evaluated in "push" mode (as in the first example above), there is no need for multithreading: the selected nodes are written directly to the current output destination.

Note that a tree is built for each selected node, and its subtree. Trees are also built for all nodes selected by the path expression, whether or not the satisfy the filter (if they do not satisfy the filter, they will be immediately discarded from memory). The saving in memory comes when these nodes are processed one at a time, because each subtree can then be discarded as soon as it has been processed. There is no benefit if the stylesheet needs to perform non-serial processing, such as sorting. There is also no benefit if the path expression selects a node that contains most or all of the source document, for example its outermost element.

Saxon can handle expressions that select nested nodes, for example //section where one section contains another. However, the need to deliver nodes in document order makes the pipeline somewhat turbulent in such cases, increasing memory usage.

Serial processing in this way is not actually faster than conventional processing (in fact, when multithreading is required, it may only run at half the speed). Its big advantage is that it saves memory, thus making it possible to process documents that would otherwise be too large for XSLT to handle. There may also be environments where the multithreading enables greater use of the processor capacity available. To run without this optimization, either change the xsl:copy-of instruction to xsl:sequence, or set saxon:read-once to "no".

Next