.. meta::
    :description: Infrastructure Reports

.. _reports:

Reports
=======

The task of report generation consists of producing a PDF document from a set of data.
This data is generally obtained as a result of a task or workflow.

The role of the plugin (workflow) is to generate:

#. an XML data file, following a specific structure, and containing all the required data that may be used in a
   report;
#. all the required image files that may be included in the report.

The generation of a PDF document from a list of XML data files is done through this procedure:

#. the content of the XML data files is merged into a single XML data tree;
#. the merged XML data tree is transformed through an XSLT file to obtain a valid LaTeX document;
#. the LaTeX document and all the image files specified by the XML data files are processed with a LaTeX compiler to
   produce a PDF document.

The generation of a PDF document from a list of XML data files through a given XSLT UUID is done by calling the
method :meth:`ORSServiceClass.xslt.xsltutil.XSLTUtil.generatePDFFromXML`.

Structure of the XML data files
-------------------------------

The XML data files that can be provided to be processed should follow this structure::

    <?xml version="1.0" encoding="UTF-8" ?>
    <orsresult>
        <application>
            <name>Dragonfly</name>
            <version>2021.3</version>
            <company>Object Research Systems (ORS), Inc.</company>
        </application>
        <pluginresult>
            <pluginname>APluginClassName_aba92fe8276711ecb98af8ac6585ab6e</pluginname>
            <reportkind>General</reportkind>
            <reportversion>1.0</reportversion>
            (any other fields)
        </pluginresult>
        (any other <pluginresult> nodes)
    </orsresult>

Such XML data files may be generated by subclassing :class:`ORSServiceClass.report.abstractreportgenerator.AbstractReportGenerator`.

Node <orsresult>
^^^^^^^^^^^^^^^^

It acts as the root node of the XML data document.
It contains a single ``<application>`` node, then any (but usually 1) number of ``<pluginresult>`` nodes.

Node <application>
^^^^^^^^^^^^^^^^^^

This node contains the details of the application that was used to generate the results.
It contains the name, the version and the name of the company corresponding to the application.

Node <pluginresult>
^^^^^^^^^^^^^^^^^^^

This node contains a set of fields describing a given result.
It contains at least of the nodes ``<pluginname>``, ``<reportkind>`` and ``<reportversion>``.
Any other fields may be added to contain the data of the result, for example the value of a measurement or the name
of an image file::

    <?xml version="1.0" encoding="UTF-8" ?>
    <orsresult>
        <application>
            <name>Dragonfly</name>
            <version>2021.3</version>
            <company>Object Research Systems (ORS), Inc.</company>
        </application>
        <pluginresult>
            <pluginname>TestReports_2a03860d260411ecaba4f8ac6585ab6e</pluginname>
            <reportkind>General</reportkind>
            <reportversion>1.0</reportversion>
            <objectType>Fibers</objectType>
            <sliceZIndex>50</sliceZIndex>
            <classificationLimits>
                <classificationLimitMinAcceptableAspectRatio>0.2</classificationLimitMinAcceptableAspectRatio>
                <classificationLimitMinAcceptableAspectRatioFormattedAsPercentage>20.00</classificationLimitMinAcceptableAspectRatioFormattedAsPercentage>
            </classificationLimits>
            <classificationResults>
                <classificationResult>
                    <label>1</label>
                    <value2DArea>0.060200000000000004</value2DArea>
                    <value2DAspectRatio>0.2801645295226413</value2DAspectRatio>
                    <isAccepted>True</isAccepted>
                </classificationResult>
                <classificationResult>
                    <label>2</label>
                    <value2DArea>0.055325000000000006</value2DArea>
                    <value2DAspectRatio>0.3509453642901375</value2DAspectRatio>
                    <isAccepted>True</isAccepted>
                </classificationResult>
                <classificationResult>
                    <label>3</label>
                    <value2DArea>0.045875</value2DArea>
                    <value2DAspectRatio>0.1849959000552852</value2DAspectRatio>
                    <isAccepted>False</isAccepted>
                </classificationResult>
                <classificationResult>
                    <label>4</label>
                    <value2DArea>0.07365000000000001</value2DArea>
                    <value2DAspectRatio>0.5893092092531802</value2DAspectRatio>
                    <isAccepted>True</isAccepted>
                </classificationResult>
                <classificationResult>
                    <label>5</label>
                    <value2DArea>0.08332500000000001</value2DArea>
                    <value2DAspectRatio>0.44606048263179615</value2DAspectRatio>
                    <isAccepted>True</isAccepted>
                </classificationResult>
                <classificationResult>
                    <label>6</label>
                    <value2DArea>0.10942500000000001</value2DArea>
                    <value2DAspectRatio>0.6245362263039491</value2DAspectRatio>
                    <isAccepted>True</isAccepted>
                </classificationResult>
                <classificationResult>
                    <label>7</label>
                    <value2DArea>0.129425</value2DArea>
                    <value2DAspectRatio>0.5796558703103682</value2DAspectRatio>
                    <isAccepted>True</isAccepted>
                </classificationResult>
            </classificationResults>
            <isSliceAccepted>False</isSliceAccepted>
            <imageWorkingSliceWithSegmentation>
                <figure>
                    <file>Fibers_50_2021_10_07_08_46_09_imageWorkingSliceWithSegmentation_1.png</file>
                    <caption></caption>
                </figure>
            </imageWorkingSliceWithSegmentation>
            <iconAccepted>
                <figure>
                    <file>iconAccepted_1.png</file>
                    <caption></caption>
                </figure>
            </iconAccepted>
            <iconRejected>
                <figure>
                    <file>iconRejected_1.png</file>
                    <caption></caption>
                </figure>
            </iconRejected>
        </pluginresult>
    </orsresult>

Node <pluginname>
^^^^^^^^^^^^^^^^^

This node contains the name of the plugin that generated the result data.
The purposes are essentially:

- for a human reader of the data file, to know what workflow this data has been generated from;
- to identify the expected set of nodes;
- to determine if a given XSL transform is applicable.

Node <reportkind>
^^^^^^^^^^^^^^^^^

This node contains a name, for the purpose of supporting multiple kinds of reports from a single plugin.

Node <reportversion>
^^^^^^^^^^^^^^^^^^^^

This node contains a version number, for the purpose of knowing if a given XSL transform is applicable.
It should be incremented when the set of nodes is changed and previous versions of XSL transformation would not be
applicable.

Structure of the merged XML data
--------------------------------

When generating a PDF document from a list of XML data files, the contents of the XML data files are merged into a
single XML data tree to be transformed by the XSLT file.
The merging is done by concatenating all ``<orsresult>`` nodes from the XML data files into a single
``<orsresults>`` node, as::

    <orsresults>
        <orsresult sourceDataLocation="D:/reports/slice_00000/">
            <application>
                <name>Dragonfly</name>
                <version>2021.3</version>
                <company>Object Research Systems (ORS), Inc.</company>
            </application>
            <pluginresult>
                <pluginname>TestReports_2a03860d260411ecaba4f8ac6585ab6e</pluginname>
                <reportkind>General</reportkind>
                <reportversion>1.0</reportversion>
                <objectType>Fibers</objectType>
                <sliceZIndex>0</sliceZIndex>
                (...)
            </pluginresult>
        </orsresult>
        <orsresult sourceDataLocation="D:/reports/slice_00050/">
            <application>
                <name>Dragonfly</name>
                <version>2021.3</version>
                <company>Object Research Systems (ORS), Inc.</company>
            </application>
            <pluginresult>
                <pluginname>TestReports_2a03860d260411ecaba4f8ac6585ab6e</pluginname>
                <reportkind>General</reportkind>
                <reportversion>1.0</reportversion>
                <objectType>Fibers</objectType>
                <sliceZIndex>50</sliceZIndex>
                (...)
            </pluginresult>
        </orsresult>
        <orsresult sourceDataLocation="D:/reports/slice_00100/">
            <application>
                <name>Dragonfly</name>
                <version>2021.3</version>
                <company>Object Research Systems (ORS), Inc.</company>
            </application>
            <pluginresult>
                <pluginname>TestReports_2a03860d260411ecaba4f8ac6585ab6e</pluginname>
                <reportkind>General</reportkind>
                <reportversion>1.0</reportversion>
                <objectType>Fibers</objectType>
                <sliceZIndex>100</sliceZIndex>
                (...)
            </pluginresult>
        </orsresult>
    </orsresults>

.. note::
    The method :meth:`ORSServiceClass.xslt.xsltutil.XSLTUtil.generateMergedXMLs` can be used to generate such a merged
    XML from a set of input files, for the purpose of developing a XSL transformation file.

Structure of the XSL transformation files
-----------------------------------------

The XSL transformation files should follow this structure::

    <?xml version="1.0" encoding="UTF-8"?>
    <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                    xmlns:ors="http://theobjects.com"
                    version="1.0">

      <ors:bibliography>
        <title>General report to test the reports</title>
        <documentation>Report for the TestReports plugin.</documentation>
        <authors>ORS Team</authors>
        <contact>http://theobjects.com</contact>
        <email>info@theobjects.com</email>
        <organization>Object Research Systems (ORS), Inc.</organization>
        <address>460 Ste-Catherine Ouest, suite 600, Montréal, Québec, Canada, H3B 1A7</address>
        <copyright>Object Research Systems (ORS), Inc. All rights reserved 2021.</copyright>
        <creationdate>Oct 05 2021 17:00</creationdate>
        <version>1.0.0</version>
        <uuid>fa032a43262111ecaba7f8ac6585ab6e</uuid>  <!-- Should be unique -->
      </ors:bibliography>

      <ors:support>
        <pluginresult>
          <pluginname>TestReports_2a03860d260411ecaba4f8ac6585ab6e</pluginname>
          <reportkind>General</reportkind>
          <reportversion>1.0</reportversion>
        </pluginresult>
      </ors:support>

      <xsl:output method="text"/>

      <!-- xsl:template: root -->
      <xsl:template match="/orsresults">
        <xsl:text>Some text to be put in the LaTeX document.&#x0a;</xsl:text>
        <!-- (any other xsl command) -->
      </xsl:template>

    </xsl:stylesheet>

Node <ors:bibliography>
^^^^^^^^^^^^^^^^^^^^^^^

This node contains the description of this XSL transformation (for example, what kind of report it produces).
This node contains also the node ``<uuid>``, used to identify uniquely this transformation.

.. warning::
    The node ``<uuid>`` should be unique amongst all XSLT files found by the application.

Node <ors:support>
^^^^^^^^^^^^^^^^^^

This node describes the supported data. Each ``<pluginresult>`` child node should contain this set of nodes:

- ``<pluginname>``: name of the plugin that generated the data;
- ``<reportkind>``: name of the report kind;
- ``<reportversion>``: report version number.

Validation of the transformation of a list of XML data files through an XSLT file can be done by calling the method
:meth:`ORSServiceClass.xslt.xsltutil.XSLTUtil.isXSLTransformationSupported`.

An XSL transformation is considered as supported if any of the ``<pluginresult>`` node found in the XSL file
matches any ``<pluginresult>`` node found in the XML data file.


Node <xsl:template match="/orsresults">
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This node contains the actual processing of the transformation of the XML data into a LaTeX document.

Tutorials
---------

Here are some links to tutorials and references about XPath and XSLT:

- `XPath Tutorial <https://www.w3schools.com/xml/xpath_intro.asp>`_
- `XSLT Tutorial <https://www.w3schools.com/xml/xsl_intro.asp>`_
- `XSLT Elements Reference <https://www.w3schools.com/xml/xsl_elementref.asp>`_
- `XSLT, XPath, and XQuery Functions <https://www.w3schools.com/xml/xsl_functions.asp>`_
- `XML Path Language (XPath) Version 1.0 <https://www.w3.org/TR/1999/REC-xpath-19991116/>`_
- `XSL Transformations (XSLT) Version 1.0 <https://www.w3.org/TR/1999/REC-xslt-19991116>`_


.. TODO Documentation video on making reports:
    - plugin to generate a report (plugin development, XML data structure, LaTeX file, XSL development) from the analysis of a dataset
    - activate logging in this plugin, make a macro to generate a single report from a dataset
    - macro builder to batch reports over a set of datasets in folders
    - use report generator to do summary report (XSL development from existing XML files)
    - in plugin, add (logging) method to take a list of report files, generate a XML report file of summary plot, add
      this file in list of XML files to use and send it to report generator with XSL adapted to include summary plot
    - to generate/debug an XSL file:
    -- make a working LaTeX example
    -- look at tutorial/reference pages (XPath, XSLT, references) on W3Schools
    -- generate the merged XML file (as it would be provided to the XSL transform file) with XSLTUtil.generateMergedXMLs or OrsReportGenerator/exportCurrentInputFilesAsMergedXML
    -- use the JetBrains IntelliJ IDEA (Community Edition), Visual Studio, or PyCharm (with XSLT plugins) to debug XSL by using breakpoints
    -- output file may be viewed in TeXnicCenter for syntax or comparison to generate the PDF
    -- use report generator to perform the full task of generating the report
    -- break in XSLTUtil/generatePDFFromXML code to see the output log of the LaTeX compilation and run it again
    - output (in documentation web page) should contain:
    -- plugin;
    -- macro (single dataset);
    -- macro (batch);
    -- XSL (single dataset);
    -- XSL (summary);
    -- datasets.
    - how to debug when the PDF is not generated because of an error in the LaTeX file?
    -- break in PyCharm to keep the temporary files alive (XML data, images)(.tex, .log)
    -- in Notepad++, examine the .log file for errors, look in .tex file at corresponding line;
    -- copy text from .tex file in a temporary project in TeXnicCenter an run. Image files are put with their absolute
       path (in a temporary folder), so the temporary folder should be kept alive, or the contents may be copied and all
       paths in .tex file changed for the new path
    -- LaTeX compilation .log file shown in PyCharm console
    -- instead of making a temporary directory for the data (plugin side), the XML data and images files may be
       generated directly in the output folder.
    -- Mechanism (class variable CLEANUP_TEMPORARY_FILES=True in XSLTUtil, changeable in startup_script or from ScriptRunner)
       to keep the temporary folders (the one from the plugin and the one of the LaTeX generated files), so that the
       LaTeX file can be generated again with the .xml and image files still alive in their temporary folder and the
       LaTeX log file showing the full output.