Section:

Intelligent Data Extraction on Server/Desktop

Installation Instructions

Using PIP with Python

When using Python on Windows or Linux you can install the package via PIP with this command:

sh

1pip install --extra-index-url=https://pypi.apryse.com apryse-data-extraction

Using NPM with Node.js

When using Node.js on Windows or Linux you can install the package via NPM with this command:

sh

1npm install @pdftron/data-extraction

Installing directly on other platforms

For Windows, just copy DataExtractionModuleWindows.zip in your PDFNetC folder, then extract it locally. You should have files like

  • Lib\Windows\StructuredOutput.exe
  • Lib\Windows\OCRModule.exe
  • Lib\Windows\TabularData\TabularData.dll
  • Lib\Windows\AIPageObjectExtractor\AIPageObjectExtractor.dll

For Linux, just copy DataExtractionModuleLinux.tar.gz in your PDFNetC directory, then extract it locally. You should have files like

  • Lib/Linux/StructuredOutput
  • Lib/Linux/OCRModule
  • Lib/Linux/TabularData/TabularData
  • Lib/Linux/AIPageObjectExtractor/AIPageObjectExtractor

JSON Output Specification

Please refer to the below specifications to learn more about the output JSON format.

Usage

If you are using PIP or NPM, you may skip setting AddResourceSearchPath. Otherwise, follow the directions below.

The first thing to set up before the module can be used is the location of the Lib directory under which the external add-ons are installed, so that the SDK knows where to look for them. This is achieved via the PDFNet AddResourceSearchPath function. If a relative path is used, it is based on the end-user executable.

1PDFNet.AddResourceSearchPath("../../../../../Lib/");

Note: do not specify the actual Windows, Linux, MacOS directory, where the individual executables are, but its parent folder.

For error handling purposes, it is generally advisable to test whether the module is available via the IsModuleAvailable function. Since the Data Extraction suite consists of multiple modules, an extra parameter is used to clarify the component to test.

1if (!DataExtractionModule.IsModuleAvailable(DataExtractionModule.DataExtractionEngine.e_tabular))
2{
3   // Unable to run Data Extraction: PDFTron SDK Tabular Data module not available.
4}
5if (!DataExtractionModule.IsModuleAvailable(DataExtractionModule.DataExtractionEngine.e_doc_structure))
6{
7   // Unable to run Data Extraction: PDFTron SDK Structured Output module not available.
8}
9if (!DataExtractionModule.IsModuleAvailable(DataExtractionModule.DataExtractionEngine.e_form))
10{
11   // Unable to run Data Extraction: PDFTron SDK AIFormFieldExtractor module not available.
12}

If you have the module installed but the function still returns false, please double check that the correct path was used in AddResourceSearchPath earlier.

Tabular Data Extraction

In this mode of operation, text on each page will become a part of a single table, like in a spreadsheet document. Headers, footers, paragraphs, lists, page numbers all come out of a combined tabular structure.

There is a choice between two types of output formats, either a JSON or an Excel document. JSON is more suitable for programmatic content discovery, natural language processing, statistical analysis, artificial intelligence. Excel is ideal for standard spreadsheet operations, calculations, formulas and charting.

Extract tabular data as JSON file

Specify the name of the input PDF file and the name of the output JSON file, then select the Tabular engine:

1DataExtractionModule.ExtractData("table.pdf", "table.json", DataExtractionModule.DataExtractionEngine.e_tabular);

Extract tabular data as JSON string

If you are going to parse the JSON right away, you may as well retrieve it as an in-memory string, instead of an external file.

Specify the name of the input PDF file, then select the Tabular engine:

1string json = DataExtractionModule.ExtractData("financial.pdf", DataExtractionModule.DataExtractionEngine.e_tabular);

Extract tabular data as Excel file

Specify the name of the input PDF file and the name of the output XLSX file:

1DataExtractionModule.ExtractToXLSX("table.pdf", "table.xlsx");

Extract tabular data as Excel stream

Specify the name of the input PDF file and an output filter, such as MemoryFilter:

1MemoryFilter output_xlsx_stream = new MemoryFilter(0, false);
2DataExtractionModule.ExtractToXLSX("financial.pdf", output_xlsx_stream);

Document Structure Recognition

In this mode of operation, the full logical structure is discovered, including paragraphs, lists, tables, headers, footers, images, graphics, like in a typical word processor. Section columns are differentiated from table columns, and paragraph text is separated from tables, although table cells may contain further paragraphs.

While Tabular Data Extraction is more focused on cells, financial tables, spreadsheets, formulas, currency signs, percentages and calculations on cells, Document Structure Recognition aims at the visual accuracy of documents, such as indentation, gaps, line spacing, alignment, borders and colors.

Extract document structure as JSON file

Specify the name of the input PDF file and the name of the output JSON file, then select the Doc Structure engine:

1DataExtractionModule.ExtractData("paragraphs_and_tables.pdf", "paragraphs_and_tables.json", DataExtractionModule.DataExtractionEngine.e_doc_structure);

Extract document structure as JSON string

If you are going to parse the JSON right away, you may as well retrieve it as an in-memory string, instead of an external file.

Specify the name of the input PDF file, then select the Doc Structure engine:

1string json = DataExtractionModule.ExtractData("tagged.pdf", DataExtractionModule.DataExtractionEngine.e_doc_structure);

Form Field Identification

In this mode of operation, the input is assumed to be a form. We currently offer 2 Form Field Identification Engines: "Form Field Detection" and "Form Field Key-Value Extraction".

Both engines require GLIBC 2.31 or newer on Linux, such as Debian 11 or Ubuntu 10.04 or newer

Form Field Detection Engine

Our Form Field Detection engine surveys the page layout and finds the most probable arrangement of the individual form fields, along with the type of the identified fields, such as text fields or check boxes.

The output is presented in a straightforward JSON format, where each field is made up of a type, confidence value and bounding box coordinates.

Form Field Key-Value Extraction Engine

Our Form Field Key-Value Extraction engine provides all the same information as the Form Field Detection engine, with some additional data provided for fields. For each form field in the document, an attempt is made to find the corresponding key (the label or question associated with a field) and value (the data within the field).

The output is presented in a JSON format, similar to the output of the Form Field Detection engine, with extra data fields for field key, label, and value text.

Extract form fields as JSON file

Specify the name of the input PDF file and the name of the output JSON file, then select the Form engine:

1DataExtractionModule.ExtractData("formfields-scanned.pdf", "formfields-scanned.json", DataExtractionModule.DataExtractionEngine.e_form);

Alternatively, you can select the Form Key-Value Extraction engine:

1DataExtractionModule.ExtractData("formfields-scanned.pdf", "formfields-scanned.json", DataExtractionModule.DataExtractionEngine.e_form_key_value);

Extract form fields as JSON string

If you are going to parse the JSON right away, you may as well retrieve it as an in-memory string, instead of an external file.

Specify the name of the input PDF file, then select the Form engine:

1string json = DataExtractionModule.ExtractData("formfields.pdf", DataExtractionModule.DataExtractionEngine.e_form);

Alternatively, you can select the Form Key-Value Extraction engine:

1string json = DataExtractionModule.ExtractData("formfields.pdf", DataExtractionModule.DataExtractionEngine.e_form_key_value);

Extract form fields and add to PDF

You can automatically add detected forms to a PDF in a single step.

Java

1PDFDoc doc = new PDFDoc("formfields.pdf");
2DataExtractionModule.detectAndAddFormFieldsToPDF(doc);
1PDFDoc doc = new PDFDoc("formfields.pdf");
2DataExtractionModule.DetectAndAddFormFieldsToPDF(doc);

By default, this function uses the Form Field Detection engine. To use the Form Field Key-Value Extraction engine, see below.

Generic Key-Value Extraction

NEW FEATURE

New in 11.3!

BETA

This engine is in beta: we expect the quality of the output to increase in subsequent releases, and there may be minor changes to the output schema.

The Generic Key-Value Extraction engine automatically identifies and extracts meaningful key-value pairs from textual documents. This allows users to quickly structure information from documents with minimal manual effort. The engine scans the document text for keys, which are typically important terms, roles, or categories, and their corresponding values, which provide specific details or identifiers (we call this type of relationship hyponymic, where the value (or hyponym) is a specific instantiation of the more general key (or hypernym)).

Extract key-value pairs as JSON file

Specify the name of the input PDF file and the name of the output JSON file, then select the Generic Key Value engine:

1DataExtractionModule.ExtractData("newsletter.pdf", "newsletter.json", DataExtractionModule.DataExtractionEngine.e_generic_key_value);

Extract key-value pairs as JSON string

Specify the name of the input PDF file, then select the Generic Key Value engine:

1string json = DataExtractionModule.ExtractData("newsletter.pdf", DataExtractionModule.DataExtractionEngine.e_generic_key_value);

Data Extraction Options

Although the default options will satisfy most common use cases, we offer a couple of options to customize the extraction behavior and unlock lesser-used functionality.

The options object is passed as the last parameter to any extraction function, as shown below.

Select OCR Language

Use the Language option to set the preferred OCR language(s). If you work with scanned documents in languages other than English, specify one or more 3-letter ISO 639-2 language codes, separated by spaces. For example, "eng deu spa fra" for English, German, Spanish, French. You may also use comma or plus as a separator.

Supported languages:

  • eng: English
  • deu or ger: German
  • fra or fre: French
  • ita: Italian
  • rus: Russian
  • spa: Spanish

Note: Listing too many languages at once may hurt performance and accuracy. If you know the exact language, it is always best to use that single setting.

1DataExtractionOptions options = new DataExtractionOptions();
2options.SetLanguage("fra spa"); // French and Spanish
3DataExtractionModule.ExtractData("table.pdf", "table.json", DataExtractionModule.DataExtractionEngine.e_tabular, options);

Specify PDF Password

Use the PDFPassword option to specify a PDF password if one is required.

Encrypted PDF files that are protected by a password may only be opened when the password is specified in addition to the filename. No password is necessary for files that can be viewed without any authentication.

1DataExtractionOptions options = new DataExtractionOptions();
2options.SetPDFPassword("password123"); // password for input PDF
3DataExtractionModule.ExtractData("table.pdf", "table.json", DataExtractionModule.DataExtractionEngine.e_tabular, options);

Select a Page Range

Use the Pages option to restrict the extraction to a selected range of pages.

This can be a single page number (such as "1" for the first page), or a range separated by a dash (such as "1-5", or "7-" for 7 and beyond). An empty string means all pages are extracted.

1DataExtractionOptions options = new DataExtractionOptions();
2options.SetPages("1"); // extract page 1
3DataExtractionModule.ExtractData("table.pdf", "table.json", DataExtractionModule.DataExtractionEngine.e_tabular, options);

Specify Regions of Interest

NEW FEATURE

New in 11.3!

You can specify regions to include or exclude from analysis for each page in a document using the Inclusion Zone and Exclusion Zone options for a page. These options specify rectangles in user-space coordinates that allow developers to either include or exclude a region from analysis. For example, if a document has a table that you don't want to analyze, you could specify it's bounding box as an exclusion zone, or if a document has only one paragraph that you care about, you could use an inclusion zone. If no zones are specified for a page, the entire page is included in analysis.

Inclusion and exclusion zones can be combined to create complex regions of interest. Inclusions zones are combined by union, and exclusion zones are subtracted.

This option is only supported for the Form, FormKeyValue, and GenericKeyValue engines at this time.

Inclusion and Exclusion example

1DataExtractionOptions options = new DataExtractionOptions();
2
3RectCollection p4InclusionZones = new RectCollection();
4RectCollection p4ExclusionZones = new RectCollection();
5p4InclusionZones.AddRect(30, 432, 562, 684);
6p4ExclusionZones.AddRect(30, 657, 295, 684);
7options.AddInclusionZonesForPage(p4InclusionZones, 4);
8options.AddExclusionZonesForPage(p4ExclusionZones, 4);
9
10DataExtractionModule.ExtractData("newsletter.pdf", "newsletter_key_val_with_zones.json", DataExtractionModule.DataExtractionEngine.e_generic_key_value, options);

Deep Learning Assist

Specifies if Deep Learning is used with table recognition in the DocStructure engine. Table recognition accuracy improves at the cost of increased processing time. This only affects the DocStructure engine.

1DataExtractionOptions options = new DataExtractionOptions();
2options.SetDeepLearningAssist(true); // Enable Deep learning assistant
3DataExtractionModule.ExtractData("table.pdf", "table.json", DataExtractionModule.DataExtractionEngine.e_DocStructure, options);

Preserve existing form fields when adding to PDF

When automatically detecting form fields and adding them to a document, you can force the module to preserve any existing form annotations that are already present in the document, only adding newly detected fields.

1PDFDoc doc = new PDFDoc("formfields.pdf");
2DataExtractionOptions options = new DataExtractionOptions();
3options.SetOverlappingFormFieldBehavior("KeepOld");
4DataExtractionModule.DetectAndAddFormFieldsToPDF(doc, options);

Use Key-Value Extraction engine with DetectAndAddFormFieldsToPDF

By default, DetectAndAddFormFieldsToPDF uses the Form Field Detection engine. You can force the function to use the Form Field Key-Value Extraction engine using the "Form Extraction Engine" option.

1PDFDoc doc = new PDFDoc("formfields.pdf");
2DataExtractionOptions options = new DataExtractionOptions();
3options.SetFormExtractionEngine("FormKeyValue");
4DataExtractionModule.DetectAndAddFormFieldsToPDF(doc, options);

NOTE

This option only has an effect on the `DetectAndAddFormFieldsToPDF` function. Passing this option to `ExtractData` will have no effect, as the `engine` parameter will take precedence.

Did you find this helpful?

Trial setup questions?

Ask experts on Discord

Need other help?

Contact Support

Pricing or product questions?

Contact Sales