Work with Fixed Form Document Extraction

The Fixed Form feature uses optical character recognition or optical character reader (OCR) in Hero Platform_ letting users convert images of typed or printed text into machine-encoded text.

Using Fixed Form, users can spend less time and avoid risks associated with human error turning forms into data that can be processed.

Considerations

Open Documents

Open AI from the navigation menu and select Documents.

Document Overview Page

View the saved document models.

This includes Fixed Form, Invoice, and Semi-structured models.

The documents overview page displays all of the saved document models.

You can view the:

• Document model names - Name given to identify the document model.
• Number of versions - The number of different versions for the document model.
  • Individual versions can be accessed within the Document Studio above the layout template image.
    
• Created - The date the document model was created.
• Settings:
  • Edit -  Open the model in Document Studio to edit the model.
  • Clone - Copy the document model.
  • Delete -  Delete the document model.
  • Configuration - Edit the name of the model.

Create a Fixed Form Model

To create a new Fixed Form model:

1. After opening Documents in Hero Platform_, click Create Document Model.
   
   
2. Enter a name for the Fixed form model and click Next.
   
   
3. Select Fixed Forms and click Next.
   
   

4. Select a layout template from the file browser by clicking Choose file...
   Click Save & Preview.
   

   A Fixed Form layout template is the reference image to base the OCR readings on for future Input images.
   
   Supported file types:

   • .jpg
   • .png
   • .pdf

   
   
   

5. The layout template image is displayed in the Document Studio.
   The Document Studio allows users to define the fields where the text to convert is located.
   
   
6. Click Field in the toolbar to define a field to convert the text within.
   Click and drag on the layout template image to create a field in which to convert the text within.
   
   
7. Configure the Add Field form
   

   Enter a name for the field.
   Select a field type:
   

   • Typed text - The field searches and registers typed text. (String)
     • Supports input for unlimited characters.
     • Supports the English language per default.
       • Select a different language(s) in the Document Studio settings.
       • Supported languages:
         
         • Arabic, Bulgarian, Chinese (Simplified), Chinese (Traditional), Czech, Dutch/Flemish, German, Greek, English, French, Hindi, Croatian, Hungarian, Indonesian, Italian, Japanese, Korean, Norwegian, Polish, Portuguese, Russian, Slovak, Slovenian, Spanish/Castilian, Thai, Turkish, Ukrainian
     • Supports Knowledge Graphs.
   • Handwritten text - The field searches and registers handwritten text. (String)
     • Supports input for up to 100 characters.
     • The handwriting engine expects input to be one line of text. Use multiple inputs for multiple lines of handwritten text.
     • Supports the English language.
     • Supports Knowledge Graphs.
     • Supports letterbox fields.
       
       • Letterbox fields are printed boxes expecting individual characters. 
       • For best results, do not include the letterbox horizontal line in the drawn field.
       
       
   • Barcode - The field searches and registers a barcode or QR code. 
     • Supports input for EAN-13/UPC-A, UPC-E, EAN-8, Code 128, Code 39, Codabar, Interleaved 2 of 5, and QR Code.
   • Checkbox - The field searches and registers a single checkbox. (Boolean)

     • Supports any marking in the selected checkbox.

   • Numbers (Typed/Handwritten) - The field searches for numbers that could be typed or handwritten. (String)
     • Supports input for up to 30 characters. Recognize numbers and .-/ characters.
     • Supports letterbox fields.
       
       • Letterbox fields are printed boxes expecting individual characters. 
       • For best results, do not include the letterbox horizontal line in the drawn field.
       
       
   
   Click OK to save.
   
   
   
   
8. Repeat the process of creating fields for each field on the layout template image.
   
   
9. Click Save after all needed fields have been created. 

Knowledge Graph

The Knowledge Graph is an Automation Hero Context Aware feature. Our OCR engines look at the possible expected field values in your configured Knowledge Graph to make better predictions when reading a field. This results in better field-level accuracy than a standard dictionary lookup or validation methods.

What is a Knowledge Graph

A Knowledge Graph is a designated bucket stored in the Data Store. The Fixed Form Data Extractor uses those expected/similar values in the Knowledge Graph to help the OCR engine make predictions. This increases the probability of the OCR engine producing accurate results. 

Creating a Knowledge Graph

Before starting to create a Knowledge Graph to use in Hero Platform_, it is required that you have a data source that includes a field(s) of as many possible expected/similar values that the OCR model may read from the fields later created in the Document Studio.

General guidelines for creating a Knowledge Graph:

• The more distinctive each entry in your Knowledge Graph is, the greater the possibility the Knowledge Graph can improve the accuracy of the results.
  • Limitation: If all the Knowledge Graph values are very similar (e.g., long strings with only one character differences or series of sequential numbers) there is a higher chance of false positive results.
• There is not a minimum or maximum amount of values required for a Knowledge Graph.
  • Try creating Knowledge Graphs with as many values expected to be read by OCR as possible.
  • Knowledge Graphs with more values take longer to process than those with fewer values.

Adding a Knowledge Graph 

1. Add a list(s) of values by importing a file directly to the Data Store or create an Input from a data source and then use the Automatic Output to store the file in the Data Store.
2. After the data is located within the Data Store, Open the Data Store, click the settings icon of the file, and select Enable Knowledge Graph.

This data file can now be selected as a Knowledge Graph in the Document Studio when configuring fields for a Fixed Form data extraction model. 

Use a Knowledge Graph in a Fixed Form Model

Configure the Fixed Form model to use a Knowledge Graph when adding type text and handwritten text fields.

1. Draw a field on the sample PDF or image in the Document Studio, give the field a name and select the field type. Typed text and Handwritten text fields support use of a Knowledge Graph.
2. Select a Knowledge Graph from the drop-down list which contain expected field values that correspond to the drawn field.
3. Select the corresponding field from that Knowledge Graph which contains the values to compare with the drawn field.
4. Select the Match strategy:
   1. Word - Select this option when a field's value tries to match an individual word from the Knowledge Graph.
   2. Phrase - Select this option when a field's value tries to match multiple words from a Knowledge Graph.
5. The results of the OCR engine using the Knowledge Graph are displayed in the results preview section of the Document Studio with either two or three returned values.
   1. The predicted value of the OCR engine alone. (Always returned)
   2. If a result was found, the predicted value of the OCR engine using the supplied Knowledge Graph. (Returned if found)
   3. The predicted "best_match". The best match value is the Knowledge Graph value. If no Knowledge Graph value was returned, the best match value is the OCR engine value. (Always returned)

Results Preview

The results preview section in the Document Studio displays the created field's names, the OCRed text values, and the confidence score indicating the accuracy of the prediction.

If a Knowledge Graph was utilized, the additional results are displayed in a drop-down under the original OCR engine predictions for the field value.

When using a Fixed Form function in a Flow, each field's results, including multiple Knowledge Graph results, are displayed in a separate field. 

View the results preview

The results preview is located to the right of the layout template image and can be toggled on/off with the Preview button in the toolbar.

Edit, Delete, and Filter Fields

Fields created in the Document Studio can be edited or deleted from the panel on the right under the fields heading.

Edit a field

To edit a field:

1. Locate the field name in the fields panel and click the edit icon.
   
2. The field configuration form is displayed and can be edited.
   
3. Click OK to save the field configuration.

Delete a field

To delete a field:

1. Locate the field name in the fields panel and click the X icon.
   Confirm to remove the field.
   or
   Mark the box to the left of the field name and then click Remove selected fields.
   Confirm to remove the field.
   
2. The field is deleted from the Document Studio layout template image.

Filter Fields

Filter displayed fields by name:

1. Locate the field name in the fields panel.
2. In the "Filter by field name" field, enter the filter string.
   1. Fields matching the filter string are displayed.
   2. View all fields by removing the filter string.

Filter displayed fields by page:

1. Locate the field name in the fields panel.
2. In the "Filter by page number" field, add or remove page numbers from the drop-down list.
   1. Fields associated with the page number are displayed.
   2. View all fields by adding all page numbers to the filter.

Layout Template Images with Multiple Pages

Document Studio supports multi page images.

Navigate to different pages of a template in the Document Studio by scrolling up/down within the image area.

The page a field is located on is shown on the fields panel.

Delete a page

To delete a page:

1. Locate the page number in the field panel and click the X icon.
2. Click Yes, delete.

The page and all fields residing on the page are permanently deleted. 

Settings

Click the Settings heading to display the files selected for the Fixed Form model.

AI Page Identification

A Fixed Form Data Extraction model in a Flow assumes that all the input images match both the order and number of images in the model as created in the Document Studio.

Turning on the AI page identification features allows for processing input images in a Flow even if they don't match the order and number of the sample images from the Document Studio model. 

This feature requires processing power. If it is known that all the input images have the same order and number of images as defined in the model, it is recommended not to use this feature. This feature is turned off by default. 

Languages

Select a language used for values in typed text fields that are created for the model.

List of supported languages. 

Delete Sample Images

Remove/delete a Fixed Form image by clicking the X icon.

Document Studio Toolbar and Controls

Toolbar

At the top of the Document Studio, a toolbar is displayed.

• Save - Select to save the Fixed Form model at the current progression. 
  • Saved Fixed Form models are displayed on the OCR overview page.
• Undo - Select to revert the previous action made in the Document Studio.
• Redo - Select to progress from the previous undo action made in the Document Studio.
• Field - Select to create a field on the layout template image.
• Reset View - Reverts to the default view of the document in Document Studio.
• Zoom in - Zoom in on the layout template image.
  
  • Keyboard shortcut - option+mouse wheel
• Zoom out - Zoom out on the layout template image.
  • Keyboard shortcut - option+mouse wheel
• Preview - Toggles the results preview panel and resets the layout template image to the starting position.
• Close - Select to close the Document Studio.

Use a Fixed Form Model in a Flow

After a Fixed Form model has been saved, it can be used as a function in a Flow.

To use a Fixed Form model in a Flow:

1. Open and start creating a Flow in the Flow Studio.
2. View the Document functions in the element browser.
3. Click and drag a Fixed Form model from the element browser onto the Flow Studio canvas.
4. Connect the Fixed Form model using a cable from an element in the Flow.
5. Select the version number for the Fixed Form model.

6. Select the coordinates type:

   Relative coordinates is the recommended option. Relative coordinates are more stable and can adjust for document scaling while absolute coordinates may require adjustments for document scaling changes. Support for absolute coordinates will be removed in a future release.

   • Absolute Coordinates - returns (output field) the position of a value box on a document by pixel location on a document.
     • Metadata (Tuple) containing x, y, w, h (Long) values

   • Relative Coordinates - returns (output field) the position of a value box on a document by percentage space on that document.

     • Page_bounding_box (Tuple) containing boundingBox (Tuple) containing left, top, width, height (Double) values.

7. Configure/review the fields for the Fixed Form model's containerized function deployment.

   1. Capture logs - Select if the containerized function should capture logs.
   2. RAM - Adjust the sliding bar for memory (RAM) allocation for the function.
   3. vCPU - Adjust the sliding bar for CPU consumption. (by cores)
   4. Attempt timeout(s) - Enter the timeout setting (in seconds).
   5. Initial Delay - Enter the initial delay value in seconds for amount of time to between when container starts and when the Flow begins to use it.
   6. Retry attempts - Enter the max retry attempts before failing.

   Automation Hero recommends leaving the container settings at the default levels unless problems arise. 

   Examples of when raising the default settings may be beneficial:

   • The documents being processed are very large.
   • The Fixed Form Extraction model has a large number of fields and/or pages.
   • The Fixed Form Extraction model heavily uses the handwriting engine.   
8. Add an argument(s)
   1. Binary and List(Binary) data types are supported.
9. Click OK to finish adding the Fixed Form model to the Flow. 

Viewing the Results

Results for a field are displayed in a List data type with four separate elements:

Example: [Smith, 0.93, MEMORY{size=2698}, [1377, 744, 139, 52, 1]]

• Element 1: The value OCR predicts is read.
• Element 2: The confidence rating that the prediction was correct.
• Element 3: The size of the snippet of the bounding box on the binary file in bytes.
• Element 4: The location of the field on the binary.

Fixed Form Snippet Outputs

After adding a Fixed Form model in a Flow, the fields created in the Document Studio are available to use an output field in the Output. This type of field is called a snippet.

A snippet is an image (Binary data type) of what OCR captures from the fields created in the Document Studio.