New Relic

Introduction

In Transact version 2020.1.04, we introduced support for New Relic. New Relic is a widely used third-party monitoring tool that allows you to gather data based on custom events in Transact. You can use custom events to measure and monitor the following information:

  • The number of documents processed.
  • The number of pages processed per minute.
  • The average size of the documents being processed.

Prerequisites

To use New Relic with Transact, the following prerequisites must be in place:

  • You must be on Transact 2020.1.04 or above.
  • You must have a New Relic account.

Note: If you do not already have a New Relic account, you can sign up for a free trial on the New Relic site.

Feature Overview

New Relic is an observability platform that allows you to bring in data from any digital source so that you can fully understand your system and how to improve it. Data is reported in the form of events, which have multiple attributes (key-value pairs). Event data will be reported to your New Relic platform, where you can query to retrieve data.

The following actions are now being posted as New Relic custom events:

  • User logging into Transact. The following user login types are tracked:
    • AD
    • LDAP
    • SAML-based SSO
    • Default Tomcat authentication
  • Batch is picked up for processing. Processing through the following ingestion methods are tracked:
    • Upload Batch page
    • Web Scanner
    • Email Import
    • UNC Folder
    • CMIS Import
  • Batch arrives at the review state
  • Batch arrives at the validation state
  • Batch processes successfully
  • Batch fails to process

Using these events, you can query for the following information:

  • Number of documents processed during a specific time period
  • Pages processed per minute
  • Average size of documents being processed

New Relic Startup Configuration

The following steps outline how to configure New Relic for on-premises Transact licenses.

  1. Download the latest version of the newrelic-java.zip file from New Relic.
  2. Extract the ZIP file.
  3. Update the newrelic.yml file.
    1. Locate the following line: license_key: '<%= license_key %>'
    2. Replace '<%= license_key %>' with your New Relic license key.
    3. Locate the following line: app_name: My Application
    4. Replace the line ‘My Application’ with the name of your application.
  4. Copy the New Relic folder to [Ephesoft_Directory]\JavaAppServer on the Transact server.
  5. Go to [Ephesoft_Directory]\JavaAppServer\bin
  6. Right-click EphesoftTransact.exe and select Run as administrator
  7. Click the Java tab and add the following param:
    1. -javaagent:C:\Ephesoft\JavaAppServer\newrelic\newrelic.jar
  8. Restart the Transact service.

To access your data, go to one.newrelic.com and click the query your data icon, displayed in the screenshot below.

New Relic Custom Events - Batch Events

This section will cover the custom batch events that are supported for New Relic.

Batch Picked Up

When a batch is picked up for processing, a custom event will be sent to New Relic containing the following information.

EventName: BATCH_PICKED_UP_EVENT {
appId: <type number> (assigned by New Relic),
appName: <type String> "(app_name Defined in New Relic configuration file(newrelic.yml)),
batchClassIdentifier: <type String> ,
batchClassName: <type String> ,
batchInstanceIdentifier: <type String> ,
batchName: <type String> ,
batchStartTime: <type number> (Epoch Unix Time Stamp),
batchStatus: <type String> ,
entityGuid: <type String> (assigned by New Relic),
host: <type String> ,
realAgentId: <type number> (assigned by New Relic),
timestamp: <type number> (assigned by New Relic)
}

Sample Query

SELECT * from BATCH_PICKED_UP_EVENT SINCE 30 days ago

A query can be raised against appName like

SELECT * from BATCH_PICKED_UP_EVENT where appName = 'APPLICATION-NAME' SINCE 30 days ago

SELECT * from BATCH_PICKED_UP_EVENT where appName ='APPLICATION-NAME' and batchInstanceIdentifier = 'BATCH IDENTIFIER' SINCE 30 days ago

Batch Ready For Review

When a batch is in Review, a custom event will be sent to New Relic containing the following information.

EventName: BATCH_READY_FOR_REVIEW_EVENT
{
appId:<type number>(assigned by New Relic),
appName:<type String>"(app_name Defined in New Relic configuration file(newrelic.yml)),
batchClassIdentifier:<type String>,
batchClassName:<type String>,
batchDocumentsSize:<type number>(In Bytes),
batchInstanceIdentifier:<type String>,
batchName:<type String>,
batchProcessingTime:<type number>(millisecond),
batchSource:<type String>,
batchStartTime:<type number>(Epoch Unix Time Stamp),
batchStatus:<type String>,
entityGuid:<type String>(assigned by New Relic),
host:<type String>,
numDocuments:<type number>,
numPages:<type number>,
numSourceDocuments:<type number>,
realAgentId:<type number>(assigned by New Relic),
timestamp:<type number>(assigned by New Relic)
}

Sample Query

Find all batches that entered the READY_FOR_REVIEW state in the last 30 days:

SELECT * from BATCH_READY_FOR_REVIEW_EVENT SINCE 30 days ago

Find all batches that entered the READY_FOR_REVIEW state in the last 30 days for a specific Transact Instance:

SELECT * from BATCH_READY_FOR_REVIEW_EVENT where appName = 'APPLICATION-NAME' SINCE 30 days ago

Find whether a specific batch entered the READY_FOR_REVIEW state in the last 30 days:

SELECT * from BATCH_READY_FOR_REVIEW_EVENT where appName ='APPLICATION-NAME' and batchInstanceIdentifier = 'BATCH IDENTIFIER' SINCE 30 days ago

Batch Ready For Validation

When a batch is in Validation, a custom event will be sent to New Relic containing the following information.

EventName: BATCH_READY_FOR_VALIDATION_EVENT
{
appId:<type number>(assigned by New Relic),
appName:<type String>"(app_name Defined in New Relic configuration file(newrelic.yml)),
batchClassIdentifier:<type String>,
batchClassName:<type String>,
batchDocumentsSize:<type number>(In Bytes),
batchInstanceIdentifier:<type String>,
batchName:<type String>,
batchProcessingTime:<type number>(millisecond),
batchSource:<type String>,
batchStartTime:<type number>(Epoch Unix Time Stamp),
batchStatus:<type String>,
entityGuid:<type String>(assigned by New Relic),
host:<type String>,
numDocuments:<type number>,
numPages:<type number>,
numSourceDocuments:<type number>,
realAgentId:<type number>(assigned by New Relic),
timestamp:<type number>(assigned by New Relic)
}

Batch Finished

When a batch finishes successfully, a custom event will be sent to New Relic containing the following information.

EventName: BATCH_FINISHED_EVENT
{
appId:<type number>(assigned by New Relic),
appName:<type String>"(app_name Defined in New Relic configuration file(newrelic.yml)),
batchClassIdentifier:<type String>,
batchClassName:<type String>,
batchDocumentsSize:<type number>(In Bytes),
batchFinishTime: <type number>(Epoch Unix Time Stamp),,
batchInstanceIdentifier:<type String>,
batchName:<type String>,
batchProcessingTime:<type number>(millisecond),
batchSource:<type String>,
batchStartTime:<type number>(Epoch Unix Time Stamp),
batchStatus:<type String>,
entityGuid:<type String>(assigned by New Relic),
host:<type String>,
numDocuments:<type number>,
numPages:<type number>,
numSourceDocuments:<type number>,
realAgentId:<type number>(assigned by New Relic),
timestamp:<type number>(assigned by New Relic)
}

Sample Query

SELECT * from BATCH_FINISHED_EVENT SINCE 30 days ago

SELECT * from BATCH_FINISHED_EVENT where appName = 'APPLICATION-NAME' SINCE 30 days ago

SELECT * from BATCH_FINISHED_EVENT where appName ='APPLICATION-NAME' and batchInstanceIdentifier = 'BATCH IDENTIFIER' SINCE 30 days ago

The number of ingested documents number documents processed (classified documents):

SELECT numSourceDocuments,numDocuments from BATCH_FINISHED_EVENT where appName ='APPLICATION-NAME' and batchInstanceIdentifier = 'BATCH IDENTIFIER' SINCE 30 days ago

The number of documents processed against the given duration:

SELECT count(numDocuments) from BATCH_FINISHED_EVENT where appName ='APPLICATION-NAME' SINCE 30 days ago

The number of documents processed against batch class within a given duration:

SELECT count(numDocuments) from BATCH_FINISHED_EVENT where appName ='APPLICATION-NAME' FACET batchClassName SINCE 30 days ago

Pages per minute (PPM) processed:

SELECT sum(numPages)/((max(batchFinishTime) - min(batchStartTime))/60) FROM BATCH_FINISHED_EVENT where appName ='APPLICATION-NAME' limit 2 Since 1 hours ago

Batch Failed

When batch processing fails, a custom event will be sent to New Relic containing the following information.

EventName: BATCH_FAILED_EVENT
{
appId:<type number>(assigned by New Relic),
appName:<type String>"(app_name Defined in New Relic configuration file(newrelic.yml)),
batchClassIdentifier:<type String>,
batchClassName:<type String>,
batchDocumentsSize:<type number>(In Bytes),
batchInstanceIdentifier:<type String>,
batchName:<type String>,
batchProcessingTime:<type number>(millisecond),
batchSource:<type String>,
batchStartTime:<type number>(Epoch Unix Time Stamp),
batchStatus:<type String>,
entityGuid:<type String>(assigned by New Relic),
host:<type String>,
numDocuments:<type number>,
numPages:<type number>,
numSourceDocuments:<type number>,
realAgentId:<type number>(assigned by New Relic),
timestamp:<type number>(assigned by New Relic)
}

Sample Query

SELECT * from BATCH_FAILED_EVENT SINCE 30 days ago

SELECT * from BATCH_FAILED_EVENT where appName = 'APPLICATION-NAME' SINCE 30 days ago

SELECT * from BATCH_FAILED_EVENT where appName ='APPLICATION-NAME' and batchInstanceIdentifier = 'BATCH IDENTIFIER' SINCE 30 days ago

Events can be queried using the sending event name; For example, to query the record for the finished batch.

SELECT host,appName,batchClassIdentifier,batchSource,numDocuments FROM BATCH_FINISHED_EVENT Since 30 days ago

The number of documents processed against the host/app name can be queried as

SELECT sum(numDocuments) as 'documents processed' FROM BATCH_FINISHED_EVENT FACET host Since 30 days ago

The number of documents processed against host and batch source can be queried as

SELECT sum(numDocuments) as 'documents processed' FROM BATCH_FINISHED_EVENT FACET host,batchSource Since 30 days ago

New Relic Custom Events - User Events

When you log in, a custom event will be sent to New Relic containing the following information. The AD, LDAP, SSO, and Tomcat (default) login methods will be counted.

EventName: USER_LOGIN_EVENT
{
appId:<type number>(assigned by New Relic),
appName:<type String>"(app_name Defined in New Relic configuration file(newrelic.yml)),
entityGuid:<type String>(assigned by New Relic),
host:<type String>,
userRoles:<String>,
realAgentId:<type number>(assigned by New Relic),
timestamp:<type number>(assigned by New Relic),
userName:<String>
}

Example: AD instance

EventName: USER_LOGIN_EVENT
{
"appId": 123456789,
"appName": "APPLICATION-NAME",
"entityGuid": "67e03f59c9b54d3988818cdfd47f09f6",
"host": "A1BC23DEF",
"userRoles": ["Tax Digital Transformation SecG1"],
"realAgentId": 123456789,
"timestamp": 1600343402576,
"userName": "engineering"
}

Example: SSO instance

EventName: USER_LOGIN_EVENT
{
"appId": 123456789,
"appName": APPLICATION-NAME,
"entityGuid": "67e03f59c9b54d3988818cdfd47f09f6",
"host": "A1BC23DEF",
"realAgentId": 123456789,
"timestamp": 1600970940583,
"userName": "ephesoft",
"userRoles":"["CAPTURE-Administrator","CAPTURE-SystemAdministrator","CAPTURE-User","Administrator"]"
}

Example: Tomcat instance

EventName: USER_LOGIN_EVENT
{
"appId": 123456789,
"appName": "APPLICATION-NAME",
"entityGuid": "67e03f59c9b54d3988818cdfd47f09f6",
"host": "A1BC23DEF
"realAgentId": 123456789,
"timestamp": 1600972596106,
"userName": "ephesoft",
"userRoles": "engineering"
}

The number of users logged in against the host/app name can be queried as

SELECT count(*) as nummber_of_users_logged_in from USER_LOGIN_EVENT where appName = <Your Application Name> SINCE 30 days ago

Conclusion

This completes an overview of integrating New Relic in Transact.

 

by kelseybarkis