Creating an OIA ETL JPP scheduler

After implementing OIA on a number of real-world engagements, Ive slowly grown tired of the 3rd party clover ETL scheduler configured OOTB for data imports. From my experience, many organizations have a centralized ETL framework for manipulations so theres no logical justification to segregate OIAs required ETL efforts. But what if theyre missing an ETL framework? Were talking about the minority of companies here though heres an alterative to scheduling imports into OIA.
March 22, 2013
OIA-11g

As clover ETL is a 3rd party technology, you are required to purchase at an independent license, probably the GUI desktop version for 1 system, costing a whopping $4000.

As an alterative, I have stewed more toward the concept of java pre-processors (JPPs) to accommodate the code modification and conformance checking. I find Individuals also prefer the scalability of java as a technology to clover.

https://technicalconfessions.com/images/postimages/postimages/_30_3_JPP_concept.png

The image presents the concept of a JPPs becoming the intermediate from converting any data feeds into the acceptable OIA format, which can then be dropped into the OIA staging repository in preparation to importing them into OIA.

https://technicalconfessions.com/images/postimages/postimages/_30_4_basic_ETL_scheduler.png

With this in mind, I have been suggesting this alternative to companies though I only recently came across the idea of java classes as java beans and allowing OIAs spring framework to schedule the alteration.

https://technicalconfessions.com/images/postimages/postimages/_30_5_web_xml_alteration_in_OIA.png

For scheduling purposes, the java file should extend the TimerTask package which will make the manipulations required to then populate this into the OIA staging as a new file.

https://technicalconfessions.com/images/postimages/postimages/_30_6_explaination_of_bean.png

I have created a new file called jpp-etl.xml, which resides within the WEB-INF directory and altered the web.xml (to reference the jpp-etl.xml file). Once configured, OIA can then take the responsibility on the ETL effort without the dependency of OOTB clover graphs.

https://technicalconfessions.com/images/postimages/postimages/_30_7_adding_line_in_webxml.png

About the author

Daniel is a Technical Manager with over 10 years of consulting expertise in the Identity and Access Management space.
Daniel has built from scratch this blog as well as technicalconfessions.com
Follow Daniel on twitter @nervouswiggles

Comments

Other Posts

ForgeRock OpenIDM - InvalidCredentialException: Remote framework key is invalid

ICFIDMOpenIDMOpenICF

November 8, 2017
Created by: Daniel Redfern
In the past, the similar error occurred though for the Oracle Identity Management solution. invalidcredentialexception remote framework key is invalid Because they all share the ICF connector framework, the error/solution would be the same.
Read More...

org.forgerock.script.exception.ScriptCompilationException: missing ; before statement

IDMsync.confforgerockopenidm

November 8, 2017
Created by: Daniel Redfern
org.forgerock.script.exception.ScriptCompilationException: missing ; before statement
Read More...

ForgeRock IDM - org.forgerock.script.exception.ScriptCompilationException: missing ; before statemen

OpenIDMsync.confForgeRock

September 17, 2017
Created by: Daniel Redfern
ForgeRock IDM - org.forgerock.script.exception.ScriptCompilationException: missing ; before statement
Read More...

Caused by: org.forgerock.json.resource.BadRequestException: Target does not support attribute groups

OpenIDMForgeRockICFConnector

September 17, 2017
Created by: Daniel Redfern
When performing the attempt of a reconciliation from ForgeRock IDM to Active Directory, I would get the following error
Read More...

ForgeRock OpenIDM - InvalidCredentialException: Remote framework key is invalid

OpenIDMForgeRockICFConnectorAD

September 17, 2017
Created by: Daniel Redfern
In the past, the similar error occurred though for the Oracle Identity Management solution. invalidcredentialexception remote framework key is invalid Because they all share the ICF connector framework, the error/solution would be the same.
Read More...

ERROR Caused by com.google.api.client.auth.oauth2.TokenResponseException 400 Bad Request - invalid_g

OpenIDMIDMGoogleGoogle-AppsICFreconciliation

September 12, 2017
Created by: Daniel Redfern
During the reconcilation from OpenIDM to the ICF google apps connector, the following error response would occur. ERROR Caused by com.google.api.client.auth.oauth2.TokenResponseException 400 Bad Request - invalid_grant
Read More...

forgerock-openidm-encryptedjwt-error

OpenIDMIDMForgeRockJWTIAM

August 29, 2017
Created by: Daniel Redfern
Received the JWT error
Read More...

Unexpected character ('¾' (code 190)): expected a valid value

ForgeRock-OpenIDMOpenIDMIDMKeystore

June 25, 2017
Created by: Daniel Redfern
Unexpected character occurred when the IP addresses changes and the virtual instance was migrated into a separate network subnet.
Read More...
E_WARNING Error in file posts.php at line 464: fopen(http://www.technicalconfessions.com/images/postimages/postIcons/pp444.png): failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found E_WARNING Error in file posts.php at line 464: fclose() expects parameter 1 to be resource, boolean given

OpenDJ Error - Connect Error Result Code: 91 (Connect Error)

OpenDJLDAPldapsearchForgeRock

June 5, 2017
Created by: Daniel Redfern
When trying to connect, I would then receive the following error "Connect Error Result Code: 91 (Connect Error)"
Read More...

Tomcat NioEndpoint$SocketProcessor.doRun java.lang.NullPointerException error

TomcatJava-8PKICAS

June 5, 2017
Created by: Daniel Redfern
When initiating the Tomcat instance, the cas-stderr log file will log a SEVERE error logging multiple times every few seconds
Read More...