CloverETL wiki

best_practicles

2009-09-16T12:27:17+02:00

CloverETL graphs use algorithms for transformation or copying of data between clover fields or records. There are two basic requirements that every algorithm have to satisfy, the first is low memory requirement and the second is a fast speed of processing, this document deals with source code optimization, its main part concerns how to use (or write) some of clover classes, their right and wrong using.

code_guide

2009-09-16T12:27:17+02:00

---------- The purpose of this document is to give some insight into Clover's source code structure. It explains packages and some key classes within them. For more detailed infromation, please look at JavaDoc documentation generated from Clover's source code.

companytransactionstutorial

2009-09-16T12:27:17+02:00

This project aims to present an overview of how users can process the information about transactions made by their company. Remember that you must set Java memory size to at least 512MB in order to run these graphs. All of the graphs presented in the project are sorted by the numbers that are included in their names in ascending order. And this is exactly the order in which these graphs should be run. The basic files of the projects are the following three: Transactions.dat, Children.dat a…

compile_time_constants

2010-05-11T12:12:29+02:00

CloverETL framework can be configured through defaultProperties resource file which is located in “org/jetel/data/” subdirectory of CloverETL engine. Various parameters stored in this file are loaded by the engine at run-time and are used during the transformation execution. These parameters can not be altered once the engine is running. These predefined values stored in the file can be overloaded by custom configuration file on startup, where could be predefined a subset of the properties. Path…

complex_examples

2009-09-16T16:02:28+02:00

To download example package go to download page. Complex examples This examples combines several components to perform useful transformation similar to one you could use in a real life. Join example This graphs combines information about orders made and information about individual items/products purchased within each order. It also adds information about which customer ordered the goods. To get all this information, it needs to join data from ORDERS.DBF (a dBase table) ODETAILS.DBF and Cu…

component_building

2009-09-16T12:27:17+02:00

Introduction Components (or nodes in graph terminology) are independent units which perform simple (or sometimes complicated) transformations on data. Each component has 0 to N inputs and 0 to N outputs through which flows data. These inputs and outputs are represented by ports - Input Ports for reading data into component nad Output Ports for writing data out of component. Each component runs as one thread (but of course can start other, if neccessary). Even though it shares memory with othe…

database_examples

2009-09-16T16:01:28+02:00

To download example package go to download page. We assume that examples are unzipped to the CloverETL home directory. When executing examples from command line you should be in the examples/ExtExamples subdirectory (eg. /home/user/cloverETL/examples/ExtExamples). (Older examples are not divided into four subdirectories and you should run them within their examples directory. Thus, in the commands the groups of caracters (../../) should be replaced by ../ only.) Remember that (on Windows)…

database_using_jdbc_how-to

2009-09-16T12:27:17+02:00

How CloverETL works with databases To simplify things, CloverETL uses JDBC to work (talk to) with databases. If your database of heart has JDBC driver, CloverETL can be used to unload data stored within database table or can populate database table with internal data.

embedding_clover

2012-03-08T16:20:00+02:00

Note: if you are interested in CloverETL Engine internals, check out Engine Overview presentation. The TransformationGraph class Transformation graph is both abstraction and class which performs some operation. Graph keeps track of all Nodes, Edges, metadata objects. It is also accompanied by class which enables reading the definition of graph from XML file and building everything dynamically.

examples

2009-09-16T16:01:53+02:00

To download example package go to download page. Simple Examples SimpleCopy example This example illustrates usage of SimpleCopy and Trash components. It reads employees.dat file, parses the records, sends them to SimpleCopy component which in turn replicates them to several Trash components connected to SimpleCopy component. One of the Trash components is configured to dump all incoming records into text file Trash1_output.txt - mostly for debugging purpose. The Trash component normally dis…

examples_index

2009-09-16T16:02:55+02:00

To download example package go to download page. Index of all examples graphAggregateSorted.grf This graph illustrates usage of Aggregate component. graphAggregateUnsorted.grf This graph illustrates usage of Aggregate component. graphAproximativeJoin.grf In this example data are read from from file customers0.dat and from database table employee; then for both flows there is generated matching key consisting of 4 letters of last name (flat file: lname, database: last_name) and 3 letters o…

file_reading

2009-09-16T12:27:17+02:00

If you want to read a flat file or an XML file as one string field, use UniversalDataReader and set the metadata on its first output port as follows:

function_building

2010-05-19T12:47:01+02:00

Introduction In addition to the prepared CTL functions, users can build their own CTL functions. To do that, they need to write their own code defining the custom CTL functions and specify its plugin. Code of Custom CTL Functions Each custom CTL function library must be derived/inherited from org.jetel.interpreter.extensions.TLFunctionLibrary class. Each custom CTL function must be derived/inhereted from org.jetel.interpreter.extensions.TLFunctionPrototype class. These classes have s…

graph_xml_schema

2010-02-25T14:41:45+02:00

This file was created to allow you to test if a CloverETL graph is valid or not. You can find several XSD validators online or write your own in Java (or Perl, Python, C++, etc.). This file should be correct for 99% of graphs, but there may be some bugs. When you validate with this file, you must specify that the graph is in the CloverETL namespace by specifying the xmlns attribute in the Graph element:

hrads_page

2009-09-16T12:27:18+02:00

just testing to see how fast the network connection is seems to be working very well!

navigation

2011-03-08T18:26:36+02:00

---------- CloverETL Tool ---------- Transformation graph Running & debugging ---------- Graph elements * Components * Connections * Edges * Lookup tables * Metadata * Properties * Sequences ---------- Graph components

parser_formatter_building

2009-09-16T12:27:17+02:00

The CloverETL engine provides support for reader and writer development. The developer can utilize the benefits that the parser and formatter interfaces offer to create their own component faster and more efficiently. These interfaces have many functions that other Clover classes use. The easiest way to understand how to use the parser and formatter is to look at the Clover DataReader or DataWriter source code. This document introduces some examples and suggest a variety of formats to use.

running

2012-05-16T11:33:40+02:00

We assume that you have downloaded a binary package of CloverETL framework and Clover Examples or you have built one from sources. Extract the examples to the engine directory. Thus, if you have your engine in the pathtoyourengine/cloverETL/ directory, after extracting the examples to this directory, it will contain the following subdirectories: bin/, docs/, examples/, lib/ and plugins/. The examples can be found in the examples/ subdirectory.

simple_examples

2009-09-16T16:00:23+02:00

To download example package go to download page. We assume that examples are unzipped to the CloverETL home directory. When executing examples from command line you should be in the examples/SimpleExamples subdirectory (eg. /home/user/cloverETL/examples/SimpleExamples). (Older examples are not divided into four subdirectories and you should run them within their examples directory. Thus, in the commands the groups of caracters (../../) should be replaced by ../ only.) Remember that (on Wi…

start

2010-09-14T10:27:28+02:00

CloverETL is a Java based ETL framework which can be used to transform structured data, and to some degree free-form data. Its complete platform independence is achieved by the exclusive use of Java technology. It can be used standalone as an application launched from a command line or can be embedded into your own application. CloverETL is released under the LGPL license.

test

2009-09-16T12:27:17+02:00

tlsyntaxdiagrams

2009-09-16T12:27:17+02:00

transf_java_preprocess

2009-09-16T12:27:17+02:00

!!! IMPORTANT !!! Transformation Language Lite is now considered obsolete. Use Transformation Language (CTL) instead ! The purpose for this concept is to handle parsing Java transformation code enhanced with CloverETL Transformation Language Lite syntax. This allows using CloverETL specific transformation code which is then unrolled to fully compilable Java source. Thus constructs in the Transformation Language Lite can be used without loosing the speed of compiled byte code.

transf_lang_examples

2010-03-22T14:21:07+02:00

Code examples Code Output int i=10; int j=9; date date1=2006-01-01;print_err(date1); date date2=2006-02-01;print_err(date2); boolean result; boolean compareDates=date1<=date2; print_err(compareDates); if (date1<=date2) { print_err('before if (i

transf_lang_functions

2010-05-14T12:25:17+02:00

Listed below are all Clover ETL TL language functions which are part of the standard release. Conversion functions base64byte (since 2.4) Converts string containing BASE64 encoded data into binary representation - bytearray. bytearray base64byte(string src);

transf_lang_syntax

2010-03-23T11:14:30+02:00

Program structure Program in Transform Language consist of two parts. The first part is declarations of variables, function declarations and statements. They don't have to be in this order, but variables and functions are visible after their declaration. You can interspersed these parts. In the second part, there is a mapping, where you can assign values to output data records. After mapping there are not allowed declarations nor statements.

transformation_concept

2010-02-25T14:43:49+02:00

Transformation graph (further referred to only as graph) describes how to transform data from one form to another. Graph consists of at least three elements, Nodes (perform various simple transformations), Edges (connect Nodes and pass data around) and Metadata (describe data structure that is defined at every Nodes and Edges).