CloverETL and the best practices

CloverETL graphs use algorithms for transformation or copying of data between clover fields or records. There are two basic requirements that every algorithm have to satisfy, the first is low memory requirement and the second is a fast speed of processing, this document deals with source code optimization, its main part concerns how to use (or write) some of clover classes, their right and wrong using.

DataRecord and DataField

DataRecord represents one data record with structure based on provided metadata object. Fields of this data record are created only once and updated when it is needed.

// create new DataRecord
DataRecord record = new DataRecord(metadata);
 
// init record - creates data fields, values of fields are setted to java default value,
// it means the string = "", long = 0, ...
record.init();
 
// reset record - sets clover default values according to metadata
record.reset();
 
// create the same record from another data record
// HINT: this is the fastest technique how to clone the record
DataRecord recordDuplicated = record.duplicate();
 
// copy all field values from anotherRecord to record
// HINT: this is the fastest technique how to copy values
//       (all fields from anotherRecord must be in record too)
record.copyFrom(anotherRecord);
 
// get field from record
// HINT: this is the fastest technique how to get value from many records
Map names = record.getMetadata().getFieldNames();
String[] fieldsForCopy = {"CustomerId", "LastName", "FirstName"};
DataField[] fields = new DataField[fieldsForCopy.length];
for (int i=0; i<fieldsForCopy.length; i++) {
	fields[i]=record.getField(((Integer)names.get(fieldsForCopy[i])).intValue());
}
// or get indices of data fields
int[] indices = record.getMetadata().fieldsIndices(fieldsForCopy);
for (int i=0; i<indices.length; i++) {
	fields[i]=record.getField(indices[i]);
}
 
...
// and get value from field in some record iteration
for (int i=0; i<fieldsForCopy.length; i++) {
	fields[i].getValue();
}
 
// a wrong code for getting value from many records
// String[] fieldsForCopy = {"CustomerId", "LastName", "FirstName"};
// ...
// for (String fieldName : fieldsForCopy) {
// 	record.getField(fieldName).getValue();
// }
 
// An operation which sets/resets field to its initial value (just after it was created by JVM)
//  - it varies depending on type of field. Nullable fields are set to NULL, non-nullable are zeroed,
//  if they have default value defined, then default value is assigned.
// HINT: this is useful to use if we copy values into some fields, another fields can have values from 
//       previous records.
fields[x].reset();