Spécifications

Node	Attribute	Type	Sub node name	Comments
input			column
	id*	String		ID of the input. Must be unique across all root steps.
	alias	String		Alias of the input. Must be unique across all root steps.
	buffer	Boolean		Enables row buffering (automatically activated when the input is consumed by multiple references).
	name	String
	description	String
column	id*	String		ID of the column.
	type	Type		Type of the column’s content. Defaults to `String`.

Root steps - Step

Generic step container that can contain multiple sub steps.

Node	Attribute	Type	Sub node name	Comments
step	id*	String
	source*	String		ID of the step to read from.
	alias	String
	buffer	Boolean		Enables row buffering (automatically activated when the step is consumed by multiple references).
	name	String
	description	String
	enabled	(Boolean)
			project
			cube
			break
			sort
			filter
			union
			remove
			update
			skip
			join
			notify
			script
			step

Normal steps - Base step

Simple container step that can be used to group sub-steps.

Node	Attribute	Type	Sub node name
step	name	String
	description	String
	enabled	(Boolean)
			project
			cube
			break
			sort
			filter
			union
			remove
			update
			skip
			join
			notify
			script
			step

Normal steps - Projection

This step allows to select columns to be kept and/or to create new columns.

Node	Attribute	Type	Sub node name	Comments
project	id	String
	enabled	(Boolean)
	insert	String		before,after,position,exclude (by default: `exclude`).
	index	int		only if insert is ‘position’ and `column` attribute is not set.
	column	String		only if insert is ‘position’ and `index` attribute is not set.
			copy	Equivalent to `column`. See Copy processor.
			column	See Copy processor.
			value	See Value processor.
			script	See Script processor.
			expr	See Expr processor.
			compute	See Compute processor.
			expression	See Expression processor.
			foreach	See ForEach processor.

Normal steps - Update

This step allows to modify the values of columns.

Node	Attribute	Type	Sub node name	Comments
update	id	String
	enabled	(Boolean)
	buffer	int		Defines the number of input/output rows to hold in memory (accessible with `inputRows` and `outputRows`).
			copy	Equivalent to `column`. See Copy processor.
			column	See Copy processor.
			value	See Value processor.
			script	See Script processor.
			expr	See Expr processor.
			compute	See Compute processor.
			expression	See Expression processor.
			foreach	See ForEach processor.

Normal steps - Remove

This step allows to remove some columns.

Node	Attribute	Type	Sub node name	Comments
remove	id	String
	enabled	(Boolean)
			column
column	id*	String		ID of the column to remove.

Normal steps - Notification

This step will send notifications to registered listeners. Listeners can be added with blueIron.addListener(listener).

Node	Attribute	Type	Comments
notify	id	String
	enabled	(Boolean)
	code	String	Code of the notification event.
	input	String	ID of the column of the notification event.
	type	String	block,row (by default: `row`)

Normal steps - Union

This step will make a union between different input steps.

Node	Attribute	Type	Sub node name	Comments
union	id	String
	enabled	(Boolean)
			source
source	id*	String		ID of the step to read from.

Normal steps - Skip

This step will forward rows that match a filter to the substep, otherwise the substep processing will be skipped (but all the rows will be outputted anyway).

Node	Attribute	Type	Sub node name	Comments
skip	id	String
	enabled	(Boolean)
			filter	Filter to apply on the input rows.
			step	Step to apply when a row matches the filter.

Normal steps - Filter

This step will output only the rows matching the filter.

Node	Attribute	Type	Sub node name	Comments
filter	id	String
	enabled	(Boolean)
	matching	String		all,any (by default: `all`). Defines if all the criteriums must match (`all`) or only one of them (`any`).
			column	Column processor criterium
			script	Script processor criterium
			expr	Expr processor criterium

Normal steps - Script

Executes a JavaScript or JEXL script. The init script (if present) runs once before processing the rows and can modify the output schema; the each script (if present) runs for each row.

At least one of init or each must be defined.

Node	Attribute	Type	Sub node name	Comments
script	id	String
	type*	String		`javascript` or `jexl`.
	input	String		ID of the column exposed as `inputValue` in the script.
			init	Script run once at startup, before any row processing.
			each	Script run for each row.

Normal steps - Sort

Reads all the input rows and outputs them sorted.

Node	Attribute	Type	Sub node name	Comments
sort	id	String
	enabled	(Boolean)
			column	Sort criterion based on an existing column.
			value	Sort criterion based on a literal value.
			script	Sort criterion based on a JavaScript expression.
			expr	Sort criterion based on a JEXL expression.
column	id*	String		ID of the column to sort on.
	direction	(String)		`asc` (default) or `desc`.
	nullValue	(String)		`before` or `after` (default). Position of `null` values.
	emptyString	(String)		`string` (default), `null`, `before` or `after`. Behavior for empty strings.
value, script, expr
	type	Type		Type of the produced value (defaults to `String`).
	direction	(String)		`asc` (default) or `desc`.
	nullValue	(String)		`before` or `after` (default).
	emptyString	(String)		`string` (default), `null`, `before` or `after`.

Normal steps - Break

Generates title and total lines on breaks.

Node	Attribute	Type	Sub node name	Comments
break	id	String
	enabled	(Boolean)
	data	(Boolean)		If `true` (default), data lines are emitted between title and total.
	forceDataType	(Boolean)		If `true`, forces inner rows to type `DATA`.
	firstRowsToSkip	(Integer)		Number of first rows to skip in output (0 by default).
	hideIfNoData	(Boolean)		If `true`, hides titles/totals when no data line exists.
	inheritsComputations	Boolean		If `true`, clones the parent break’s `computations` into this break.
	resetComputations	String		`never` or `always` (default). Reset behaviour of `computations` processors.
			condition	Condition that determines if a row belongs to the current break.
			force	Forces a new break even if `condition` is unchanged.
			restrict	Restricts the break to an additional sub-condition.
			display	Determines if the whole break must be displayed.
			titleDisplay	Determines if the title row must be emitted.
			title	Title row of the break (see TitleTotal below).
			data	Sub-pipeline applied to break rows before emission.
			totalDisplay	Determines if the total row must be emitted.
			total	Total row of the break (see TitleTotal below).
			computations	Processors that accumulate values across break rows.
			break	Nested sub-break.
condition, force, restrict, display, titleDisplay, totalDisplay
	type	String		`column` (default), `value`, `script` or `expression`.
	id	String		Column id when `type="column"`.
title, total
	enabled	(Boolean)		Enables/disables this title/total row.
	row	String		`first` (default for title), `first_data`, `last` (default for total) or `last_data`.
	type	String		Forces the row type: `title`, `data` or `total`.

Normal steps - Cube

Creates a data cube given a pivot.

Node	Attribute	Type	Sub node name	Comments
cube	id*	String		Required: identifies the cube for later `foreach` references.
			key	Pivot column definition.
			group	Grouping columns.
			values	Processors producing the pivoted values.
key	id	String		ID of the pivot column in the input schema.
			value	Computes the pivot key’s technical value.
			label	Computes the pivot key’s displayed label.
value, label
	type*	String		`value`, `script` or `expression`.
group			column
column	id*	String		ID of the grouping column.

Normal steps - Join

Joins the current step with another one.

Node	Attribute	Type	Sub node name	Comments
join	id	String
	enabled	(Boolean)
	with*	String		ID of the root step to join (right side). Automatically buffered.
	type*	String		`inner`, `left`, `right` or `linear`.
			leftIndex	Join key on the left side.
			rightIndex	Join key on the right side.
			constraint	Additional `ON` predicate (multiple allowed).
leftIndex, rightIndex, constraint
	type	String		`column` (default), `value`, `script` or `expression`.
	id	String		Column id when `type="column"`.

Processors - Copy/Column

Data based on an existing column.

Node	Attribute	Type	Sub node name	Comments
copy	id*	String		ID of the column to put the value to.
	source	String		ID of the column to copy the value from. It will be the same as `id` if not specified.

Note: the node copy can also be named column.

Processors - Value

Allows data combination of other columns with static values. See the documentation.

Node	Attribute	Type	Sub node name	Comments
value	id*	String		ID of the column to put the value to.
	type	Type		Type of the column, if different from the one specified by `id`.

Processors - Script

Scripted value using JavaScript.

Node	Attribute	Type	Sub node name	Comments
script	id*	String		ID of the column to put the value to.
	type	Type		Type of the column, if different from the one specified by `id`.

Processors - Expr

Scripted value using JexlScript.

Node	Attribute	Type	Sub node name	Comments
expr	id*	String		ID of the column to put the value to.
	type	Type		Type of the column, if different from the one specified by `id`.

Processors - Compute

Extended specification of Script.

Node	Attribute	Type	Sub node name	Comments
compute	id*	String		ID of the column to put the value to.
	input	String		ID of the column to put in the `inputValue` variable.
	type	Type		Type of the column, if different from the one specified by `id`.
			filter	Filter expression. Any row that matches this expression will be skipped.
			pre	Expression to execute before the processing starts.
			each	Expression to execute on each row.
			post	Expression to execute after the processing completed.

Processors - Expression

Extended specification of Expr.

Node	Attribute	Type	Sub node name	Comments
expression	id*	String		ID of the column to put the value to.
	input	String		ID of the column to put in the `inputValue` variable.
	type	Type		Type of the column, if different from the one specified by `id`.
			filter	Filter expression. Any row that matches this expression will be skipped.
			pre	Expression to execute before the processing starts.
			each	Expression to execute on each row.
			post	Expression to execute after the processing completed.

Processors - ForEach

Loops on the selected columns. Useful to treat dynamic columns created columns by a Cube.

Node	Attribute	Type	Sub node name	Comments
foreach	id*	String		ID of the cube to loop on.

Misc - Type

A type is a String attribute that can contain the following values:

String
Boolean
Integer
BigInteger
Double
BigDecimal
Date
Long

Misc - Option

The attribute having their type defined between parenthesis are meant to contain an option. For instance an attribute with the type (Boolean) could contain ‘true’, ‘false’ or an option content.

An option content has always the syntax #{...} and can contain (very) simple expressions that use external variables, for instance: #{myVariable == 'value'}. Note that the syntax has to be stricly respected: spaces between operator and operands is mandatory and only quote is supported to denote a string (you can still put a number).

External variables can be injected programmaticaly in BlueIron with the following code:

BlueIron bir = ...
Map<String, Object> options = bir.getOptions();
options.put("myVariable", "value");

The supported operators are described in the table below. Since an option will be contained in an XML attribute, alternate syntax for each of them is allowed to keep readability and XML-compliance.

Operator	Alternate	Description
==	eq	Equal
<	lt	Less than
<=	le	Less or equal than
>	gt	Greater than
>=	ge	Greater or equal than
!=	ne	Not equal

Both syntax are the same: #{value < 4.55} and #{value lt 4.55}.

Misc - Value content

This type is a very simple and straightforward format meant to easily mix data from different columns together. Basically, it is a string with references with the format ${refName} where refName could be the id of a column or a computation/expression.

Assuming that the columns FIRSTNAME and LASTNAME exist, the following step will create a new column named FULLNAME with both values separated by a space:

<project>
  <column id="FULLNAME">${FIRSTNAME} ${LASTNAME}</column>
</project>

Since the reference can be of any supported type, it is possible to specify some arguments, separated by commas:

<project>
  <column id="DESC">${FIRSTNAME} ${LASTNAME} was born on ${BIRTHDATE,dddd}, the ${BIRTHDATE,dd/MM/yyyy}</column>
</project>

The arguments are handled by a Transformer which is implemented by the class DefaultTransformer in BlueIron (see here). It provides the following features:

Type	Argument	Description
String	action	- lower : Makes the string lower case.

                                                         -   upper : Makes the string upper case.
                                                         -   fupper : Makes the first char upper case.
                                                         -   substring : Takes the substring of the string from a start index and an end index.
                                                         -   substr : Takes the substring of the string from a start index and a length.        |

Misc - Javascript

BlueIron provides advanced script features by using the JDK JavaScript engine.

The input row is always described by the variable inputRow or the most recent alias. There are two ways to access the row’s input: either by calling the column’s name as a function, or by using the standard get function.

<root>
  <input id="employees" alias="e">
    <column id="ID" type="String" />
    <column id="FIRSTNAME" type="String" />
    <column id="LASTNAME" type="String" />
  </input>
  <step id="final" description="Simple projection" source="employees">
    <project insert="after">
      <script id="FULLNAME1" type="String"><![CDATA[e.FIRSTNAME()+" "+e.LASTNAME()]]></script>
      <script id="FULLNAME2" type="String"><![CDATA[e.get("FIRSTNAME")+" "+e.get("LASTNAME")]]></script>
    </project>
  </step>
</root>

Performance tip: prefer the usage of JexlScript. Even if the JavaScript scripts are compiled, BlueIron will perform much better with JEXL when a lot of rows are involved.

Misc - JexlScript

Provides basic expressions parsing with boolean operators. Jexl is an open source Apache Project.

Compared with a JavaScript expression, Jexl is much more readable and performant, especially for simple computations. However, Jexl is limited to simple expressions, hence if you need ‘real’ language features (loops for instance), you’ll need to fallback to JavaScript.

The script below shows the difference between a Jexl expression and a JavaScript computation:

<root>
  <input id="salaries" alias="e">
    <column id="ID" type="String" />
    <column id="SALARY" type="BigDecimal" />
    <column id="FEES" type="BigDecimal" />
  </input>
  <step id="final" description="Simple projection" source="salaries">
    <project insert="after">
      <expr id="SUM1">SALARY + FEES</expr> <!-- by default the type is BigDecimal -->
      <script id="SUM" type="BigDecimal">e.get("SALARY").add(e.get("FEES"))</script> <!-- by default the type is String -->
    </project>
  </step>
</root>

Misc - Jexl functions

All the functions below are callable directly in any jexl script.

Global functions

unique(str), Integer

Returns 1 if str has already been seen, else 0.

 unique("H"); //1 (H was not seen)
 unique("H"); //0 (H was seen)
 unique("B"); //1 (B was not seen)
 unique("C"); //1 (C was not seen)
 unique("B"); //0 (B was seen)

distinct(str), Integer

Returns the number of distinct str that have been seen.

 distinct("A") //1 (one value has been seen)
 distinct("B") //2 (two values have been seen)
 distinct("A") //2 (A was already seen)
 distinct("C") //3 (three values have been seen)
 distinct("B") //3 (B was already seen)

count(str), Integer

Returns the number of times str has been seen.

 count("A"); //1 (A was not seen)
 count("B"); //1 (B was not seen)
 count("A"); //2 (A was already seen one time)
 count("C"); //1 (C was not seen)
 count("A"); //3 (A was already seen two times)
 count("C"); //2 (C was already seen one time)

list(str, [separator]), String

Returns a list of all str that have been seen, separated by separator (comma by default).

 list("A"); //A
 list("B"); //A,B
 list("A"); //A,B,A
 list("C"); //A,B,A,C
 list("A"); //A,B,A,C,A
 list("C"); //A,B,A,C,A,C

set(str, [separator]), String

Returns a list of all str that have been seen, separated by separator (comma by default).

 set("A"); //A
 set("B"); //A,B
 set("A"); //A,B
 set("C"); //A,B,C
 set("A"); //A,B,C
 set("C"); //A,B,C

log([level], str, params…), void

Logs the specified str, merged with params at the specified level (SEVERE by default).

Parse functions (namespace: parse)

parse:asString(value, [pattern], [decimalSeparator], [groupSeparator]), String

Transforms the specified value to a String. The parameters depend of the type of value.

parse:asString(aDate); //31.12.2017
parse:asString(aDate, "dd/MM/yyyy HH:mm:ss"); // 31/12/2017 23:59:59
parse:asString(11700.1234); //11700.12
parse:asString(11700.1234, "#,###.00"); //11'700.12
parse:asString(11700.1234, "#,###.00", ",", "."); //11.700,12
parse:asString(1234); //1234
parse:asString(1234, "#,###"); //1'234
parse:asString("some string"); //some string
parse:asString(true); //true

parse:asInteger(value), int

Parse the value as Integer.

parse:asInteger("1234"); //1234

parse:asDouble(value), double

Parse the value as Double.

parse:asDouble("1234.56"); //1234.56

parse:asBigDecimal(value), BigDecimal

Parse the value as BigDecimal.

parse:asBigDecimal("1234.56"); //1234.56

parse:asBigInteger(value), BigInteger

Parse the value as BigInteger.

parse:asBigInteger("1234"); //1234

parse:asDateTime(value), Date

Parse the value as DateTime. The format is ‘dd.MM.yyyy HH:mm:ss’.

parse:asDateTime("31.12.2017 23:59:59");

parse:asDate(value, [format]), Date

Parse the value as Date. The default format is ‘dd.MM.yyyy’.

parse:asDate("31.12.2017");
parse:asDate("31/12/2017 15:30", "dd/MM/yyyy HH:ss");

Date functions (namespace: date)

Note that theses functions are not builtin in BlueIron but are provided by the SAINet integration.

date:addXXX(date), Date

Add one unit of time (depending on the called function) to the specified date. The date is left untouched and a new Date instance will be returned.

newDate = date:addMillis(myDate);
newDate = date:addSecond(myDate);
newDate = date:addMinute(myDate);
newDate = date:addHour(myDate);
newDate = date:addDay(myDate);
newDate = date:addMonth(myDate);
newDate = date:addYear(myDate);

date:addXXX(date, nb), Date

Add nb unit of time (depending on the called function) to the specified date. The date is left untouched and a new Date instance will be returned.

newDate = date:addMillis(myDate, 500);
newDate = date:addSeconds(myDate, 10);
newDate = date:addMinutes(myDate, 5);
newDate = date:addHours(myDate, 2);
newDate = date:addDays(myDate, 1);
newDate = date:addMonths(myDate, 1);
newDate = date:addYears(myDate, 1);

date:setTime(date, hour, minute, second, [millis]), Date

Set the specified hour,minute,second,millis (or zero if unspecified) to date. The date is left untouched and a new Date instance will be returned.

newDate = date:setTime(myDate, 15, 30, 0);

date:setDate(date, dayOfMonth, month, year), Date

Set the specified day of month,month and year to date. The date is left untouched and a new Date instance will be returned.

newDate = date:setDate(myDate, 31, 12, 2017);

date:getDate(year, month, dayOfMonth), Date

Returns a new Date with the specified year,month and day of month.

myDate = date:getDate(2017, 12, 31);
myDate = date:date(2017, 12, 31);

date:getDateTime(year, month, dayOfMonth, hour, minute, second), Date

Returns a new Date with the specified year,month,day of month,hour,minute and second.

myDate = date:getDateTime(2017, 12, 31, 23, 59, 59);
myDate = date:date(2017, 12, 31, 23, 59, 59);
myDate = date:create(2017, 12, 31, 23, 59, 59);

date:date(year, month, dayOfMonth, [hour], [minute], [second]), Date

Simple alias for getDate and getDateTime.

date:create(year, month, dayOfMonth, hour, minute, second), Date

Simple alias for getDateTime.

date:now(), Date

Returns the current date.

currentDate = date:now();

date:startOfDay(date), Date

Returns a new Date instance that starts at 00:00 of the day.

midnight = date:startOfDay(myDate);

date:startOfMonth(date), Date

Returns a new Date instance that starts at 00:00 the first of the month.

firstOfMonth = date:startOfMonth(myDate);

date:startOfYear(date), Date

Returns a new Date instance that starts at 00:00 the first of the year.

firstOfYear = date:startOfYear(myDate);

date:endOfDay(date), Date

Returns a new Date instance that ends at 23:59:59’999 of the day.

endDay = date:endOfDay(myDate);

date:endOfMonth(date), Date

Returns a new Date instance that ends at 23:59:59’999 of the last day of the month.

endMonth = date:endOfMonth(myDate);

date:endOfYear(date), Date

Returns a new Date instance that ends at 23:59:59’999 of the last day of the year.

endYear = date:endOfYear(myDate);

date:XXX(date), Date

Returns the date value, according to the invoked function.

//consider myDate being Sunday, 31.12.2017 23:59:59'999
date:year(myDate); //2017
date:month(myDate); //12
date:date(myDate); //31
date:dayOfWeek(myDate); //7
date:weekOfYear(myDate); //52
date:hour(myDate); //23
date:minute(myDate); //59
date:second(myDate); //59
date:millis(myDate); //999

date:daysInMonth(year, month), int

Returns the number of days in the specified month of year.

nbDays = date:daysInMonth(2017, 2); //28

date:daysInYear(year), int

Returns the number of days in the specified year.

nbDays = date:daysInYear(2017); //365

date:weeksInYear(year), int

Returns the number of weeks in the specified year.

nbWeeks = date:weeksInYear(2017); //52

date:XXXDiff(date1, date2), int

Returns the differences between date1 and date2, according to the called function. The returned value will be floored.

nbYears = date:yearsDiff(d1, d2); //2
nbMonths = date:monthsDiff(d1, d2);
nbDays = date:daysDiff(d1, d2);
nbHours = date:hoursDiff(d1, d2);
nbMinutes = date:minutesDiff(d1, d2);
nbSeconds = date:secondsDiff(d1, d2);
nbMillis = date:millisDiff(d1, d2);

date:XXXBetween(date1, date2), double

Returns the differences between date1 and date2, according to the called function.

nbYears = date:yearsBetween(d1, d2); //2.34
nbMonths = date:monthsBetween(d1, d2);
nbDays = date:daysBetween(d1, d2);
nbHours = date:hoursBetween(d1, d2);
nbMinutes = date:minutesBetween(d1, d2);
nbSeconds = date:secondsBetween(d1, d2);
nbMillis = date:millisBetween(d1, d2); //same as date:millisDiff(d1,d2)

Misc - Magic variables

Describes variables that are automatically injected in execution context of the expression/script/values.

Name	Step	Context	Description
`computations`	Break	computations	Used in to access other computations data
`nbDataRows`	Break	display,displayTitle,data,displayTotal,total	Defines the number of data rows that have been read.
`nbInnerRows`	Break	display,displayTitle,title,data,displayTotal,total	Defines the number of output rows before `data` processing.
`nbOutputRows`	Break	display,displayTitle,title,displayTotal,total	Defines the number of output rows after `data` processing (all rows that will be shown in the break).
`nbOutputDataRows`	Break	display,displayTitle,title,displayTotal,total	Defines the number of output rows after `data` processing (all data rows that will be shown in the break).
`subBreak`	Break	display,displayTitle,title,displayTotal,total,computations	Access to the computations of the direct subbreak. This is an array containing all the (direct) sub-breaks computations.
`breakRow`	Break	displayTitle,title,data,displayTotal,total	Defines the break row (for the next break). Might be null.
`rowTypeIndex`	Break	displayTitle,displayTotal	Defines the index linked to the type of the row. (see here)
`titleRowDisplayed`	Break	displayTotal	Defines if the title row is displayed. Only available if one `<title>` has been declared (syntax sugar).
`titleRowDisplayed_`n	Break	displayTitle,displayTotal	Defines if the title row with the `rowTypeIndex` n is displayed. The n is an index starting from 0.
`totalRowDisplayed_`n	Break	displayTotal	Defines if the total row with the `rowTypeIndex` n is displayed. The n is an index starting from 0.
`currentKey`	Break	condition,force,displayTitle,title,data,displayTotal,total	Defines the break value (defined by the condition).
`previousKey`	Break	condition,force,displayTitle,title,data,displayTotal,total	Defines the previous break value.
`inputRows`	Update	expr	Defines the input rows (only if a buffer value is set).
`outputRows`	Update	expr	Defines the output rows (only if a buffer value is set).
`inputValue`		compute,expression	Represents the value of the column defined in the `input` attribute.
`previousValue`		script,expr,compute,expression	Contains the last returned value.
`nbRows`		compute,expression	Number of rows to process.
`nbMatchingRows`		compute,expression	Number of rows that matched the filter (available in ).
`allInputRows`		compute,expression	Array of the input rows.
`options`		script,expr,compute,expression	Defines the map of the options.
`inputRow`		script,compute	Represents the current input row (might be null).
`inputMetadata`		script,compute	Represents the current input metadata (might be null).
`outputRow`		script,compute	Represents the output row (might be null).
`outputMetadata`		script,compute	Represents the output metadata (might be null).
`currentGroup`	Cube	values	List containing the group values.
`currentKey`	Cube	values	Defines the key value.
`currentLabel`	Cube	values	Defines the key label.
`context`			Defines the context in which the script is running. This is a map that can be used to share variable across different run.