With the 2.1 release, Apache Flink also now supports Process Table Functions (PTFs), the most powerful kind of function for Flink SQL and Table API, the PMC said. Conceptually, a PTF is a superset of all other user-defined functions, mapping zero, one, or multiple tables to zero, one, or multiple rows. This enables implementing user-defined operators that can be as feature-rich as built-in operations, the PMC said. PTFs have access to Flink’s managed state, event-time, table change logs, and timer services.

Apache Flink 2.1 also adds VARIANT as a data type for semi-structured data such as JSON. This new type supports storing any semi-structured data including ARRAY, MAP (with STRING keys), and scalar types while preserving field type information in a JSON-like structure. Unlike the ROW and STRUCTURED types, VARIANT provides superior flexibility for handling deeply nested and evolving schemas. Users can use PARSE_JSON or TRY_PARSE_JSON to convert JSON-formatted VARCHAR data to VARIANT.

Also in Apache Flink 2.1: