SQL Macros - Game Changing Feature for SQL Developers?

1. SQL Macros – Game Changing Feature
für SQL Entwickler?
Andrej Pashchenko
@Andrej_SQL https://blog.sqlora.com

2. About me
• Working at Trivadis Germany, Düsseldorf
• Focusing on Oracle:
• Data Warehousing
• Application Development
• Application Performance
• Course instructor „Oracle New Features for
Developers“
@Andrej_SQL blog.sqlora.com

4. Why SQL macros?

5. Motivation
It’s a good idea to encapsulate
some complex logic and make
it reusable
We have enough tools to do
so, don’t we?
• Views
• WITH-Subqueries
• PL/SQL Functions
• Polymorphic Table Functions
Why do we need something
else?

6. Motivation
• Performance
• Context switch with PL/SQL functions
• The same data structures are often accessed from different PL/SQL functions called
from the same SQL
• The optimizer having no clue what happens within PL/SQL
• Lack of control
• No parameters for views
• Binds or literals used
• Read-Consistency:
• SQL calling PL/SQL calling SQL calling PL/SQL calling SQL calling PL/SQL …

7. Motivation
• SQL Macros are functions where your (SQL) logic goes in
• will NOT be executed at runtime (no context switch!)
• will be executed once at parse time and return a piece of SQL code which will be
incorporated in your SQL query.
• enable to create reusable building blocks used to build complex SQL statements
• hiding the unnecessary complexity from developer
• but exposing all useful information to the optimizer

8. Motivation

9. SQL Macro Basics

10. • SQL macro is just a function returning text (CHAR, VARCHAR2, CLOB)
• Whatever your complex logic inside the function is, you return a piece of SQL code at the end
SELECT e.*
, FLOOR(MONTHS_BETWEEN (SYSDATE, hiredate)/12) as years
FROM emp e
WHERE FLOOR(MONTHS_BETWEEN (SYSDATE, hiredate)/12) > 38;
CREATE OR REPLACE FUNCTION job_duration RETURN VARCHAR2
SQL_MACRO(SCALAR) AS
BEGIN
RETURN 'FLOOR(MONTHS_BETWEEN (SYSDATE, hiredate)/12)';
END;
First SQL Macro
• How can I hide the calculation?

11. SQL> SELECT ename
, job_duration
as years
FROM emp e
WHERE job_duration > 38
First SQL Macro
• Calling SQL macro function
SQL> SELECT ename
, FLOOR(MONTHS_BETWEEN (SYSDATE, hiredate)/12)
as years
FROM emp e
WHERE FLOOR(MONTHS_BETWEEN (SYSDATE, hiredate)/12 > 38
ENAME YEARS
---------- ----------
SMITH 39
ALLEN 39
WARD 39
3 rows selected.
SQL you write
SQL macro
executed while
parsing
SQL is
translated
into
SQL actually executed

12. SQL> SELECT ename
, FLOOR(MONTHS_BETWEEN (SYSDATE, hiredate)/12) as years
, FLOOR(MOD(MONTHS_BETWEEN (SYSDATE, hiredate),12)) as months
FROM (SELECT * FROM emp WHERE job = 'ANALYST' ) e
WHERE FLOOR(MONTHS_BETWEEN (SYSDATE, hiredate)/12) > 38
GROUP BY ename
, FLOOR(MONTHS_BETWEEN (SYSDATE, hiredate)/12)
, FLOOR(MOD(MONTHS_BETWEEN (SYSDATE, hiredate),12))
HAVING SUM(sal) > 1000
ORDER BY FLOOR(MONTHS_BETWEEN (SYSDATE, hiredate)/12)
+ FLOOR(MOD(MONTHS_BETWEEN (SYSDATE, hiredate),12))
SQL macro 1
SQL macro 2
SQL macro 3
SQL macro 4
Where and how do I call SQL Macro?
• In general, everywhere in a SQL statement where a function call is allowed

13. Where and how do I call SQL Macro?
• Give the function meaningful names and here you go:
SQL> SELECT ename
, duration_years as years
, duration_months as months
FROM emp_only_analysts e
WHERE duration_years > 38
GROUP BY ename
, duration_years
, duration_months
HAVING sum_sal > 1000
ORDER BY duration_years
+ duration_months

14. Won’t work!
Column alias,
condition
Won’t work!
More than one
expression
Won’t work!
Condition
Where and how do I call SQL Macro?
• You cannot “break” the structure
• Only take up as much of SQL as a normal function
SQL> SELECT ename
, FLOOR(MONTHS_BETWEEN (SYSDATE, hiredate)/12) as years
, FLOOR(MOD(MONTHS_BETWEEN (SYSDATE, hiredate),12)) as months
FROM (SELECT * FROM emp WHERE job = 'ANALYST' ) e
WHERE FLOOR(MONTHS_BETWEEN (SYSDATE, hiredate)/12) > 38
GROUP BY ename
, FLOOR(MONTHS_BETWEEN (SYSDATE, hiredate)/12)
, FLOOR(MOD(MONTHS_BETWEEN (SYSDATE, hiredate),12))
HAVING SUM(sal) > 1000
ORDER BY FLOOR(MONTHS_BETWEEN (SYSDATE, hiredate)/12) +
FLOOR(MOD(MONTHS_BETWEEN (SYSDATE, hiredate),12))

15. SQL> SELECT ename
, job_duration as years
FROM emp e
WHERE job_duration > 38
ENAME YEARS
---------- ----------
SMITH 39
ALLEN 39
WARD 39
3 rows selected.
What Types of SQL Macros are there?
• Used in SELECT, WHERE, GROUP BY, …
SELECT e.hash_diff
, e.empno
, e.ename
FROM add_hash_columns (emp) e;
HASH_DIFF EMPNO ENAME
---------- ---------- ----------
42CB6932B4 7369 SMITH
AA63299F72 7499 ALLEN
27332DD16B 7521 WARD
3 rows selected.
• Used in FROM clause
Scalar SQL macro Table SQL macro (default)

16. CREATE FUNCTION job_duration (<optional parameters>)
RETURN VARCHAR2 | CHAR | CLOB
SQL_MACRO(SCALAR|TABLE)
AS
BEGIN
<.. Your business logic here .. >
RETURN 'FLOOR(MONTHS_BETWEEN (SYSDATE, hiredate)/12)';
END;
How to define a SQL macro function?
• The function must return CHAR, VARCHAR2 or CLOB
• Using a new keyword SQL_MACRO
This makes a
regular function
a SQL macro
The result of a function becomes
a part of the calling SQL
TABLE is default
and can be
omitted

17. When can I use SQL Macros?
• SQL Macros have been introduced in Oracle 20c
• Only Preview Version of 20c in Oracle Cloud as of now
• (Only) Table SQL Macros have been backported to 19c (19.8)! You can start to test them!
• Officially documented and supported
• You cannot specify the type of a macro, but since TABLE is default – no problem migrating
to 20c later
CREATE FUNCTION job_duration (<optional parameters>)
RETURN VARCHAR2 | CHAR | CLOB
SQL_MACRO(SCALAR|TABLE)
AS
BEGIN
RETURN 'FLOOR(MONTHS_BETWEEN (SYSDATE, hiredate)/12)';
END;

18. Parameters, Parsing

19. Passing Parameters
• You can pass scalar parameters and use them in return string
• But don’t concatenate
CREATE FUNCTION duration (p_date IN DATE)
RETURN VARCHAR2
SQL_MACRO(SCALAR)
AS
BEGIN
RETURN 'FLOOR(MONTHS_BETWEEN (SYSDATE, duration.p_date)/12)';
END;
exec dbms_output.put_line( duration (DATE '2020-02-26') );
PL/SQL procedure successfully completed.
FLOOR(MONTHS_BETWEEN (SYSDATE, duration.p_hiredate)/12)
What is a parameter inside a
quoted string good for?
Calling outside of SQL it just
makes no sense, but…

20. Passing Parameters
• When calling in SQL, the parameter you pass seems just to be substituted
in the return statement of the function (result string)
SQL> SELECT ename, duration (hiredate) as years
FROM emp e
WHERE duration (hiredate) > 38
...
SQL> select * from dbms_xplan.display_cursor()
...
--------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
--------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | | 3 (100)| |
|* 1 | TABLE ACCESS FULL| EMP | 1 | 14 | 3 (0)| 00:00:01 |
--------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
1 - filter(FLOOR(MONTHS_BETWEEN(SYSDATE@!,INTERNAL_FUNCTION("HIREDATE"))/12)>38)

21. Passing Parameters
• Passing a column name
SQL> CREATE OR REPLACE FUNCTION new_func (p_param IN VARCHAR2)
RETURN VARCHAR2 SQL_MACRO(SCALAR) AS
BEGIN
RETURN 'UPPER(p_param)';
END;
Function created.
SQL> SELECT count(*) FROM emp e WHERE new_func(ename) = 'SCOTT'
...
SQL> select * from dbms_xplan.display_cursor()
...
Predicate Information (identified by operation id):
---------------------------------------------------
2 - filter(UPPER("ENAME")='SCOTT')

22. Passing Parameters
• Passing a literal: the optimizer may see that the condition can never be true
SQL> SELECT count(*) FROM emp e WHERE new_func('ADAMS') = 'SCOTT'
...
SQL> select * from dbms_xplan.display_cursor()
--------------------------------------------------------------------
| Id | Operation | Name | Rows | Cost (%CPU)| Time |
--------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | 1 (100)| |
| 1 | SORT AGGREGATE | | 1 | | |
|* 2 | FILTER | | | | |
| 3 | INDEX FULL SCAN| PK_EMP | 14 | 1 (0)| 00:00:01 |
--------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - filter(NULL IS NOT NULL)

23. Passing Parameters
• Passing bind variables: the bind variable is used in a calling SQL
SQL> SELECT count(*) FROM emp e WHERE new_func(:bind) = 'SCOTT'
...
SQL> select * from dbms_xplan.display_cursor()
--------------------------------------------------------------------
| Id | Operation | Name | Rows | Cost (%CPU)| Time |
--------------------------------------------------------------------
| 0 | SELECT STATEMENT | | | 1 (100)| |
| 1 | SORT AGGREGATE | | 1 | | |
|* 2 | FILTER | | | | |
| 3 | INDEX FULL SCAN| PK_EMP | 14 | 1 (0)| 00:00:01 |
--------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - filter(UPPER(:BIND)='SCOTT')

24. Passing Parameters
• don’t concatenate function parameter into the result string like
RETURN 'UPPER('||p_param||')'
• don’t use parameter as bind variable in a result string like
RETURN 'UPPER(:p_param)'
• just reference the parameter in a literal string to be returned, you can optionally
prefix it with a function name if conflicting with other names
RETURN 'UPPER(new_func.p_param)'
• this reference will be replaced with column name, bind variable, function or literal,
whatever was used to invoke a SQL macro function

25. Parameters inside a function
• To protect against SQL injection all string parameters are NULL inside a function
CREATE OR REPLACE FUNCTION f_test_par (p_num NUMBER, p_str VARCHAR2)
RETURN VARCHAR2 SQL_MACRO(SCALAR) IS
BEGIN
DBMS_OUTPUT.PUT_LINE('p_num '||p_num);
DBMS_OUTPUT.PUT_LINE('p_str '||p_str);
RETURN 'p_str||to_char(p_num)' ;
END;
/
SELECT f_test_par(20, 'Oracle ') FROM dual;
F_TEST_PA
---------
Oracle 20
p_num 20
p_str <-- the string 'Oracle’ is not visible inside the function

26. SQL Macros and Parsing
• What does it mean?
• SQL Macro function will not be called every time the SQL is executed
• It will be executed occasionally when SQL statement gets hard parsed, which you cannot control:
• DDL on involved database objects
• SQL aged out from shared pool, etc.
• You must ensure its deterministic behavior
• Don’t try to make the return string depend on the state in the database
• More on this: https://blog.sqlora.com/en/sql-macros-part-2-parameters-and-parsing/

27. Table SQL Macros

28. Table SQL Macros
CREATE OR REPLACE FUNCTION filter_dept RETURN VARCHAR2 SQL_MACRO(TABLE)
IS
BEGIN
RETURN 'SELECT * FROM scott.emp WHERE deptno = 30';
END;
/
• Table SQL macros can be called in FROM clause only
SELECT empno, deptno
FROM filter_dept();
SELECT empno, deptno
FROM (SELECT * FROM emp WHERE deptno = 30);
• Acts just like inline view in this query
• Or even like a regular view because the query is saved in the database
• But in contrast to regular views we can use parameters!

29. Table SQL Macros – Using Scalar Parameters
CREATE OR REPLACE FUNCTION filter_dept( p_deptno IN NUMBER)
RETURN VARCHAR2 SQL_MACRO(TABLE)
IS
BEGIN
RETURN 'SELECT * FROM emp WHERE deptno = p_deptno';
END;
/
• Just use function parameters in the return string
SELECT empno, deptno
FROM filter_dept(10);
SELECT empno, deptno
FROM (SELECT * FROM emp WHERE deptno = 10);
A literal substitution
of parameters in the
return string

30. Table SQL Macros – Examples Using Scalar Parameters
SELECT empno, deptno
FROM filter_dept(10)
SELECT empno, deptno FROM
(SELECT * FROM emp WHERE deptno = 10)
SELECT empno, deptno
FROM filter_dept(empno)
SELECT empno, deptno FROM
(SELECT * FROM emp WHERE deptno = empno)
SELECT empno, deptno
FROM filter_dept(:P_DEPT)
SELECT empno, deptno FROM
(SELECT * FROM emp WHERE deptno =:P_DEPT)
SELECT empno, deptno
FROM filter_dept(my_func(100))
SELECT empno, deptno FROM
(SELECT * FROM emp WHERE deptno = my_func(100))
SELECT empno, deptno
FROM filter_dept((SELECT 10
FROM dual))
SELECT empno, deptno
FROM (SELECT * FROM emp
WHERE deptno = ((SELECT 10 FROM dual))
How do I know what’s actually executed?

31. Table SQL Macros – How to See the Real SQL Statement
SQL> DECLARE
2 l_clob CLOB;
3 BEGIN
4 DBMS_UTILITY.expand_sql_text (
5 input_sql_text => 'SELECT empno, deptno FROM filter_dept(:P_DEPT)',
6 output_sql_text => l_clob );
7 DBMS_OUTPUT.put_line(l_clob);
8 END;
9 /
SELECT "A1"."EMPNO" "EMPNO","A1"."DEPTNO" "DEPTNO" FROM (SELECT "A2"."EMPNO"
"EMPNO","A2"."ENAME" "ENAME","A2"."JOB" "JOB","A2"."MGR" "MGR","A2"."HIREDATE"
"HIREDATE","A2"."SAL" "SAL","A2"."COMM" "COMM","A2"."DEPTNO" "DEPTNO" FROM (SELECT
"A3"."EMPNO" "EMPNO","A3"."ENAME" "ENAME","A3"."JOB" "JOB","A3"."MGR"
"MGR","A3"."HIREDATE" "HIREDATE","A3"."SAL" "SAL","A3"."COMM" "COMM","A3"."DEPTNO"
"DEPTNO" FROM "SCOTT"."EMP" "A3" WHERE "A3"."DEPTNO"=:B1) "A2") "A1"
PL/SQL procedure successfully completed.
• You can use DBMS_UTILITY.expand_sql_text
• But only for table macros!

32. Table SQL Macros – Polymorphic Views
Using parameters for views is
fine, but must the query
always remain hardcoded?
No! Just use table and
column parameters to create
polymorphic views!
Polymorphic Views and
Polymorphic Table Functions
– is it the same?
No, don’t confuse them. PTF means PL/SQL
execution at runtime, Polymorphic Views via
SQL macros are only executed at parse time
So no need to learn about the
PTF’s right now?
At least you should get to know the table
and column parameters, because they were
introduced and documented in the context
of PTF's

33. Table SQL Macros – Polymorphic Views
• Pass tables (views, named subqueries) as parameter of type DBMS_TF.TABLE_T
a table of
• E.g. to get the data type of the first
column:
<Param>.column(1).description.type

34. SELECT *
FROM my_sql_macro_func( emp, COLUMNS(EMPNO, ENAME));
Table SQL Macros – Polymorphic Views
• Pass column lists as a parameter of type DBMS_TF.COLUMNS_T
• The right way to do it is using a variadic pseudo-operator COLUMNS (18c)
Visible as TABLE_T structure
with all column description
etc.
Visible as array "EMPNO",
"ENAME"

35. CREATE OR REPLACE FUNCTION top_n (p_tab IN DBMS_TF.TABLE_T, p_limit IN NUMBER
, p_order IN DBMS_TF.COLUMNS_T)
RETURN VARCHAR2 SQL_MACRO IS
v_order_list VARCHAR2(2000);
BEGIN
-- turn PL/SQL table to comma separated list for ORDER BY clause
SELECT LISTAGG(column_value,',') INTO v_order_list FROM TABLE (p_order);
RETURN 'SELECT * FROM top_n.p_tab ORDER BY '||v_order_list||
' FETCH FIRST top_n.p_limit ROWS ONLY';
END;
Table SQL Macros – Polymorphic Views
• Implement reusable SQL macro to get Top-N rows from any table using defined sort order
SQL> SELECT deptno, dname, loc
2 FROM top_n(scott.dept, 2, COLUMNS(loc));
DEPTNO DNAME LOC
---------- -------------- -------------
40 OPERATIONS BOSTON
30 SALES CHICAGO
SQL> SELECT empno, ename
2 FROM top_n(scott.emp, 2, COLUMNS(ename));
EMPNO ENAME
---------- ----------
7876 ADAMS
7499 ALLEN

36. Use Cases

37. SELECT e.hash_diff
, e.empno
, e.ename
FROM add_hash_columns (emp) e;
HASH_DIFF EMPNO ENAME
---------- ---------- ----------
42CB6932B4 7369 SMITH
AA63299F72 7499 ALLEN
27332DD16B 7521 WARD
3 rows selected.
Generate Hash Keys
• Build MD5 digest of the whole row, often used for row comparison during ETL processes
• There are complex rules to follow, thus one PL/SQL function across all use cases is preferrable
• The most efficient way to generate MD5-Hash is STANDARD_HASH, not available in PL/SQL
CREATE OR REPLACE FUNCTION add_hash_columns(t DBMS_TF.TABLE_T
, key_cols DBMS_TF.COLUMNS_T)
RETURN VARCHAR2 SQL_MACRO(TABLE)
AS
v_hdiff clob ;
v_hkey clob ;
v_str varchar2(200);
v_delimiter varchar2(9):= '||''#''||';
v_name dbms_id;
BEGIN
FOR I IN 1..t.column.count LOOP
v_name := t.column(i).description.name;
IF t.column(i).description.type = dbms_tf.type_varchar2 THEN
v_str := v_name;
ELSIF t.column(i).description.type = dbms_tf.type_number THEN
v_str := 'to_char('||v_name||')';
ELSIF t.column(i).description.type = dbms_tf.type_date THEN
v_str := 'to_char('||v_name||',''YYYYMMDD'')';
END IF;
v_hdiff := v_hdiff || v_delimiter || v_str;
IF v_name MEMBER OF key_cols THEN
v_hkey := v_hkey || v_delimiter || v_str;
END IF;
END LOOP;
v_hdiff := LTRIM(v_hdiff,'|''#');
v_hkey := LTRIM(v_hkey,'|''#');
RETURN 'SELECT STANDARD_HASH('||v_hkey||',''MD5'') hash_key, '||
' STANDARD_HASH('||v_hdiff||',''MD5'') hash_diff, '||
' t.* FROM t';
END;
https://blog.sqlora.com/en/building-hash-keys-using-sql-macros-in-oracle-20c/
IF t.column(i).description.type =
dbms_tf.type_varchar2 THEN
v_str := v_name;
ELSIF t.column(i).description.type =
dbms_tf.type_number THEN
v_str := 'to_char('||v_name||')';
ELSIF t.column(i).description.type =
dbms_tf.type_date THEN
v_str := 'to_char('||v_name||',''YYYYMMDD'')';
END IF;
Data type check at the core

38. Parameterized Views
• An example for querying versioned data, typical for data warehouse use cases
• Point-in-time and range queries
• PL/SQL function overloading
https://blog.sqlora.com/en/parameterized-views-in-oracle-no-problem-with-sql-macros/

39. Temporal Joins
• An example of hiding complex query syntax in a SQL Macro function
• Multiple input tables
• Using MATCH_RECOGNIZE
https://blog.sqlora.com/en/temporal-joins-with-sql-macros-in-oracle-20c/

40. Dynamic PIVOT
• One of “most wanted” features
• But a poor example for SQL Macros
• You can’t make it deterministic
• Only works in 20c
https://blog.sqlora.com/en/dynamic-pivot-with-sql-macros-in-oracle-20c/

41. Links
• Oracle Documentation
• https://blog.sqlora.com/en/tag/sql-macros/
• Asktom Office Hours
• SQL Macros Have Arrived in Autonomous Database
• LiveSQL

42. Summary
• Encapsulated reusable logic without typical performance issues
• Show the whole picture to the optimizer without having to maintain complex SQL
statements
• Power-developer can provide efficient building blocks using modern SQL for those who
aren't confident enough to go beyond SQL-92
• Full control over binds or literals used in “parameterized” views
• No read consistency issues with nested SQL-PL/SQL-SQL
• Table macros available in 19c!
Try it yourself!