Effective and Efficient Python with Oracle Database

Effective and Efficient Python
with Oracle Database
Christopher Jones
Data Access Development
Oracle Database
Copyright © 2019 Oracle and/or its affiliates.
christopher.jones@oracle.com
@ghrd
blogs..oracle.com/opal

The following is intended to outline our general product direction. It is intended for information purposes
only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development,
release, timing, and pricing of any features or functionality described for Oracle’s products may change
and remains at the sole discretion of Oracle Corporation.
Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and
prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed
discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and
Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q
under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website
at http://www.oracle.com/investor. All information in this presentation is current as of September 2019
and Oracle undertakes no duty to update any statement in light of new information or future events.
Safe Harbor

Oracle Database for the Developer
Third Party Open Source Drivers
Oracle Open Source Drivers
Oracle Call Interface (OCI)
Typically from Oracle Instant Client ZIPs or RPMs
ODPI-C
Also used by custom apps
Python
cx_Oracle
Node.js
node-oracledb
Go
goracle
Julia
Oracle.jl
Rust
rust-
oracle PHP
OCI8
PHP
PDO_OCI
R
ROracle
Go
go-oci8
Erlang
oranif Perl
DBD::
Oracle
Ruby
ruby-
oci8
Oracle Call Interface, Oracle C++ Call Interface, ODBC, JDBC, ODP.NET,
Pro*C, Pro*COBOL, SQLJ, OLE DB, OLE DB for OLAP
Oracle Proprietary Drivers

Themes
• Introduction & Installation
• cx_Oracle Feature Dive
• High Availability
• Towards Data Science

Introduction

Python in Four Points
• General purpose scripting language
• Huge package library with sophisticated packages
• “Everyone” is using it
• Great Oracle Database (and TimesTen) support

cx_Oracle
• Open Source package for Python access to Oracle Database
• Covers all of Python Database API specification
• Many additions to support advanced Oracle features
• 50+ releases covering Oracle 8i through 19c
• Begun by Anthony Tuininga in 1998 for Oracle 8i and Python 1.5
• Latest release is cx_Oracle 7.2.3 (September 2019)

cx_Oracle Features
Open Source
Maintained by Oracle
Hosted on PyPI and GitHub
Binary 'wheel' packages
SQL and PL/SQL Execution
Scrollable cursors
Extensive data type support
Binding to SQL objects, PL/SQL
index-by tables, and records
Fetching of large result sets
REF CURSORs
Oracle Database JSON datatype
Large Objects: CLOBs and BLOBs
Simple Oracle Document Access
(SODA)
Transaction Management
Session Pooling
Database Resident Connection
Pooling (DRCP)
Privileged Connections and
Database startup/shutdown
External Authentication
Advanced Queuing (AQ)
Continuous Query Notification
(CQN)
Client Result Caching
End-to-end authentication and
tracing
Implicit Results
Transaction Guard
Edition Based Redefinition
Row Prefetching
Statement Caching
Application Context
Sharded Databases
Oracle Database High Availability
Features
Oracle Net Features including
Encryption

10
cx_Oracle Architecture
app.py
cx_Oracle
Oracle Call Interface
in Oracle Client
libraries
Oracle DBPythonUsers

Installation in One Slide
1. Install Python 2.7 or 3.5+
2. Install cx_Oracle:
python -m pip install cx_Oracle --upgrade
--user
--proxy=http://proxy.example.com:80
3. Add Oracle 11.2 – 19c client libraries to the library search path
• Windows: PATH
• Linux: LD_LIBRARY_PATH or use ldconfig
• macOS: put files in ~/lib or /usr/local/lib
• Use Instant Client if your database is on a remote computer or the cloud

Ok, Two Slides
4. For Cloud DBs, download wallet.zip
5. Extract and put files in instantclient_XX_Y/network/admin
cwallet.sso
sqlnet.ora
tnsnames.ora
6. Still need username/password at connection with cloud wallets

Connecting to the
Database

Standalone Connections
import cx_Oracle
with cx_Oracle.connect("user", "password", "localhost/orclpdb1",
encoding="UTF-8") as connection:
cursor = connection.cursor()
for row in cursor.execute("select * from SomeTab"):
print(row)
Python DB

Pooled Connections
pool = cx_Oracle.SessionPool("user", "password", "localhost/orclpdb1",
min=20, max=20, increment=0,
threaded=True,
encoding="UTF-8")
. . .
with pool.acquire() as connection:
for row in cursor.execute("select * from SomeTab"):
print(row)
Copyright © 2019 Oracle and/or its affiliates. Python DB

Many Apps Set Session State
cursor.callproc("set_app_state")
do_work(connection)
Round-trip required each
time connection is
acquired from the pool
Set NLS settings, time
zone, PL/SQL package
state, etc.
"A server round-trip is defined as the trip from the client to the
database server and back to the client."

Session Callback: All Connections with
Same State
def init_session(connection, requestedTag):
cursor.callproc("set_app_state")
pool = cx_Oracle.SessionPool("user", "password", "dsn", min=5,
max=20, increment=3, sessionCallback=init_session)
do_work(connection)
Only invoked for new
sessions or when the
requested tag does not
match the connection tag
Connection returned from
pool.acquire() already has
the right state

Getting Data Out of
the Database

Fetching Rows (in Batches)
cursor.arraysize = 500
cursor.execute("select * from SomeTab")
while True:
rows = cursor.fetchmany()
if rows:
do_something(rows)
if len(rows) < cursor.arraysize:
break
Default fetchmany() value.
Internal tuning buffer size for
fetchone() and fetchall() too.

Fetching LOBs
cursor.execute("select ClobVal from SomeTab")
for lob in cursor:
lobData = lob.read() # or str(lob)

Fetching LOBs as Strings/Bytes
def handler(cursor, name, defaultType, size, precision, scale):
if defaultType == cx_Oracle.CLOB:
return cursor.var(cx_Oracle.LONG_STRING, arraysize=cursor.arraysize)
elif defaultType == cx_Oracle.BLOB:
return cursor.var(cx_Oracle.LONG_BINARY, arraysize=cursor.arraysize)
cursor.outputtypehandler = handler
cursor.execute("select LobVal from SomeTab")
for lobData in cursor:
...

Fetching Numbers as Decimals
def handler(cursor, name, defaultType, size, precision, scale):
if defaultType == cx_Oracle.NUMBER:
return cursor.var(decimal.Decimal, arraysize=cursor.arraysize)
cursor.outputtypehandler = handler
for value, in cur.execute("select Data from DecimalTab"):
print("Value:", value, "* 3 =", value * 3)
Default output: Output type handler output:
Value: 0.1 * 3 = 0.30000000000000004
Value: 3.1 * 3 = 9.3
Value: 7.1000000000000005 * 3 = 21.3
Value: 0.1 * 3 = 0.3
Value: 3.1 * 3 = 9.3
Value: 7.1 * 3 = 21.3

Putting Data Into the
Database

Binding Data
sql = "insert into MyTab (Id, Data) values (:idVal, :dataVal)"
# bind by position using a sequence (list or tuple)
cursor.execute(sql, [1, "String 1"])
cursor.execute(sql, (1, "String 1"))
# bind by name using a dictionary
cursor.execute(sql, {"idVal": 1, "dataVal": "String 1"})
# bind by name using keyword arguments
cursor.execute(sql, idVal=1, dataVal="String 1")

Batch Execution
Execute one DML or PL/SQL
statementwith many data values
rows = [ (1, "First" ),
(2, "Second" ),
(3, "Third" ),
(4, "Fourth" ),
(5, "Fifth" ),
(6, "Sixth" ),
(7, "Seventh" ) ]
cursor.setinputsizes(none, 20)
cursor.executemany("insert into MyTab (Id, Data) values (:idVal, :dataVal)", rows)
Saves “round-trips” between Python and Oracle Database

Batch Errors
dataToInsert = [
(1016, 10, 'Red'),
(1018, 20, 'Blue'),
(1018, 30, 'Green’), # duplicate key
(1022, 40, 'Yellow'),
(1021, 75, 'Orange'), # parent does not exist
]
cursor.executemany("insert into ChildTable values (:1, :2, :3)", dataToInsert,
batcherrors=True)
for error in cursor.getbatcherrors():
print("Error", error.message, "at row offset", error.offset)
Error ORA-00001: unique constraint (PYTHONDEMO.CHILDTABLE_PK) violated at row offset 2
Error ORA-02291: integrity constraint (PYTHONDEMO.CHILDTABLE_FK) violated - parent key not
found at row offset 4

Oracle Objects and
Collections

Binding Named Object Types
create table TestGeometry (
IntCol number(9) not null,
Geometry SDO_GEOMETRY not null
);
describe SDO_GEOMETRY
Name Null? Type
----------------------------------------- -------- ----------------------------
SDO_GTYPE NUMBER
. . .
SDO_ELEM_INFO MDSYS.SDO_ELEM_INFO_ARRAY
SDO_ORDINATES MDSYS.SDO_ORDINATE_ARRAY
. . .

Binding Named Object Types (continued)
typeObj = conn.gettype("SDO_GEOMETRY")
elementInfoTypeObj = conn.gettype("SDO_ELEM_INFO_ARRAY")
ordinateTypeObj = conn.gettype("SDO_ORDINATE_ARRAY")
obj = typeObj.newobject() # or typeObj()
obj.SDO_GTYPE = 2003
obj.SDO_ELEM_INFO = elementInfoTypeObj.newobject()
obj.SDO_ELEM_INFO.extend([1, 1003, 3])
obj.SDO_ORDINATES = ordinateTypeObj.newobject()
obj.SDO_ORDINATES.extend([1, 1, 5, 7])
cur = conn.cursor()
cur.execute("insert into TestGeometry values (1, :objbv)", objbv=obj)

Fetching Objects
def dumpobject(obj, prefix = " "):
if obj.type.iscollection:
print(prefix, "[")
for value in obj.aslist():
if isinstance(value, cx_Oracle.Object):
dumpobject(value, prefix + " ")
else:
print(prefix + " ", repr(value))
print(prefix, "]")
else:
print(prefix, "{")
for attr in obj.type.attributes:
value = getattr(obj, attr.name)
if isinstance(value, cx_Oracle.Object):
print(prefix + " " + attr.name + " :")
dumpobject(value, prefix + " ")
else:
print(prefix + " " + attr.name + " :", repr(value))
print(prefix, "}")

Advanced Queuing

Enqueuing
SQL> call dbms_aqadm.create_queue_table('RAW_QUEUE_TAB', 'RAW’);
SQL> call dbms_aqadm.create_queue('RAW_QUEUE', 'RAW_QUEUE_TAB’);
SQL> call dbms_aqadm.start_queue('RAW_QUEUE');
data = [ "Message 1", "Message 2", ... ]
queue = connection.queue("RAW_QUEUE")
# enqueue one at a time
for payloadData in data:
queue.enqOne(connection.msgproperties(payload=payloadData))
# enqueue many at a time
messages = [connection.msgproperties(payload=d) for d in data]
queue.enqMany(messages)

Dequeuing
queue = connection.queue("RAW_QUEUE")
# dequeue one at a time
message = queue.deqOne()
print(message.payload.decode(connection.encoding))
# dequeue many at a time
maxNumMessages = 8
for message in queue.deqMany(maxNumMessages):
print(message.payload.decode(connection.encoding))

SODA

Simple Oracle Document Access (SODA)
• NoSQL-style APIs to create and access documents
• Documents are often JSON
• Query-by-example makes data operations easy
• SODA calls are mapped to managed tables, indexes and sequences
• Production in cx_Oracle 7 with Oracle Database 18.5 or 19.3
• SODA APIs also exist in PL/SQL, Node.js, C and Java

Creating SODA Documents
SQL> grant SODA_APP to pythondemo;
conn = cx_Oracle.connect("pythondemo/welcome@localhost/orclpdb1")
conn.autocommit = True
soda = conn.getSodaDatabase()
coll = soda.createCollection("mycollection")
content = {'name': 'Matilda', 'address': {'city': 'Melbourne'}}
coll.insertOne(content)
content = {'name': 'Mary', 'address': {'city': 'Madrid'}, 'postcode': 3207}
coll.insertOne(content)
content = {'name': 'Sally', 'address': {'city': 'Singapore'}}
doc = coll.insertOneAndGet(content)
keyForSally = doc.key

Querying SODA Collections
doc = coll.find().key(keyForSally).getOne()
print("Document content:", doc.getContent())
print()
docs = coll.find().filter({'name': {'$like': 'Ma%'}}).getDocuments()
names = [d.getContent()["name"] for d in docs]
print("Names matching 'Ma%':", names)
print()
count = coll.find().count()
print('Collection has', count, 'documents')
Document content: {'name': 'Sally', 'address': {'city': 'Singapore'}}
Names matching 'Ma%': ['Matilda', 'Mary']
Collection has 3 documents

High Availability

Oracle Network Configuration
• sqlnet.ora and tnsnames.ora on the ‘client’
• instantclient_XX_Y/network/admin
• $TNS_ADMIN
• ~/.sqlnet.ora
• Useful sqlnet.ora options
SQLNET.OUTBOUND_CONNECT_TIMEOUT – limit the time to establish a connection
DISABLE_OOB=ON – if network drops/in-lines out-of-band breaks (auto-detected in 19c)
• Useful tnsnames.ora options (18c+)
EXPIRE_TIME – send packets for dead server detection / firewall timeout

Oracle Client 19c Easy Connect Plus
Security, Proxy and Description parameters without config files
"mydbmachine.example.com/orclpdb1?connect_timeout=1&expire_time=3"
[[protocol:]//]host1{,host2}[:port1]{,host2:port2}
[/[service_name][:server_type]
[/instance_name]][?parameter_name=value{&parameter_name=value}]

Execution Timeout
cx_Oracle Connection.callTimeout
• milliseconds any round-trip to the DB may take
• returns error to the app

Oracle Database High Availability
• Premise: a database instance is always available
• RAC – Real Application Clusters
• DG – Data Guard
• ADG – Active Data Guard
• Oracle Technologies:
• Fast Application Notification (FAN)
• Application Continuity (AC)
• Transparent Application Continuity (TAC)
• . . .

Fast Application Notification (FAN)
• Enable events mode in Python
cx_Oracle.SessionPool(..., events=True)
• Configure database service
srvctl modify service –db hr -service hasvc-notification TRUE
(RAC)
dbms_service.modify_service('hasvc', aq_ha_notfications => TRUE);
(non-RAC)
• Messages that are sent
• Down – notify apps that an instance is down
• Planned Down – notify apps of planned maintenance
• Up – notify apps that an instance is back up again

Graceful Planned Maintenance Scenario
Python app uses: cx_Oracle connection pool
DBA runs: srvctl relocate|stop service –db ... –service ... –drain_timeout ...
(not using –force)
Connections drain New work is redirected by listeners
Idle connections in the pool are released
Active connections are released when returned to pools
After drain
timeout, DBA
completes
shutdown
Wait to allow connections to drain, e.g. 10-30 minutes
exec dbms_service.disconnect_session('svcname', ...,
DBMS_SERVICE.POST_TRANSACTION); (optional)
shutdown immediate

Application Continuity
• On error, application automatically replays queries and
transactions to another DB instance
• Masks most hardware, software, network, storage errors and
outages
• Available with Oracle Database 12.2+
• Transparent Application Continuity with Oracle Database 18+
• RAC, RAC One, & Active Data Guard
• Enabled by a DB service setting

Application Continuity – Two Phases
Automatic Capture
1. Pool marks start of
request
2. Client captures
original calls, their
inputs, and validation
data
3. Server decides which
can and cannot be
replayed
4. At successful end of
request, client purges
captured calls
Replay on Failure
1. Client checks replay is
enabled
2. Creates a new
connection
3. Client verifies
timeliness
4. Server checks replay
database is valid
5. Server checks if
committed, rolls back
if not
Replay (cont.)
6. Client replays
captured calls
7. Server and client
ensure results
returned to
application match
original results
8. On success, client
returns control to the
application

Towards Data Science

• Pandas
• Jupyter Notebook
• Oracle Spatial
• OML4Py*
* Coming soon

Time for a Demo !

oracle.com/gbtour
New Free Tier Always Free
Oracle Cloud Infrastructure
Services you can use for unlimited time
30-Day Free Trial
Free credits you can use for more services
+

Resources
Mail: christopher.jones@oracle.com
Twitter: @ghrd
Blog: blogs.oracle.com/opal
Home: https://oracle.github.io/python-cx_Oracle
Always Free: oracle.com/gbtour

Thank You
Christopher Jones
christopher.jones@oracle.com
Data Access Development
Oracle Database

Effective and Efficient Python with Oracle Database

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

Effective and Efficient Python with Oracle Database