N1QL = SQL + JSON. N1QL gives developers and enterprises an expressive, powerful, and complete language for querying, transforming, and manipulating JSON data. We begin with a brief overview. Couchbase 5.0 has language and performance improvements for pagination, index exploitation, integration, and more. We’ll walk through scenarios, features, and best practices.

1. N1QL : WHAT’S NEW
IN COUCHBASE 5.0
Keshav Murthy
Senior Director, Couchbase R&D

2. AGENDA
01
02
N1QL: Overview -- 5 minutes, hopefully!
N1QL Features in 5.0 -- Rest of the time

3. 1 N1QL: OVERVIEW

4. 4
ResultSet
Relations/Tuples

5. 5
{
"Name" : "Jane Smith",
"DOB" : "1990-01-30",
"Billing" : [
{
"type" : "visa",
"cardnum" : "5827-2842-2847-3909",
"expiry" : "2019-03"
},
{
"type" : "master",
"cardnum" : "6274-2842-2847-3909",
"expiry" : "2019-03"
}
],
"Connections" : [
{
"CustId" : "XYZ987",
"Name" : "Joe Smith"
},
{
"CustId" : "PQR823",
"Name" : "Dylan Smith"
}
{
"CustId" : "PQR823",
"Name" : "Dylan Smith"
}
],
"Purchases" : [
{ "id":12, item: "mac", "amt": 2823.52 }
{ "id":19, item: "ipad2", "amt": 623.52 }
]
}
LoyaltyInfo
Results
Orders
CUSTOMER
• NoSQL systems provide specialized APIs
• Key-Value get and set
• Each task requires custom built program
• Should test & maintain it

6. 6
Find High-Value Customers with Orders > $10000
Query customer
objects from
database
• Complex codes and logic
• Inefficient processing on client side
For each customer
object
Find all the order
objects for the
customer
Calculate the total
amount for each
order
Sum up the grand
total amount for all
orders
If grand total
amount > $10000,
Extract customer
data
Add customer to
the high-value
customer list
Sort the high-value
customer list
LOOPING OVER MILLIONS OF CUSTOMERS IN APPLICATION!!!

7. 7
{
"Name" : "Jane Smith",
"DOB" : "1990-01-30",
"Billing" : [
{
"type" : "visa",
"cardnum" : "5827-2842-2847-3909",
"expiry" : "2019-03"
},
{
"type" : "master",
"cardnum" : "6274-2842-2847-3909",
"expiry" : "2019-03"
}
],
"Connections" : [
{
"CustId" : "XYZ987",
"Name" : "Joe Smith"
},
{
"CustId" : "PQR823",
"Name" : "Dylan Smith"
}
{
"CustId" : "PQR823",
"Name" : "Dylan Smith"
}
],
"Purchases" : [
{ "id":12, item: "mac", "amt": 2823.52 }
{ "id":19, item: "ipad2", "amt": 623.52 }
]
}
LoyaltyInfo
ResultDocuments
Orders
CUSTOMER

8. 8
N1QL = SQL + JSON
Give developers and enterprises an
expressive, powerful, and complete language
for querying, transforming, and manipulating
JSON data.

9. 9
Why SQL for NoSQL?

10. 10
N1QL : Data Types from JSON
Data Type Example
Numbers { "id": 5, "balance":2942.59 }
Strings { "name": "Joe", "city": "Morrisville" }
Boolean { "premium": true, "balance pending": false}
Null { "last_address": Null }
Array { "hobbies": ["tennis", "skiing", "lego"]}
Object { "address": {"street": "1, Main street", "city":
Morrisville, "state":"CA", "zip":"94824"}}
MISSING
Arrays of objects of arrays [
{
"type": "visa",
"cardnum": "5827-2842-2847-3909",
"expiry": "2019-03"
},
{
"type": "master",
"cardnum": "6274-2542-5847-3949",
"expiry": "2018-12"
}
]

11. 11
N1QL: Data Manipulation Statements
• SELECT Statement-
• UPDATE … SET … WHERE …
• DELETE FROM … WHERE …
• INSERT INTO … ( KEY, VALUE ) VALUES …
• INSERT INTO … ( KEY …, VALUE … ) SELECT …
• MERGE INTO … USING … ON …
WHEN [ NOT ] MATCHED THEN …
Note: Couchbase provides per-document atomicity.

12. 12
N1QL: SELECT Statement
SELECT *
FROM customers c
WHERE c.address.state = 'NY'
AND c.status = 'premium'
ORDER BY c.address.zip
Project Everything
From the bucket customers
Sort order
Predicate

13. 13
N1QL: SELECT Statement
SELECT customers.id,
customers.NAME.lastname,
customers.NAME.firstname
Sum(orderline.amount)
FROM orders UNNEST orders.lineitems AS orderline
INNER JOIN customers ON KEYS orders.custid
WHERE customers.state = 'NY'
GROUP BY customers.id,
customers.NAME.lastname,
customers.NAME.firstname
HAVING sum(orderline.amount) > 10000
ORDER BY sum(orderline.amount) DESC
• Dotted sub-document
reference
• Names are CASE-
SENSITIVE
UNNEST to flatten the arrays
JOINS with Document KEY of
customers

14. 14
N1QL: SELECT Statement Highlights
• Querying across relationships
• JOINs
• Subqueries
• Aggregation
• MIN, MAX
• SUM, COUNT, AVG, ARRAY_AGG [ DISTINCT ]
• Combining result sets using set operators
• UNION, UNION ALL, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL

15. 15
N1QL : Query Operators [ 1 of 2 ]
• USE KEYS …
• Direct primary key lookup bypassing index scans
• Ideal for hash-distributed datastore
• Available in SELECT, UPDATE, DELETE
• JOIN … ON KEYS …
• Nested loop JOIN using key relationships
• Ideal for hash-distributed datastore
• Current implementation supports INNER and LEFT OUTER joins
• ANSI JOINS
• We’re working on it. Be part of BETA for the next release.

16. 16
N1QL : Query Operators [ 2 of 2 ]
• NEST
• Special JOIN that embeds external child documents under their parent
• Ideal for JSON encapsulation
• UNNEST
• Flattening JOIN that surfaces nested objects as top-level documents
• Ideal for decomposing JSON hierarchies
• JOIN, NEST, and UNNEST can be chained in any combination

17. 17
N1QL : Expressions for JSON
Ranging over collections
• WHERE ANY c IN children SATISFIES c.age > 10 END
• WHERE EVERY r IN ratings SATISFIES r > 3 END
Mapping with filtering • ARRAY c.name FOR c IN children WHEN c.age > 10 END
Deep traversal, SET,
and UNSET
• WHERE ANY node WITHIN request SATISFIES node.type = “xyz” END
• UPDATE doc UNSET c.field1 FOR c WITHIN doc END
Dynamic Construction
• SELECT { “a”: expr1, “b”: expr2 } AS obj1, name FROM … // Dynamic
object
• SELECT [ a, b ] FROM … // Dynamic array
Nested traversal • SELECT x.y.z, a[0] FROM a.b.c …
IS [ NOT ] MISSING • WHERE name IS MISSING

18. 18
Architecture
App App
Couchbase
App App
Couchbase
App App
Couchbase
App App
Couchbase
Couchbase Cluster
App App
Couchbase
Couchbase Single
Node Deployment
Couchbase Cluster Deployment

19. 19
Couchbase Server Cluster Service Deployment
STORAGE
Couchbase Server 1
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed Cache
Cluster
ManagerCluster
Manager
Managed Cache
Storage
Data
Service STORAGE
Couchbase Server 2
Managed Cache
Cluster
ManagerCluster
Manager
Data
Service STORAGE
Couchbase Server 3
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed Cache
Cluster
ManagerCluster
Manager
Data
Service STORAGE
Couchbase Server 4
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed Cache
Cluster
ManagerCluster
Manager
Query
Service STORAGE
Couchbase Server 5
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed Cache
Cluster
ManagerCluster
Manager
Query
Service STORAGE
Couchbase Server 6
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed Cache
Cluster
ManagerCluster
Manager
Index
Service
Managed Cache
Storage
Managed Cache
Storage Storage
STORAGE
Couchbase Server 6
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed Cache
Cluster
ManagerCluster
Manager
Index
Service
Storage
Managed Cache Managed Cache
SDK SDK

20. 20
N1QL: Query Execution Flow
Clients
1. Submit the query over REST API 8. Query result
2. Parse, Analyze, create Plan 7. Evaluate: Documents to results
3. Scan Request;
index filters
6. Fetch the documents
Index
Service
Query
Service
Data
Service
4. Get qualified doc keys
5. Fetch Request,
doc keys
SELECT c_id,
c_first,
c_last,
c_max
FROM CUSTOMER
WHERE c_id = 49165;
{
"c_first": "Joe",
"c_id": 49165,
"c_last": "Montana",
"c_max" : 50000
}

21. 21
N1QL: Inside the Query Service
Client
FetchParse Plan Join Filter
Pre-Aggregate
Offset Limit ProjectSortAggregateScan
Query Service
Index
Service
Data
Service

22. * N1QL: PATH OF PROGRESS

23. 23
N1QL features in Couchbase
releases
23
Couchbase 4.0: N1QL GA
Query language for JSON,
Integrated Query Service, Global
Secondary Index, REST API,
Simba ODBC, JDBC Drivers
Couchbase 4.1:
INSERT, UPDATE, DELETE, MERGE
Covering Index Optimization
Couchbase 4.1.1:
Index JOINs
Couchbase 4.5:
Array Indexes, Workbench, CBQ Shell++, INFER
Memory Optimized Index, IndexScanCount
YCSB Performance Optimizations++, Language++
Couchbase 4.5.1:
Pretty=false; Fetch; SUFFIXES;
Index Selection; UPDATE
improvement
Oct 2015
Dec
2015
March 2016
June 2016
Sep 2016

24. 24
N1QL features in Couchbase
releases.
24
Couchbase 4.6.1:
TOKENS (Simple Search/Faster LIKE), Optimizer improvements
Couchbase 5.0:
Subqueries over nested data ; Pagination; RBAC; Curl,
Super Charged Indexing; Monitoring & Profiling; New
workbench, UI, monitoring, profiler, visual EXPLAIN
Performance++, Bitwise functions
Q1 2017
4Q 2017
Couchbase 4.6.2:
Optimizer improvements, intersect scan performance
Q2 2017

25. N1QL FEATURES IN 5.0
01
02
04
List of Features
Query-Indexing Enhancements
Optimizer
03 Language & Infrastructure
05 Flexible Indexing
06 Security
07 Monitoring & Profiling
08 Developer Tools
09 Performance

26. 1 LIST OF FEATURES

27. 27
Couchbase N1QL and GSI features
Query-Indexing Features
• Large Indexing Keysize
• Index key collation: ASC, DESC on each key
• Index replicas, just like data replication
• New storage engine: Plasma
Query Language & Infrastructure
• Subquery Expressions
• Additional Date & time functions
• Bitwise functions
• CURL() within N1QL
Query Optimizer
• Complex Filters Pushdown
• Pagination optimization
• Optimization for ASC, DESC keys
• Query-Index API optimization (projection, etc.)
• Index projections, Intersect scans
• Adaptive Indexes
Security, Administration & Functionality
• Security: RBAC: Statement level security
• Query Monitoring, Profiling with UI
• Query work bench and UI: Fully upgraded
• Query UI: Visual Explain
• Query on Ephemeral buckets
• Application Continuity, Seamless Upgrade
Performance
• Core daily workload
• YCSB
• YCSB-JSON for Engagement Database
http://query.couchbase.com

28. 2 QUERY-INDEX FEATURES

29. 29
Query-Indexing Enhancements
Index key collation: ASC, DESC on each key
• Prior to 5.0, each index key was sorted and kept in ASCENDING order only
• To sort the key in descending order, you did
• CREATE INDEX i1 ON t(c1 ASC, -c2, -c3)
• SELECT * FROM t WHERE c1 = 10 and -c2 < -20 ORDER BY c1, -c2
• Query formulations becomes confusing
• Cannot use this trick on all data types and expressions
In Couchbase 5.0:
• CREATE INDEX i1 ON t(c1 ASC, c2 DESC, c3 DESC)
• SELECT * FROM t WHERE c1 = 10 and c2 < 20 ORDER BY c1,c2 DESC
• You need to create an index to match the ORDER BY order
• Reverse scans are still unsupported

30. 30
Query-Indexing Enhancements
Large Indexing Keysize
• Prior to 5.0, the sum of index key size could be up to 4096 bytes
• This was controlled by the setting
• For ARRAY keys, sum of all array key sizes could be up to 10240.
• This is controlled by the setting max_array_seckey_size
In Couchbase 5.0:
• The total keysize could be pretty high – high up to 20 MB
• This is true for single key, composite key, expressions and array indexes as well.
• Simply do nothing, except create the index and issue the query.
• The index entries that exceed 20MB will still generate error in the index log

31. 31
Query-Indexing Enhancements
Index replicas, just like data replication
• Prior to 5.0, you could create multiple indexes with same keys & condition
• This is needed for load balancing and index high availabilitt
CREATE INDEX i1 ON t(c1, c2, c3)
CREATE INDEX i2 ON t(c1, c2, c3)
CREATE INDEX i3 ON t(c1, c2, c3)
• Indexer automatically recognizes these to be equivalent and does load balancing on all o these.
In Couchbase 5.0:
• Simply create one index and set the num_replica at CREATE or ALTER time
• CREATE INDEX i1 ON t(c1, c2, c3) WITH {"num_replica":2}
• Number of replicas can be up to number of nodes in the cluster
• You can ALTER the number of replica dynamically

32. 32
Query-Indexing Enhancements
New storage engine: Plasma
• Uses lock-free skip list
• All the performance benefits of MOI – Memory Optimized Index
• Automatically does IO as needed
• From usage point of view:
• Choose the standard secondary Index during installation
• simply create any kind of index and use it.
• More on Plasma in the Indexing Talk this afternoon.

33. 3 QUERY LANGUAGE &
INFRASTRUCTURE

34. 34
Query Language & Infrastructure
Subquery Expressions
• Sub-queries are N1QL expressions that can embed a full N1QL SELECT
query
• Can be used in projection, FROM and WHERE clause
• Before 5.0, the table expression in FROM-clause is limited to:
• Keyspace name or Sub-query
• constants, expressions, functions, nested paths are disallowed
• Correlated Sub-queries on nested objects is expensive
• Inner sub-query is evaluated for every document from outer query
• Same document is accessed/read for every level of the query nesting

35. 35
Query Language & Infrastructure
Subquery Expressions
• Provides rich functionality and Powerful subquery-expressions
• Can be used in FROM-clause, projection, LET/WHERE-clauses etc.,
SELECT word, cnt
FROM ARRAY split(i) FOR i IN (SELECT raw name
FROM `travel-sample`
WHERE type = "hotel") END AS words
UNNEST words w
GROUP BY w LETTING cnt = COUNT(w)
ORDER BY cnt DESC;

36. 36
Query Language & Infrastructure
Additional Date, time, timestamp function
• JSON does not directly support date and time related functions
• Store the date and time in extended ISO 8901 format
• "2017-10-16T18:44:43.308-07:00”
• Need extract, conversion and arithmetic functions
• Detailed article with all the functions and Oracle to Couchbase mapping
https://dzone.com/articles/comparing-oracle-and-n1ql-support-for-the-date-tim
• If you can’t do something, let us know!

37. 37
Query Language & Infrastructure
CURL() within N1QL
• CURL (URL, [options])
• The first argument is the URL, which represents any URL that points to a JSON
endpoint.
• Only URLs with the http:// or the https:// protocol are supported.
• Redirection is disabled.
• The second argument is a list of options.
• This is a JSON object that contains a list of curl options and their corresponding
values.
• For a full list of options that we support, please refer to the Dzone article on
CURL in N1QL by Isha Kandaswamy
•

38. 38
Query Language & Infrastructure
CURL() within N1QL
• Search for Santa Cruz in Spain using my Google dev api key
• SELECT
CURL("GET","https://maps.googleapis.com/maps/api/geocode/json",
{"data":"address=santa+cruz&components=country:ES&key=AIzaSyCT6n
iGCMsgegJkQSYasfoLZ4_rSO59XQQ"}) ;
• Search for Half Moon Bay
• SELECT
CURL("GET","https://maps.googleapis.com/maps/api/geocode/json",{
"data":"address=Half+Moon+Bay"} ).results rs;
•

39. 39
Query Language & Infrastructure
BITWISE Functions
• All bitwise functions can only take a number. All numbers are 64 bit signed numbers
(integers). If the Number is not an integer and for other data types, we throw an error.
• When looking at the value in binary form, bit 1 is the Least Significant Bit (LSB) and bit
32 is the Most Significant Bit. (MSB) Bit 32 → 0000 0000 0000 0000 0000 0000 0000
0000 ← Bit 1 (LSB)
• Supported functions :
• BitAND
• BitOR
• BitNOT
• BitXOR
• BitSHIFT
• BitSET
• BitCLEAR
• BitTEST/ IsBitSET

40. 4 QUERY OPTIMIZER &
EXECUTION

41. 41
Query Optimizer & Execution: Stable Scans
• IndexScan use to do single range scan (i.e single Span)
• If the query has multiple ranges (i.e. OR, IN, NOT clauses) Query service use to
do separate IndexScan for each range.
• This causes Indexer can use different snapshot for each scan (make it unstable
scan)
• Number of IndexScans can grow and result increase in index conneconnections
• In 5.0.0 multiple ranges are passed into indexer and indexer uses same
snapshot for all the ranges.
• This makes stable Scan for given IndexScan (i.e. IndexScan2 in the EXPLAIN).
• This will not make stable scan for query due to Subqueries, Joins etc
• Example:
• create index ix1 on default(k0);
• EXPLAIN SELECT META().id FROM default WHERE k0 IN [10,12,13];

42. 42
Query Optimizer & Execution: Pushdown Composite Filters
• For composite Index the spans that pushed to indexer contains
single range for all composite keys together.
• Indexer will not applying range for each part of the key separately.
This result in lot of false positives.
• In 5.0.0 with IndexScan2 we push the each index key range
separately and indexer will apply keys separately.
• This results in no/less false positives and aides push more
information to indexer.
• Example:
• CREATE INDEX ix1 ON default(k0,k1);
• EXPLAIN SELECT meta().id FROM default WHERE k0 BETWEEN 0 AND
100 AND k1 = 200;

43. 43
Query Optimizer: ORDER, OFFSET, LIMIT pushdown
• Pagination queries can contain any combination of ORDER, LIMIT, OFFSET
clauses.
• Performance of these queries are critical to applications.
• When Predicates are completely and exactly pushed to indexer, by pushing
offset, limit to indexer can improve query performance significantly. If that
happened IndexScan2 section of EXPLAIN will have limit,offset.
• If query ORDER BY matches index key order query can avoid index sort and
performance can be improved significantly. If that happened order operator is
not present in the EXPLAIN.
• Example:
• CREATE INDEX ix1 ON default(k0,k1);
• EXPLAIN SELECT meta().id FROM default WHERE k0 > 10 AND k1 > 20 ORDER
BY k0 LIMIT 10 OFFSET 100;

44. 44
Query Optimizer: MAX pushdown
• If the MAX arguments matched with Index leading key exploit
index order for MAX.
• MAX can only DESC on index key.
• MIN can only use ASC on index key.
• Example :
• CREATE INDEX ix5 ON default(k0 DESC);
• SELECT MAX(k0) FROM default WHERE k0 > 10;
• Above query able to exploit index order. In that case IndexScan2
section of EXPLAIN will have “limit” 1.

45. 45
Query Optimizer: Index Projection
• The index can have many keys but query might be interested only
subset of keys.
• By only requesting required information can save lot of network
transportation, memory, cpu, backfill etc. All this can help in
performance and scaling the cluster.
• The requested information can be found in “IndexScan2” Section
of EXPLAIN as “index_projection”
"index_projection": {
"entry_keys": [ xxx,....... ],
“primary_key”: true
}
•

46. 46
Query Optimizer: Index Projection
CREATE INDEX ix1 ON default(k0,k1);
Covered query
SELECT k0 FROM default WHERE k0 = 10 AND k1 = 100;
"index_projection": {
"entry_keys": [0,1]
}
SELECT k0 FROM default WHERE k0 = 10;
"index_projection": {
"entry_keys": [0]
}
SELECT k0 ,META().idFROM default WHERE k0 = 10;
"index_projection": {
"entry_keys": [0],
“primary_key”: true
}
Non-covered query
SELECT k0 ,k5 FROM default WHERE k0 = 10 AND k1 = 100;
"Index_projetion": { “primary_key”: true }

47. 47
Query Execution: CAS & Expiration
• In 5.0.0 META().cas, META().expiration can be indexed and used
in queries.
• Example:
• CREATE INDEX ix1 ON default( meta().id, meta().cas,
meta().expiration);
• SELECT meta().id , meta().cas, meta().expiration FROM
default where meta().id > ""
• Note: META().expiration will work in covered queries. For
non covered queries it gives 0

48. 48
Query Execution: COUNT (DISTINCT expr)
• If the expr matched with Index leading key COUNT DISTINCT can
be pushed to indexer
• Complete predicate needs to pushed to indexer exactly
• No false positives are possible
• No group or JOIN
• Only single projection
• Example :
• CREATE INDEX ix5 ON default(k0);
• SELECT COUNT(DISTINCT k0) FROM default WHERE k0 > 10;
• Above query uses IndexCountDistinctScan2

49. 5 N1QL: FLEXIBLE INDEXING

50. 50
Customer Scenario
• Customer document has 100 fields
• They have multiple business entities sharing the same data
• Each entity want to FILTER, GROUP, ORDER on distinct criteria
• For Index selection, order of the keys in the composite index is important.
Fields: c1 through c100
Filter fields: c1 through c50
Group, order and projection: Any from c1 through c100
SELECT c1, c2, c3, COUNT(c10), SUM(c5)
FROM CUSTOMER
WHERE c4 = "CXT-MULTI"
AND c8 = "iPhone6"
AND c9 BETWEEN 10 IN 20
GROUP BY c1, c2, c3;
SELECT c12, COUNT(c19), SUM(c15)
FROM CUSTOMER
WHERE c44 = "CXT-MULTI"
AND c18 = "Gpixel 2"
AND c29 BETWEEN 10 IN 20
GROUP BY c12;

51. 51
Customer Scenario
• What indexes to create for this?
SELECT c1, c2, c3, COUNT(c10), SUM(c5)
FROM CUSTOMER
WHERE c4 = "CXT-MULTI"
AND c8 = "iPhone6"
AND c9 BETWEEN 10 IN 20
GROUP BY c1, c2, c3;
CREATE INDEX i1 ON CUSTOMER(c8, c4, c9)
CREATE INDEX i1 ON CUSTOMER(c8, c4, c9, c1, c2, c3, c10, c5; For Covering the query
What about this?
SELECT c12, COUNT(c19), SUM(c15)
FROM CUSTOMER
WHERE c44 = "CXT-MULTI"
AND c18 = "Gpixel 2"
AND c29 BETWEEN 10 IN 20
GROUP BY c12;

52. 52
Large, wide, composite indexes
Filter fields: c1 through c50
To support all combinations of 50 predicates via composite indexes, you’ll need LOT of
indexes.
50!
=304140932017133780436126081660647688443776415689605
12000000000000

53. 53
Customer Scenario
Solution: Intersection
• Option 1
• Create indexes on individual fields
• Scan individual indexes
• Apply the full set of predicates (boolean expression from WHERE clause)
• Then do the post processing.
CREATE INDEX i1 on CUSTOMER(c1);
CREATE INDEX i2 on CUSTOMER(c2);
CREATE INDEX i3 on CUSTOMER(c3);
• Option 2
• Too many indexes to maintain and manage.
• Don’t even talk about equivalent indexes for each of these.
CREATE INDEX i1to50 on CUSTOMER(DISTINCT PAIRS({c1, c2, c3, c4,
c5,c6, c7, c8, c9, c10, c11, c23, c13, c14, …});

54. 54
Solution: Intersection
• Option 3
• Too many keys to manage/specify
• The document is flexible. I want the index to be flexible.
CREATE INDEX ixpairon CUSTOMER(DISTINCT PAIRS(self));
SELECT * FROM CUSTOMER WHERE a = 10 and b < 20 and c between 30 and 40;
"#operator": "IntersectScan",
"scans": [
{
"#operator": "DistinctScan",
"scan": {
"#operator": "IndexScan2",
"index": "ixpair",
"index_id": "466c0c5c4c3b21c1",
"index_projection": {
"primary_key": true
},
"keyspace": "test",
"namespace": "default",
"spans": [
{
"exact": true,
"range": [
{
"high": "["a", 10]",
"inclusion": 3,
"low": "["a", 10]"
}
"range": [
{
"high": "["b", 20]",
"inclusion": 1,
"low": "["b", false]"
}
"range": [
{
"high": "[successor("c")]",
"inclusion": 1,
"low": "["c", 30]"
}
]

55. 55
Flexible Indexing
• This is not a silver bullet, yet.
• TRY THIS OUT
• SIZING is a concern because we {“Key“:“value“}
• Give us feedback

56. 6 SECURITY

57. 57
SECURITY
• Query_select, query_insert, query_update, query_delete roles
• Parameterized: query_select[customers] or query_insert[*]
• Query_manage_index[foo]
• Create, delete, build indexes on bucket foo
• Query_system_catalog
• Full access to the system tables (which are controlled now)
• Query_external_access
• Allows access to CURL() function (disabled by default)
• GRANT cluster_admin TO spock
• GRANT query_select ON default TO kirk
• REVOKE query_insert, query_delete ON bridge, engineering FROM mccoy, scotty

58. 7 MONITORING & PROFILING

59. 59
Monitoring in UI
59Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.

60. 60
Profiling in UI
60Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.

61. 61
Profiling
• We can collect execution timings and document processed on a per operator basis
• If the functionality is turned on, timings are reported
• with the metrics at the end of execution
• in system:active_requests
• in system:completed_requests
• Profiling is turned on
• at the request level via the “profile” REST API parameter, EG from cbq:
• set –profile timings;
• at the node level via the “profile” command line parameter or admin settings REST API parameter
• takes 3 values, “off”, “phases”, “timings”
• “phases” supplies total times for each operator class
• “timings” supplies detailed information for each operator

62. 62
Profiling
cbq> select * from `travel-sample` where source-airport is not missing;
…
"executionTimings": {
"~children": [
{
"#operator": "IndexScan2",
"#stats": {
"#itemsOut": 24024,
"#phaseSwitches": 96099,
"execTime": "55.370283ms",
"kernTime": "5.397199311s"
},
"index": "def_sourceairport",
"index_id": "29702e564c9d2ca4",
"index_projection": {
"primary_key": true
},
"keyspace": "travel-sample",
"namespace": "default",
"spans": [
{
"exact": true,
"range": [
{
"inclusion": 1,
"low": "null"

63. 8 DEVELOPER TOOLING

64. 64
Developer Tooling

65. 9 PERFORMANCE

66. 66
N1QL Performance: 5.0 vs. 4.5
• Run internally
• YCSB is the public YCSB
• other queries are written on Couchbase dataset
• 50% higher throughput in YCSB workload E
• 10-40x faster pagination queries
• 10-30x better performance of queries with composite filters
• 10-40x faster queries with COUNT function
• 6-9x better performance of basic queries (Q1 & Q2)
• 55x faster queries with UNNEST clause

67. 67
N1QL Performance: 5.0 vs. 4.5
• Up to 10x faster array indexing
• Fast text search with TOKENS()
• 10x better performance of lookup and index joins
• Query performance on Windows is on par with Linux
• Up to 100K index scans per second in DGM scenarios

68. 68
SUMMARY of N1QL features in Couchbase 5.0
Query-Indexing Features
• Large Indexing Keysize
• Index key collation: ASC, DESC on each key
• Index replicas, just like data replication
• New storage engine: Plasma
Query Language & Infrastructure
• Subquery Expressions
• Additional Date & time functions
• Bitwise functions
• CURL() within N1QL
Query Optimizer
• Complex Filters Pushdown
• Pagination optimization
• Optimization for ASC, DESC keys
• Query-Index API optimization (projection, etc.)
• Index projections, Intersect scans
• Adaptive Indexes
Security, Administration & Functionality
• Security: RBAC: Statement level security
• Query Monitoring, Profiling with UI
• Query work bench and UI: Fully upgraded
• Query UI: Visual Explain
• Query on Ephemeral buckets
• Application Continuity, Seamless Upgrade
Performance
• Core daily workload
• YCSB
• YCSB-JSON for Engagement Database
http://query.couchbase.com