Patrick Premont of BoldRadius presented this talk at Scala By The Bay 2015. Why do data structure lookups often return Options? Could we safely eliminate all the recovery code that we hope is never called? We will see how Scala’s type system lets us express referential integrity constraints to achieve unparalleled reliability. We apply the technique to in-memory data structures using the Total-Map library and consider how to extend the benefits to persisted data.

1. TOWARDS
RELIABLE LOOKUPS
Scala by the Bay 2015, August 14th
Patrick Prémont - @ppremont
Functional Programming Architect

2. MAP LOOKUPS
trait Map[K, V]{
def get(key: K): Option[V]
}

3. OPTION.GET

4. DEFENSIVE
PROGRAMMING
.getOrElse(
// Not sure how this
// can ever happen,
// but just in case…
“Unknown”
)()

5. INVALID REFERENCES
Incorrect judgements yield waste or bugs!
Impossible: Ignore/terminate
Possible:
Handle appropriately or
revise program
?

6. INVALID REFERENCES
WITH TOTAL-MAP
Impossible: Types prove it!
?

7. RELATIONAL DATABASES
Foreign Key
quantity partId
10 102
100 102
25 101
id name
101 Monitor
102 Keyboard
Shipment Part
Brittle insertion Brittle removal

8. @ppremont
CURRENT PRACTICE
case class Data(
parts: Map[Int, String],
shipments: List[(Int, Int)])

9. @ppremont
DISTINCT IDTYPES
case class PartId(id: Int)
case class Data(
parts: Map[PartId, String],
shipments: List[(Int, PartId)])

10. TOTAL MAPS
trait Map[K, V]{
def get(key: K): Option[V]
}
trait Total[V]{
type Id
def apply(id: Id): V
}

11. PATH-DEPENDENT TYPE
trait Data {
val parts: Total[String]
val shipments: List[(Int, parts.Id)]
}

12. SAFER UPDATES TOO
trait Total[V]{
type Id
def update(id: Id, f: V=>V)
}

13. IDENTIFIER TYPES
• As we insert, need types with more values
• As we remove, need types with fewer values

14. INSERTION
def insert(value: V): {
val total:
Total[V] {
type Id >: Total.this.Id
}
val newId: total.Id
}

15. REMOVAL
def remove(removedKey: Id): {
val total:
Total[V] {
type Id <: Total.this.Id
}
def filter(k: Total.this.Id):
Option[total.Id]
}

16. COVARIANT
ID TYPE PARAMETERS
case class Shipment[+P](
quantity: Int,
part: P)
val parts: Total[String]
val shipments:
List[Shipment[parts.Id]]

17. MULTIPLE CONSTRAINTS
case class Country(name: String)
case class Address[+CountryId](country: CountryId)
case class Customer[+AddressId](address: AddressId)
trait Data {
val countries : Total[Country]
val addresses : Total[Address[countries.Id]]
val customers : List[Customer[addresses.Id]]
}

18. CYCLICAL CONSTRAINTS
case class User[+U](
name: String,
follows: List[U])
trait Data {
val users : Total[User[users.Id]]
}

19. BREAKING THE CYCLE
case class User[+U](
name: String,
follows: List[U])
trait Data {
val userSet : Total[Unit]
val users : userSet.Map[User[userSet.Id]]
}

20. COMPLEMENT TYPES
trait Total {
type Id
type Comp
def allocate: Comp
def insertAt(id: Comp) : Total {
type Id >: Total.this.Id
}
}

21. ID TYPE FAMILY
Nothing {}
Option[Nothing] {None}
Option[
Option[Nothing]]
{None,
Some(None)}
Option[
Option[
Option[Nothing]]]
{None,
Some(None),
Some(Some(None))}

22. SELECTIVE SUBSETS
Nothing {}
Either[Unit,
Nothing]
{Left(())}
Either[Unit,
Either[Unit,
Nothing]]
{Left(()),
Right(Left(()))}
Either[Nothing,
Either[Unit,
Nothing]]
{Right(Left(()))}

23. BINARY TREE
Nothing {}
Id2[Unit,
Nothing,
Nothing]
{Element}
Id2[Unit,
Id2[Unit,Nothing,Nothing],
Nothing]
{Element,
Left(Element)}
Id2[Unit,
Id2[Unit,Nothing,Nothing],
Id2[Unit,Nothing,Nothing]]
{Element,
Left(Element),
Right(Element)}

24. CONSTANT SPACE
Nothing {}
Id2[Unit,
Nothing,
Nothing]
{Id2(0)}
Id2[Unit,
Id2[Unit,Nothing,Nothing],
Nothing]
{Id2(0),
Id2(1)}
Id2[Unit,
Id2[Unit,Nothing,Nothing],
Id2[Unit,Nothing,Nothing]]
{Id2(0),
Id2(1),
Id2(2)}

25. PERFORMANCE
• Total maps are binary trees: O(log n) operations
• No rebalancing: identifiers designate fixed paths
to a node
• n is the highest index in use

26. PERFORMANCE
DEGRADATION
Total.empty
.insertAll(0 to size)
(1 to size)
.map(i => (i, i))(breakOut)
: Map[Int, Int]
us per inserted element
0
0.5
1
1.5
100 000 1 000 000
Total Map

27. BENEFITS
• Safe lookups, updates, inserts and removals
• Descriptive identifier types (parts.Id)

28. LIMITATIONS
• Traversal on removal
• Narrowing identifier types generates garbage
• Advanced, path-dependent types
• Type inference

29. ORTHOGONAL
PERSISTENCE
• Run a non-persistent program in a persistent
manner.
• Referential integrity must be built into the
program

30. TRANSACTIONS
type Txn = Data => (Data, A)
def run(txn: Txn) : IO[A] = IO{
val in = db.read[Data]
val (out, a) = txn(in)
db.write(out)
a
}

31. TOTAL MAP LIBRARY
github.com/boldradius/total-map

32. TOWARDS
RELIABLE LOOKUPS
Scala Symposium 2015 Paper:
Referential Integrity with Scala Types
github.com/boldradius/total-map
Patrick Prémont - @ppremont