This slide deck describes the most successful software project of my career: automating a 40-person human compiler yielding a reduction in project time from months to hours and cost from $1,000,000 to $100s for our client. The system is still in operation years later and has never had a single bug filed against it.
2. Jon Harrop
Started programming in 1981
BASIC, Pascal, 6502, ARM, C, C++
At British Petroleum in 1995
UFI
Physics and Chemistry at Cambridge
Fortran, SML, OCaml, Mathematica, F#
Co-founded Flying Frog in 2005
◦ Sell books, software and consultancy services
◦ Over 1,100 clients in industry using OCaml & F#
3. One of the world’s largest insurance
companies.
4. One of the world’s largest insurance
companies.
Actually, almost everyone else too…
5. 1. An ordinary delivery process
2. Streamlining the delivery process
3. Technical aspects of the solution
11. 1. Business idea!
2. Model idea: MS Excel.
3. Build idea: C++/Java/C#.
4. Test idea: NUnit (feedback to 2)
12. 1. Business idea!
2. Model idea: MS Excel.
3. Build idea: C++/Java/C#.
4. Test idea: NUnit (feedback to 2)
5. Deploy production code.
Room for improvement?
13. Room for improvement in:
Classic waterfall approach.
◦ Iterations are an absolute killer.
Delivery speed and cost.
◦ This is a long journey.
So what does an iteration look like?
15. First iteration highlights issues.
◦ Civilized discussion.
Second iteration, more issues.
◦ Stressful disagreements.
16. First iteration highlights issues.
◦ Civilized discussion.
Second iteration, more issues.
◦ Stressful disagreements.
Later iterations.
◦ Wild fighting.
17. Big problems:
Accuracy.
◦ Excel/Word are too free-form.
Complexity.
◦ Models are complicated, lots of dispatch.
Latency.
◦ Complected testing.
20. See the forest for the trees.
◦ Don’t just rewrite code in language du jour!!!
◦ Common mistake when adopting new technologies.
Overhaul the delivery process…
22. Observations
Separate people/jobs
◦ One person uses Tool to turn ideas into products.
◦ Another person maintains Tool.
Separate tests
◦ Tests for the model.
◦ Unit test for the Tool.
Advantages
23. Advantages:
Business people ship products themselves!
Less contention
◦ Faster iterations
◦ Quicker time-to-market
◦ Cheaper delivery
◦ Happier employees
Scalable delivery process
◦ Add more tool users
◦ Add more tool developers
24. Challenges:
Designing bespoke Tool
◦ Hard to gather requirements:“Define the language
you dream in?”
Training
◦ Users
◦ Developers
Plumbing
Deployment
25. Build a new tool for business users that:
Has a great user experience.
◦ Graphical application.
◦ Interoperates with Excel.
Creates precise specifications.
◦ Catch errors as early as possible.
Mirrors the production environment.
◦ Code reuse.
Integrates testing.
Is tuned to their needs.
26. DSLs have a rich history but GUIs are notably
absent…
DSL Patterns by Martin Fowler
27. No need to ship a DSL that looks worse than
this:
28. No need to ship a DSL that looks worse than
this:
1974
29. Started with a working demo
Developed in one week!
Slick looking
Easy to use
Bespoke language
Integrated testing
Their own worked examples included
Demo runs on their machines (WPF)
30. Final solution:
Development effort
◦ 3 months to design
◦ 3 months to build
Code size
◦ 10kLOC of F#
Performance
◦ 20,000 msg/s
◦ Under 35ms door-to-door latency
31. A different 3 month project:
114µs latency.
200,000 tps.
With fault tolerance!
Async message passing over Infiniband
Low-latency allocationless F# code
32. Training
◦ On-the-fly
◦ 30 people
◦ 5 teams
Emergency
◦ Valuable change came in late
◦ Estimated 9 months and £1m
◦ Completed in 24 hours
Maintenance
◦ All done by client now
◦ After one week of F# training
◦ Ordinary developers
33. “Pattern matching is quite neat, as is the partial
application of functions”
“rapidly coming to appreciate the benefits
when it comes to refactoring code”
“Our plans are getting bigger each week as we
can see more and more flexibility, it’s great!”
35. F# is good for programmatic GUIs
Approach described in the F# Journal
Custom Control union type
Compiled to WPF Control
Custom Editable Control
Reuse for editable strings, unions etc.
Compose into a UI tree
Peek Value property to get the “rules”
45. type Control =
| Gap
| Label of string
| Title of string
| Dock of (Position * Control) list
| Ctrl of FrameworkElement
| …
46. let rec make : Control -> FrameworkElement = function
| Gap ->
let ctrl = new FrameworkElement()
ctrl.HorizontalAlignment <- HorizontalAlignment.Stretch
ctrl.VerticalAlignment <- VerticalAlignment.Stretch
ctrl
| Label s -> upcast Controls.Label(Content=s)
| Title s -> upcast Controls.TextBlock(Text=s, FontSize=24.0)
| Dock xs ->
let panel = Controls.DockPanel()
for p, ctrl in xs do
let ctrl = make ctrl
Controls.DockPanel.SetDock(ctrl, p.AsWPF)
panel.Children.Add ctrl |> ignore
upcast panel
| …
47. let ui =
Dock
[ Top, Title "Bespoke business rules engine"
Bottom,
Tabs
[ Label "Inputs",
Dock[ Top, Label "This version was…"
Bottom, Scroll inputs ]
Label "Rules",
Dock[ Top, Label "The following rules…"
Bottom, Scroll(Ctrl rulesView) ]]]
48. let StringValue(value: Value) =
let label = Controls.Label()
let textbox = Controls.TextBox()
let setContent = function
| VString text ->
label.Content <- text
textbox.Text <- text
| _ -> failwith "Invalid value"
let ctrl =
Editable<Value>(value, label, textbox, setContent)
textbox.TextChanged.Add(fun _ ->
ctrl.SetValue(Some textbox.Text
|> Option.map VString))
ctrl
49. let UnionValue(allCases, value: Value) =
let label = Controls.Label()
let combo = Controls.ComboBox()
let itemOf case = case, Controls.ComboBoxItem(Content=case)
let items = Map[for case in allCases -> itemOf case]
let setContent = function
| VCase case ->
label.Content <- case
combo.SelectedItem <- items.[case]
| value -> failwith "Invalid value"
let ctrl = Editable<Value>(value, label, combo, setContent)
combo.ItemsSource <- [ for KeyValue(_, ctrl) in items -> ctrl ]
for KeyValue(case, x) in items do
x.Selected.Add(fun _ -> ctrl.SetValue(Some(VCase case)))
ctrl
50. The expression language:
type Expr =
| EBool of bool
| ENumber of decimal
| EString of string
| EDate of System.DateTime
| ETuple of Expr list
| EVar of string
| …
51. The expression interpreter:
let rec eval vars expr =
match expr with
| EBool b -> VBool b
| ENumber x -> VNumber x
| EString s -> VString s
| EDate d -> VDate d
| ETuple fs -> VTuple(List.map (eval vars) fs)
| EVar v ->
match Map.tryFind v vars with
| None -> failwithf "Unknown variable '%s'" v
| Some x -> x
| …
52. Challenges
Slow startup times
Poor GUI performance (grid of numbers)
Minor perf bugs
Brittle .NET serialization
55. GUI performance
Problem:
The WPF Grid control is:
◦ very slow to start up
◦ very slow to scroll
Solution:
Custom FastGrid control…
Low-level rendering in F#
◦ Calls DrawGlyphs
56. GUI performance
16x faster startup than vanilla WPF.
Solution in the F# Journal.
Still a lot of room for improvement…
58. Serialization
Problem:
.NET serialization not great for F#.
Bloated format (includes types).
Brittle when loading (incompatible types).
Solution:
Custom structural serialization library.
Handles values of all F# types.
Solution in the F# Journal.