I've been around hundreds of Oracle performance projects since 1989, and my colleagues have been around thousands more. In candid after-work talks about our experiences, there has been remarkable symmetry in what we believe makes a project successful. People everywhere seem to be discovering the same secret on their own, through the Agile, Lean Startup, and Design Thinking movements. The secret? Shorter feedback loops. But shortening your feedback loops in Oracle projects is easier said than done. It all depends on how well you Test and Measure. But how do you overcome the political and technical barriers to making great testing and measuring a part of your project plan?
11. @CaryMillsap 11
A STORY BY LessonsLearned.FAA.gov
ECO15 PRESENTS A METHOD R PRODUCTION
“THE KEGWORTH AIR DISASTER: THE TRAGIC STORY OF BRITISH MIDLAND FLIGHT 92” TOLD BY CARY MILLSAP
The Kegworth
Air Disaster
The Tragic Story of British Midland Flight 92
12. @CaryMillsap 12@CaryMillsap
Approximately 13 minutes after takeoff from
London's Heathrow Airport, on a flight planned to
Belfast, Ireland, the outer portion of a fan blade on
the #1 engine failed as the airplane was climbing
through 28,000 feet. The fan blade failure resulted in
high levels of airframe vibration, a series of
compressor stalls in the left engine, fluctuation of the
left engine parameters, and smoke and fumes in the
flight deck.
The flight crew, believing that the right engine had
failed, reduced thrust on that engine, and
subsequently shut it down. The airframe vibration
ceased as soon as thrust was reduced on the right
engine, reinforcing the crew's identification as the
right engine having been the engine that had failed.
The crew initiated a diversion to East Midlands
Airport, which progressed normally until, at 2.4
nautical miles from the runway, a fire warning and
abrupt thrust loss occurred on the left engine.
Attempts to restart the right engine were
unsuccessful, and the airplane crashed approximately
one-half mile short of the airport.
Source: LessonsLearned.FAA.gov
13. @CaryMillsap
39 passengers died in the accident, and 8 others died
later due to their injuries. Of the other 79 occupants,
74 suffered serious injury.
The primary event in the accident sequence was the
flight crew's misidentification of the #2 engine as the
failed engine, and its subsequent shutdown.
Source: LessonsLearned.FAA.gov
13@CaryMillsap http://lessonslearned.faa.gov/ll_main.cfm?TabID=3&LLID=62&LLTypeID=2#null
14. @CaryMillsap
After the accident, [the commander] stated that he had judged the #2 engine to
be at fault from his knowledge of the aircraft air conditioning system. His
reasoning was that he thought the smoke and fumes were coming forward from
the passenger cabin; the air for the cabin came mostly from the #2 engine;
therefore the trouble lay in that engine.
Whilst this reasoning might have applied fairly well to other [Boeing 737-300]
aircraft he had flown, it was flawed in this case because some of the conditioning
air for the passenger cabin of the Boeing 737-400 comes from the #1 engine.
Source: LessonsLearned.FAA.gov
14@CaryMillsap
16. @CaryMillsap
Brief recap:
There are important facts about the
future of your project that you
cannot know right now.
Not understanding this
just makes it worse.
Now what?
16
23. @CaryMillsap 23h p://spo squee.blogspot.com/2007/03/today-on-hot-stove-ma -co on-schaub.html@CaryMillsap
If only it had burned
when they touched it…
24. @CaryMillsap 24
Imagine if it took 300+ days to
discover that you had designed a
problem into your system...
26. @CaryMillsap 26
Requirements Design Code Development
test
Acceptance
test
Operation
0
50
100
150
200
Phase in which error was detected and corrected
Relativecosttofixerror
Boehm, B. 1981. Software Engineering Economics. Englewood Cliffs NJ: Prentice-Hall
20×
...And it is expensive.
33. @CaryMillsap
When you hire, find people
who know how to
test, measure, learn, and prioritize.
Not people who claim to already
know everything.
33
Changing how you hire...
34. @CaryMillsap
Plan to test continually, and
leave time to redo parts of
your project in response to
what you learn from the tests.
34
Changing how you plan...
39. @CaryMillsap
I’ve chosen a rule that some sequences
of four numbers obey, and some do not.
Your job is to guess what the rule is.
1. You’ll suggest the next number in the following sequence.
2. I’ll tell you whether the resulting sequence follows the rule.
3. When you have enough information, say "I wish to guess."
4. And then guess the rule.
39http://www.nytimes.com/interactive/2015/07/03/upshot/a-quick-puzzle-to-test-your-problem-solving.html
2, 4, 6, __
48. @CaryMillsap
If your project is destined to fail,
you need to know now.
That doesn’t mean you want it to fail.
It means you want to know the truth.
48
49. @CaryMillsap
ou should be glad that
bridge collapsed. I was
planning to build a dozen
more to the same design.
—I. K. Brunel
49
Y
51. @CaryMillsap
Testing is where you try as hard as
you can to break your project.
Because if your project is destined to fail, when would you rather know?
Now? Later? In production?
51
52. @CaryMillsap
When really smart people
demonstrate that the only way
they can break your system is by
hitting it harder than anybody will
need to hit it in production, ...
That is when you have you
tested your system.
52
59. @CaryMillsap 59
response time = call count × call latency
how your code is written
how often people run it
dependson
your hardware
your call count
depends on
61. “
@CaryMillsap
The First Rule of Program Optimization:
Don’t do it.
The Second Rule of Program Optimization (for experts only!):
Don’t do it yet.
— Michael A. Jackson
61
63. @CaryMillsap
Keep your call counts small.
Embed this philosophy everywhere in your system.
business processes · architecture · user interface · data · code · everywhere
63
64. @CaryMillsap
Measure your call counts.
Embed this philosophy everywhere in your system.
business processes · architecture · user interface · data · code · everywhere
64
69. @CaryMillsap
Is your goal to have an awesome system?
Or to save face?
...Not deciding is also a decision.
69
The 90% political part
70. @CaryMillsap
Eliminate calls (code path).
Shorten feedback loops.
Create feedback continually with
adversarial tests.
Plan these tasks into your project.
70
The 10% technical part