http://flink-forward.org/kb_sessions/flinkspector-taming-the-squirrel/ The costs of logic errors in production for streaming applications are higher than for batch processing systems. Depending on the setup, errors cannot be rectified or have already influenced important decisions. The goal of Flinkspector is to improve the test process of Apache Flink streaming applications in order to detect streaming application logic errors early during development. It features dedicated mechanics for test setup, execution, and evaluation. While Flinkspector’s streamlined API keeps testing overhead small. The framework is able to handle non-terminating and parallelized data flows involving windowing. The lightweight integration-tests enabled by Flinkspector allow Flink applications to be included into the continuous integration and deployment process. The talk introduces the core functionality of Flinkspector. In addition, background concepts of the runtime and the evaluation algorithms are presented. https://github.com/ottogroup/flink-spector

1. Flinkspector
Taming the Squirrel

2. Alexander Kolb
otto group BI
@lofifnc
Alexander Kolb
Otto Group BI
@lofifnc

3. Alexander Kolb - 2016 - otto group
Agenda
• Introduction
• Testing Apache Flink Applications
• Flinkspector
• Concept
• Input
• Expectations
• Execution
• Conclusion
• Outlook

4. Alexander Kolb - 2016 - otto group
Introduction

5. Alexander Kolb - 2016 - otto group
• Multichannel Retail
• Financial Services
• Services
• 30 countries
• 123 companies

6. Alexander Kolb - 2016 - otto group
Problem
• Transparency
• Traceability
• Reproducibility

7. Alexander Kolb - 2016 - otto group
Solutions
• Visualisation
• Datalineage graph / Message tracing
• Unit Testing

8. Alexander Kolb - 2016 - otto group
Testing Apache Flink
Applications

9. Alexander Kolb - 2016 - otto group
Apache Flink Application
final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<String> text = env.socketTextStream("localhost", 9999);
DataStream<Tuple2<String, Integer>> words =
text.flatMap(new WordCount.LineSplitter());
DataStream<Tuple2<String, Integer>> aWords =
words.filter(new StartsWithAFilter());
DataStream<Tuple2<String, Integer>> counts =
aWords.keyBy(0).timeWindow(Time.of(2, TimeUnit.MINUTES)).sum(1);
counts.print();
env.execute("Wordcount Example");

10. Alexander Kolb - 2016 - otto group
User Defined Functions
public final class StartsWithAFilter
implements FilterFunction<Tuple2<String,Integer>> {
@Override
public boolean filter(Tuple2<String, Integer> t)
throws Exception {
return t.f0.startsWith("a");
}
}

11. Alexander Kolb - 2016 - otto group
User Defined Functions
@Test
public void testAFilter() throws Exception {
StartsWithAFilter filter = new StartsWithAFilter();
Assert.assertEquals(true,
filter.filter(Tuple2.of("aTest",1)));
Assert.assertEquals(false,
filter.filter(Tuple2.of("bTest",1)));
}

12. Alexander Kolb - 2016 - otto group
Stream Transformations
public static DataStream<Tuple2<String,Integer>>
countAWords(DataStream<Tuple2<String,Integer>> aWords) {
return aWords
.keyBy(0)
.timeWindow(Time.of(2, TimeUnit.MINUTES))
.sum(1);
}

13. Alexander Kolb - 2016 - otto group
Stream Transformations
@Test
public void aWordCountTest() {
StreamExecutionEnvironment env =
StreamExecutionEnvironment.createLocalEnvironment();
env.setParallelism(1);
DataStream<Tuple2<String, Integer>> words =
env.fromCollection(Arrays.asList(
Tuple2.of("a1", 1),
Tuple2.of("a2", 1),
Tuple2.of("a1", 1)
));
DataStream<Tuple2<String, Integer>> results =
SocketTextStreamWordCount.countAWords(words);
Iterator<Tuple2<String, Integer>> output =
DataStreamUtils.collect(results);
Assert.assertEquals(output.next(), Tuple2.of("a1", 1));
Assert.assertEquals(output.next(), Tuple2.of("a2", 2));
Assert.assertEquals(output.hasNext(), false);
}

14. Alexander Kolb - 2016 - otto group
Flinkspector

15. Alexander Kolb - 2016 - otto group
Concept
1. Specify input.
2. Aquire data stream from the input specification.
3. Define expectations for the resulting data stream.
4. Apply the expectations to the produced data stream.

16. Alexander Kolb - 2016 - otto group
TestStreamEnvironment
Runtime
Subscriber
Test Source Input
Filter
KeyedWindowReducer
Concept
Verifier
Test Sink Publisher
Trigger
(default)
paired

17. Alexander Kolb - 2016 - otto group
Expectations

18. Alexander Kolb - 2016 - otto group
Input
EventTimeInput<Tuple2<String,Integer>> input =
EventTimeInputBuilder.startWith(Tuple2.of("a1", 1))
.emit(Tuple2.of("a2", 1), after(1, minutes))
.emit(Tuple2.of("a3", 1), before(1, seconds), times(2))
.emit(Tuple3.of(“a4”, 1), intoWindow(4, seconds))
.repeatAll(after(1, minutes), times(3));

19. Alexander Kolb - 2016 - otto group
List Based
ExpectedRecords.create(asList(1,2,3,4));
ExpectedRecords.create(asList(1,2,3,4))
.refine().only();
ExpectedRecords.create(asList(1,2,3,4))
.refine().sameFrequency();
(4,1,2,3,3,5)
(4,1,2,3,3,5)
(4,1,2,3,3,5)

20. Alexander Kolb - 2016 - otto group
List Based
(4,1,2,3,3,5)
(4,1,2,3,3,5)
(4,1,2,3,3,5)
ExpectedRecords.create(asList(1,2,3,4));
ExpectedRecords.create(asList(1,2,3,4))
.refine().only();
ExpectedRecords.create(asList(1,2,3,4))
.refine().sameFrequency();

21. Alexander Kolb - 2016 - otto group
List Based
(4,1,2,3,3,5)
(4,1,2,3,3,5)
(4,1,2,3,3,5)
ExpectedRecords.create(asList(1,2,3,4));
ExpectedRecords.create(asList(1,2,3,4))
.refine().only();
ExpectedRecords.create(asList(1,2,3,4))
.refine().sameFrequency();

22. Alexander Kolb - 2016 - otto group
List Based
(4,1,2,3,3,5)
(4,1,2,3,3)
(4,1,2,3,3,5)
ExpectedRecords.create(asList(1,2,3,4));
ExpectedRecords.create(asList(1,2,3,4))
.refine().only();
ExpectedRecords.create(asList(1,2,3,4))
.refine().sameFrequency();

23. Alexander Kolb - 2016 - otto group
List Based
(4,1,2,3,3,5)
(4,1,2,3,3)
(4,1,2,3,3,5)
ExpectedRecords.create(asList(1,2,3,4));
ExpectedRecords.create(asList(1,2,3,4))
.refine().only();
ExpectedRecords.create(asList(1,2,3,4))
.refine().sameFrequency();

24. Alexander Kolb - 2016 - otto group
List Based
(4,1,2,3,3,5)
(4,1,2,3,3)
(4,1,2,3,5)
ExpectedRecords.create(asList(1,2,3,4));
ExpectedRecords.create(asList(1,2,3,4))
.refine().only();
ExpectedRecords.create(asList(1,2,3,4))
.refine().sameFrequency();

25. Alexander Kolb - 2016 - otto group
List Based
ExpectedRecords.create(asList(1,2,3,4))
.refine().inOrder(notStrict).all();
ExpectedRecords.create(asList(1,2,3,4))
.refine().inOrder(strict).from(1);
List(4,1,2,3,3,5)
List(5,2,3,4,1)

26. Alexander Kolb - 2016 - otto group
List Based
List(4,1,2,3,3,5)
List(5,2,3,4,1)
ExpectedRecords.create(asList(1,2,3,4))
.refine().inOrder(notStrict).all();
ExpectedRecords.create(asList(1,2,3,4))
.refine().inOrder(strict).from(1);

27. Alexander Kolb - 2016 - otto group
List Based
List(1,2,3,3,5,4)
List(5,2,3,4,1)
ExpectedRecords.create(asList(1,2,3,4))
.refine().inOrder(notStrict).all();
ExpectedRecords.create(asList(1,2,3,4))
.refine().inOrder(strict).from(1);

28. Alexander Kolb - 2016 - otto group
List Based
List(1,2,3,3,5,4)
List(5,2,3,4,1)
ExpectedRecords.create(asList(1,2,3,4))
.refine().inOrder(notStrict).all();
ExpectedRecords.create(asList(1,2,3,4))
.refine().inOrder(strict).from(1);

29. Alexander Kolb - 2016 - otto group
List Based
List(1,2,3,4,1)
MatcherBuilder
only
sameFrequency
order
ExpectedRecords.create(asList(1,2,3,4))
.refine()
.only()
.sameFrequency()
.inOrder(strict).from(1);

30. Alexander Kolb - 2016 - otto group
List Based
ExpectedRecords.create(asList(1,2,3,4))
.refine()
.only()
.sameFrequency()
.inOrder(strict).from(1);
List(1,2,3,4,1)
MatcherBuilder
only
sameFrequency
order

31. Alexander Kolb - 2016 - otto group
List Based
List(1,2,3,4,1)
MatcherBuilder
only
sameFrequency
order
ExpectedRecords.create(asList(1,2,3,4))
.refine()
.only()
.sameFrequency()
.inOrder(strict).from(1);

32. Alexander Kolb - 2016 - otto group
List Based
List(1,2,3,4,1)
MatcherBuilder
only
sameFrequency
order
ExpectedRecords.create(asList(1,2,3,4))
.refine()
.only()
.sameFrequency()
.inOrder(strict).from(1);

33. Alexander Kolb - 2016 - otto group
List Based
List(2,3,4,1)
MatcherBuilder
only
sameFrequency
order
ExpectedRecords.create(asList(1,2,3,4))
.refine()
.only()
.sameFrequency()
.inOrder(strict).from(1);

34. Alexander Kolb - 2016 - otto group
List Based
List(2,3,4,1)
MatcherBuilder
only
sameFrequency
order
ExpectedRecords.create(asList(1,2,3,4))
.refine()
.only()
.sameFrequency()
.inOrder(strict).from(1);

35. Alexander Kolb - 2016 - otto group
Assertion Based
new MatchTuples<Tuple2<String,Integer>>("word","count")
.assertThat("word", startsWith("a"))
.assertThat("count", greaterThan(3))
.onEachRecord();

36. Alexander Kolb - 2016 - otto group
Assertion Based
new MatchTuples<Tuple2<String,Integer>>("word","count")
.assertThat("word", startsWith("a"))
.assertThat("count", greaterThan(3))
.oneOfThem()
.onEachRecord();

37. Alexander Kolb - 2016 - otto group
Assertion Based
.assertThat(A,…)
.assertThat(B,…)
.assertThat(C,…)
.eachOfThem()
.onAnyRecord()
A B C

38. Alexander Kolb - 2016 - otto group
Assertion Based
A B C
.assertThat(A,…)
.assertThat(B,…)
.assertThat(C,…)
.eachOfThem()
.onAnyRecord()

39. Alexander Kolb - 2016 - otto group
Assertion Based
.assertThat(A,…)
.assertThat(B,…)
.eachOfThem()
.onAnyRecord()
A B C

40. Alexander Kolb - 2016 - otto group
Assertion Based
.assertThat(A,…)
.assertThat(B,…)
.assertThat(C,…)
.eachOfThem()
.onAnyRecord()
A B C

41. Alexander Kolb - 2016 - otto group
Assertion Based
.assertThat(A,…)
.assertThat(A,…)
.assertThat(A,…)
.atLeastNOfThem(2)
.onEach()
A B C

42. Alexander Kolb - 2016 - otto group
Assertion Based
.assertThat(A,…)
.assertThat(B,…)
.assertThat(C,…)
.atExactlyNOfThem(2)
.onEach()
A B C

43. Alexander Kolb - 2016 - otto group
Assertion Based
public class Each<T> extends UntilCombineMatcher<T> {
public Each(Iterable<Matcher<? super T>> matchers) {
super(matchers);
}
@Override
public String prefix() {
return "each of";
}
@Override
public boolean validWhen(int matches, int possibleMatches) {
return matches == possibleMatches;
}
@Factory
public static <T> Each<T> each(Iterable<Matcher<? super T>> matchers) {
return new Each<T>(matchers);
}
}

44. Alexander Kolb - 2016 - otto group
TestStreamEnvironment
Runtime
Subscriber
Test Source Input
Filter
KeyedWindowReducer
Execution
Verifier
Test Sink Publisher
Trigger
(default)
paired

45. Alexander Kolb - 2016 - otto group
Execution
Test SourceInput
Filter
KeyedWindowReducer
Test Sink Publisher
Test Source Input
Filter
KeyedWindowReducer
Test Sink Publisher
1
1
1
1
2
2
2
2
LocalCluster
TestStreamEnvironment
Runtime
Subscriber
Verifier Trigger
(default)

46. Alexander Kolb - 2016 - otto group
Execution
LocalCluster
TS I1
F
KWR
TSi P1
1
1
TS I2
2
F
2
KWR
1
1
2
TSi P2
2
TestStreamEnvironment
Runtime
Subscriber
Verifier Trigger
(default)
Timeout

47. Alexander Kolb - 2016 - otto group
Execution
TestStreamEnvironment
Runtime
Subscriber
Verifier Trigger
(default)
LocalCluster
TS I1
F
KWR
TSi P1
1
1
TS I2
2
F
2
KWR
1
1
2
TSi P2
2
Timeout

48. Alexander Kolb - 2016 - otto group
Execution
TestStreamEnvironment
Runtime
Subscriber
Verifier Trigger
(default)
LocalCluster
TS I1
F
KWR
TSi P1
1
1
TS I2
2
F
2
KWR
1
1
2
TSi P2
2
Timeout

49. Alexander Kolb - 2016 - otto group
LocalCluster
TSI1
F
KWR
TSiP1
1
1
TSI2
2
F
2
KWR
1
1
2
TSiP2
2
Execution
TestStreamEnvironment
Runtime
Subscriber
Verifier Trigger
(default)
Timeout
CLOSE 1
CLOSE 2

50. Alexander Kolb - 2016 - otto group
TSI1
F
KWR
TSiP1
1
1
TSI2
2
F
2
KWR
1
1
2
TSiP2
2
Execution
TestStreamEnvironment
Runtime
Subscriber
Verifier Trigger
(default)
LocalCluster

51. Alexander Kolb - 2016 - otto group
Execution
TestStreamEnvironment
Runtime
Subscriber
Verifier Trigger
LocalCluster
TS I1
F
KWR
TSi P1
1
1
TS I2
2
F
2
KWR
1
1
2
TSi P2
2
TimeoutTimeout

52. Alexander Kolb - 2016 - otto group
Execution
TestStreamEnvironment
Runtime
Subscriber
Verifier Trigger
LocalCluster
TS I1
F
KWR
TSi P1
1
1
TS I2
2
F
2
KWR
1
1
2
TSi P2
2

53. Alexander Kolb - 2016 - otto group
Execution
TestStreamEnvironment
Runtime
Subscriber
Verifier Trigger
(default)
LocalCluster
TS I1
F
KWR
TSi P1
1
1
TS I2
2
F
2
KWR
1
1
2
TSi P2
2
TimeoutTimeout

54. Alexander Kolb - 2016 - otto group
Execution
TestStreamEnvironment
Runtime
Subscriber
Verifier Trigger
(default)
LocalCluster
TS I1
F
KWR
TSi P1
1
1
TS I2
2
F
2
KWR
1
1
2
TSi P2
2
TimeoutTimeout

55. Alexander Kolb - 2016 - otto group
Conclusion

56. Alexander Kolb - 2016 - otto group
Conclusion
• Small and concise test cases.
• Extensible dedicated runtime.
• Integration into existing test
platforms.
• ProcessingTimeWindows.
• Input specification too
expansive.

57. Alexander Kolb - 2016 - otto group
Outlook

58. Alexander Kolb - 2016 - otto group
Outlook
• Scala (Test) support.
• Symbolic testing integration.
• System tests on the actual cluster.

59. Alexander Kolb - 2016 - otto group
https://github.com/ottogroup/flink-spector

60. Alexander Kolb - 2016 - otto group
ottogroup.com
WE ARE
HIRING!