It is common to have jobs running periodically, especially in asynchronous and distributed systems. If the service is scaled horizontally (i.e. there are multiple instances of the same service), you often only want a single node to handle the task. In this session I will demonstrate how to manually setup Spring to have custom logic in scheduling configuration and perform recurring tasks only on a single node. This requires keeping notation of the leader and persisting the selection. The key takeaway of this session is how to implement distributed locking and how simple it is to run Spring application on top of it. In this talk you will learn how to mitigate challenges that arise when you use traditional declarative approach for scheduling and how to switch to a more flexible programmatic approach.

1. Building resilient scheduling in
distributed systems with Spring
Marek Jeszka
@logic_marc

2. Agenda
一 Background
一 Approach
一 Results
一 Conclusion

3. Use-case
一 asynchronous communication
一 reliable processing
一 better visibility

5. @Component
public class SimpleService {
@Scheduled(cron = "0 * * * * *")
public void runQuiteOften() {
// process events
}
}

6. Distributed
Systems
一 vertical scaling
• actually it doesn’t scale…
一 horizontal scaling
• cost-efficiency
• higher reliability
• easier to expand

7. Running on a single node
一 How to select the node?
一 Where to keep information
about the selected node?

8. Instance 1 Instance 2 Instance 3

9. Instance 1 Instance 2 Instance 3
Am I a leader?

10. Leader election
in Spring-based
application

11. @Component
public class SimpleService {
@Scheduled(cron = "0 * * * * *")
public void runQuiteOften() {
// process events
}
@RunIfLeader

12. @Retention(RetentionPolicy.RUNTIME)
@Target({ ElementType.METHOD })
public @interface RunIfLeader {
}

13. Aspect
Oriented
Programming
with Spring
一 Aspect - crosscutting concern
• Logging
• Transaction management
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
一 Enabled with dependency:

14. Types of
advices
一 After
一 Around
一 Before

15. @Aspect
public class RunIfLeaderAspect {
@Around("@annotation(com.n26.RunIfLeader) && execution(void *(..))")
public void annotatedMethod(ProceedingJoinPoint joinPoint)
throws Throwable {
if (isLeader()) {
joinPoint.proceed();
}
// do not execute
}

16. Why we didn’t
like it?
一 No clear separation between
business and scheduling logic
一 Hard to test
一 Scheduled jobs spread across
the application

17. Issue with the
@SqsListener
@SqsListener(value = "eventsQueue",
deletionPolicy = ON_SUCCESS)
@RunIfLeader
void onEvent(String eventAsJson) {
// process event
}

18. Selecting a leader in
programmatic
approach

19. Programmatic
approach
一 SchedulingConfigurer from
org.springframework.scheduling.annotation
public interface SchedulingConfigurer {
void configureTasks(
ScheduledTaskRegistrar taskRegistrar);
}

20. @Configuration
@EnableScheduling
public class SchedulingConfig implements SchedulingConfigurer {
@Override
public void configureTasks(ScheduledTaskRegistrar taskRegistrar) {
taskRegistrar.addCronTask(
new CronTask(() -> {
// process events
}, "0 * * * * *"));
}

21. @Autowired
private Runnable processEventsTask;
@Override
public void configureTasks(ScheduledTaskRegistrar taskRegistrar) {
taskRegistrar.addCronTask(
new CronTask(processEventsTask, "0 * * * * *"));
}
@Component
public class ProcessEventsTask implements Runnable {
@Override
public void run() {
// process events
}
}

22. What are the
benefits of
programmatic
approach?
一 Tasks are scheduled in one place
一 Custom executor service

23. @Configuration
@EnableScheduling
public class SchedulingConfig implements SchedulingConfigurer {
@Override
public void configureTasks(ScheduledTaskRegistrar taskRegistrar) {
taskRegistrar.setScheduler(taskScheduler());
}
@Bean(destroyMethod = "shutdown")
public ExecutorService taskScheduler() {
return Executors.newScheduledThreadPool(
4, // pool size
new ThreadFactoryBuilder()
.setNameFormat("scheduler-thread-%d").build());
}
}

24. What are the
benefits of
programmatic
approach?
一 Tasks are scheduled in one place
一 Custom executor service
一 Convenient testing

25. @RunWith(MockitoJUnitRunner.class)
public class SchedulingConfigTest {
@InjectMocks
private SchedulingConfig underTest;
@Mock
private ScheduledTaskRegistrar taskRegistrarMock;
@Mock
private ProcessEventsTask processEventsTaskMock;
@Test
public void schedulesCronTask() {
underTest.configureTasks(taskRegistrarMock);
verify(taskRegistrarMock)
.addCronTask(processEventsTaskMock, "0 * * * * *");
}

26. @RunWith(MockitoJUnitRunner.class)
public class SchedulingConfigTest {
@InjectMocks
private SchedulingConfig underTest;
@Mock
private ScheduledTaskRegistrar taskRegistrarMock;
@Test
public void usesScheduledThreadPoolExecutor() {
ArgumentCaptor<ScheduledThreadPoolExecutor> captor =
forClass(ScheduledThreadPoolExecutor.class);
underTest.configureTasks(taskRegistrarMock);
verify(taskRegistrarMock).setScheduler(captor.capture());
assertThat(captor.getValue().getCorePoolSize()).isEqualTo(4);
}

27. @Configuration
@EnableScheduling
public class SchedulingConfig implements SchedulingConfigurer {
@Override
public void configureTasks(ScheduledTaskRegistrar taskRegistrar) {
taskRegistrar.addCronTask(
new CronTask(() -> {
if (isLeader()) {
// process events
}
}, "0 * * * * *"));
}

28. @Configuration
@EnableScheduling
public class SchedulingConfig implements SchedulingConfigurer {
@Autowired
private Runnable processEventsTask;
@Override
public void configureTasks(ScheduledTaskRegistrar taskRegistrar) {
Runnable leaderAwareTask =
new LeaderAwareTaskDecorator(processEventsTask);
taskRegistrar.addCronTask(
new CronTask(leaderAwareTask, "0 * * * * *"));
}

29. public final class LeaderAwareTaskDecorator implements Runnable {
private Runnable delegate;
public LeaderAwareTaskDecorator(Runnable delegate) {
this.delegate = delegate;
}
@Override
public void run() {
if (isLeader()) {
delegate.run();
}
}

31. Resiliency
一 What if the response didn’t come?
一 Can we safely repeat?
• Duplicate entries created
一 Is the action idempotent?
• One or multiple identical
requests give the same result

32. Improvements
一 Distribute the jobs
SELECT * FROM events FOR UPDATE SKIP LOCKED;

33. Further
improvements
SQS queue
Instance 1 Instance 2 Instance 3

34. What have we
learned?
一 Annotation-driven development is hard
一 Keep (code) consistency
一 Increase resilience & predictability
一 Think about observability

35. Thank you Questions?

36. References
一 AOP:
https://docs.spring.io/spring/docs/2.5.x
/reference/aop.html
一 SchedulingConfigurer:
https://docs.spring.io/spring-
framework/docs/current/javadoc-
api/org/springframework/scheduling/a
nnotation/SchedulingConfigurer.html
一 Postgresql select:
https://www.postgresql.org/docs/9.5/s
ql-select.html