2. 23/03/2016 Janet Network DDoS Experience
Tim Kidd
Executive director, Jisc technologies
Tim.Kidd@jisc.ac.uk
What happened in early December
3. To set the scene…
»I will say more than we have said publicly
»There is a police investigation ongoing
»Confidentiality
23/03/2016 Janet Network DDoS Experience
4. Timeline
» Tuesday 1 Dec 11:15 - 1 hour
Attack directed at NW institution then infrastructure
» Friday 4 Dec 13:58 - 40 minutes
Initial blocks in place at 14:35 with attack blocked
» Friday 4-Dec 15:54 - 20 minutes
Initial blocks at 16:02 but little impact, attack blocked at 16:16
» Monday 7 Dec 09:11 - 1 hour 10 minutes
Initial blocks at 09:47 but little impact, attack blocked at 10:18
» Monday 7 Dec 11:17 - 25 minutes
Attack blocked at 11:40
» Tuesday 8 Dec 09:10 - 3 hours 30 minutes
Blocked at 10:10 but further problems due to defensive blocks
23/03/2016 Janet Network DDoS Experience
Engineers prepared next level of blocks to install Monday morning
Jisc website hit 11:39 coincidence?
5. Communication
»Declared a major incident;
used web page andTwitter
@JiscMI
»In accordance with major
incident procedure, staff were
moved from normal duties to
bolster the Janet Service Desk
but still more calls than we
could handle
23/03/2016 Janet Network DDoS Experience
6. External border protection
»≈50 routers to configure
»Blocked IP fragments to
all infrastructure
»PolicedTCP, UDP and
ICMP to core
infrastructure
»Site access link
infrastructure
under way
23/03/2016 Janet Network DDoS Experience
7. Lessons Learned
»BBC DDoS attack on 31 December caused people to think Janet
was being attacked
»A malicious attack feels very different from other major
incidents
»Potential misuse of public updates viaTwitter – use SMS directly
to nominated people
»A more nuanced response (bronze, silver, gold) and difference
between Major Incident and High Impact Incident
»Accelerate the DDoS element of our security programme
»Secure the infrastructure address space
23/03/2016 Janet Network DDoS Experience
8. 23/03/2016 Janet Network DDoS Experience
Steve Kennett
Head of operational services
Steve.Kennett@jisc.ac.uk
Responding to a changing threat landscape
10. What’s changing in the threat landscape?
»Janet and customer infrastructure has now been directly targeted
»Attacks appear to be more reactive to countermeasures we deploy
»An effective attack now only requires a credit card
»The cost of launching an attack continues to drop
23/03/2016 Janet Network DDoS Experience
11. The challenge of dealing with large scale DDoS
» Requires coordinated action between customer and Janet operations:
› Impacts the weakest link between where attacks enter Janet and the
target system
› Depending on scale can disrupt customer, regional or even national
infrastructures
› Once customer access link capacity is overloaded you have limited
options
› Providing advice on likely duration and impact of event(s)
› Multiple internet connections do not necessarily help depending on
nature and sophistication of attack
› Asymmetry of costs between attackers and defenders
23/03/2016 Janet Network DDoS Experience
12. Impact of mitigation (I)
»We have to detect attacks in order to apply mitigation –
reactive function
»Traffic will have to be re-routed to apply mitigation
»Some traffic latency will be introduced
»Mitigation is not 100% effective – some ’attack’ traffic will
still get through
»Can create false positives – blocking genuine traffic
»Legitimate traffic flows look similar to large scale DDoS –
improved awareness and coordination required
23/03/2016 Janet Network DDoS Experience
13. Impact of mitigation (II)
»Greater automation is required to free up resources, control costs
and support response time
»Mitigation capacity is expensive to deliver and operate
»Organisations under persistent attack can be kept in mitigation -
but capacity is limited
»Arms race in capacity terms is likely
»System complexity
23/03/2016 Janet Network DDoS Experience
1/12: Affected global transit in the North. Started as standard DDoS then moved to infrastructure as we blocked the attack.
01-12-15 11:15 1 hour
Attack was directed through global transit links into Telecity Manchester. Blocks in place by 12:13 UTC with connectivity stabilising throughout network. This attack mainly affected NNW, CNL and YHR.
04-12-15 13:58 40 min & 04-12-15 15:54 20 min
Attack starts at 13:58 UTC on Friday 4th December. All sites using Northern Global Connectivity were affected.
First blocks implemented at 14:35 with connectivity restored.
Second attack starts via changed vector at 15:54 UTC.
Further blocks in place at 16:02 but have little effect.
Attack stops at 16:16 UTC
07-12-15 09:11 1 hr 10 min & 07-12-15 11:17 25 min
Attack starts at 09:11 UTC on Monday 7th December. All sites using Northern Global Connectivity were affected.
Blocks placed at 09:47 have little impact.
Further blocks placed at 10:18 UTC which stabilises connectivity.
08-12-15 09:10 3 hr 30 min
Attack starts at 09:10 UTC on Tuesday 8th December. Attack via global transit links into Telehouse North and Telecity Manchester. Whole network affected.
First blocks identified at 09:20
Attack stopped at 10:10 UTC but connectivity issues remained due to defensive blocks that we had put in place. These blocks were inspected and modified a number of times and finally restored connectivity at 12:40 UTC.
Web Submit advise JISC main website is also a target and is offline from 11:39 UTC.
08-12-15 13:47 10 min
Attack directed at West Notts College affecting the whole of the East Midlands Region at 13:47 UTC.
Blocks applied by 13:54 UTC. Connectivity stable.
Expect to secure infrastructure address space by the end of March. We will be writing to sites that appear to be using it very shortly and will help them to move off this.
As we heard in the introduction from Bob we are not having to deal with a changing threat landscape
If your being targeted then multiple connections does not necessary help – if an attacker knows you have two or three connections these can be targeted
If your organisation is near an attack targeting an organisation or regional infrastructure then multiple connection might help if your infrastructure is sufficiently resilient
Be clear that traffic needs to be re-routed
Data centre access and cloud services
Expect to be able to fund capital for this exercise from available funds – but are recurrent costs that will go into overall operating costs