How we found a firewall vendor bug using Teleport as a bastion jump host
Teleport is an SSH system which we’ve fallen in love with. There are some great security features, of course: - two factor authentication right out of the box - acts as ssh certificate authority issuing short-lived credentials - commercial options for role-based access control But the features which we find most compelling are the ones you can’t get as easily with the likes of OpenSSH: - session recording which can be used for audit or to refer back to from troubleshooting tickets - session sharing so that our customers or junior staff can learn-by-doing, just like having dual controls on a car - NAT-piercing to help manage devices within customer networks that do not have direct Internet connectivity We have been using Teleport on a number of projects and with several customers: - a remote probe deployment to debug a strange, intermittent connectivity problem (given as a talk at UKNOF 40 in conjunction with David Farrar of Exa Networks) - training sessions with customers’ technical staff to show them a slightly unusual systems administration request — and the resulting session recording is an excellent reference for next time their staff encounter a similar request for changes - paralleling pair programming we have been able to “observe” or “navigate” while junior staff “drive” the console to perform systems or network adminstration for the first time I’ve evangelised Teleport because I feel its use fits with our philosophy of openness. Teleport could complement the knowledge sharing that goes on within network operations teams, and help senior staff work out the playbooks and improve operational procedures for their junior staff. At least one service provider was inspired by my longer Teleport presentation at NetMcr and set their junior staff the background task of moving all out-of-band access to their POP infrastructure to Teleport. I hope that their use of this tool empowers their junior engineers to take on more work, while satisfying any regulatory or audit requirements that security staff worry about.
Recommended
Recommended
More Related Content
What's hot
What's hot (18)
Similar to How we found a firewall vendor bug using Teleport as a bastion jump host
Similar to How we found a firewall vendor bug using Teleport as a bastion jump host (20)
More from Faelix Ltd
More from Faelix Ltd (8)
Recently uploaded
Recently uploaded (20)
How we found a firewall vendor bug using Teleport as a bastion jump host
- 1. USING TELEPORT AS A BASTION JUMP HOST HOW WE FOUND A FIREWALL VENDOR BUG https://faelix.link/uknof40 — 2Mb
- 2. " THE INTERNET ISN'T WORKING! " Calls to Exa Networks' support team THE PROBLEM
- 3. IT GETS WORSE THE MORE YOU LOOK AT IT THE TASK AT HAND ▸ Isn't affecting all customers ▸ Initial suspicion of leased-line backhaul congestion… ▸ …but we found some DSL customers with problems too ▸ Intermittent, but not quite a heisenbug ▸ Manifested as timeout loading web pages ▸ All affected customers use Exa's new filtering service
- 4. WE NEED MORE DATA AIM TO TEST FROM CUSTOMERS' NETWORKS ▸ Put some probe devices in some customer networks ▸ …to be able to "ssh" into them, run measurements. ▸ Don't want customers to have to open ports on routers. ▸ Some sort of NAT-piercing required. ▸ Security is vital: ▸ Don't want probe to be an attack vector into customer. ▸ Team of staff need access.
- 5. IOT SECURITY (NETMCR #11) @kooky_uk Tim Bray SHOUT OUT #1
- 6. SSH CERTIFICATES (NETMCR #13) @TimJDFletcher Tim Fletcher SHOUT OUT #2
- 7. IOT SECURITY WITH PI.PE (NETMCR #17) @steely_glint Tim Panton SHOUT OUT #3
- 8. RIPE ATLAS PROBE SECURITY (AQL IOT ROUNDTABLE) @kistel Robert Kisteleki SHOUT OUT #4
- 9. STANDING ON THE SHOULDERS OF GIANTS RIPE ATLAS ▸ Plug it in, gets address/DNS by DHCP ▸ Connects to RIPE bastion hosts using ssh (with provisioning) ▸ Creates tunnels to itself for telemetry, read all about it: ▸ https://www.uknof.org.uk/uknof18/Kisteleki-Atlas.pdf ▸ Security rep is pretty good, e.g. ▸ https://www.mdsec.co.uk/2015/09/an-introduction-to- hardware-hacking-the-ripe-atlas-probe/
- 10. STANDING ON THE SHOULDERS OF GIANTS SSH BASTION HOSTS, WITH SSH CA ▸ The big players are doing it: ▸ https://code.facebook.com/posts/365787980419535/ scalable-and-secure-access-with-ssh/ ▸ https://github.com/Netflix/bless ▸ How to apply this pattern to our "IoT" probe project?
- 11. WHO NEEDS ANOTHER SSHD? WHY BOTHER USING TELEPORT? ▸ ssh CA out of the box, compatible with OpenSSHd ▸ 2FA out of the box (TOTP or U2F), no google_authenticator.pam ▸ ssh through-the-web out of the box ▸ Compliance Officer's dream: session recording jumphost. ▸ …and with "session_recording: proxy" it can do this for legacy sshd implementations too! [caveat: Security Officer] ▸ Free OSS < $aa$_startup_pricing_model < enterpri$$$e ▸ $paid_editions feature include RBAC, LDAP/SASL integration
- 12. IN A BREAK FROM TRADITION, OPS LOBS THE PROBLEM AT DEVS THE SOLUTION ▸ Ansible script #1: ▸ Deploys Teleport on a VM for bastion host ▸ Ansible script #2: ▸ Installs Teleport on a Raspberry Pi ▸ Preconfigures Teleport (as trusting cluster of bastion host) ▸ Bunch of Raspberry Pi / case / SD card combos ▸ Ship to customers with instructions about placement ▸ Exa's software team can now run diagnostics
- 14. you! proxy auth node client (tsh or https)
- 15. you! proxy auth node client (tsh or https) all-in-one
- 16. you! proxy auth node client (tsh or https) bastion node(s)
- 17. you! proxy auth node client (tsh or https) all separated
- 18. you! proxynode client (tsh or https) RBAC ("enterprise") auth LDAP SAML etc
- 19. you! proxies auth node client (tsh or https) HA auth etcd / dynamodb load bal
- 20. you! auth + proxy & trusted cluster auth node trusting cluster client (tsh or https) bastionanother node
- 21. you! auth + proxy & trusted cluster auth trusting cluster client (tsh or https) bastionnode
- 22. you! auth + proxy & trusted cluster auth trusting cluster client (tsh or https) bastionnode
- 23. tsh login --proxy teleport.example.com --user networkmoose ssh-key -A LOGGING IN
- 27. "…AND YOU WON'T BELIEVE WHAT HAPPENED NEXT…" e: marek@faelix.net t: @maznu https://faelix.link/uknof40