Is observability good for your brain?
•Download as PPTX, PDF•
1 like•753 views
This talk was given during Monitorama EU 2018. Observability, like other ops practices, has hard and soft benefits. No logs - no root cause, that’s a hard benefit. A soft benefit is when we have more confidence in an observable system. Then we can be more productive in developing it. The trouble with soft benefits like confidence, is how to measure them. Does observability actually make us more productive? How about other activities, such as post-mortems? Why is alert fatigue so bad? Turns out, there are plenty of studies about the impact of such activities on our brain, our behavior, our productivity. In this session, we’ll explore what [neuro]science says about such practices so that: We turn soft benefits into hard benefits We can encourage a culture where we get the benefits and avoid the traps Be prepared for surprises, as some “best practices” aren’t “best” at all.
Recommended
More Related Content
What's hot
What's hot (10)
Similar to Is observability good for your brain?
Similar to Is observability good for your brain? (20)
More from Sematext Group, Inc.
More from Sematext Group, Inc. (20)
Recently uploaded
Recently uploaded (20)
Is observability good for your brain?
- 1. Is observability good for your brain? How about post - mortems? Radu Gheorghe & Rafał Kuć Sematext Group, Inc.
- 3. Agenda Interacting with tools monitoring, alerts... Learning new tools methods for IT Interacting with people meetings, feedback... Daily structure when are we the most productive?
- 4. Tools: Monitor all the things!
- 5. Alert all the things!
- 6. Use the best tool for the job. Automate all the things!
- 7. Ahhh, one more thing ;)
- 8. Some history
- 9. The brain 1 0 1 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 0 1 0 0 1 1 1 1 0 0 0 0 0 0 1 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 0 1 0 0 1 1 1 1 0 0 0 0 0 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 0 1 0 0 1 1 1 1 0 0 0 0 0 1 0 1 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 0 1 0 0 1 1 1 1 0 0 0 0 0 0 1 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 0 1 0 0 1 1 1 1 0 0 0 0 0 1 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 0 1 0 0 1 1 1 1 0 0 0 0 0 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 0 1 0 0 1 1 1 1 0 0 0 0 0 1 0 1 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 1 0 1 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 0 1 1 0 1 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 0 1 0 0 1 0 1 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 0 1 0 1 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 0 1 1 0 1 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 0 1 0 1 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 1 0 1 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 0 1 0
- 10. Learning methods suitable for IT Curious child Apprentice Crash test Filling card Dancer Place switch Parrot Mastering Box king Cetacan Memory palace Monitorama EU login: _ TutorImmersion
- 13. Dancer Method Theory -> Practice Repeat N Times
- 15. Parrot Method
- 16. Mastering Method
- 17. Box King Method
- 19. People: Meetings
- 21. Sprint planning and review
- 22. Feedback
- 23. Micromanagement
- 24. Using the F word
- 25. Structure Eat. Sleep. $INTERESTING_THINGS. Repeat.
- 26. Morning Emails
- 27. Plan all the things I adore spontaneity. Providing it is carefully planned.
- 28. Thank You! Radu Gheorghe radu.gheorghe@sematext.com @radu0gheorghe Rafał Kuć rafal.kuc@sematext.com @kucrafal Containers, Monitoring, Logging? Talk to us at the Booth! We are hiring! https://sematext.com/jobs/
Editor's Notes
- Enables certainty and autonomy. We like patterns. Certainty: I know the site crashes every day at 7PM, when users come online. This is much better than the site crashing randomly, because that would be more stressful. Autonomy: I can spin up an instance at 6AM, fiddle with the config, etc. This lowers the stress, like with rats.
- It takes a long time for the brain to achieve peak performance on a difficult task. Up to 30 minutes. Easily distracted, though. Which makes sense, 10K years ago, if you’d focus to make a fire and a tiger shows up, you’d better be easily distracted. Inhibiting distractions (e.g. it’s just Rafal in his orange t-shirt), takes a lot of energy from the brain. And we need that energy. Is the alert actionable? Is it tiger-actionable?
- Learning tools and automation is not only about time, but also energy. Prefrontal cortex (thinking, decision-making) is slow and eats a lot of sugar. When I say slow, I mean: 10-20KB/s memory throughput 3-4 elements in lower caches Compare two things at once Once we know stuff, we involve other parts of the brain, it’s much easier. Resistance to learning may come from anxiety (we detect a status threat - am I stupid for not knowing Perl?). Can be a major blocker in learning.
- One of us should say that learning is a crucial part of IT work. What is important is that we learn every day - we learn new tools, we learn new technologies, how we interact with them. The crucial part is that learning should be fun and as easy as it can be, because otherwise the exhaustion will come sooner or later especially nowadays where new tools are coming up every day.
- The history of school is not as bright as we may think. In Austria in the 18th century teenagers and young man were causing trouble. Because of that the authorities came up with an idea to force young people to learn. They were put in schools for the major part of the day and given so much material that they didn't have "stupid" ideas. So it was not about expanding knowledge, but about keeping youngsters under control. Because of that the teaching system was not developed well and you would be surprised how many schools actually don't teach people how to learn new knowledge in an efficient way. We could ask how many of the audience actually tried to learn geography or history repeating the material over and over again. After that question we may ask how long they actually remembered details about what they were learning.
- We need to say that brain encodes the information. We have two types of memory - the one that is responsible for short term storage and one that is responsible for keeping the "data" for long. We can easily access the short term storage memory and we accept all information there. The information that seems most important is sorted by our brain during sleep and we can access but it requires more effort. It is crucial to connect information with emotions, repetitions of common stuff, so that pieces of information in the long term storage can be easily connected with everyday life and thus easier accessible.
- We should say that some of the methods are really suitable for IT and those are...
- One of the common methods that is used to bring programmers closer together when it comes to their knowledge, i.e. during pair programming. One person from the pair is more experienced, while the other is less experienced. That way the less experienced programmer can gain knowledge and experience quicker. Requirements: * friendly team * team members willing to share knowledge and not worrying about losing control The good sides of that approach are: * knowledge sharing * no single point of failure when it comes to knowledge about the system * quicker introduction of new people to the organization * faster problem solving by having two engineers working on the same problem looking into it from different angles * responsibility sharing The not so good sides of that approach are: * need of use double the time in some cases * responsibility sharing
- The crash test method is one of the best approaches to learn by example, especially if you want to gain knowledge about a certain technology - just go deeper. One of the examples of usage of this method are performance tests. They require certain, at least basic, knowledge about the system and when they are performed we learn about strong and weak points of the system, its limits, how it scales and so on. Of course a single performance test won't give us all the answers and we need to do a dozen of them to get some useful insights, etc. The good side of that approach: * learning by example allows to get more knowledge * person can learn about strong and weak points of the technology in a controlled environment, without being stressed and learning "the hard way" - in production when things fail, under time pressure The not so good sides of that approach: * requires time and additional effort * requires multiple repetitions to get useful insight * requires basic knowledge about the system to get useful conclusions - without the basic knowledge the conclusions may be misleading * the more complicated the test will be the more knowledge is required
- As they say "In theory, theory and practice is the same, in practice they are not". This is what the researchers say about our brain and its ability to encode information. Most of us can't read a book from top to bottom and start using all the knowledge from that book in practice. That is not doable in majority of cases. What works though is repeating smaller cycles of learning and practicing, just like dancing. You read or you are shown a move and then you repeat that until you know it. Then the next move and next and so on. Once you know the basic moves you combine them into a dance and you practice. That way you improve. The good things about the method: * allows for very good encoding as theoretical knowledge is encoded while using physical moves (even clicking) * taking smaller steps helps noticing what is not clear and allows to fully dig into the topic The not so good things about the method: * requires time * requires patience
- We all like to stay inside our comfort zone - it is nice there, we feel good about it, almost nothing can surprise us, etc. However such situation is the very opposite to good when it comes to learning new stuff. Because of that the place switch method suggest that people should be switched between projects, take different responsibilities - all of course within or close to their skills - you can't put devops person as a Java architect, unless you really want the project to fail drastically. The good things about this method: * encourages people to learn new things * stops people from sticking in a single position helping them develop their own skills * avoids having single points of failure inside the company as multiple people will eventually have similar knowledge about certain parts which makes the team being less stressed * works well when combined with tutor method, i.e. pair programming or pair problem solving The not so good things about this method: * some people get angry when being forced to get out of their comfort zone, which can introduce conflicts in the team * after the switch of places the overall productivity of the team will drop for a while until the knowledge is gathered
- Method not exactly for developers, but in general for people interested in sharing their knowledge by giving talks. The idea is that when you need to give a speech, demo, talk or anything similar one of the ways to prepare is rehearsal. However, the rehearsal should be done in a certain way. By introducing emotions and trying to visualise the place where you'll be giving the talk, presentation or product demo will help you connect the emotions to the thing that you want to present. By doing that your mind will connect the emotions with the things you want to say or demo and help you not fall into pitfalls when doing the actual thing in front of the audience. The good things about the method: * allows to prepare upfront * reduces stress of the person that is doing the rehearsals * allows to prepare really well for the upcoming task, i.e. Steve Jobs was doing hundreds of rehearsals on the actual stage before giving the talk during Apple keynote The not so good things about the method: * requires time and preparation upfront * not suitable for people doing everything at the last possible time
- Use pen & paper. Make notes of everything that you think is important - you have a new command that will be useful, note that down. You see a programming language structure that you think will be of use - note that down. You see a one-liner that you think you will reuse - note that down. If you think something is or may be important, just make a note and keep those notes. Having such notes will help you get back to the topic, command, etc. and quickly remind yourself about the topic and help memorize that and encode into the long term memory. The good things about that method: * additional resources needed on the desk ;) * degree of being self-organized is needed * analyze while you learn/read/practice is the key The not so good things about that method: * may become copy+paste method if not used correctly
- Don't try learning everything about a single topic in one step, don't go too deep. For a novice person, one trying to learn how to use linux, it doesn't make sense to learn everything about kernel functions, they only need to know how to configure their desktop, setup security properly and so on. Take it easy on yourself and your brain, try to switch from topic to topic without going too deep. The depth of knowledge will come gradually when you will get more and more experience with a given technology, infrastructure piece and so on. Of course that doesn't mean that you can omit things - after all there are always methods like apprentice method where you can learn from a more experienced colleague. The good things about that method: * encourages taking small steps * small steps help memorize knowledge better * very good when combined with dancer method or apprentice * helps avoiding stressful situations with too much knowledge to be learned at the same time The not so good things about that method: * very bad for people who are easily distracted * not good when you have a lot on your mind and you do a lot of context switching (distraction)
- One of those methods that is very useful for some of us. When you are learning new things try explaining the thing that you just learned like you would have to teach someone. That way you repeat things that you've learned, you ask additional questions and of course you answer. You see what things are super clear and which are not and that makes you dig more and try to understand the topic better. This can also work when you do something in a team - if someone asks a question you may want to answer it and explain, which will develop more consistent knowledge inside your brain. The good things about that method: * very well suited for people who want to learn all about a given topic, because it encourages you to ask and dig into the topic * encourages asking even the hard questions * encodes information very well helping in long distance learning The not so good things about that method: * can be abused by those who like to say that they know everything * if someone wants to just slide through the topic that method won't stop one from doing that
- Context-switching reduces brain activity. If the meeting also isn’t interesting, is soon after lunch, we may fall asleep :) If the meeting creates a status threat (e.g. someone tells me what to do => I’m not feeling that significant), I may get defensive (either being too passive, or somewhat rebellious). If the meeting is about meeting new people, it’s probably more useful. First impression matters (we tend to quickly classify someone new as friend/foe, and if we don’t get positive stuff, we default to foe). And we’re more likely to see more negative stuff if we can’t see facial expressions. If it’s about solving problems, usually collaboration makes learning easier (it’s a reason why pair programming is efficient). Language structures thoughts and we can use visuals to make processing even faster. One class of problem solving is planning, so let’s talk more about that.
- We like patterns, so if we’re not planning something upfront, the brain will constantly try to mold this frame of reference, taking resources in the background. Effectively making it hard to focus. Planning itself is resource-intensive, that’s why it’s often better to do it in the morning, when our brains are still fresh. It’s also nice to do some planning at the end of the day, to tie loose ends, but that serves a different purpose: we tend not to like unfinished stuff.
- Let’s talk about estimations. It’s best to make them conservative. If we don’t meet what we plan for, we get an away response from the brain (the fight-flight mechanism trying to stay away from danger => procrastinate, hard to focus). We also have a tendency to overfit for the plan, so basically we’ll rush, make mistakes, cut out stuff that may be important, etc. If we meet expectations, we get a bit of a towards response (some dopamine, basically making us a bit “addicted” to accomplishing tasks. Not bad, ha?). If we exceed expectations, we get a strong towards response. Dividing into small bits helps get a sense of progress (more stuff may be met). Don’t assume you can multitask thinking. I can make an omelet and fries at the same time, but I can’t fix a bug and decide where to have my vacation at the same time.
- Sprint review is a form of feedback, so let’s talk about that, and hopefully cover things like post mortems, pull request comments, yearly performance reviews and whatnot. Our brain constantly looks for what’s wrong (the tiger). And wants to control things. So it comes naturally to say: you did this wrong, do it my way, and wrap it into “constructive feedback”. Which often fails because it creates a status threat. We’re really good at picking up when our status feels challenged and become defensive. It’s why we say that stage fright is greater than fear of death. This anxiety activates the fight-flight mechanism and inhibits the CPU. The simple presence of a person that’s perceived as having a higher status triggers this system in most of us. That’s why micromanagement is bad.
- Instead of appreciating success/failure, it’s often better to appreciate effort. When we appreciate our own effort we get dopamine, so we’re more likely to raise to the next challenge, because we’re addicted to doing that. Evaluation of success creates status threats. Rather than telling someone what to do, it’s often better to ask questions and facilitate their own insights, help them make a change. We can also facilitate change and lower status threats by sharing our own insights (provided they don’t count as bragging) and… I’m going to use the F word here: feelings.
- Sharing feelings invites empathy. Empathy lowers the status threat. One thing about feelings: they happen anyway. Suppressing them is expensive (energy-wise), and it’s more efficient to take them up into consciousness and think about their significance. Then, we can use this information to control behavior. This shifts activity from our limbic system (fight-flight mechanism) to our prefrontal cortex (the CPU).
- Sleep is important. Too little sleep means the amygdala larger, which in turn makes us more anxious but less able to focus. Brain eats glucose. Though some studies suggest that it can eat fat if it constantly gets too little glucose, which might actually be better. Either way, the brain needs energy, up to 20% of the body’s energy, though it’s only up to 2% of the mass. That’s why in the morning, after an hour or two from when we wake up, it’s easiest to focus. That’s when it’s best to do the important stuff. It’s usually recommended not to wake up suddenly, not to snooze the alarm clock, otherwise we disturb the natural process of waking up. Before lunch, we tend to run out of glucose, so it’s harder to focus, so we should do something more relaxing. After lunch, we need something more stressful to get “activated”. Then, towards the end of the day, we tend to get tired again, so more relaxing activities are better.
- Usually bad, because it tends to take more time and energy than anticipated, and we’re left with little to do the important stuff. Plus, our brain is an emergency junkie. We’re quick to label something easy as urgent and deal with it right away, delaying what’s important. Even if we refrain from doing those things, it may hurt, because they still linger there, like idle threads that still have to be visited by the CPU. Some Emails may not be bad, may help with planning. If we account for urgency as a factor of importance, and still end up with a decent plan, we should be good to go.
- Low brain activity (e.g. from switching activities), if it’s under low stress, stimulates creativity. That’s when we’re more likely to have insights. That’s why we sometimes “park” a topic or “let it bake”, or “sleep on it”.