Using Cognitive Dimensions Questionnaire to Evaluate the Usability of Security APIs
•Download as PPTX, PDF•
2 likes•245 views
This was presented by me at the 28th annual gathering of Psychology of Programmers Interest Group (PPIG). Usability issues that exist in security APIs cause programmers to embed those security APIs incorrectly to the applications they develop. This results in introduction of security vulnerabilities to those applications. One of the main reasons for security APIs to be not usable is currently there is no proper method by which the usability issues of security APIs can be identified. We conducted a study to assess the effectiveness of the cognitive dimensions questionnaire based usability evaluation methodology in evaluating the usability of security APIs. We used a cognitive dimensions based generic questionnaire to collect feedback from programmers who participated in the study. Results revealed interesting facts about the prevailing usability issues in four commonly used security APIs and the capability of the methodology to identify those issues.
Recommended
Recommended
More Related Content
What's hot
What's hot (13)
Similar to Using Cognitive Dimensions Questionnaire to Evaluate the Usability of Security APIs
Similar to Using Cognitive Dimensions Questionnaire to Evaluate the Usability of Security APIs (20)
More from Chamila Wijayarathna
More from Chamila Wijayarathna (19)
Recently uploaded
Recently uploaded (20)
Using Cognitive Dimensions Questionnaire to Evaluate the Usability of Security APIs
- 1. Using Cognitive Dimensions Questionnaire to Evaluate the Usability of Security APIs Chamila Wijayarathna, Nalin A. G. Arachchilage, Jill Slay Australian Centre for Cyber Security School of Engineering and IT University of New South Wales - Canberra
- 2. Developer is the ENEMY? • Wurster, G., & van Oorschot, P. C. (2009). The developer is the enemy. In Proceedings of the 2008 workshop on new security paradigms (pp. 89–97). Source : https://au.pinterest.com/pin/129548926751300227
- 3. • Wurster, G., & van Oorschot, P. C. (2009). The developer is the enemy. In Proceedings of the 2008 workshop on new security paradigms (pp. 89–97). Solution? Educate Programmers Improve usability of Tools
- 4. “Easily usable security libraries are less prone to erroneous implementation and therefore less subject to introducing vulnerabilities in the application.” Mindermann (2016) “APIs are also often used incorrectly, resulting in bugs and sometimes significant security problems.” Myers and Stylos (2016) “It is not good enough for an API to be technically correct, it has to also be usable by other programmers, or failures will result.” Weber (2016) Usability of Security APIs • K. Mindermann. Are easily usable security libraries possible and how should experts work together to create them? In Proceedings of the 9th International Workshop on Cooperative and Human Aspects of Software Engineering, pages 62-63.ACM, 2016. • B. A. Myers and J. Stylos. Improving api usability. Communications of the ACM,59(6):62-69, 2016 • S. Weber. Empirical evaluation of api usability and security. https://insights.sei.cmu.edu/sei_blog/2016/01/empirical-evaluation-of-api-usability-and-security.html, 2016. Accessed: 2016-09-08.
- 5. Cognitive Dimensions Questionnaire Method • S. Clarke. Measuring api usability. Doctor Dobbs Journal, 29(5):S1-S5, 2004. • Blackwell, Alan F., and Thomas RG Green. "A Cognitive Dimensions questionnaire optimised for users." Proceedings of the Twelfth Annual Meeting of the Psychology of Programming Interest Group. 2000.
- 6. • Abstraction level • Learning style • Working framework • Work step unit • Progressive evaluation • Premature commitment • Penetrability • API elaboration • API viscosity • Consistency • Role expressiveness • Domain correspondence • Hard to misuse • End user protection • Testability Cognitive Dimensions for Security APIs • Wijayarathna, C., Arachchilage, N. A. G., & Slay, J. (2017). A Generic cognitive dimensions questionnaire to evaluate the usability of security apis, in Proceedings of the 19th International Conference on Human-Computer Interaction (to appear), 2017
- 7. Research Questions • Identify capability of evaluating usability of security APIs through the cognitive dimensions questionnaire method. • Is proposed 15 cognitive dimensions complete for describing the usability of security APIs? • Is the proposed questionnaire effective and efficient for identifying usability issues exist in security APIs? • Identify the capability of cognitive dimensions questionnaire method for evaluating the usability of APIs in general.
- 8. Methodology Participant sign up and complete demographic questionnaire and consent form We assign a programming task for each programmer Participant complete programming task while thinking aloud and recording their screen Participant complete cognitive dimensions questionnaire based on their experience with the API Identify issues by analyzing recordings, code artifacts and questionnaire responses
- 9. Results
- 10. Highlights • Average 74% from the total issues identified by each user have been revealed from his/her responses to the questionnaire. • Out of these 44 issues identified through observation and code analysis, only 20 (45%) were revealed by the questionnaire answers. However, questionnaire answers gave a high-level idea about some of these issues. • Questionnaire method identified some issues that could not be identified by observation and code analysis. • Improved version of questionnaire revealed 11.6 usability issues per participant compared to Clarke (2004)’s questionnaire which revealed 8 issues per participant. • Every participant mentioned that security of the programme they developed depended on the way they completed the task • None of the participants mentioned that API they used provided any support to test the security of the programme they developed
- 11. Future Work • Achieve statistical significant and saturation through more participants. • Remove analyst bias of the results. • Describe best solution to the participant before completing the questionnaire?
- 12. Questions ?
- 14. Backup Slides 14
- 15. “Security API is an application programming interface that provides developers with security functionalities that enforce one or more security policies on the interaction between at least two entities.” Gorski and Iacono (2016) P. L. Gorski and L. L. Iacono, “Towards the usability evaluation of security apis,”
- 16. Evaluating Usability of APIs • Cognitive dimensions questionnaire based methodology • User studies • Heuristic evaluation • API peer reviews • API concept maps • Automated tools
- 17. 4 different programming tasks with 4 different APIs Used • Google Authentication API • Bouncy castle Light weight Crypto API • OWASP ESAPI output encorder • A proprietary Java SSL API
Editor's Notes
- Good morning, Thank you for giving me the opportunity to present our work and I apologize for not being able to be there in person. I am going to talk about our current work on Using cognitive dimensions questionnaire method To evaluate the usability of security APIs
- Errors made by developers in the software development process have been identified as one of the main reasons for security vulnerabilities exist in software applications. Because of this, some researchers called developers as the enemy of computer security. However, it is important to identify why developers have failed to develop their applications securely. Security is a complex concept compared to other concepts in software development. Also, many developers who are involved in software development are not security experts and they are not familiar with security concepts. therefore they may not follow security practises when developing their applications.
- Wurster and van oorschot identified two solutions that could be taken to address this issue rather than blaming the developers. Those solutions are educating programmers on how to ensure security of applications they develop And the other thing is Improving usability of programming tools that developers use for developing applications, so they will make minimum mistakes while using them, that will result in security vulnerabilities. Our research is following this approach, we are focusing on a special set of tools which are security APIs.
- It has been identified that many security vulnerabilities get introduced in to applications, due to usability issues exist in security APIs, that are used while developing those applications. Many researchers have agreed with the fact, That developers are not to be blamed for such vulnerabilities. They have emphasized the importance of usable security APIs, for developing secure software, To develop and deliver usable security APIs, It is important to evaluate and identify usability issues exist in security APIs, Before they are delivered as a final products.
- There have been several methods used to evaluate the usability of general APIs. In our study, we are particularly focusing on the cognitive dimensions questionnaire based usability evaluation methodology. This was initially proposed by prof. Alan Blackwell and Prof. Thomas Green, and Steven Clarke from Microsoft adapted it to use to evaluate the usability of APIs. There are few main reasons why we used this methodology, instead of other existing API usability evaluation methods. This method takes feedback from programmers after they used API to do some useful task. So compared to a methodologies such as heuristic evaluation, this method will reveal actual issues that programmers came up with while using the API which made them make mistakes while they used the API. Also, the feedback collection method, which is the cognitive dimensions questionnaire, also has some advantages over Conducting a user study, where issues will be identified by observing programmers while they use the API and from their think aloud results. In this method, since programmer is reporting issues through his responses to the questionnaire, the involvement of the experimenter, Who might most probably be a developer of the API in the real world scenario will be less. So the influence of the experimenter for the issues identified will be much less in this case. Furthermore, this method has been tested and practically used than most other methods to evaluate the usability of APIs. Clarke claims that they are using this method at Microsoft to evaluate their APIs, and there are few researches which use this methodology or variations of this, to evaluate the usability of APIs
- Clarke introduced a cognitive dimensions framework which consists of 12 dimensions To describe the usability of APIs. However, recent work on the usability of security APIs suggest that These dimensions won’t be enough to describe the usability of security APIs. Those suggest that there are more aspects that need to be considered when talking about the usability of security APIs. By considering these facts, we proposed an enhanced version of this framework To be used in the usability evaluations of security APIs. There are 5 main improvements that we suggested. In Clarke’s framework, “Learning Style” described “what are the learning requirements posed by the API, and what are the learning styles available to a targeted developer”, When it comes to security APIs, it has been suggested that a security API should be easily learnable Even without a cryptographic or security background. Therefore, we suggested to improve learning style in our enhanced version to cover this aspect also. Similar improvements were done to the penetrability dimension as well. In Clarke’s version, penetrability referred to “How the API facilitates exploration, analysis, and understanding of its components, and how targeted developers go about retrieving what is needed.” This does not cover, to which extend does the API communicate its security related specifics to programmers who use it. So we suggested to improve “penetrability” dimension to cover this aspect as well. In addition to improving these 3 dimensions, we also included 3 new dimensions in our proposed version of cognitive dimensions framework, Which are “hard to misuse”, “End user protection” and “testability” The work based on the framework we proposed have been published in HCI international 2017 conference. We also improved the questionnaire used by Clarke to cover these newly suggested aspects.
- Since this framework and the questionnaire is proposed by only referring to previous work on the domain, We don’t know whether it works practically or not In this work, we are conducting an empirical investigation to identify the applicability of this proposed framework. We are mainly trying to answer four main research gaps through this study. First we are trying identify the applicability of this methodology to evaluate the usability of security APIs, Also, we are trying to identify whether the cognitive dimensions framework we proposed is complete. Furthermore, we are trying to evaluate the efficiency and effectiveness of the questionnaire we developed for evaluating the usability of security APIs. The questionnaire Clarke proposed for usability evaluation of general APIs is not backed up by empirical evidence, through this study we are planning to give some insights about Clarke’s version of questionnaire as well.
- This is the methodology we are following to answer these questions. We recruited programmers who have industrial or open source software development experience in java. Once they signed up, we assigned each of them a programing task from a pool of programming task. We used four programming tasks that required to implement some security critical code using a security API. These tasks had to use either Google authentication API, bouncycastle crypto API, OWASP ESAPI encorder or java secure socket extension API. When selecting APIs we tried to select APIs so that they will cover many different domains and to be different from each other as much as possible as suggested in previous API usability studies. Then each participant has to complete the task. While completing the task, they had to think aloud and record their screens. Once they completed the task, they had to complete the cognitive dimensions questionnaire. Then I analyzed the questionnaire responses provided by each participant and identified usability issues they reported. Then I separately analyzed screen recording with think aloud results and code artifacts they produced, and identified usability issues of API.
- This table shows the results we collected from our first 7 participants, It lists total number of issues identified by each participant in both methods, Amount of issues identified through observation and code analysis, Amount of issues identified by cognitive dimensions questionnaire, And also I have listed number of issues identified by only Clarke’s questions
- Now let’s see what are the main observations or highlights we can see in these results. Each user response to the questionnaire revealed an average 74% of the issues that were revealed by that particular user from both methods, If we consider all issues identified using both methods by that user are approximately equal to all the issues he encounter while using the API, Questionnaire has revealed average 74% of the issues that each user came up with. By observing recordings and analysing source codes provided, we identified 44 potential issues of the 4 APIs, out of these 44 issues, only 20 issues were identify through the questionnaire method. You can see that this is a relatively low number. Even though, questionnaire method fail to identified about half of the issues identified through observation, Questionnaire answers gave a high level idea about some of the issues. For example, by observing screen recordings, I observed that “parameters of Bouncycastle Scrypt.generate() method are not obvious” Eventhough this was not revealed by the questionnaire answers, those pointed out that “API does not reveal information about function parameters and what they return”. So we’ll have to consider that also when talking about this number. Questionnaire method also revealed some issues that could not be identified using observation and code analysis, Specially questionnaire method was more sensitive in identifying issues related to progressive evaluation, premature commitment, API elaboration, consistency, end user protection and testability. As expected, enhanced questionnaire identified more issues compared to Clarke’s version. All participants mention that security of the developed application depends on the way they completed the task, Eventhough previous studies have mentioned that security of the end user who uses an application should not depend on the developers who develop the application and should solely depend on the security APIs that are used, Results imply that this is something that is difficult to achieve practically. Also, none of the participants mention that the API gave any help to test the security of the code they developed using the API, This suggest that current API designs does not consider this aspect. These are some insights we gained in the early results of the experiment, We will investigate more on these while we progress with our study.
- When progressing with the study, Achieving statistical significance and data saturation is one of the main objectives. We plan to use 40 participants in total, That is 10 participants per each task. Another main limitation is analyst bias of the results, We are planning to use triangulation to overcome this. Another observation we did was that, Some participants completed the task incorrectly, And when completing the questionnaire, They did it thinking that their solution was correct, If they knew the correct solution, They would’ve know the reason for the mistakes they did And might have reported more usability issues. We need to do further studies on this as well.
- Modern software development is mainly an API driven development, without implementing each and every functionality from the scratch, programmers use existing functionalities developed by other developers and embed them in to applications they are developing, through APIs An API is a interface that two software use to communicate with each other. A software can expose its data and functions to outside using an API, so other software can make use of them. Security APIs are a special set of APIs which provide security functionalities Such as authentication, authorization, encryption, decryption, etc