Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Securing Search Index with Searchable Encryption

A searchable encryption plugin for encrypting search indices of enterprise search platforms, Apache Solr and Elasticsearch.

Related Books

Free with a 30 day trial from Scribd

See all
  • Login to see the comments

Securing Search Index with Searchable Encryption

  1. 1. © Hitachi Solutions America, Ltd. 2017. All rights reserved. Zero-Knowledge Technologies for the CloudJune 7, 2017 NYC Apache Lucene/Solr Meetup
  2. 2. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 1 Security Needs for Search Engine  Gathered in one place  Pre-processed in useful form  Shows relativity between data Index Data is Valuable
  3. 3. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 2 Data Breach is Real  $4 million cost in average  29% increased cost since 2013  Only 25% is human error Security Needs for Search Engine
  4. 4. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 3 Index Data Encryption  Thicker walls are not enough to protect credential data any more.  Effective protection even after the attackers break in is required. Security Needs for Search Engine
  5. 5. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 4 Security Needs for Search Engine Use FDE platform Customization by engine integrator Lucene 6966 Existing Encryption Solution
  6. 6. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 5 Security Needs for Search Engine Keys are accessible for server Server side encryption Decrypt for every search Problem of Existing Solution
  7. 7. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 6 Overview of Credeon SFS is a product developed by Hitachi Solutions in Japan, based on an advanced cryptographic scheme born in the research lab of Hitachi. Credeon Secure Full-text Search
  8. 8. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 7 Overview of Credeon SFS Keys are inaccessible for server Client side encryption Search without decryption Main Goal of Credeon SFS
  9. 9. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 8 Overview of Credeon SFS How dose Credeon SFS Work Credeon Codec Credeon Plugin Update Handler Search Component Request Handler Postings Format Stored Fields Format
  10. 10. © Hitachi Solutions America, Ltd. 2017. All rights reserved. Client Side Server Side 9 Scheme like Lucene 6966 Encryption Scheme Upload Create Index Decrypt Encrypt Return Result QUERY ? QUERY ?Send Query Search
  11. 11. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 10 Scheme of Credeon SFS Encryption Scheme Key Management Server Client Side Server Side QUERY ? QUERY ? Create Index Create Query Decrypt Result Decrypt Upload Search
  12. 12. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 11 A Patent Cryptographic Technology Developed by Hitachi R&D Searchable Encryption Secure Practical
  13. 13. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 12 Searchable Encryption How Secure×Practical  AES 256 (FIPS 140-2)  Probabilistic Encryption  AES Encryption Speed  None Linear to Data Size Probabilistic Plain Encrypted Apple Banana Banana Banana Coconut $k24J$jX K#$#J%K^ G%jQ%K9( HFv9hbvn !=v[h-u Terms in Index SearchTime Plain Text Search Encrypted Search
  14. 14. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 13 Mechanism Searchable Encryption
  15. 15. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 14 Current Model As a Product text keyword result Searchable Storage Storage Client Server App Server OS No Encryption OS Encryption Simple Encryption with ES/Solr Client-Side Credeon SFS with ES/Solr Credeon SFS with ES/Solr Security Level No Security Decryption at storage layer  Difficult to separate key Decrypt first and match  Very slow  plaintext in memory Match first and decrypt  Key separation  High Performance  Semantically Secure  plaintext in memory Client Client Decryption at client-side  Key separation  High Performance  Semantically Secure  no plaintext on server Storage value Storage value text keyword result Client ES/Solr/Lucene ES/Solr/Lucene File System Encryption value text keyword result ES/Solr/Lucene Simple Encryption Client value text keyword result ES/Solr/Lucene Encryption Client value text keyword result Searchable Encryption Storage ES/Solr/Lucene IndexIndexIndexIndexIndex
  16. 16. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 15 Performance As a Product Plain Credeon Credeon/Plain Indexing Time (s) 1,643 4,835 2.9 Search Time (ms) 16 19 1.2 Index Size (MB) 2,941 5,011 1.7 CPU Core i5 2.4GHz Mem 8GB OS Ubuntu 14.04 LTD Engine App Solr 5 Data Wikipedia Pages 10GB (13,800,000 terms) TestEnvironmentResult
  17. 17. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 16 Limitation As a Product Type Feature Encrypted Field Plain Field Search Schema Core type, Copy field, Dynamic field Yes Yes Index Indexing, Delete, Update Yes Yes Search N-Gram, Lower case, Phonetic, Stop, Trim, Filter Query, Boost, Boolean, Stemmer, Phrase Yes Yes Pattern, Wildcard, Range, Grouping, Spatial search No Yes Result Relevance, Sort*1, Highlighting*2, Pagination, Yes Yes Spell check, Suggester, More like this No Yes Facet Query faceting, Field-value faceting, Range faceting No Yes Caching Filter cache, Query result cache Yes Yes Field cache No Yes Analyzer, Tokenizer, Filter, Transform, Response writer Yes*3 Yes Cloud Yes Yes Encryption Terms (.tim/.tip/.doc/.pos/.pay) Encrypted N/A Stored field (.fdt) Encrypted N/A Meta data Plain N/A Search keyword Encrypted N/A Transaction log Plain N/A *1: Only with Score *2: Except FastVectorHighlighter *3: Need to evaluate
  18. 18. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 17  Setup  Upload  Search  Select Encryption Fields  Use Multiple Keys  Others Demo
  19. 19. © Hitachi Solutions America, Ltd. 2017. All rights reserved. 18 Thanks for Listening Tong Ye tye@hitachisolutions.com Harry Ochiai hochiai@hitachisolutions.com @credeon https://psg.hitachi-solutions.com/credeon/secure-full-text-search

×