Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Oracle cloud data interface

333 views

Published on

Oracle Cloud Data Interface

Published in: Data & Analytics
  • Login to see the comments

  • Be the first to like this

Oracle cloud data interface

  1. 1. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Confidential – Oracle Internal/Restricted/Highly Restricted Oracle Cloud Data Interface 김태완 @taewanme https://taewan.kim
  2. 2. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 데이터 라이프 사이클 2 Ingestion streaming batch replication DataStore Object store data lake Hadoop/HDFS NOSQL/Cassandra Transform Data pipelines ML pipelines Data quality Data Augmentation Analyze Python/R Spark Hive Interactive BI Learn Machine learning AI Data science
  3. 3. Copyright © 2018,Oracle and/orits affiliates. All rights reserved. Oracle Cloud Data Lake Oracle Object Storage BigData Cloud Event Hub Cloud DIPC Database Data Hub IaaS VM NoSQL OAC ADW *ADW: Autonomous Data warehouse *DIPC: Data Integration Platform Cloud * OAC: Oracle Analytics Cloud Object Storage
  4. 4. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 4 OCI Object Storage Case 01: Desktop & OCI Object Storage
  5. 5. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle OCI Object Storage 특징 5
  6. 6. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | OCI Object Storage 6 참조: http://bit.ly/2Z0lyA3
  7. 7. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | OCI Object Storage: bucket 7 참조: http://bit.ly/2Z0lyA3
  8. 8. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | OCI Object Storage: Connection Info 8 참조: http://bit.ly/2Z0lyA3
  9. 9. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | OCI Object Storage: Connection Info 9 참조: http://bit.ly/2Z0lyA3
  10. 10. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | OCI Object Storage: CloudBerry S3 Explorer 10 테넌트이름.compat.objectstorage.리전이름.oraclecloud.com 참조: http://bit.ly/2Z0lyA3
  11. 11. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | OCI Object Storage: CloudBerry S3 Explorer 11 참조: http://bit.ly/2Z0lyA3
  12. 12. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 12 OCI Object Storage 현재 위치
  13. 13. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 13 OCI Object Storage Case 02: OCI CLI & OCI Object Storage oci cli
  14. 14. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli 설치 14 참조: http://bit.ly/2Z0lyA3 bash -c "$(curl -L https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh)" Linux / Mac OS powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://raw.githubusercontent.com/oracle/oci- cli/master/scripts/install/install.ps1'))" Window
  15. 15. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli 설치 확인 15 참조: http://bit.ly/2Z0lyA3
  16. 16. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli command 16
  17. 17. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli 설정: info 17 참조: http://bit.ly/2Z0lyA3 user OCID
  18. 18. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli 설정: info ➔ Tenancy OCID, Region ID 18
  19. 19. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli 설정: oci setup config 19
  20. 20. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli 설정: key files 20
  21. 21. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli 설정: public key 등록 21
  22. 22. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli 설정: public key 등록 22
  23. 23. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli 설정: public key 등록
  24. 24. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli 설정: Connection Test
  25. 25. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli: Bucket 생성 25 oci cli os bucket --name --compartment-id
  26. 26. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli: Bucket 생성 26
  27. 27. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli: 파일 업로드 27
  28. 28. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli: 파일 업로드 28
  29. 29. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 29 OCI Object Storage 현재 위치 oci cli
  30. 30. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 30 OCI Object Storage Case 03: 대용량 파일 업로드 oci cli
  31. 31. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli: 데모 파일 생성 31
  32. 32. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli: 10GB 파일 업로드 32
  33. 33. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | oci cli: 10GB 파일 업로드 33
  34. 34. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 34 OCI Object Storage 현재 위치 oci cli
  35. 35. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 35 OCI Object Storage Case 04: Java를 이용한 파일 업로드 다운로드 OCI Java SDK
  36. 36. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle OCI Java SDK 36 https://search.maven.org/search?q=g:com.oracle.oci.sdkhttps://github.com/oracle/oci-java-sdk
  37. 37. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle OCI Java SDK: Maven 37
  38. 38. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle OCI Java SDK: Code 38
  39. 39. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle OCI Java SDK: Code 39
  40. 40. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle OCI Java SDK: Connection 40
  41. 41. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle OCI Java SDK: Bucket 생성 41
  42. 42. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle OCI Java SDK: 파일 추가 42
  43. 43. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 43 OCI Object Storage 현재 위치 oci cli VM OCI Java SDK
  44. 44. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 44 OCI Object Storage Case 05: OCI SDK & Object storage
  45. 45. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle OCI SDK & Rest API 45 https://github.com/oracle?utf8=%E2%9C%93&q=oci+sdk&type =&language=
  46. 46. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle OCI SDK & Rest API 46 https://github.com/oracle?utf8=%E2%9C%93&q=oci+sdk&type =&language=
  47. 47. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle OCI SDK Python: 설치 47
  48. 48. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle OCI SDK Python: Code 48
  49. 49. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle OCI SDK Python: Code 49
  50. 50. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle OCI SDK Python: Code 50
  51. 51. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle OCI SDK Python: Object 생성 결과 51
  52. 52. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle OCI SDK Go-lang: Install 52 > go get -u github.com/oracle/oci-go-sdk import ( "github.com/oracle/oci-go-sdk/common" "github.com/oracle/oci-go-sdk/identity” ) https://github.com/oracle/oci-go-sdk
  53. 53. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 53 OCI Object Storage 현재 위치 oci cli VM OCI Java SDK VM VM OCI GO SDK VM OCI Ruby SDK OCI Python SDK
  54. 54. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 54 OCI Object Storage Case 06: Hadoop & Object Storage HDFS Connector for Object Storage
  55. 55. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 55 Hadoop & Object Storage https://github.com/oracle/oci-hdfs-connector/releases
  56. 56. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 56 HDFS Connector for OCI: 설치 https://github.com/oracle/oci-hdfs-connector/releases 속성 타입 설명 fs.oci.client.auth.tenantId String tenancy의 OCID fs.oci.client.auth.userId String 사용자 OCID fs.oci.client.auth.fingerprint String 사용자 Fingerprint fs.oci.client.auth.pemfilepath String PEM 파일 위치 fs.oci.client.auth.passphrase String 암호화 문자열 core-site.xml
  57. 57. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 57 HDFS Connector for OCI: 설치 scala> sc.wholeTextFiles("oci://PipedUploadTest@internalbriangustafson/")
  58. 58. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 58 OCI Object Storage 현재 위치 oci cli VM OCI Java SDK VM VM OCI GO SDK VM OCI Ruby SDK OCI Python SDK HDFS Connector for Object Storage
  59. 59. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 59 OCI Object Storage Case 07: Kafka & Object Storage
  60. 60. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 60 OCI Object Storage Case 07: Kafka & Object Storage Apps NoSQL Search RDBMS Monitoring RT Analysis DWH Hadoop
  61. 61. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | + = Managed Kafka Service Oracle Event Hub CS
  62. 62. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
  63. 63. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
  64. 64. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
  65. 65. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
  66. 66. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
  67. 67. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. |
  68. 68. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | https://www.confluent.io/product/connectors/ - Scalable Framework - Open Source - Only Configuration
  69. 69. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | OEHCS의 Kafka Connector 지원
  70. 70. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Streaming Data + Database Oracle GoldenGate 추출 트랜잭션 추출 실시간 이관 JMS Capture Trail Pump Route Deliver
  71. 71. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 71 OCI Object Storage 현재 위치 oci cli VM OCI Java SDK VM VM OCI GO SDK VM OCI Ruby SDK OCI Python SDK HDFS Connector for Object Storage
  72. 72. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Case 10: ADW와 Object Storage 72
  73. 73. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Object Storage에 데이터 저장 73 parquet df.write.parquet("oci://devmeetup@apackrsct01/sales_extended.parquet")
  74. 74. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Object Storage에 데이터 뷰 74 parquet df.write.parquet("oci://devmeetup@apackrsct01/sales_extended.parquet") $ parquet-tools schema sales_extended.parquet message hive_schema { optional int32 prod_id; optional int32 cust_id; optional binary time_id (UTF8); optional int32 channel_id; optional int32 promo_id; optional int32 quantity_sold; optional fixed_len_byte_array(5) amount_sold (DECIMAL(10,2)); optional binary gender (UTF8); optional binary city (UTF8); optional binary state_province (UTF8); optional binary income_level (UTF8); }
  75. 75. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Object Storage에 데이터 뷰 75 parquet df.write.parquet("oci://devmeetup@apackrsct01/sales_extended.parquet") $ parquet-tools schema sales_extended.parquet message hive_schema { optional int32 prod_id; optional int32 cust_id; optional binary time_id (UTF8); optional int32 channel_id; optional int32 promo_id; optional int32 quantity_sold; optional fixed_len_byte_array(5) amount_sold (DECIMAL(10,2)); optional binary gender (UTF8); optional binary city (UTF8); optional binary state_province (UTF8); optional binary income_level (UTF8); } https://github.com/Guavus/parquet-mr- 1/tree/master/parquet-tools
  76. 76. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Object Storage에 데이터 뷰 76 parquet df.write.parquet("oci://devmeetup@apackrsct01/sales_extended.parquet") https://github.com/Guavus/parquet-mr- 1/tree/master/parquet-tools
  77. 77. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | ADW의 External Table 77
  78. 78. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | ADW와 Parquet External Table 78 External Table
  79. 79. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 79 OCI Object Storage 현재 위치 oci cli VM OCI Java SDK VM VM OCI GO SDK VM OCI Ruby SDK OCI Python SDK HDFS Connector for Object Storage
  80. 80. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Case 11: 다양한 인터페이스? ➔ 데이터 가상화 80 Hadoop Security ACL’s | Sentry | HDFS Encryption | Encryption in Motion Big Data SQL Cloud Service
  81. 81. Copyright © 2018,Oracle and/orits affiliates. All rights reserved. Oracle Cloud Data Lake Oracle Object Storage BigData Cloud Event Hub Cloud DIPC Database Data Hub IaaS VM NoSQL OAC ADW *ADW: Autonomous Data warehouse *DIPC: Data Integration Platform Cloud * OAC: Oracle Analytics Cloud Object Storage
  82. 82. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | Oracle Data Platform 82
  83. 83. Copyright © 2019,Oracle and/orits affiliates. All rights reserved. | 83

×