More Related Content Similar to 커머스 스타트업의 효율적인 데이터 분석 플랫폼 구축기 - 하지양 데이터 엔지니어, 발란 / 강웅석 데이터 엔지니어, 크로키닷컴 :: AWS Summit Seoul 2021 (20) More from Amazon Web Services Korea (20) 커머스 스타트업의 효율적인 데이터 분석 플랫폼 구축기 - 하지양 데이터 엔지니어, 발란 / 강웅석 데이터 엔지니어, 크로키닷컴 :: AWS Summit Seoul 20211. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
K O R E A | M A Y 1 1 - 1 2 , 2 0 2 1
2. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
커머스 스타트업의 효율적인
데이터 분석 플랫폼 구축기
하지양
데이터 엔지니어
발란
강웅석
데이터 엔지니어
크로키닷컴
3. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
커머스 스타트업, AWS를 만나다
다양한 데이터 분석 사례를 중심으로
4. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Who is HAJI? 발표자 소개
What does BALAAN do? BALAAN 회사 소개
STEP UP
Amazon QuickSight; 대시보드
Elasticsearch의 형태소 분석기; 검색 고도화
Amazon Kinesis/Firehose -> Glue -> Athena; 로그 수집/ 분석
Amazon Personalize; 개인화 추천
Amazon SageMaker; 이미지분류 (ML)
Agenda
5. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
JAVA backend developer
시스템 모니터링
DATA engineer
동영상 데이터
DATA full stack scientist
명품 커머스 플랫폼
HAJI
아무것도 없는 바닥에서 시작하는 것은 어렵지만,
그보다 더 배울 수 있는 것은 없을 지도 모른다
6. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
BALAAN
7. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
8. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Business Decision Customer Data-based Data First
비지니스
정보 분석
전략적
의사결정
고객 데이터
분석
데이터 기반
서비스 제공
데이터
중심 사고
The way business need data
9. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
전체 구조도 AWS Cloud
Amazon Kinesis
Data Firehose
AWS Glue
Amazon Athena
Amazon Elasticsearch
Service
Amazon Forecast
Amazon Personalize
database
Amazon
QuickSight
server
Client
Amazon Simple Storage
Service (Amazon S3)
Amazon SageMaker
Will be
10. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon QuickSight 활용하기
11. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon QuickSight 대시보드
거래액 증감
percentDifference
작업/ 파라미터 기능 연동
금주 예상 거래액
동적 파라미터 연동
Super Easy Auto Refresh
12. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon QuickSight 대시보드
13. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
검색 서비스를 위한 과정
DB 검색 자동완성 기능 제공 ES 색인 기반 검색 검색 결과
정렬 최적화
동의어 처리
형태소 분석기 도입
단어 분석기 적용
오타 교정
ngram 적용
연관 검색어 고도화
14. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
지금, 발란
15. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
2020. 07
2020. 11
2020. 12
2021. 01
모바일 검색 기능 강화
- SKU 검색
- 띄어쓰기 검색
- 최근 검색어 기능
추천 검색어 개편
- 카테고리/브랜드 추천 검색어
- 영어 추천 검색어
형태소 분석기 도입
- 검색 속도 개선
- 형태소 분석기 도입
검색 UI 개편
- 검색 도우미 기능 도입
- 자동완성 기능 개선
- 검색 화면 개선
검색 기능 개선 과정
16. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 발란의 브랜드, 카테고리, 상품에 대한 사용자 정의 사전을 만들고 관리
- 명품 브랜드에 최적화된 검색 기능
Elasticsearch의 형태소 분석기 적용
스톤아일랜드
구찌
몽클레어
메종마르지엘라
톰브라운
프라다
막스마라
….
힘들었던 부분
- 사용자의 오타
- 로퍼 / 퍼
- 발렌시아가
17. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Elasticsearch의 형태소 분석기 적용
발란의 브랜드, 카테고리, 상품에 대한 사용자 정의 사전을 만들고 관리
18. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Firehose 및 Athena 활용
Amazon Athena 사용 쿼리
cross join unnest(transform(
CAST(json_parse(result) AS ARRAY(MAP(varchar, varchar))),
x -> cast(x['goodsno'] as integer)
) ) att(goodsno)
json_extract_scalar(params, '$.keyword')
array_join(array_agg(inputed_word), ',')
array_join(filter(array_agg("keyword_type"), a -> a IS NOT
NULL),',')
array_join(array_distinct(array_agg("gender")),',')
이후, 전처리에서 집계되도록 변경 필요
19. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
검색 퍼널 분석(AASRR)
방문&
검색진입
28>
이탈
상품구매
35>
재검색
29>
이탈
Acquisition; 방문
Approach Search; 검색 진입
Search; 검색 시도
Re-search; 결과 내 재검색
Revenue; 상품 구매
검색 37>
Continuous Development….
20. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
검색 키워드 분류
기타
60%
브랜드
21%
브랜드+키워드
14%
직접입력
71%
자동완성
6%
인기 브랜드
8%
최근 검색어
10%
자동완성 >
브랜드
65.9%
자동완성 >
키워드
33.6%
인기 키워드
1%
추천 키워드
4%
자동완성 >
카테고리
0.2%
자동완성 >
이벤트
0.3%
카테고리
5%
브랜드 검색이 전체 검색 중 27%
21. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Personalize를 이용한 개인화 추천
22. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Personalize를 이용한 개인화 추천
BALAAN
- 신규 유저의 비율 : 47.4%
- 로그인 사용자 비율 : 44.2%
- 월간 입고되는 신규 상품 비율 : 34.4%
Amazon Personalize
- 8시간 만에 구축 가능
- 초기 구축 장벽이 낮음
- 학습/ 서비스 비용
- 하이퍼 파라미터 학습
- HRNN 세션기반 큐레이팅 알고리즘 이해
23. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker를 이용한 이미지분류
24. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
앞으로 가야할 목표
검색
추천
ML
이미지 분류,
상품 품질 판단…
BI
상관관계
분석
25. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
커머스 스타트업, AWS를 만나다
대용량 로그 수집 플랫폼 구축을 중심으로
26. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Key takeaways: Importance and Challenges For Log Collection
• Building The Log Collection Platform With Minimal Effort and Cost
• Use Collected Events In Various Ways
• Conclusion
Agenda
27. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key takeaways:
Importance and Challenges
for Log Collection
28. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• 내 마음에 드는 상품을 즐겨, 찾기, 쉽게
• 누적 다운로드 2800만+
• 월간 활성 사용자 (MAU) 3백만+
• 누적 거래액 2조 4천억원
지그재그 - No. 1 여성 쇼핑몰 모음앱
29. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
지그재그 데이터 소개
Amazon S3
100TB+
외부 데이터 (third-party)
내부 데이터 (DB)
행동 데이터 (Log)
30. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
지그재그 데이터 소개
일간 5억+ 건
피크 타임시 5분에 3백만+ 건
31. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
로그 수집의 중요성과 어려움
• 모든 것을 수치화할 수 있는 세상의 핵심은 데이터와 로그
• 버튼을 어떻게 배치하면 사용자들이 더 좋아할까?
32. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
로그 수집의 중요성과 어려움
• 모든 것을 수치화할 수 있는 세상의 핵심은 데이터와 로그
• 버튼을 어떻게 배치하면 사용자들이 더 좋아할까?
33. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
로그 수집의 중요성과 어려움
• 로그를 요구사항에 맞게 수집하는 일은 점점 어려워지고 있다 (1)
34. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
로그 수집의 중요성과 어려움
• 로그를 요구사항에 맞게 수집하는 일은 점점 어려워지고 있다 (1)
• Monolithic to MSA
• Polyglot logging modules
35. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
로그 수집의 중요성과 어려움
• 로그를 요구사항에 맞게 수집하는 일은 점점 어려워지고 있다 (1)
• Monolithic to MSA
• Polyglot logging modules
36. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
로그 수집의 중요성과 어려움
• 로그를 요구사항에 맞게 수집하는 일은 점점 어려워지고 있다 (2)
37. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
로그 수집의 중요성과 어려움
• 로그를 요구사항에 맞게 수집하는 일은 점점 어려워지고 있다 (2)
• 비즈니스가 커짐에 따라 늘어나는 요구사항들
• Pre/Post-processing
• Deduplication
• Schema, Field Validation
• Fault-tolerance
• Near real-time
38. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building the Log Collection Platform
with Minimal Effort and Cost
39. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Suggestion: Reactive Microservices Architecture
40. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
지그재그의 로그 수집 아키텍처
41. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Clients
42. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gateway / Log Agents
43. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event Brokers / Consumers
44. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Use Collected Events in
Various Ways
45. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
수없이 많은 데이터 활용 사례들
46. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Conclusion
47. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
로그 수집 플랫폼이 꼭 갖춰야할 역량
• Fault-Tolerant
• High Availability
• Monitoring
• Integrity
48. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
로그 수집 플랫폼이 꼭 갖춰야할 역량
• Fault-Tolerant ☑️
• High Availability ☑️
• Monitoring ☑️
• Integrity ☑️
• Cheapness
• Effortless
49. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
로그 수집 플랫폼이 꼭 갖춰야할 역량
Amazon
Kinesis Data
Stream
Amazon Managed
Streaming for
Apache Kafka
Amazon
CloudWatch
Amazon
Kinesis
Firehose
Amazon
S3
50. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWSKRUG 데이터 사이언스 소모임
• 데이터 분석 및 기계 학습 분야의 다양한 정보 공유
https://www.awskr.org
51. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
여러분의 소중한 피드백을 기다립니다.
강연 종료 후, 강연 평가에 참여해 주세요!
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
52. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
감사합니다
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.