Submit Search
Upload
Deep Learningによる超解像の進歩
•
44 likes
•
29,898 views
H
Hiroto Honda
Follow
deep learningベースの超解像手法についてのまとめ
Read less
Read more
Technology
Report
Share
Report
Share
1 of 36
Download now
Download to read offline
Recommended
SSII2022 [SS1] ニューラル3D表現の最新動向〜 ニューラルネットでなんでも表せる?? 〜
SSII2022 [SS1] ニューラル3D表現の最新動向〜 ニューラルネットでなんでも表せる?? 〜
SSII
[DL輪読会]Graph R-CNN for Scene Graph Generation
[DL輪読会]Graph R-CNN for Scene Graph Generation
Deep Learning JP
【メタサーベイ】Neural Fields
【メタサーベイ】Neural Fields
cvpaper. challenge
[DL輪読会]NVAE: A Deep Hierarchical Variational Autoencoder
[DL輪読会]NVAE: A Deep Hierarchical Variational Autoencoder
Deep Learning JP
深層強化学習の分散化・RNN利用の動向〜R2D2の紹介をもとに〜
深層強化学習の分散化・RNN利用の動向〜R2D2の紹介をもとに〜
Jun Okumura
Generative Models(メタサーベイ )
Generative Models(メタサーベイ )
cvpaper. challenge
【メタサーベイ】数式ドリブン教師あり学習
【メタサーベイ】数式ドリブン教師あり学習
cvpaper. challenge
backbone としての timm 入門
backbone としての timm 入門
Takuji Tahara
Recommended
SSII2022 [SS1] ニューラル3D表現の最新動向〜 ニューラルネットでなんでも表せる?? 〜
SSII2022 [SS1] ニューラル3D表現の最新動向〜 ニューラルネットでなんでも表せる?? 〜
SSII
[DL輪読会]Graph R-CNN for Scene Graph Generation
[DL輪読会]Graph R-CNN for Scene Graph Generation
Deep Learning JP
【メタサーベイ】Neural Fields
【メタサーベイ】Neural Fields
cvpaper. challenge
[DL輪読会]NVAE: A Deep Hierarchical Variational Autoencoder
[DL輪読会]NVAE: A Deep Hierarchical Variational Autoencoder
Deep Learning JP
深層強化学習の分散化・RNN利用の動向〜R2D2の紹介をもとに〜
深層強化学習の分散化・RNN利用の動向〜R2D2の紹介をもとに〜
Jun Okumura
Generative Models(メタサーベイ )
Generative Models(メタサーベイ )
cvpaper. challenge
【メタサーベイ】数式ドリブン教師あり学習
【メタサーベイ】数式ドリブン教師あり学習
cvpaper. challenge
backbone としての timm 入門
backbone としての timm 入門
Takuji Tahara
Skip Connection まとめ(Neural Network)
Skip Connection まとめ(Neural Network)
Yamato OKAMOTO
Swin Transformer (ICCV'21 Best Paper) を完璧に理解する資料
Swin Transformer (ICCV'21 Best Paper) を完璧に理解する資料
Yusuke Uchida
Attentionの基礎からTransformerの入門まで
Attentionの基礎からTransformerの入門まで
AGIRobots
Transformer 動向調査 in 画像認識
Transformer 動向調査 in 画像認識
Kazuki Maeno
畳み込みニューラルネットワークの研究動向
畳み込みニューラルネットワークの研究動向
Yusuke Uchida
近年のHierarchical Vision Transformer
近年のHierarchical Vision Transformer
Yusuke Uchida
PRML学習者から入る深層生成モデル入門
PRML学習者から入る深層生成モデル入門
tmtm otm
GAN(と強化学習との関係)
GAN(と強化学習との関係)
Masahiro Suzuki
[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...
[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...
Deep Learning JP
敵対的生成ネットワーク(GAN)
敵対的生成ネットワーク(GAN)
cvpaper. challenge
Transformer 動向調査 in 画像認識(修正版)
Transformer 動向調査 in 画像認識(修正版)
Kazuki Maeno
動作認識の最前線:手法,タスク,データセット
動作認識の最前線:手法,タスク,データセット
Toru Tamaki
【DL輪読会】High-Resolution Image Synthesis with Latent Diffusion Models
【DL輪読会】High-Resolution Image Synthesis with Latent Diffusion Models
Deep Learning JP
[DL輪読会]Neural Radiance Flow for 4D View Synthesis and Video Processing (NeRF...
[DL輪読会]Neural Radiance Flow for 4D View Synthesis and Video Processing (NeRF...
Deep Learning JP
[DL輪読会]A System for General In-Hand Object Re-Orientation
[DL輪読会]A System for General In-Hand Object Re-Orientation
Deep Learning JP
Deeplearning輪読会
Deeplearning輪読会
正志 坪坂
[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models
[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models
Deep Learning JP
畳み込みニューラルネットワークの高精度化と高速化
畳み込みニューラルネットワークの高精度化と高速化
Yusuke Uchida
Point net
Point net
Fujimoto Keisuke
【メタサーベイ】基盤モデル / Foundation Models
【メタサーベイ】基盤モデル / Foundation Models
cvpaper. challenge
Recent Progress on Single-Image Super-Resolution
Recent Progress on Single-Image Super-Resolution
Hiroto Honda
SeRanet introduction
SeRanet introduction
Kosuke Nakago
More Related Content
What's hot
Skip Connection まとめ(Neural Network)
Skip Connection まとめ(Neural Network)
Yamato OKAMOTO
Swin Transformer (ICCV'21 Best Paper) を完璧に理解する資料
Swin Transformer (ICCV'21 Best Paper) を完璧に理解する資料
Yusuke Uchida
Attentionの基礎からTransformerの入門まで
Attentionの基礎からTransformerの入門まで
AGIRobots
Transformer 動向調査 in 画像認識
Transformer 動向調査 in 画像認識
Kazuki Maeno
畳み込みニューラルネットワークの研究動向
畳み込みニューラルネットワークの研究動向
Yusuke Uchida
近年のHierarchical Vision Transformer
近年のHierarchical Vision Transformer
Yusuke Uchida
PRML学習者から入る深層生成モデル入門
PRML学習者から入る深層生成モデル入門
tmtm otm
GAN(と強化学習との関係)
GAN(と強化学習との関係)
Masahiro Suzuki
[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...
[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...
Deep Learning JP
敵対的生成ネットワーク(GAN)
敵対的生成ネットワーク(GAN)
cvpaper. challenge
Transformer 動向調査 in 画像認識(修正版)
Transformer 動向調査 in 画像認識(修正版)
Kazuki Maeno
動作認識の最前線:手法,タスク,データセット
動作認識の最前線:手法,タスク,データセット
Toru Tamaki
【DL輪読会】High-Resolution Image Synthesis with Latent Diffusion Models
【DL輪読会】High-Resolution Image Synthesis with Latent Diffusion Models
Deep Learning JP
[DL輪読会]Neural Radiance Flow for 4D View Synthesis and Video Processing (NeRF...
[DL輪読会]Neural Radiance Flow for 4D View Synthesis and Video Processing (NeRF...
Deep Learning JP
[DL輪読会]A System for General In-Hand Object Re-Orientation
[DL輪読会]A System for General In-Hand Object Re-Orientation
Deep Learning JP
Deeplearning輪読会
Deeplearning輪読会
正志 坪坂
[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models
[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models
Deep Learning JP
畳み込みニューラルネットワークの高精度化と高速化
畳み込みニューラルネットワークの高精度化と高速化
Yusuke Uchida
Point net
Point net
Fujimoto Keisuke
【メタサーベイ】基盤モデル / Foundation Models
【メタサーベイ】基盤モデル / Foundation Models
cvpaper. challenge
What's hot
(20)
Skip Connection まとめ(Neural Network)
Skip Connection まとめ(Neural Network)
Swin Transformer (ICCV'21 Best Paper) を完璧に理解する資料
Swin Transformer (ICCV'21 Best Paper) を完璧に理解する資料
Attentionの基礎からTransformerの入門まで
Attentionの基礎からTransformerの入門まで
Transformer 動向調査 in 画像認識
Transformer 動向調査 in 画像認識
畳み込みニューラルネットワークの研究動向
畳み込みニューラルネットワークの研究動向
近年のHierarchical Vision Transformer
近年のHierarchical Vision Transformer
PRML学習者から入る深層生成モデル入門
PRML学習者から入る深層生成モデル入門
GAN(と強化学習との関係)
GAN(と強化学習との関係)
[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...
[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...
敵対的生成ネットワーク(GAN)
敵対的生成ネットワーク(GAN)
Transformer 動向調査 in 画像認識(修正版)
Transformer 動向調査 in 画像認識(修正版)
動作認識の最前線:手法,タスク,データセット
動作認識の最前線:手法,タスク,データセット
【DL輪読会】High-Resolution Image Synthesis with Latent Diffusion Models
【DL輪読会】High-Resolution Image Synthesis with Latent Diffusion Models
[DL輪読会]Neural Radiance Flow for 4D View Synthesis and Video Processing (NeRF...
[DL輪読会]Neural Radiance Flow for 4D View Synthesis and Video Processing (NeRF...
[DL輪読会]A System for General In-Hand Object Re-Orientation
[DL輪読会]A System for General In-Hand Object Re-Orientation
Deeplearning輪読会
Deeplearning輪読会
[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models
[DL輪読会]Transframer: Arbitrary Frame Prediction with Generative Models
畳み込みニューラルネットワークの高精度化と高速化
畳み込みニューラルネットワークの高精度化と高速化
Point net
Point net
【メタサーベイ】基盤モデル / Foundation Models
【メタサーベイ】基盤モデル / Foundation Models
Similar to Deep Learningによる超解像の進歩
Recent Progress on Single-Image Super-Resolution
Recent Progress on Single-Image Super-Resolution
Hiroto Honda
SeRanet introduction
SeRanet introduction
Kosuke Nakago
Small Deep-Neural-Networks: Their Advantages and Their Design
Small Deep-Neural-Networks: Their Advantages and Their Design
Forrest Iandola
小數據如何實現電腦視覺,微軟AI研究首席剖析關鍵
小數據如何實現電腦視覺,微軟AI研究首席剖析關鍵
CHENHuiMei
Urs Köster Presenting at RE-Work DL Summit in Boston
Urs Köster Presenting at RE-Work DL Summit in Boston
Intel Nervana
Scaling Up AI Research to Production with PyTorch and MLFlow
Scaling Up AI Research to Production with PyTorch and MLFlow
Databricks
Operationalizing SDN
Operationalizing SDN
ADVA
Software Defined Visualization (SDVis): Get the Most Out of ParaView* with OS...
Software Defined Visualization (SDVis): Get the Most Out of ParaView* with OS...
Intel® Software
Deep Learning Hardware: Past, Present, & Future
Deep Learning Hardware: Past, Present, & Future
Rouyun Pan
PointNet
PointNet
PetteriTeikariPhD
Distributed deep learning_over_spark_20_nov_2014_ver_2.8
Distributed deep learning_over_spark_20_nov_2014_ver_2.8
Vijay Srinivas Agneeswaran, Ph.D
"Designing CNN Algorithms for Real-time Applications," a Presentation from Al...
"Designing CNN Algorithms for Real-time Applications," a Presentation from Al...
Edge AI and Vision Alliance
Synthetic dialogue generation with Deep Learning
Synthetic dialogue generation with Deep Learning
S N
Hao hsiang ma resume
Hao hsiang ma resume
Eliot Ma
MCL303-Deep Learning with Apache MXNet and Gluon
MCL303-Deep Learning with Apache MXNet and Gluon
Amazon Web Services
Introduction to Deep Learning and neon at Galvanize
Introduction to Deep Learning and neon at Galvanize
Intel Nervana
GTC Europe 2017 Keynote
GTC Europe 2017 Keynote
NVIDIA
(Research Note) Delving deeper into convolutional neural networks for camera ...
(Research Note) Delving deeper into convolutional neural networks for camera ...
Jacky Liu
Convolutional neural network
Convolutional neural network
Yan Xu
PIES_Profile_INDIA
PIES_Profile_INDIA
Piengsol India
Similar to Deep Learningによる超解像の進歩
(20)
Recent Progress on Single-Image Super-Resolution
Recent Progress on Single-Image Super-Resolution
SeRanet introduction
SeRanet introduction
Small Deep-Neural-Networks: Their Advantages and Their Design
Small Deep-Neural-Networks: Their Advantages and Their Design
小數據如何實現電腦視覺,微軟AI研究首席剖析關鍵
小數據如何實現電腦視覺,微軟AI研究首席剖析關鍵
Urs Köster Presenting at RE-Work DL Summit in Boston
Urs Köster Presenting at RE-Work DL Summit in Boston
Scaling Up AI Research to Production with PyTorch and MLFlow
Scaling Up AI Research to Production with PyTorch and MLFlow
Operationalizing SDN
Operationalizing SDN
Software Defined Visualization (SDVis): Get the Most Out of ParaView* with OS...
Software Defined Visualization (SDVis): Get the Most Out of ParaView* with OS...
Deep Learning Hardware: Past, Present, & Future
Deep Learning Hardware: Past, Present, & Future
PointNet
PointNet
Distributed deep learning_over_spark_20_nov_2014_ver_2.8
Distributed deep learning_over_spark_20_nov_2014_ver_2.8
"Designing CNN Algorithms for Real-time Applications," a Presentation from Al...
"Designing CNN Algorithms for Real-time Applications," a Presentation from Al...
Synthetic dialogue generation with Deep Learning
Synthetic dialogue generation with Deep Learning
Hao hsiang ma resume
Hao hsiang ma resume
MCL303-Deep Learning with Apache MXNet and Gluon
MCL303-Deep Learning with Apache MXNet and Gluon
Introduction to Deep Learning and neon at Galvanize
Introduction to Deep Learning and neon at Galvanize
GTC Europe 2017 Keynote
GTC Europe 2017 Keynote
(Research Note) Delving deeper into convolutional neural networks for camera ...
(Research Note) Delving deeper into convolutional neural networks for camera ...
Convolutional neural network
Convolutional neural network
PIES_Profile_INDIA
PIES_Profile_INDIA
Recently uploaded
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
LoriGlavin3
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
Kari Kakkonen
A Framework for Development in the AI Age
A Framework for Development in the AI Age
Cprime
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
LoriGlavin3
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
marketing932765
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
Curtis Poe
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Alkin Tezuysal
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App Framework
Pixlogix Infotech
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
Neo4j
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
ThousandEyes
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Mark Goldstein
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
itnewsafrica
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
BookNet Canada
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
Lonnie McRorey
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
itnewsafrica
How to write a Business Continuity Plan
How to write a Business Continuity Plan
Databarracks
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
Wes McKinney
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
LoriGlavin3
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
LoriGlavin3
Recently uploaded
(20)
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
A Framework for Development in the AI Age
A Framework for Development in the AI Age
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App Framework
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
Glenn Lazarus- Why Your Observability Strategy Needs Security Observability
How to write a Business Continuity Plan
How to write a Business Continuity Plan
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
Deep Learningによる超解像の進歩
1.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Deep Learningによる 超解像の進歩
2.
Copyright © DeNA Co.,Ltd. All Rights Reserved. ⾃⼰紹介 2 Hiroto Honda @hirotomusiker n メーカー研究所
→ 2017/1 DeNA n ETH Zurich CVLにて客員(2013-2014) n CVPR NTIRE Workshop Program Committee n DeNA AI研究開発エンジニア n 現職:Object Detection (OSS: https://github.com/DeNA/Chainer_Mask_R-CNN ) n 前職:Low-Level Vision, Computational, Sensor LSI
3.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Contents n 超解像は試しやすい n 初期のSISRネットワーク ⁃
SRCNN, ESPCN, VDSR ⁃ Upsampling⼿法– deconv or pixelshuffle n ベースライン⼿法:SRResNet ⁃ SRResNet, SRGAN, and EDSR n 超解像とperception ⁃ 復元結果とロス関数の関係 ⁃ Perception – Distortion Tradeoff n まとめ 3
4.
Copyright © DeNA Co.,Ltd. All Rights Reserved. 超解像とは n 低解像度画像 n ⾼解像度画像 4 復元
5.
Copyright © DeNA Co.,Ltd. All Rights Reserved. 超解像は試しやすい! 5 original(HR) LR resize train アノテーションが不要な Self-supervised learningの⼀種
6.
Copyright © DeNA Co.,Ltd. All Rights Reserved. 超解像の進歩 6 https://github.com/jbhuang0604/SelfExSRPSNR* [dB] (over bicubic) on Set5 dataset, x4 +1.86 +2.93 +2.06 +3.63 A+0.0 bicubic 2015 20172014 2016 +4.20 +2.48 PSNR data from:5) SRCNN
VDSR SRResNet EDSRESPCN 超解像の精度は年々向上している * PSNR = 10 log10 (2552 / MSE ) when max value is 255
7.
Copyright © DeNA Co.,Ltd. All Rights Reserved. 超解像ネットワークの学習 n 正解画像からpatchをcropする HR n
patchをダウンサンプルする LR = g(HR) n バッチを編成する {LR}, {HR} n ネットワークfを学習する ロス関数は: MSE(HR, f(LR)) n ...以上! 7 LR=g(HR) f(LR) HR f MSE e.g. bicubic down-sampling
8.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Non-deep⼿法: 辞書ベースのアルゴリズム 8 = 係数を最適化する 8 ベースライン: A+ (2014) http://www.vision.ee.ethz.ch/~timofter/publications/Timofte-ACCV-2014.pdf = 学習済みの辞書 x 0 + x 0 + x 0.8 + x 0.8 + x 0.05 + x 0.05 + LR patch HR patch
9.
Copyright © DeNA Co.,Ltd. All Rights Reserved. n 初期のSISR networks ⁃
SRCNN, ESPCN, VDSR ⁃ Upsampling⼿法 – deconv or pixelshuffle 9
10.
Copyright © DeNA Co.,Ltd. All Rights Reserved. 最初のDeep超解像– SRCNN 10 Kernel size: 9 – 1 –
5 or 9 – 3 – 5 or 9 – 5 – 5 from:1) ⾮常にシンプルで計算量も少ない bicubic x2
11.
Copyright © DeNA Co.,Ltd. All Rights Reserved. VDSR: ディープなSRCNN 11 from:3) 3x3, 64 ch D= 5 to 20
12.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Efficient sub-pixel CNN (ESPCN) 12 SRCNNと違い、LR画像をconvするので効率的 Kernel size 5 – 3 – 3 from:2)
13.
Copyright © DeNA Co.,Ltd. All Rights Reserved. SRCNN / VDSR とESPCNの違い n Post-upsamplingのほうが効率的だが、1.6倍 といった⾮整数の upsamplingができない 13 SRCNN, VDSR ESPCN bicubic x2
output input Pixel shuffle x2 ch h w
14.
Copyright © DeNA Co.,Ltd. All Rights Reserved. CNNによるアップスケール - Deconvolution or PixelShuffle? n
Deconvolution 14 https://distill.pub/2016/deconv-checkerboard/ 位置ごとに関与する画素数が均⼀ではないため 格⼦パターンが出てしまう
15.
Copyright © DeNA Co.,Ltd. All Rights Reserved. CNNによるアップスケール - Deconvolution or PixelShuffle? n
resize – convolutionしては? 15 格⼦パターンはなくなる Resize(low-pass)により情報が失われる可能性があるので、 Nearest neighborで埋める⽅法も
16.
Copyright © DeNA Co.,Ltd. All Rights Reserved. CNNによるアップスケール - Deconvolution or PixelShuffle? n
Sub-pixel convolution (aka. PixelShuffle) 16 各位置でチャネルの情報をタイルする e.g. 9 channels -> 3x3 サブピクセル 格⼦ノイズフリーではない from:2)
17.
Copyright © DeNA Co.,Ltd. All Rights Reserved. n ベースライン⼿法:SRResNet ⁃ SRResNet,
SRGAN, and EDSR 17
18.
Copyright © DeNA Co.,Ltd. All Rights Reserved. SRResnet and SRGAN – twitter CVPR’17 18 Skip
connection pixel shuffle x2 MSE MSE Discriminator Trained VGG Perceptual Loss Discriminator Loss MSE Loss from:4) pixel shuffle x2 ch h w ・3種類のロス関数 ・MSEのみを使⽤する場合SRResNetと呼ぶ 24 residual blocks, 64 ch
19.
Copyright © DeNA Co.,Ltd. All Rights Reserved. SRResnet* and SRGAN – ネットワーク詳細 19 ・resblockとskip connection ・pixel shuffle upsampling from:4)
20.
Copyright © DeNA Co.,Ltd. All Rights Reserved. さらに⾼精度に特化したEnhanced Deep Super Resolution (EDSR) ソウル⼤ 20 32 residual blocks,
256 ch Skip connection x2 x2 l1 l1 Loss from:5)
21.
Copyright © DeNA Co.,Ltd. All Rights Reserved. PSNRと⾒た⽬ 21 from:5) 20dB台で1dB違うと明らかに⾒た⽬が変わる
22.
Copyright © DeNA Co.,Ltd. All Rights Reserved. n 超解像とPerception ⁃ 復元結果とロス関数の関係 ⁃
Perception – Distortion Tradeoff 22
23.
Copyright © DeNA Co.,Ltd. All Rights Reserved. 主観評価とPSNR 23 Original SRResNet 25.53dB SRGAN 21.15dB bicubic 21.59dB Method→ PSNR → from: 4)
24.
Copyright © DeNA Co.,Ltd. All Rights Reserved. SRResnet and SRGAN – lossでこんなに違う 24 MSE loss
● ● Perceptual loss using VGG ● Discriminator loss ● ● from:4) PSNRが 最も⾼い
25.
Copyright © DeNA Co.,Ltd. All Rights Reserved. 3タイプのロス関数 ①l1/l2 loss ②perceptual loss ③GAN
loss 25 generated image real / fake ground truth multi-scale feature matching VGG discrimi- nator generated image ground truth generated image ground truth Low Distortion Good Perception
26.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Perception-Distortion Tradeoff どの⼿法も、low distortionとgood perceptual qualityを 同時に満たせない → tradeoff把握が⼤事 26 from:8)
27.
Copyright © DeNA Co.,Ltd. All Rights Reserved. 超解像の⽬的はなにか? 27 Accurate Plausible 正確な復元 ⾃然な復元 どちらを選ぶかは、⽤途次第!! 引⽤元:4)
28.
Copyright © DeNA Co.,Ltd. All Rights Reserved. n まとめ 28
29.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Progress on SISR – 精度と速度 29 PSNR [dB] (over bicubic) on Set5 dataset, x4 +1.86 +2.93 +2.06 +3.63 A+ SRCNN
VDSR SRResNet EDSR0.0 bicubic 2015 20172014 2016 +4.20 ESPCN +2.48 0.44 0.04 0.74 1.33 40.7 ・CNNを通る画像サイズ ・中間レイヤのチャネル数 で計算量が⼤きく変化する PSNRデータ引⽤元:5) Mega-Multiplication per one input pixel for x2 restoration
30.
Copyright © DeNA Co.,Ltd. All Rights Reserved. NTIRE 2017 超解像コンペでのベンチマーク詳細 30 EDSR SRResNet VDSR ESPCN SRCNN A+ from: 9)
31.
Copyright © DeNA Co.,Ltd. All Rights Reserved. まとめ n 超解像はdeepが主流、⾼精度だが計算量が⼤きい n resblock連結
+ skip connectionや、pixel shuffle upsamplingが重要 n SRResNetベースの⼿法がベースライン n ʻAccurateʼ か ʻPlausibleʼ かは⽤途次第。 31
32.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Appendix: Residual Dense Network for Super-Resolution 32 DenseNetベースのSRResNet from: 6)
33.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Appendix: Deep Back-Projection Networks For Super-Resolution (best PSNR in NTIRE ʼ18 x8 bicubic downsampling track) 33 from: 7)
34.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Datasets n DIV2K dataset
(train, val) https://data.vision.ee.ethz.ch/cvl/DIV2K/ n Set5 dataset (test) http://people.rennes.inria.fr/Aline.Roumy/results/SR_BMVC12.html n B100 dataset (test) https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/ n Urban100 dataset (test) https://sites.google.com/site/jbhuang0604/publications/struct_sr 34
35.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Competitions n NTIRE2017: New Trends
in Image Restoration and Enhancement workshop and challenge on image super- resolution in conjunction with CVPR 2017 http://www.vision.ee.ethz.ch/ntire17/ report: http://www.vision.ee.ethz.ch/~timofter/publications/Timofte-CVPRW-2017.pdf n NTIRE2018: New Trends in Image Restoration and Enhancement workshop and challenge on super-resolution, dehazing, and spectral reconstructionin conjunction with CVPR 2018 http://www.vision.ee.ethz.ch/ntire18/ report: http://openaccess.thecvf.com/content_cvpr_2018_workshops/papers/w13/Timofte_NTIRE_2018 _Challenge_CVPR_2018_paper.pdf n PIRM2018: Workshop and Challenge on Perceptual Image Restoration and Manipulation in conjunction with ECCV 2018 https://www.pirm2018.org/ 35
36.
Copyright © DeNA Co.,Ltd. All Rights Reserved. References 1) Dong et
al., Image Super-Resolution Using Deep Convolutional Networks, https://arxiv.org/abs/1501.00092 2) Shi et al., Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network, https://arxiv.org/abs/1609.05158 3) Kim et al., Accurate Image Super-Resolution Using Very Deep Convolutional Networks, https://arxiv.org/pdf/1511.04587 4) Ledig et al., Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , https://arxiv.org/abs/1609.04802 5) Lim et al., Enhanced Deep Residual Networks for Single Image Super-Resolution, https://arxiv.org/abs/1707.02921 6) Zhang et al., Residual Dense Network for Image Super-Resolution, https://arxiv.org/abs/1802.08797 7) Haris et al., Deep Back-Projection Networks For Super-Resolution, https://arxiv.org/pdf/1803.02735.pdf 8) Blau et al., Perception Distortion Tradeoff, https://arxiv.org/abs/1711.06077 9) Timofte et al., NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results , http://www.vision.ee.ethz.ch/~timofter/publications/Timofte-CVPRW-2017.pdf
Download now