Open in app

Search Write

Deep Learning + IPFS + Ethereum

Blockchain in practice
Liqiao Ying · Follow
Published in Coinmonks · 8 min read · Jun 7, 2018

136 2

Recently, ONVIF the leading global standardization initiative for IP-based

physical security products, hosted the ONVIF Open Source Spotlight
Challenge, a technology competition to develop open source applications for
innovative IP video streaming applications to help resolve global security
issues. I signed up the competition and built “cam X” a proof of concept iOS
project written in Swift to demonstrate how Deep Learning, IPFS and
Ethereum technologies can be applied together in practice to equip the
normal IP cameras with superpowers. You can get the project from GitHub.

What issues the current security market is facing

Video Analytics is complex to configure, resources consuming and

single-tasking in general.

Traditional Video Management Solution data storage space is finite &

expensive, backup & maintenance are extremely painful and has single
point failure risk.

Valuable data mostly stores centralized without encryption, under the

risk of being tampered and has privacy concern.

What is cam X’s magic solution

All you need is a cellphone! cam X provides a serverless solution equips with
two general purpose object detection engines Tiny Yolo which detects 20
object types in real-time and Yolo 2 a bit slower version which detects 80
object types and has better accuracy. cam X turns any IP camera to an AI
beast within the snap of a finger. IPFS & Ethereum is a seamless solution to
keep data encrypted and place the immutable fingerprints into blockchain
transactions.

How cam X works

Step 1: Camera Management

App initial view is a camera list. There’re three sections: ONVIF, IP and
Mobile. ONVIF contains some demo cameras provided by competition
sponsors use the latest Profile T protocol. IP cameras are network cameras
use the http or rtsp protocol. Mobile is the iOS device build-in camera. I
prepopulated some default cameras for each section.

Click “+” icon on the top left navibar to show add camera dialog

Click “-” icon on the top right navibar to show delete icon ahead of each
camera row

Click “i” icon at the end of each camera row to show camera edit dialog

Click camera name to start playing live streaming

Step 2: Object Detection Video Analytics Configuration

Go to Settings -> Object Detection -> Model to pick a video analytics engine
or turn off video analytics. cam X equips two deep learning pre-trained
object detection models: Yolo 2 and Tiny Yolo. Yolo 2 detects and localizes 80
object types. It runs in 2–3 FPS on iPhone 8 or iPad Pro. Version 2 has better
accuracy. Tiny Yolo detects and localizes 20 object types. It runs in nearly
real-time (20–30 FPS) on iPhone 8 or iPad Pro. Tiny version has worse
accuracy but faster detection speed. Both neural networks run on CoreML.
Yolo 2 is the default engine.
YOLO: Real-Time Object Detection
You only look once (YOLO) is a state-of-the-art, real-time object detection
system. Unlike other classifier-based detection systems like R-CNN apply the
model to an image at multiple locations and scales, YOLO applies a single
neural network to the image. It divides the image into regions and predicts
bounding boxes and probabilities for each region. Bounding boxes are
weighted by the predicted probabilities. It also makes predictions with a
single network evaluation which is extremely fast.

Comparison to Other Detectors

Performance on the COCO Dataset
 Yolo 2 Demo Video

Navigate to Object Filters tab, a list of object types depends on the engine
chosen in Settings displayed. You can turn on or turn off detection or alarm
for each individual object or make bulk change. Detection refers to drawing
the bounding boxes and name around the objects being detected. Alarm
refers the red bounding boxes and ALARM label shows up besides the object
name. A snapshot also being captured and kept in memory temporarily
when raising an alarm. Turning off detection would also turning off alarm
automatically since if an object can’t be detected, it can’t be alarmed.
Detection is on and alarm is off for all objects by default. The alarm
threshold setting is used to avoid taking too many alarm snapshots. Multiple
objects could raise alarms in one frame and alarm on still object could stays
for every frame.
Step 3: Live Streaming with Object Detection Video Analytics
Finally here comes the exciting part. I implemented a FFmpeg video player
to decode the video stream from network cameras. You can use FFmpeg-iOS-
build-script to build FFmpeg for iOS. Overlays include logo, camera name,
detected object type and bounding boxes, engine name and current
detection FPS. Raised alarms shown in red boxes with ALARM label besides
the object name.

Step 4: Alarm Viewer

Alarm snapshots are displayed in grid view. Users can pick alarm manually
to store in IPFS and Ethereum. When the user loads the view at the first
time, all saved alarms will be restored from IPFS and Ethereum. Saved
alarms have green border. Unsaved alarms have gray border. Selected alarm
has gray background. Double click cell to show alarm in fullscreen. Each cell
shows alarm timestamps, camera name, engine name, object name
triggered alarm and snapshot. Tap Save icon at the top navibar to save or
delete alarm and view alarm details.

Step 5: Off-Chain and On-Chain Alarm Storage

I mentioned IPFS and Ethereum for storage. What are they? What benefits
do we get to use them? I’m not going into technical details. This post is not
intended to be the tutorials for these technologies behind “cam X”.

IPFS
Ethereum
Infura provides secure, reliable, and scalable access to Ethereum APIs and
IPFS gateways. “cam X” uses swift-ipfs-api library to access IPFS via Infura
remote node and uses web3swift library to access Ethereum via via Infura
remote node. I used Ethereum Rinkeby Test Network in the project.
Wallet Address
 Smart Contract Address

Blockchain Save Alarm

1. Click Save Alarm button.

2. Alarm snapshot is saved to IPFS and returns a hash.

3. Alarm metadata along with alarm snapshot hash is saved to IPFS and
returns another hash.

4. Alarm metadata hash is added to saved alarm list file in json format
which is the alarms root file and returns the root hash. The root hash
changes whenever the file content updates.

5. Returned alarm root hash is saved to Ethereum via Smart Contract

AddHash method and returns a receipt.
6. Tx Hash is the transaction receipt returned from Ethereum.

7. When the app launches next time, Saved Alarm Hash is restored from
Ethereum via Smart Contract GetHash method.

Blockchain Delete Alarm

1. Click Delete Alarm button.

2. Alarm metadata is removed from saved alarm list and returns a updated
hash. IPFS files are permanent, so alarm snapshot file and alarm
metadata file won’t be deleted from IPFS.
 3. Returned alarm root hash is saved to Ethereum via Smart Contract
AddHash method and returns a receipt.

This is a straightforward lookup table Smart Contract used to store alarm root
hash. iOS device UUID is used as the key.

This is a straightforward lookup table Smart Contract used to store alarm root hash. iOS device UUID is used as the
key.

Let us see an alarm metadata file example. The hash link on the page is the
full size alarm snapshot file mentioned above.
Alarm root file.
Ethereum smart contract transaction receipt.
Quick demo videos on how to use “cam X”
That’s it. Hope my project inspires you to create awesome apps combine
these revolutionary technologies. Feel free to ping me or make comments.

If you like my work, you can tip me in:

ETH: 0x330533c313dC617bcfcC716A66642cc3e6E57973

Bitcoin: 3BwEbigbzkQVw1MvN4gBrh3kwyu19exLnY
Deep Learning Ipfs Ethereum Blockchain Tutorial
Written by Liqiao Ying Follow

33 Followers · Writer for Coinmonks

Senior Software Architect, Certified Ethereum Developer

More from Liqiao Ying and Coinmonks

Crypto Big Stories in Coinmonks Johnwege in Coinmonks

Investing in These Two Altcoins is Bitcoin is about To SHOCK the

Like Buying Ethereum Under $10,… World
Searching for which crypto will explode in Bitcoin is about to shock the world, and it will
2024? Check out this article to learn about… be sooner than you think. This market will te…

9 min read · Feb 3, 2024 · 6 min read · Jan 27, 2024

841 15 1K 7
 Crypto Big Stories in Coinmonks Crypto Big Stories in Coinmonks

My Crypto Wallet Got Hacked and I Have Found 5 Crypto Gems That
Drained: But This Happened! Could Be the Next Solana, Fantom…
Honestly, I feel ashamed to share this story Don’t Miss Out on These Emerging Altcoins
because I should have known better. But I… Challenging the Old Cats (Cryptos) For…

6 min read · Jan 24, 2024 7 min read · Jan 18, 2024

674 28 1.1K 14

See all from Liqiao Ying See all from Coinmonks

Recommended from Medium

CryptoWaveRider (follow me on x @cryptowaveri… Robert McMenemy

ERC-404: A Comprehensive Storing Websites on The

Analysis Blockchain Using IPFS and Solidity
NOTE: THIS IS AN EDITED VERSION OF MY Introduction
ORGINAL ARTICLE THAT WAS REVAMPED…
4 min read · Oct 12, 2023
5 min read · Feb 9, 2024

37
3

Lists

Natural Language Processing Practical Guides to Machine

1223 stories · 702 saves Learning
10 stories · 1090 saves

data science and AI My Kind Of Medium (All-Time

40 stories · 81 saves Faves)
67 stories · 223 saves

nagrarohit MESON in Meson Network

IPFS: A Journey of Decentralized Meson Network: Revolutionizing

Empowerment Web3’s Bandwidth Landscape
Introduction: Arthur Hayes recently observed a surge in
funds flowing into AI, particularly favoring…

3 min read · Oct 19, 2023 10 min read · Nov 16, 2023

193
 Rajeev kumar Spheron Staff in spheronfdn

IPFS In Blockchain System Web3 Hosting: A Beginner’s Guide

Introduction on How to Host a Decentralized…
Discover Web3 hosting transformative
potential. Learn how dHosting offers…

6 min read · Dec 9, 2023 9 min read · Nov 1, 2023

2 16 1

See more recommendations