Scribd Logo
Upload

Welcome to Scribd!

  • 226 views

    Analyzing IoT Data in Python Chapter1

    Uploaded by

    Fgpeqw

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    Analyzing IoT Data in Python Chapter1

    Uploaded by

    Fgpeqw
    226 views27 pages

    Original Title

    Analyzing IoT Data in Python chapter1

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    226 views27 pages

    Analyzing IoT Data in Python Chapter1

    Uploaded by

    Fgpeqw

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    of 27

    Introduction to IoT

    data
    A N A LY Z I N G I OT D ATA I N P Y T H O N

    Matthias Voppichler
    IT Developer
    Course overview
    Collect and analyze IoT data

    Gather data
    API Endpoints

    Data Streams

    Visualize data

    Combine datasets

    Detect patterns

    ML Model based alerts

    ANALYZING IOT DATA IN PYTHON


    What is IoT?
    IoT == Internet of Things

    Network of connected devices

    Measure and collect data

    Interact with environment

    ANALYZING IOT DATA IN PYTHON


    IoT Devices
    Connected devices Industrial connected devices

    Smart locks Connected machines

    Connected thermostats Robots / Cobots

    Temperature sensors Package tracking

    ANALYZING IOT DATA IN PYTHON


    IoT Data formats
    http / json

    plain text

    binary data

    XML

    Proprietary protocols

    ANALYZING IOT DATA IN PYTHON


    Data aquisition
    Data streams

    Gathered from a device

    API endpoints

    ANALYZING IOT DATA IN PYTHON


    Data aquisition - requests
    import requests
    url = "https://demo.datacamp.com/api/temp?count=3"
    r = requests.get(url)
    print(r.json())

    [{'timestamp': 1536924000000, 'value': 22.3},


    {'timestamp': 1536924600000, 'value': 22.8},
    {'timestamp': 1536925200000, 'value': 23.3}]

    print(pd.DataFrame(r.json()).head())

    timestamp value
    0 1536924000000 22.3
    1 1536924600000 22.8
    2 1536925200000 23.3

    ANALYZING IOT DATA IN PYTHON


    Data aquisition - pandas
    import pandas as pd
    df_env = pd.read_json("https://demo.datacamp.com/api/temp?count=3")

    print(df_env.head())

    timestamp value
    0 2018-09-14 11:20:00 22.3
    1 2018-09-14 11:30:00 22.8
    2 2018-09-14 11:40:00 23.3

    print(df_env.dtypes)

    timestamp datetime64[ns]
    value float64
    dtype: object

    ANALYZING IOT DATA IN PYTHON


    Let's Practice
    A N A LY Z I N G I OT D ATA I N P Y T H O N
    Understand the data
    A N A LY Z I N G I OT D ATA I N P Y T H O N

    Matthias Voppichler
    IT Developer
    Store data to disk
    Reasons to store IoT Data

    Limited historical data availability

    Reproducible results

    Training ML Models

    ANALYZING IOT DATA IN PYTHON


    Store data using pandas
    df_env.to_json("data.json", orient="records")

    !cat data.json
    [{'timestamp': 1536924000000, 'value': 22.3},
    {'timestamp': 1536924600000, 'value': 22.8},
    {'timestamp': 1536925200000, 'value': 23.3},
    {'timestamp': 1536925800000, 'value': 23.6},
    {'timestamp': 1536926400000, 'value': 23.5}]

    ANALYZING IOT DATA IN PYTHON


    Reading stored data
    From JSON les

    import pandas as pd
    df_env = pd.read_json("data.json")

    From CSV le

    import pandas as pd
    df_env = pd.read_csv("data.csv")

    ANALYZING IOT DATA IN PYTHON


    Validate data load
    Correct column headers

    Check Data formats

    df_env.head()

    timestamp humidity pressure sunshine temperature


    0 2018-09-01 00:00:00 95.6 1016.3 599.2 16.1
    2 2018-09-01 00:10:00 95.5 1016.4 600.0 16.1
    4 2018-09-01 00:20:00 95.2 1016.5 598.9 16.1
    6 2018-09-01 00:30:00 95.1 1016.4 600.0 16.1
    8 2018-09-01 00:40:00 95.3 1016.3 600.0 16.1

    ANALYZING IOT DATA IN PYTHON


    dataframe.info()
    df_env.info()

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 13085 entries, 0 to 13085
    Data columns (total 5 columns):
    pressure 13085 non-null float64
    humidity 13085 non-null float64
    sunshine 13083 non-null float64
    temperature 13059 non-null float64
    timestamp 13085 non-null datetime64[ns]
    dtypes: datetime64[ns](1), float64(6)
    memory usage: 1.4 MB

    ANALYZING IOT DATA IN PYTHON


    pandas describe()
    df_env.describe()

    humidity pressure sunshine temperature


    count 13057.000000 13057.000000 13057.000000 13057.00000
    mean 73.748350 1019.173003 187.794746 14.06647
    std 20.233558 6.708031 274.094951 6.61272
    min 8.900000 989.500000 0.000000 -1.80000
    25% 57.500000 1016.000000 0.000000 9.80000
    50% 78.800000 1019.700000 0.000000 13.40000
    75% 91.300000 1023.300000 598.900000 18.90000
    max 100.100000 1039.800000 600.000000 30.40000

    ANALYZING IOT DATA IN PYTHON


    Time for Practice!
    A N A LY Z I N G I OT D ATA I N P Y T H O N
    Introduction to Data
    streams
    A N A LY Z I N G I OT D ATA I N P Y T H O N

    Matthias Voppichler
    IT Developer
    What is a Data Stream
    Constant stream of Data

    Examples
    Twitter messages

    Online News Articles

    Video streams

    Sensor data (IoT)

    Market orders ( nancial)

    ANALYZING IOT DATA IN PYTHON


    What is a Data Stream
    Constant stream of Data

    Examples
    Twitter messages

    Online News Articles

    Video streams

    Sensor data (IoT)

    Market orders ( nancial)

    ANALYZING IOT DATA IN PYTHON


    MQTT
    Message protocol Message Queuing Telemetry Transport

    Publish / subscribe

    Small footprint

    Server -> Acts as a message Broker

    Client:

    Connects to a Broker

    Publishes data

    Subscribes to topics

    ANALYZING IOT DATA IN PYTHON


    Python library
    Eclipse Paho™ MQTT Python Client

    # Import MQTT library


    import paho.mqtt

    More information and the documentation available at GitHub


    https://github.com/eclipse/paho.mqtt.python

    ANALYZING IOT DATA IN PYTHON


    Single message
    import paho.mqtt.subscribe as subscribe
    msg = subscribe.simple("paho/test/simple",
    hostname="test.mosquitto.org")

    print(f"{msg.topic}, {msg.payload}")

    Output:

    paho/test/simple, {"time": 1549481572, "humidity": 77, "temp": 21}

    ANALYZING IOT DATA IN PYTHON


    Callback
    def on_message(client, userdata, message):

    print(f"{message.topic} : {message.payload}")

    Arguments

    client - client instance

    userdata - private user data

    message - instance of MQTTMessage

    ANALYZING IOT DATA IN PYTHON


    Callback
    import paho.mqtt.subscribe as subscribe

    subscribe.callback(on_message,
    topics="datacamp/roomtemp",
    hostname="test.mosquitto.org")

    ANALYZING IOT DATA IN PYTHON


    MQTT Subscribe
    import paho.mqtt.subscribe as subscribe

    def on_message(client, userdata, message):


    print("{} : {}".format(message.topic, message.payload))

    subscribe.callback(on_message,
    topics="datacamp/roomtemp",
    hostname="test.mosquitto.org")

    datacamp/roomtemp : b'{"time": 1543344857, "hum": 34, "temp": 24}'


    datacamp/roomtemp : b'{"time": 1543344858, "hum": 35, "temp": 23}'
    datacamp/roomtemp : b'{"time": 1543344860, "hum": 36, "temp": 22}'
    datacamp/roomtemp : b'{"time": 1543344946, "hum": 37, "temp": 22}'
    datacamp/roomtemp : b'{"time": 1543345010, "hum": 36, "temp": 13}'

    ANALYZING IOT DATA IN PYTHON


    Let's practice!
    A N A LY Z I N G I OT D ATA I N P Y T H O N

    You might also like