Scribd Logo
Upload

Welcome to Scribd!

  • 10 views

    Scala Spark

    Uploaded by

    Mastan
    The document discusses how functional abstractions from previous Scala courses can be mapped to computations over massive datasets on multiple machines. It notes that functional frameworks like Spark make scaling computations easier than imperative systems. Learners will analyze large data tasks like K-means functionally and see how they can be implemented in Spark. While many data science courses use languages like R, Python, Octave and MATLAB for small datasets, these don't allow scaling to large datasets without reimplementing algorithms from scratch in systems like Hadoop or Spark.

    Copyright:

    © All Rights Reserved
    Download as DOCX, PDF, TXT or read online from Scribd

    Scala Spark

    Uploaded by

    Mastan
    10 views2 pages
    The document discusses how functional abstractions from previous Scala courses can be mapped to computations over massive datasets on multiple machines. It notes that functional frameworks like Spark make scaling computations easier than imperative systems. Learners will analyze large data tasks like K-means functionally and see how they can be implemented in Spark. While many data science courses use languages like R, Python, Octave and MATLAB for small datasets, these don't allow scaling to large datasets without reimplementing algorithms from scratch in systems like Hadoop or Spark.

    Original Description:

    Original Title

    Scala_Spark

    Copyright

    © © All Rights Reserved

    Available Formats

    DOCX, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    The document discusses how functional abstractions from previous Scala courses can be mapped to computations over massive datasets on multiple machines. It notes that functional frameworks like Spark make scaling computations easier than imperative systems. Learners will analyze large data tasks like K-means functionally and see how they can be implemented in Spark. While many data science courses use languages like R, Python, Octave and MATLAB for small datasets, these don't allow scaling to large datasets without reimplementing algorithms from scratch in systems like Hadoop or Spark.

    Copyright:

    © All Rights Reserved
    Download as DOCX, PDF, TXT or read online from Scribd
    Download as docx, pdf, or txt
    10 views2 pages

    Scala Spark

    Uploaded by

    Mastan
    The document discusses how functional abstractions from previous Scala courses can be mapped to computations over massive datasets on multiple machines. It notes that functional frameworks like Spark make scaling computations easier than imperative systems. Learners will analyze large data tasks like K-means functionally and see how they can be implemented in Spark. While many data science courses use languages like R, Python, Octave and MATLAB for small datasets, these don't allow scaling to large datasets without reimplementing algorithms from scratch in systems like Hadoop or Spark.

    Copyright:

    © All Rights Reserved
    Download as DOCX, PDF, TXT or read online from Scribd
    Download as docx, pdf, or txt
    of 2

    how to map some of the functional abstractions that you've learned in previous Scala courses to

    computations on multiple machines over massive data sets.

    What is, we will see first-hand how the functional abstractions that we've covered in the previous Scala
    courses makes it easier and more user-friendly to scale computations over large clusters. Or easier, per
    se, than scalingcomputations on imperative frameworks, imperative systems fordistributedcomputation.

    we're always going to focus on analyzing large data sets. That is you'll be challenged to think about
    common data science tasks like K-means functionally, such as that they can be adopted to and
    implemented in the context of Spark.

    A functionally oriented framework for large scale data processing that's implemented in Scala

    you might beasking well, if we're going to be focusing on a lightweight data science flavor of the
    processing tasks, then why are we bothering with Scala and

    why are we bothering with Spark? After all

    if you want to learn data science in the classroom off of statistics professor's favorite languages or

    frameworks like R or Python or Octave and/or MATLAB.

    So then why should one bother running Scala or Spark which are both arguably very unlike R, Python,
    Octave and MATLAB? The answer is that these language and frameworks are good for data science in
    the small.

    Algorithms on data sets that are perhaps just a few hundred megabytes or even a few gigabytes in size.
    However, once the dataset becomes too large to fit into main memory on one computer, it suddenly
    becomes much more difficult to use one of these languages or frameworks alone.

    if your small dataset grows into a much larger data set than these languages and frameworks like
    R,Python, MATLAB, etc. They won't allow you to scale,you'll need to start completely from scratch
    reimplementing all of your algorithms using a system like Hadoop or Spark anyway. We'll need to
    manually figure out how to distribute your problem over many machines without the help of such a
    framework.

    Which is kind of a bad idea if you're not already an expert in building distributed systems.

    there's also this wholehuge massive industry shift towards data-oriented decision making. Nowadays,
    many companies across manydifferent industries have realized that by looking more closely at the data
    they'recollecting from device logs to health or genetic data, they can innovate in ways that were
    impossible before. For example, now we have all of these devices surrounding us, collecting information
    and attempting to provide all kinds of insights to enrich our day-to-day lives.

    instead, imagine hundreds of thousands of users of some device, say a smartphone or


    some wearable or something. And imagine as part of your job, you'rresponsible for providing some
    analysis or insight behind all of the data that's collected.

    You might also like