If you want to quickly develop a deep learning solution, there is a powerful combination you can use that leverages Spark and Tensorflow. Here in this article, Satyajit introduces the method of blending these two technologies to develop a deep learning toolkit.
Here is Satyajit’s description of the two tools and what they do:
Apache Spark™ is a unified analytics engine for large-scale data processing.
Speed: Run workloads 100x faster. Apache Spark achieves high performance for both batch and streaming data, using a state-of-the-art DAG scheduler, a query optimizer, and a physical execution engine.(DAG means )
Logistic regression in Hadoop and Spark
Ease of Use: Write applications quickly in Java, Scala, Python, R, and SQL.Spark offers over 80 high-level operators that make it easy to build parallel apps. And you can use it interactively from the Scala, Python, R, and SQL shells.
To learn how to write scripts performing ML in spark using python.
df = spark.read.json("logs.json")
df.where("age > 21").select("name.first").show()
#Spark's Python DataFrame API
#Read JSON files with automatic schema inference
Runs Everywhere: Spark runs on Hadoop, Apache Mesos, Kubernetes, standalone, or in the cloud. It can access diverse data sources.
TensorFlow makes it easy for beginners and experts to create machine learning models for desktop, mobile, web, and cloud. See the sections below to get started.
Easy model building: TensorFlow offers multiple levels of abstraction so you can choose the right one for your needs. Build and train models by using the high-level Keras API, which makes getting started with TensorFlow and machine learning easy.
If you need more flexibility, eager execution allows for immediate iteration and intuitive debugging. For large ML training tasks, use the Distribution Strategy API for distributed training on different hardware configurations without changing the model definition.
Satyajit Maitra, UseJournal
Robust ML production anywhere: TensorFlow has always provided a direct path to production. Whether it’s on servers, edge devices, or the web, TensorFlow lets you train and deploy your model easily, no matter what language or platform you use.
This powerful combination lets you run the solution virtually anywhere, since Spark and Tensorflow are widely supported on virtually every platform out there.
Find out more about this deep learning solution by visiting Satyajit’s post on Usejournal.
Learn more about deep learning and AI solutions.