Friday 5 August 2016

Setting Up Spark - Installation of Java JDK, Scala, SBT Build Tool, Spark


Java JDK

  • Install JDK instead of JRE
    • We gonna be needing Java also for the development, such as Maven Projects.

Scala

  • Install Scala 2.11.7
  • Scala Lang Download
  • Add New Environment Variable
  • SCALA_HOME c:\scala\bin
  • Adding to Path Variable
  • %SCALA_HOME%\bin;
  • Test with Command Prompt
  • scala

SBT Build Tool

  • Install SBT on Windows (MSI Installer)
  • Add New Environment Variable
  • SBT_HOME c:\sbt

Hadoop(Optional)

  • Download and Extract
  • Add New Environment Variable
  • HADOOP_HOME hadoop-common-2.2.0-bin-master

SPARK

  • Download and Extract
    • 1.6.2
    • Pre-built for Hadoop 2.6
    • Direct Download
  • Navigate to the Folder via Command Prompt
  • sbt package
  • Navigate to Spark\bin Folder
  • spark-shell
  • Testing the Spark and Scala Configuration
  • val textFile = sc.textFile("C:/Spark/spark-2.0.0-bin-hadoop2.7/README.md") textFile.count()
  • Running the Spark Cluster on Windows, we have to start the Master and Workers
    • Installation of Cluster on Windows
      • Navigate to Spark\bin Folder in Command Prompt
      • spark-class org.apache.spark.deploy.master.Master
      • Spark is Up and Running at LOCALHOST [http://localhost:8080/] and Master will give [spark://HOST:POST - URL]
    • Now we initiate a Worker and Connect it to a Master
      • Navigate to Spark\bin Folder in Command Prompt
      • spark-class org.apache.spark.deploy.worker.Worker spark://192.168.0.100:7077
      • Thus we receive a Worker ID
        • You can see a WORKER listed under the WORKERS heading, one can explore the jobs that are currently in progress.
    • Now connecting Application with Cluster
      • Navigate to Spark\bin Folder in Command Prompt
      • spark-shell --master spark://192.168.0.100:7077
      • You can see a App with an ID is listed under the Running Applications heading.
        • Click on the Application ID to Navigate and explore further description of that particular job.
        • Right now its Blank as we have not started any Application Task. Let start with a simple application in our command prompt and have those detailed listed here in the Browser as well.
        • Atlast we Test a Simple App
          > val textFile = sc.textFile("C:/Spark/spark-2.0.0-bin-hadoop2.7/README.md")
          > textFile.count() // Number of items in this RDD
          > textFile.first() // First item in this RDD
          > val linesWithSpark = textFile.filter(line => line.contains("Spark"))
          > textFile.filter(line => line.contains("Spark")).count()
            // How many lines contain "Spark"?
        • Back to the Browser and Reload the Localhost Page
          • Click on the Application ID to Navigate and explore further description of that particular job.
          • Some of the basics regarding the App are listed, such as: Worker which is assigned for that job, Amount of Cores and Memory consumed by that particular App and the App Status.
          • Moreover, for Deatiled description click on the Link "Application Detail UI" and we can have Event Timeline and Progress bar representation of the Tasks which were in queue or are in progress.

Cheers ;-)

Wednesday 4 May 2016

ZeroMQ (ØMQ) Distributed Messaging
[Setting up Visual Studio C++ Project for Client/Server Publisher/Subscriber Push/Pull Pair Architecture]

ZeroMQ (ØMQ) is an extension/library for Distributed Messaging on the bases of several different Architecture and Concepts.
  • Client/Server 
    • This is the most common architecture of message passing in which client sends a request with some predefined commands and server responds accordingly. Also referred as Request/Reply Pattern.
  • Publisher/Subscriber 
    • Publish/Subscribe is one of the classic pattern in which Publishers are the message senders and receivers are referred as Subscribers. Here messages are not programedto be sent directly to some specific receivers, instead messages are published without the knowledge of the receivers and depend on the subscriber of that knowledge exists.
  • Push/Pull 
    •  As also referred as Pipeline Pattern, distribute commands to multiple workers, arranged in a pipeline. Push sockets will distribute messages to its Pull clients evenly. Here, the results computed by consumer(the one pulling the message) are not sent upstream but downstream to another pull/consumer socket.
  • PAIR
    •  It provides sockets that are close in behavior to conventional sockets. Conventional sockets allow:
      • only strict one-to-one (two peers)
      • many-to-one (many clients, one server)
      • one-to-many (multicast) relationships

Setting Up ZeroMQ (ØMQ)

Let's start with a ZeroMQ (ØMQ)Installer, you can download from their Website. I will suggest you to select a latest version according to your operating system. I have used ZeroMQ version number 4.0.4 for 64Bit Windows Operating System. After the installation, navigate the your installed directory and you can find further folders namely: bin, doc, include, lib and other files as per the installation.

Setting Up the C++ Visual Studio Project

For the sake of simplicity create a New Project for C++ Console Application and then follow these following steps.
  1. Including the Headers
    • Right click Project -> Properties -> C/C++ -> General
    • Edit 'Additional Include Directories'
      • Add C:\Program Files\ZeroMQ 4.0.4\include
    • Right click 'Header Files' -> Add -> Existing Item -> C:\Program Files\ZeroMQ 4.0.4\include\zmq.h
  2. Adding the Library Directory
    •     Right click Project -> Properties -> Linker -> General
    •     Edit 'Additional Library Directories'
      •     Add C:\Program Files\ZeroMQ 4.0.4\lib\
  3. Linking to the proper Library
    •     Right click Project -> Properties -> General
    •     Check "Platform Toolset"
      • notice the version number (ex: v120)
    •     Configuration Properties -> Linker -> Input
    •     Edit 'Additional Dependencies'
      •     Add appopriate toolset library (ex: libzmq-v120-mt-gd-4_0_4.lib) "-gd-" is the debug version
  4. Ensure your App is 64-bit
    •     Build -> Configuration Manager
    •     Active solution platform -> New...
      •     Select x64
      •     Copy settings from Win32
    •     Right click Project -> Properties -> Linker -> All Options
    •     Edit "Output File"
      • set to ".\x64\Debug\whatever.exe"
  5. Include the pre-compiled Dll as a resource for Distribution
    •     Right click Project -> Properties -> Build Events -> Post-Build Event
    •     Edit "Command Line"
      •     insert copy command
      • example: copy /Y "C:\Program Files\ZeroMQ 4.0.4\bin\libzmq-v120-mt-gd-4_0_4.dll" "$(OutDir)"
Now, the setup is almost done as per the conventional setting up projects for external libraries in Visual Studio but the project is still not up and running. There's a slight addition for the .hpp file which needs to be downloaded and added explicitly as last stand alone part.

Which can be download from their Github Repository, placed in the same include directory as mentioned in Step#1 under the heading Setting up the Visual Studio Project and of course include this .hpp file as an existing item.

That's all for the setting up the ZeroMQ (ØMQ), start your application and have fun. In next few days, I will try to set up the basic client and server application on ZeroMQ (ØMQ). Cheers ;)

Tuesday 9 February 2016

Data Pre-Processing for Visual Data Mining


Most Visual Data Mining tools use interactive 2D visualization methods in delivering the meaningful information from the data but when the high number of features, attributes and observations in a data set is large, the methods do not scale well and it is hard to visually determine the data set. This leads to the development of cooperative approach using a consensus theory based on feature selection algorithm, clustering for sampling and visualization for weight assignments in order to aggregate multivariate and multidimensional data sets. This allows visualization and then visual data mining without loss of classifier accuracy, thus resulting in a successful experiments with several high dimensional data sets.

  • Feature Selection aims at choosing a subset of attributes that is sufficient to describe the data set. This process includes identifying and removing the irrelevant and redundant information. This method can be very helpful for Visual Data Mining as it reduces the load of handling large amount of data.
  • Numerosity Reduction reduces number of objects based on their redundancy and description as expected by the experiment. 
  • Dimensionality Reduction helps in reduction of attributes, this not only reduces the number of iteration linearly but also exponentially and hence the name explains it reduces the dimension that are irrelevant for the particular set of visualization or analysis that need to be performed. 
  • Discretization and Quantization reduces the number of values for a given continuous attribute by dividing the range of the attribute into intervals. Interval labels can then be used to replace actual data values. Binning techniques is the most commonly used form of Quantization.
  • Generalization is the process which abstracts a large set of task-relevant data in a database from low conceptual levels to higher ones. Generalization has some limitations like handling only dimensions of simple non-numeric data and measures of simple aggregated numeric values, lack of intelligent analysis, it can`t tell which dimensions should be used and what levels should the generalization reach. But it also includes one of the most simple way of reduction, that is; Generalization and specialization can be performed on a data cube by roll-up and drill-down.
Besides aggregation and filtering which are applied over the large data set or database technologies, can also be carried out by using Data Mining techniques. These techniques may involve database operations such as aggregation, sampling, or other method resulting into classification of data or the clusters as an output.

Wednesday 20 January 2016

IONIC WiFi Plugin Example App using Phonegap/Cordova [Cordova WIFI Plugin ]


This is a simple app to implement a Simple WiFi Application with functionality using IONIC Framework on the top of Phonegap/Cordova. This application is designed for Iphone, Android and Windows Phone but can also be deployed as Blackberry Application. I name it as WiFiApp targetting the management of WiFi and their connectivity.

Setting Up the Project

Installation Required:
  • Node JS for NPM Packages
  • Cordova
  • IONIC Framework
  • Android SDK (for the deployment to the Device/Emulator)
  • IOS (you need a MAC for the deployment to the Device/Emulator)
  • And the involvement of Third Party Plugin "WiFi Wizard"

Create Project Directory

Here's a link to the Github Repository WifiApp. You can either download a zip or simply clone it with Git Bash.
git clone https://github.com/DynamicRemo/WifiApp.git

Plugin Installation

cd IonicAppCRUD
After the navigation to the Project Directory, we can run our project with ionic serve command and the output will be displayed in the Browser.
cordova plugin add com.pylonproducts.wifiwizard
Once the plugin is added to our application, we are good to test and deploy.

Platform, Build and Deploy

ionic serve
Or we can have the App View just like the Mock Ups for IOS and Android, its a new cool feature by IONIC.
ionic serve --lab
Moreover, we can add IOS and Android Platform to our Project and can directly deploy to our Device.             
ionic platform add android
ionic platform add ios
ionic run android
ionic run ios

Application Overview

Followings are the different views of Application: Searching for WiFi, Detected WiFi(s) and the success message for the Connection.

 




















For feedbacks and discussion, please leave a comment. Thank You!

Sunday 27 December 2015

App with CRUD Functionality - User Management using IONIC Framework Phonegap/Cordova [Tutorial/An Overview]


This is a simple app to implement CRUD (Create Read Update Delete) functionality using IONIC Framework on the top of Phonegap/Cordova. This application is designed for Iphone, Android and Windows Phone but can also be deployed as Blackberry Application. I name it as AppCRUD targetting User Management. I have also added some extra perks like visualizing the User Data using Charts.js and Angular Charts.

In this application I have implemented an additional functionality which is Directive. Directives are the part of AngularJS Framework which helps developer to extend the standard functionality. In this application, use of directive is applied to the "Add/Edit User Template" to focus the first Input Element in our View. You can have a look in the "/www/js/directives.js" and yes we have to add this directives.js file to our base html file that is "index.html" as well as to our Module as a Dependency in "app.js" file.


Setting Up the Project

Installation Required:
  • Node JS for NPM Packages
  • Cordova
  • IONIC Framework
  • Android SDK (for the deployment to the Device/Emulator)
  • IOS (you need a MAC for the deployment to the Device/Emulator)

Create Project Directory

Here's a link to the Github Repository IonicAppCRUD. You can either download a zip or simply clone it with Git Bash.
git clone https://github.com/DynamicRemo/IonicAppCRUD.git

Platform, Build and Deploy

cd IonicAppCRUD
After the navigation to the Project Directory, we can run our project with ionic serve command and the output will be displayed in the Browser.
ionic serve
Or we can have the App View just like the Mock Ups for IOS and Android, its a new cool feature by IONIC.
ionic serve --lab
Moreover, we can add IOS and Android Platform to our Project and can directly deploy to our Device.
ionic platform add android
ionic platform add ios
ionic run android
ionic run ios

Application Overview

Followings are the different views of Application: User Listing, User Add/Edit, User Delete, Reordering of User Listing and the Visualization of data using Angular Charts.

For feedbacks and discussion, please leave a comment. Thank You!

Thursday 30 July 2015

Simple Serial Communication App using Phonegap/Cordova [Tutorial]


It`s a simple tutorial to make Serial Communication App using Phonegap/Cordova and basically a boiler plate application which connects your Smart Phone such as; Iphone and Android serially with your hardware device. I name it as Simple Serial Communication App.
After the successful installation of NodeJS and Phonegap/Cordova, you guys can open the Command Prompt and start with me.

Create Project Directory

To create a new project of Phonegap/Cordova, we first need to navigate to the directory in which we want to have our Project using Command Line Interface and then we create our Project using this command.
phonegap create SimpleSerialApp com.dynamicremo.simpleserialapp SimpleSerialApp
   

Adding Android as a Platform, Build and Deploy

cordova platform add android
   
Here command Cordova is used instead of Phonegap, Cordova is soon to be form of Phonegap and Adobe is about to shift totally from Phonegap to Cordova. As our project has been created and Platform has been added, now Testing the Project to run as an Android Application.
Connect your device to your Computer/Laptop and navigate to your Project Directory. In my case its "D:\SimpleSerialApp" and then Execute.
phonegap run android --device
   
You can also build your project via command line and can deploy it via .APK File also with this build command and you can get the .APK File at this hierarchy of the project "D:\SimpleSerialApp\platforms\android\ant-build".
Here`s the output of the Snapshot from my Device.
phonegap build android
   

Plugin Installation

Moving forward in the Direction of Serial Communication, we will use a Phonegap Plugin Cordovarduino. This provide a really cool and simple call backs to connect our Cross Platform Mobile Application with the Hardware via Serial Communication.
To download this plugin and enable it for the use, navigate to the Project Directory "D:\SimpleSerialApp" and run this command.
cordova plugin add https://github.com/xseignard/cordovarduino.git
   
In result of this, you will get this Plugin in your Project`s Plugin Folder and you can see the new Directory "D:\SimpleSerialApp\plugins\fr.drangies.cordova.serial".

Setting Up

Now to keep it simple and fluent, we will add a new HTML File to our "www" directory "D:\SimpleSerialApp\www". I make it and named it as "serial_communication.html". Now for the flow of the Application, after our First Build. We can see the Splash Screen Page with Phonegap Bot, this is actually an "index.html" page placed inside www folder of our Project Directory D:\SimpleSerialApp\www" and is rendered with its Javascript file named as "index.js" from the Folder "D:\SimpleSerialApp\www\js". Open this file "index.js" and add these lines of code, so that we can redirect to our newly created Page "serial_communication.html" page and can continue with our Application.
window.setTimeout(function(){
 window.location = "serial_communication.html";
}, 200);
   
This code will preserve the index.html with its functionality and will redirect us to "serial_communication.html" page after the delay of 200 milliseconds. Place it right after the DeviceReady functionality. So the final Version will look like this.
// function, we must explicity call 'app.receivedEvent(...);'
onDeviceReady: function() {
    app.receivedEvent('deviceready');

    window.setTimeout(function(){
        window.location = "serial_communication.html";
    }, 200);
}
   
Now coming back to our newly added page for Serial Communication. Add its corresponding CSS File "serial_communication.css" for Styling to the Folder "D:\SimpleSerialApp\www\css" and also the JavaScript File "serial_communication.js" to the Folder "D:\SimpleSerialApp\www\js" and link them in the serial_communication.html page. I have also added some Jquery for usability purpose in InApp Browser, just like this.
<script type="text/javascript" src="js/jquery-2.1.1.min.js"></script>
<script type="text/javascript" src="js/jquery-ui-1.11.2/jquery-ui.js"></script>
<script type="text/javascript" src="js/jquery.ui.touch-punch.min.js"></script>
   
Now, coming to the HTML Elements and Application flow. I am adding a simple HTML Button, this button will connect my Android Device with the Hardware for Serial Communication
<button id="connect_button">Serial Communication</button>
   
Implementing the functionality of this Button, Adding the Connection command to its corresponding JavaScript File "serial_communication.js".
$(document).ready(function(){
 $("#connect_button").click(function(){
  serial.requestPermission(
   function success(){
    // BaudRate for our Chip: 115200, Adruino: 9600 
    var opts = {"baudRate":115200, "dataBits":8, "stopBits":1, "parity":0, "dtr":false}
    serial.open(opts, 
     function success(){
      alert("Success");
     }, function error(evt){
      alert("Error");
     }
    );
   },
   function error(evt){
    alert("Error");
   }
  );
 });
});
   
This is the code, that need to be in your JavaScript File "serial_communication.js". Here, the Application is requesting to the device to allow Permission for the connection and then in its Success Callback we are opening the connection. I have used here BaudRate of 115200 because this is the supported rate for my ChipSet, usually there`s BaudRate with 9600 and 9600 is the one which is also suitable for the Arduino. Now, its Build Time. I will again build my Project and directly deploy to my Device with the Command.
phonegap run android --device
   
After Build, final Testing of the Communication. We will be needing: 
  • Android Device
  • Hardware ware preferably Arduino
  • Connection Wire
  • USB to Go
To Test our application, Connect Mobile Device with the Hardware via Cable and USB to Go converter and run the Application.
As soon as you are on the serial_communication Page, you can see our attached Button with the Text "Serial Communication", Click that and you should get the HTML Prompt Box which will be Requesting for the Permission to Connect.

Accept that permission and then according to our JavaScript Callback Function we must get an Alert Box with the Message "Success" in it.




That`s All, you have got your first Cross Platform Mobile Application with Serial Port Communication, Cheers :)
Further more You can continue with the changes and modification, with Connect Button, TextBox for Sending the Desired Command, associated Send Button, Close Connection button and nice logging of Sending and Receiving Messages with beautiful OnsenUI Components. Or you can simply fork/download the whole Project from my GitHub.


For feedbacks and discussion, write a comment. Thank You!