Tracing and analyzing the performance of Hadoop MapReduce

Tracing, Analyzing and Benchmarking Tools for Distributed System

Nowadays, a large-scale distributed system is widely used for several purposes. For example, science, business, government and academia. In order to implement this large system that consists of a large number of modules, the system is implemented by different developed teams, meaning that every developers may not know every part of the system. They know some modules which they took part. Moreover, each module is working separately on different machine. To analyze the performance of the distributed system is a difficult task, since all the components and services are distributed in the cluster and performs several tasks at the same time. Therefore, in order to diagnose the performance or observe the behavior of the system, the diagnosis or tracing tools are the most essential factor which makes developer work easily. One of the popular distributed system nowadays is Hadoop MapReduce.

Hadoop MapReduce is a software framework that aims to process a large amount of data simultaneously. The framework consists of 2 main parts which are Hadoop distributed file system (HDFS) and MapReduce. HDFS is a file system which manages and monitors all the file in the cluster. MapReduce is a processing module which acquires the input from the HDFS and splits the data into pieces, called input splits. All the jobs submitted to the framework are called MapReduce jobs. Each map reduce jobs is divided into small tasks and scattered to each node on the cluster in order to execute in parallel-fashion. However there are only 2 main function in Map Reduce which are map and reduce function. The frameworks takes care of the scheduling tasks, monitoring and re-executes if the task failed. Finally, the result is written back to the HDFS. However, the platform itself automatically decides the appropriate number of the input splits including the number of mappers and reduces. Therefore, it is very difficult to observe what happen inside the framework in the developer point of view. In addition, to investigate the resource usages is also a difficult tasks because the mapper and reduce are dynamically assigned on the runtime, meaning that we cannot know where are they until the job is submitted. With this reasons, it became our objective and goal of this project.

Read more about this project here.

Close Project

The Design of Mobile-sourcing Framework for the Energy Efficient Computation with Multi-objective Ant Colony Optimization

Mobile Volunteer Computing

Volunteered computing is one of the most popular distributed computing conceptsin therecent years.The basic idea is to allow computer owners to donate computing power and storage to scientific applications. Currently, the advancement of the mobile technology has madepossible the utilization ofmobile devices for serious computations. In this research, we attemptedto exploit the power of mobile devices. We believedthat by outsourcing the computation, the energy consumption at data centers can be reduced while keeping the execution time within a reasonable deadline. We then designeda mobile-sourcing framework to utilize devices’ capability and capitalize on the CPU cycles contributed from mobile users under the concept of volunteer computing. Due to the availability of volunteers, hardware limitation, and wireless network connectivity, task scheduling becomes anothercrucial process. Thus, this research also emphasizes the design and the implementation of a scheduling algorithm for the framework. We adoptedMulti-Objective Ant Colony Optimization (MOACO) to find a good balance between energy consumption and the runtime. In order to validate the effectiveness of the scheduler, we performed a set of experiments. The results showed that the accuracy of the prediction models within the scheduler was approximately above 95% high for both energy and time. In other words, when the created schedules were executed, the actual observed time and the estimated time were similar. In addition, we found that the auto-generated schedules gained about 25% quality improvement with MOACO in comparison to the round-robin technique. Lastly, the scalability of the designed platform was investigated with different sizes of scientific workflows and different numbers of volunteered devices. From the results, it can be concluded that the quality of the created schedules was sufficiently good regardless of the workflow sizes and the number of devices.

Read more about this project here.

Close Project

The Automatic Question-answering Bot for the Virtual Online Store Platform

The Question-answering Agent for the Frequently Asked Questions

In this work, we are interested in chatterbot which is capable of answer the customer’s questions of the selling products. We adopt AIML or Artificial Intelligence Markup Language in order to handle the request from the user and produce the appropriate answer for each question. The prototype was designed and developed for answering the question in the specific domain. In this case, we assume that our agent is the online sale person for the Apple products, named “Jill”.

Read more about this project here.

Close Project

SkyPACS

Medical Image Solution for the Hospital

SkyPACS is a low cost and customizable mobile solution for radiologists and medical doctors to view medical images of any type in both 2D and 3D planes. The solution is designed to be an assistive technology with the focus on simplicity and user-experience. SkyPACS can be integrated to any existing Picture Archiving and Communication System or PACS.

With SkyPACS, the customers have the flexibility to choose backend services: cloud computing and storage on public cloud, private server, or hybrid system. A full SaaS option is also available. On the front end side, devices of any type including iOS, Android, Windows 8, and Desktop machines can access SkyPACS through the Internet. With the data security as our main concerns, SkyPACS is designed to leave a zero footprint on the mobiles devices. Patient data are stored and manipulated on the server and images are streamed to devices for a display purpose only.

Main features of SkyPACS includes:

• Zero footprint
• Capable of connect to any PACS
• All standard tools for viewing and diagnosis in 2D
• 3D viewing and printing
• Automatic tumor detection and reconstruction
• Convenient tools for collaboration – SkyLink and SkySync
• Case management tools (integrated to Radiology Information System)

You can read more about SkyPACS at novitat.co.th.

Close Project

The Smart House

Intelligent space for independent living

Smart house is designed to be an assistive platform for independently living of the elderly. The living space can be equipped with customized technology modules, such as anomaly detection (fall, irregular heart rate, etc.), easy-to-use communication medium, and video surveillance.

Smart house focuses on using existing technology to which elderly are already familiar with in their everyday life; hence, they can easily live in the intelligent environment without having to learn about new sophisticated tools. Sensors used in the current version of the system include microphone, speaker, monitor, camera, Kinect, and heart rate detector.

Main features of Smart house includes:

• Daily alarm and Medicine Schedule
• Automatic Sending of SMS and Telephone Call
• Long-distance Social Interaction
• Monitoring system through cameras
• Medical consultation
• Emergency button

You can read more about Smart House at novitat.co.th.

Close Project

ThonglorExchange.com

The website for the authorized money changer

This is a website of the Thonglor Money Exchange Co., Ltd. The website provides the information of the current currency exchange rate. In addition, the currency convertor is provided for the customer in order to estimate the amount of money based on the current rate.

Visit thonglorexchange.com here.

Close Project

Sentiment Analysis and Topic Modeling on Twitter Data

Mining users' opinions on Twitter about Donald Trump

Nowadays, social media, e.g Facebook or Twitter, becomes one of the most popular channel that people use for broadcasting messages or expressing their opinions. Millions messages are broadcasted everyday via these channels. People usually share comments and expreess their opinions in different aspects on the social network sites. Therefore, this huge amount of data is very useful if we can somehow gather these data and analyze them in a practical way. For example, the Apple company want to know the feedbacks from their customers after they announced a new iPhone in WWDC event. People that attend in the event or whoever that watches the event online usually post their opinion on the social network sites. The key idea is to find out the customers’ sentiment or the polarity of their products, which we can categorize into positive or negative.

Sentiment analysis is the the task which is to find out the polarity of that particular comment from text. Basically, the sentiment can be catagorized in 3 types, positive, negative and neutral. Positive sentiment is defined the comment which expresses the positive opinion. On the other hand, negative sentiment is defined the comment that expresses the negative opinion. Neutral sentiment is the opinion that does not belong to a positive or negative opinion. It can be a fact or some definition about something. To minimize the problem, we are going to classify the sentiment into positive and negative. We can formulate the problem to be a classification problem, meaning that given any text message, the classifier should be able to classify that message to be a positive or negative comment based on the model we have.

Another interesting thing about mining these data is to discover topics that people are taking about. Let use the example based on the previous one, the Apple event. The company get opinions of their customers from mining the data. However, the company also want to know what are popular topic that their customers are talking. For example, many customers may discuss about the design of the new iPhone or they might talk about a ground breaking feature that put on the phone.

With this motivation, it becomes the objective of this project. Recently, United States presidential election is one of the most topic that everybody is talking about, since they are a lot of potential candidates that want to help and solve problems in the country. Therefore, there are 2 main goals that we want to find out. Firstly, we want to know how the sentiment changes over the time, since there will be a lot of debates and campaign rallies. We want to investigate how these events affect people’s opinion, are they changing their mind from not support this candidate to support or vice versa. Second goal is to find out the topic that people talk about this candidate.

Read more about this project here.

Close Project

Product Feedbacks Survey using Sentiment Classification and Topic Modeling

Users' Feedbacks of New MacBook 12" Rose Gold after product launch

Currently, social network sites become one of the necessary things in our people’s life. There are several kinds of these sites. For example, Facebook, Twitter, Youtube, Google Plus and so many more. People spend time on these social network sites several hours per day. Mostly, they use the sites to communicate with their friends, family, co-workers or use for entertaining themselves. One of the common feature that most of these sites has is to allow the users to post anything they want to say or think about. It can be any format, e.g. texts, photos, videos. Usually, people post the comments or their opinions on the sites, meaning that this data is very valuable since the comments express how the people think about that particular topic.

In the business perspective, the company really want to know how their users think about the new campaign or product that just released. For instance, Apple releases a new Macbook with a new model and specification. They want to know the feedback from the users (like/dislike). Then, they can use this information to do some further analysis likes estimate the revenue, or investigate which functionalities should be improved or drop out from the next product. The traditional way to get this kind of information is to make a survey and randomly pick people for doing it. However, this process takes a lot of time and wastes a lot of money since they need to create the questionnaires and spend time finding the people. In addition, the result might not accurate because the sample size might not cover the whole population.

In order to reduce the cost and time for the process, the company can figure out their users’ opinions from these social network sites, since it is already directly come from their users. To tackle the problem, sentiment analysis helps us extract the users’ sentiment from the text data.

Read more about this project here.

Close Project

Communities Detection using Genetic Algorithm

Finding disjoint communities on a graph

In many complex systems, it can be represented as a network which consists of several nodes(vertices) and edges(links). Each node in the network represents actor or users of that particular system. Each edge represents an activity or connection between each pair of vertices. The connection can be friendship connection, following/follower, comments/likes and etc. One of the useful tools in the analysis of the large complex networks is community detection. The goal is to find the parts on the graph that densely connected to each other. Community means a group of vertices that share some common properties/interests, organizations, family, nations. Each community plays a similar role within the graph. For example, we can represent Syracuse University as a big undirected graph. Each vertex on the graph represents student, professor or staff. The edge that connects between each pair of vertices represent who know whom. It is likely that most of the students in the same department, e.g. Computer Science, will have more friends in the same department than having friends in other departments like Physics or Chemistry. Thus, we can define that the community in this case is the department where the actor belongs to.

Community detection gives a lot of useful application. One concrete example is finding the communities for the group recommendation system. Since we know that people in the same group have some similar properties or preferences, meaning that if we can find this such group in the graph, we can recommend a new product that match the interest of people in this group. For instance, we might want to recommend the technology news to the people in CS department and recommend the existing of Gravitational Wave news to the people in Physics.

Read more about this project here.

Close Project