Sentiment Analysis

Every day we create over 2.5 quintillion bytes of data. This is over 20000 times the size of the English text version of Wikipedia. This is information from Facebook statuses to Tweets to product reviews to millions of different things. Now say that you are a company and you want to find out the public opinion on something be it a product, a politician, or a medical procedure. It is not easy to have a person read the equivalent of 20,000 wikipedias to decide if people like something or not. And while much of this data is irrelevant, it is often very hard to know where to look for your data. And even if you do, if you know you want to look at every tweet in the last week and find the opinion on something, well searching every tweet made in the last week would still take more man power than most countries let alone businesses can provide.

Here comes sentiment analysis. Sentiment analysis is a technique that uses computers to analyze text to judge opinion. Your initial thought might be "Well, that shouldn't be too hard, and computers work a lot faster than people." You would be slightly right and slightly wrong. Getting computers to recognize a human concept just from the words used is a very hard task. A first approach would be to look for positive or negative words in relation to your product. But what if someone says "I would hate for someone to live without this product" or "If you enjoy pleasant day of sizzling the skin off your feet or eating food so lively that you get dysentery, this vacation spot is the place for you!" Sarcasm is a complex linguistic process that many humans fail to understand, let alone machines.

And yet this is what these computers do. They analyze text to judge public opinion and companies look at the results and make decisions based on what they find. These techniques are incredibly versatile and used in a myriad of ways in the electronic world.

Time Series Modelling

Time series are incredibly useful tools for modeling systems. Time series are basically representations of variables that change over time. They can be used to model ocean currents, stocks, population, and pretty much everything that changes over time.

Construction of time series is done by analyzing past data for a number of trends. These things can be as simple as is the data cyclic as in does it repeat a pattern over some time interval. Or it can be more complex such as having various frequency dependencies that cause various smaller cycles to occur within a larger cycle.

Some time series are chaotic in nature meaning that starting out with similar but not equal initial conditions can yield large differences in their progressions over time. Many natural systems are chaotic such as water flow during a storm, double pendulum machines, or turbulence in a vortex.

Time series can also be used to model systems that change with respect to other variables over time. This way it can model things like the stock market which changes due to many variables such as inflation or earnings. Developing an accurate time series model then allows extrapolation to future events and allows for predictions to be made. This also shows some of the limitations of the theoretical uses because clearly we do not have accurate predictors of the stock market.

This occurs because there are so many variables that affect our system that we cannot perfectly model the system. Generally we settle for approximations of systems which gives us a general idea but does not give perfect results. We construct these models to allow general predictions to be made and we strive to improve our models as this gives us results that are closer and closer to reality.

SQL and You

If you have ever used the internet in any way you have interacted with SQL and probably don't know it. SQL is an amazing tool used in the backbone of the internet to access data. SQL is the communication line between data stored in a table and a user's screen where the data is wanted. Data such as your personal information on your facebook profile, your tastes as catalogued by Netflix, frequently searched terms from Google, or item types you've shown an interest in to Amazon. All of this information is stored neatly in tables in a server and SQL is the key to getting it where it needs to go.

SQL is a programming language that is a key part of website design. SQL commands can store information inputted by an individual, recall data from a table, and check various conditions in user and website variables to alter commands accordingly. SQL is highly versatile and can be incorporated directly into a websites backbone with html support. What this means is that the very code that describes the layout of a website can have pieces of SQL that handle movement of data from server to client.

SQL is heavily involved with targeted advertisements. Companies that have data on you such as facebook or google will store this information in a table. When web pages are loaded and advertisements are selected they are picked so that the advertisements have characteristics that have appealed to you in the past and have been documented in their databases.

Segmentation

Segmentation is a key goal of many service providing companies and is a large component of both my research project and my work with Axtria. Segmentation is the process of dividing up clientele into subgroups based on available characteristics. It has uses that range from identifying prospective donors to targeting ads based on an individual's interests.

Groups can be segmented in a number of ways. Say for example you have a list of prospective donors and a list of past donors with a number of characteristics for each. You may know for both groups things like age, estimated income, connections with the organization, and many other variables. And you also have information on the donations of past donors such as amount given, frequency of gifts and things like this. Segmentation would work by trying to find a correlation between characteristics we care about (donation information) and other information about the donor base so we can know which potential donors would be more probable to donate so that more time can be focused on them.

Another example would be how Netflix uses segmentation. Netflix has an immense user base which they have a variety of information on from age, to categories of interest, to individual media that they found incredible or horrific. Thus, when new people sign up for Netflix and list their interests Netflix pattern matches them toward television shows and movies that they are most likely to enjoy. As customers watch and rate more Netflix has a larger data set and can make better and better suggestions.

Segmentation is but one small tool used in data analysis but it proves very useful when applied to the right circumstances.  

Excelling with Excel

The most prominent thought that has occurred to me many times since starting my internship is the fact that Microsoft Excel is a beautiful tool to work with. While far from perfect, the number of amazing, clever, and plain useful features it brings is astounding.

At its core Excel appears to be just a sophisticated spreadsheet: a tool for putting data in nice little rows for easy consulting with a lot of high-tech looking features that are too complex for actual use. But after a little instruction, the intricacies of Excel reduce labor and minimize pain.

Excel can do basic things like sum the items in a column or multiply the values in a row but it has much more power than this. In a table listing population of European countries it can color code the largest and smallest, tell you the average population, graph the distribution of population, and tell you how many country names include the letter 'o'. With a table of financial information it can tell you what attributes contribute the most toward revenue, which factors are nearly irrelevant and how to generate the most profit.

Excel is a wonderful tool to analyze data though it does have its limits. For exceedingly large data tables Excel begins to run very slowly. While its user-friendly graphics tend to help with understanding they also require computer resources based on the number of fields entered. But even with this drawback Excel is a highly useful tool for basic analysis and it erases much of the drudgery from data crunching.

The Wonderful World of Corporate

This week I discovered that for having two parents that have spent the majority of their lives working in the corporate sphere I am remarkably ignorant of the corporate world's functionality. And I guess that after a lifetime of limited exposure to the consumer aspect of business life that this should not be such a surprise.

Over the past week I've been exposed to a number of facets of the modern business. From lectures on employee efficiency to conferences calls with heads of IT I have learned much about the internal structure of modern corporations.

This is a key feature to understanding how businesses use data analytics. Many different departments work together to produce useful output. R and D must interpret data and identify significance, Engineering must figure out how to manufacture this change, Marketing must find a way to publicize this technology. All these independent sub-units must cooperate and listen to each other to develop a useful product.

Information sharing is crucial for cooperation and it brings many issues to the table. Even the simple communication of data is complicated. Data may be in Excel Spreadsheets while the analysts might want to look at it in R. Getting many people from many organizations on the same page is a difficult task and involves much communication so that services can be smoothly rendered. And all of these things coalesce to make the corporate network a spiderweb of interconnecting chaos.

The Beginning of the End 1/21/14

As the second trimester winds down we seniors find ourselves preparing to journey far and wide to fulfill our Senior Research Projects. My journey is electronic rather than geographic as I work alongside the data analytics company Axtria and learn about the field of data analysis from the comfort of my own home as Axtria does most of their work online.

I am Ryan Smith, future student of Stanford University, passionate mathematician and novice computer scientist. Using the techniques I learn while working with Axtria will help me research how companies use data to target their customers, improve their business practices and make large sums of money. For more information on what I am researching you can view my entire proposal here: https://drive.google.com/file/d/0BzhSjAYafblcX3pzR1lxQlVyb2c/edit?usp=sharing.


Starting with the end of the second trimester on February 7th, I will start my project by beginning my internship with Axtria. I will also be investigating the various applications companies like Google, Amazon, Facebook and Netflix use to manage data and what they get out of it. This internship and research project will be my first step toward life as a college student studying mathematics and computer science at Stanford University!

You can also find more information on the company I am working with, Axtria, here: http://axtria.com/.