Indicators and using noise as signal

​How should you use an indicator? When you look at the weather report, it seems pretty straightforward. Sunny means you do not need an umbrella. High of 36 means you should wear a coat. But what about when there is a 40% chance of rain? And what if you are trying to figure out if you can go for a hike this weekend?

When talking to my friend Chris Miller recently, he was trying to predict the weather before going on an aggressive hike. He mentioned that the noise in the forecast was part of his signal to decide how seriously he should take the report. If the forecast kept changing in the few days leading up to the proposed hike, that meant there was a decent chance that the weather would be unfavorable the day-of.

This got me thinking about the standard weather forecast as an indicator of the underlying data. I had a friend who was studying to be a meteorologist, and so she would go straight to the NOAA source data to predict the weather. For everyone else the weather report is basically a black box and all we have to work with are the indicators.

And when there are no indicators that answer your question? Or if it is simply impossible to  interpret the underlying data? That is when it is time to get creative with the information you do have.

Voting and an intro to some game theory ideas

My apartment complex decided to show a movie via projector and sent out a poll with 6 options. We were asked to rank the options and told that the movie that won would be shown at the movie night.

If management only got one response, it would be easy to decide how to vote and which movie won. However, assuming there was more than one response, how should the winner be determined? And given that there will be other voters, what should your vote be? In this game, management has decided on the rules for voting (rank the 6 options) as well as the rule for which movie is selected based on those votes (which they did not tell us). The voters are then left to decide how to vote, given their guess of the rules.

The most likely way to score such a set of votes is that management could give a set number of points for each possible rank-position (i.e., 5 points for being ranked 1st, 4 points for 2nd, etc) and then take the movie with the highest sum. However, there's no reason they couldn't pick the movie with the largest number of times being ranked 1st, and then use the later places as tie breakers. And in general, there can be truly crazy sets of rules. For example, if one movie is far-and-away the favorite in general, we could handicap that movie by saying any vote for it is actually only worth half a vote (this seems ludicrous... but in auction theory counting a "high-valuation" bidder's bid as a fraction of their actual bid is a standard tool to design an "optimal" auction).

Depending on the rules, an individual voter has different incentives. Further, depending on their guess as to how everyone else will vote, they will have additional incentives. An important notion in game theory is a "Nash Equilibrium." A NE is a set of votes for everyone so no individual person will choose to switch their vote. So if you knew that everyone else was following the NE, there is no benefit to you from not following the NE. But there are a lot of assumptions that go into the NE including that it is unique, that there will be no collusion (lets both list our shared second-favorite movie as 1st), and that somehow there being a NE actually leads people to vote accordingly (here is the Wikipedia link on when that will happen).

Given all this complexity, you might wonder how anyone ever decides anything. In a world where so much is uncertain though, a lot of decisions could be the best. If I vote my genuine ranking, I'm at least giving my preferred movie it's best shot to be selected. I could vote strategically and rank my 3rd favorite as top because I think my least favorite movies are the most popular. I could also not vote at all because I think the effort involved in voting is more than the difference between the outcome when I vote or not. Which of these guesses of uncertainty is right is impossible to say until after the votes are in, which are then influenced by what everyone else is guessing to be the case.

Hopefully, this gives a taste of some of the difficulty both in designing the rules of a game, and the subsequent decisions by the voters. Early on in learning about game theory I was told "the devil is in the details," which I have found to be absolutely true. First-past-the-post voting seems sensible, until you realize the incentive issues when there are more than two choices.

Feel free to send me any follow-up game theory questions you have and I will do my best to get them answered!

Classification problems

A friend asked me for ideas of a good analogy for classification problems in machine learning. In a classification problem, we have a collection of objects, and we somehow want to separate them into groups. Ideally, when designing this analogy there are a few things we want to convey:

• Not all properties of objects are equal when it comes to classification. Some will be highly predictive, while others just help you over-fit your model.
• Some properties will be highly correlated, so it can be a waste of energy / data effectiveness to include them all.
• Your desired classification informs which properties you should use for your classification.
• What do "properties" even look like, and how do they help us get at a classification?

The example I suggested is a person deciding what to eat at a potluck. I usually have two different classification problems to worry about when I'm filling my plate. First off, I want to decide which things I want to eat. In addition, I'm one of these people who will get a main course plate, eat that, and then go get dessert later. So as I survey the food, I have to decide both which things I think will be delicious, and which things I want to get later as dessert.

When trying to figure out what will be delicious, there are a lot of criteria I could use. Since I do not like cucumbers, anything with them is immediately excluded. Other properties besides ingredients could be smell, color, how much of it is available, how much was already eaten, the temperature of the food, anything! Some of these criteria are more helpful than others. I've had a lot of delicious brown things in my life at potlucks. And when I am trying to decide if something is dessert of not, how much people ate is not going to be very informative.

How do you classify food at a potluck? 

Organized Brainstorming

When you learn about brainstorming, there is often a focus on how "spontaneous" it should be. Don't worry if an idea is good or bad, just add it to the list! During undergrad I took several of Tau Beta Pi's "Engineering Futures" classes. One of the topics was a modified approach to brainstorming.

With the mindset they taught, the quality of the ideas are still unimportant, but you go about producing them in an organized way. Say you are trying to come up with the list of things you need to buy from the store. Instead of simply writing things down, you create categories and then fill in each category. For our shopping example, you might list each room in your house as a category and then think about what things you need for each room. If you are trying to figure out the best way to solve an engineering problem at work you might have categories like "new equipment" and "better software."

What are your thoughts on brainstorming? After a quick read through this link on the topic, it looks like I am describing a more task-oriented version of brainstorming. Which makes sense for an engineering-focused training on the subject.

Using decision trees for sequential decision making

I am a planner by nature. As I get close to leaving my apartment of 6 years to go a third of the way across the country, I've found planning a move to be a complicated project. For most of the past year I have tried to avoid thinking about the move since there was a lot of uncertainty which would be resolved with time. With the move less than a month away there is still a lot of uncertainty, but very little of it will be resolved until weeks, months, or years from now.

Since my research is in stochastic optimization (making decisions under uncertainty), in principle I have a lot of tools at my disposal. But most of the complexity of a move is the sequential decision-making aspect. Initially you have a lot of degrees of freedom. With each decision you eliminate not only the alternatives to that choice, but also the feasibility of any decisions which come later.

In my research I have not come across tools to handle this aspect of empirical decision making. However, with a little thinking outside the box, decision trees can! Decision trees let you visually consider what will happen in a world after a series of decisions have been made and uncertainty resolved. For a situation where the order of decisions matters, you can then draw multiple decision trees for each of those orderings.

I only explicitly drew decision trees for a couple aspects of my move. But for the early decisions, lots of flexibility down the road was an important factor. There were also a couple cases where I did draw the tree so I could articulate the relationship between specific uncertainty and my decisions. In one case I found that the uncertainty I was worried about did not actually change my choice!

In somewhat related news, hopefully I will get back to my once-a-week schedule for posting soon.

Link on stats and journalism

I've linked to his blog before, but this post by Andrew Gelman is a great read. Whether you happen to be a journalist, someone trying to understand how to communicate statistics results to the public, or just want to know what is missing from the standard "science says x cures y!" news stories.

Back to business

After defending (and revising) my thesis late last month, I'm back to blogging. This week I was in Orlando for the INFORMS Analytics conference. During the conference they host the Edelman competition with the goal of recognizing a project with huge tangible contributions made possible by OR and analytics.

I made it to three of the six talks, and enjoyed hearing about what people are doing with data.

• The NYPD has developed "DAS" which integrates with department phones and provides real-time information about an address and the surrounding area before police even show up at the scene of an event. They also use the data to identify likely locations for crime to target extra enforcement (it seemed somewhat less sinister than minority report).
• Until very recently, UPS still had their drivers plan their routes each day. ORION changed that with major cost savings. One of the most interesting parts to me was that before they could roll out this project, they needed to collect much more accurate map data to avoid the problem of "Google maps tells me to drive a mile out of the way to make a u-turn."
• 360i is a marketing firm which specializes in paid search advertising. Keyword-based advertising at Google is allocated based on the result of an auction. However, there are literally hundreds of thousands of strings that a particular company might want to bid on. 360i developed a set of tools to improve the effectiveness of search advertising by trying to figure out what people intended to search for (their example was you don't want to waste advertising dollars on someone trying to figure out how to sort out relationship problems)

UPS was chosen as the winner this year, and will be joining an impressive list going back to Pillsbury in 1972.