Friday, January 29, 2010

More ideas and questions about Image Processing

Meeting Notes
---------------------

What about gigapan pictures?
- Would divide these images into smaller images, but not drop below a certain resolution (say, 800 x 600 for example)
- Game could be that each person tags sections of a large image.  Once they tag an image, they "collect" that image.  The collected images fill in the original large picture.  The person's end goal could be to guess what the large image is (or tag the large image with a general term like "landscape" or a specific name like "San Francisco")
- Game could have two modes and at the beginning the players choose to be either a Tagger or a Finder.  The Taggers draw squares around the objects they tag in the image.  The Finders try to find the object that has been tagged (all tagged squares are flipped over or hidden and the Finder must click on the square they think is the appropriate object to show what was underneath).


When tagging close-up images of people, do we want players to tag facial features like noses, eyes, and ears?
- If they draw a square around it, we can pull out the nose and show it in a search engine search
- We can extract out the cropped parts and show them to someone else for verification

-Use the fade-away memory game as confirmation to tags created in another game.

-Categorization of the image into - name, place, animal, thing

-How accurate do we want the tags to be?  If we use these tags in a search engine, we will be perpetuating the circulation of misinformation.

-Another mini-game could be to show players an image and the players must determine if a human is in the image or not.

-Could try converting colored images into black and white images, players then tag the objects with their appropriate colors (for example, a black and white image of the Flintstones -> players tag Fred's shirt as orange).

-a thought that crossed: the size of the objects in the real world has an impact on how the people perceive it in an image. splitting an image should be based on this.

Thursday, January 28, 2010

Advisor meeting notes - Image processing game ideas

Here are some of the ideas from John and Shirley, along with some inputs from our side

All the following ideas pertain to the Image recognition game:-

1. how about recognizing the objects based on a given category?
2. how about associating objects together in some way?
3. how about a game of memory?
        -show the picture, give some time for the people to see the image
        -cover the image now
        -ask the player to point to the place he thought that the object would be
4. how about using the silhouettes of the objects in the images in some way
5. how about the 6 difference game? two images with slight difference between them?
6. how about collaborating and creating new pictures from the objects found ?
7. how about coming up with a game based on the colour of objects? biggest object coloured red? find the objects based on their colour..
8. how about a Mystery game? sherlock holmes, clue based ? finding objects in the image?
9. how about associating the object with a piece of music? or a musical clue to guess the object?

some more tips and pitfalls:-
----------------------------------------
*-since game is seconday and the objective of problem solving is primary, is there anything other game that can be thought of. ofcourse, this can be an interesting and fun activity..
perhaps some place that people would not expect it to be
-make sure that there is confirmation/verification of the idea of the game
-we also figure out where in the image, the object would be placed
-know your audience
-think if everyone would tag the same way?
-can you use the cave? or the huge ball (whatever it is ) ?
-a game based on the advanced technology that lets computers determine where the people are seeing in the image.
-use filters to get down to what you want.

Wednesday, January 27, 2010

Project Idea Discussion - Image Processing


Team Meeting:

We discussed in length about the possibilities with the image processing. As mentioned in the client meeting, we wanted to get as many attributes as possible from the image.
So what we came up with was this:

1. A closed loop image subdivision algorithm based on tagging objects in images

Step 1: 
Lets assume we have a picture. any general picture. we decide to play a game of 20 questions. the player can identify and keep in his mind *any* object in the image.
Since 20 questions works on binary subdivision of possibilities, we are quickly able to reduce the item to what the player is thinking of:
1. this can be either done based on the kind of object itself. OR
2. it can be done by dividing the image into parts and asking the person if the object in his mind is in that part of the image.

Step 2: 
we are either going to guess the object in the players mind ( if we already have what the player sees in our database of metatags about this picture ) or we are going to ask the player to tag the object. once the player tags the object, we have a richer database to guess the subsequent player choices of objects.

we repeat Steps 1 and 2 in order for several players for the same image, thus resulting in a closed loop system that constantly betters itself by learning about the image from human players.

this is just a simplistic perspective and does not deal with the exact nature of game design. i am certain this can be made more exciting by using different lenses of game design. only the idea is being conveyed here. for example we could have an image like this tagged as shown.



there are so many objects that the brain sees that might take years of image recognition research to catch up to.

now objects are just nouns..
we could deal with actions that an object is doing.. verbs.. 
so we can have a reading lady in this image.. or static bike..  or walking man.
or perhaps adjectives like red light.. green light .. red bike.

so by taking a simple approach to this complex problem, we are generating huge amounts of meta data that can be highly relevant and useful for image search.
we could infact focus on one set of data.. like satellite imagery.. or something on that line.

as images become more dense ( more objects per inch ), this method of tagging becomes significant.. in fact the more complex an image is, the more fun the game is to play.

further we did consider the possibility of re-orienting an object in image to match a given plane.. so that we knew how the picture was w.r.t a normal plane.. but this is not a part of the core idea stated above.


Meeting Notes 2

Meeting with the Client:

- We are doing well -> we're on the right track with brainstorming.
- Interested in metatagging and assigning attributes to objects (in both images and sounds)
- In terms of defining the orientation of objects in pictures can go beyond just faces -> it can be used for architecture, cars, and other everyday objects to inform an image recognition software.
~ We could have a searchable database of images or videos; after teaching the software, we could then call up all instances of an object (like a telephone).
~ Identification of objects is key
~ Could narrow database to surveillance footage and have people identify what (who) is in it.
- In terms of tagging emotions/feelings in sounds, we could potentially tag the emotion in a voice clip.
- Not as interested in traffic flow problems, but they could have some potential -> just need to find a better optimization problem if this is the path we want to explore.
- We can define classes of objects -> can the player outline the object and then we extract the object?
- Glyph recognition -> we could then identify objects and replace them in an image with something else, or remove them entirely
~ Divide the image up into a grid - each player gets only a small piece
- The traveling salesman problem is a high-risk path to take, but with a potentially huge payoff

To Do:
- Deliverable: Write up in detail all the brainstorming and explain why we chose the path we chose.
- Categorize the ideas we've had so far
~ Look further in depth at some of the ideas we've had relating to image processing
~ Research more potential problems, potential games, etc...


Tuesday, January 26, 2010

Idea for the Project session 2

1. Identifying the classes of objects and their number in an image.

Given an image database this solutions attempts to find the objects that are present in the image, as well as the number of them present.
as humans we are good at recognising images of people or partial images. as in we can see a half cut table or a human who is facing backwards.
these seem to be things that computers would find very difficult to pattern recognize. 
hence if we give the players an image and ask them to recognize what they see in the image.. come up with a game mechanic that makes this fun, we will be generating classification data that can be used to further tag the images.

2. Tagging a feeling for a song or a radio.

we identify songs with the feelings taht they bring to us. however this is not the classification that we see on music labels.
by asking the player to classify the music he is hearing into one of the categories or a category of his own, we can have a better selection for music.
the categories can be based on feelings alone. the songs can be fed in live from the various radio stations across the globe.
this will create a useful database of information.. 
further if a music band wants to get a feedback for a song that it has composed, in terms of the feeling it intends to evoke, it could be made a part of this game.
they can get instant feedbacks on how they feel.
all this needs is an addictive game that is designed around the concept of music classification. people just hear songs and play the game.. perhaps there may not be a verbal indication of how the song feels but a visually generated world into which player transcends into.. .. .. .. 

3. Separation of noise and actual information.. voice, background vehicle sound. 

we know that dropping some item in the background when we are talking to someone does not make us turn around unless that item is large enough.. or its a person. we instantly know what is important. however a computer finds it difficult to classify what is noise and what is not. so given a random sound track it could be achallenge for a person to identify how many people are there or tag the different sounds that he can hear in the track. in essence he is eliminating the noise and is tagging only what is essential.
for example, if its a clip of a car moving on the road.. the player will rightly identify it as the car in the background.. and perhaps the sound of brakes.
humans build a context as they hear a track, but computers dont.. humans are context sensitive,hence they can analyse a given sound track in greater deths.

on my way back home i met a friend from CMU who is doing his PhD in image processing. as we talked, one of the things he mentioned caught my attention.

4. Orientation of an object in an Image

it is non-trivial to find the orientation of the image in an object. however as humans we are gifted with the ability to extrapolate the information that is not present in the image. like if there is a surveillance camera shot of a person who is walking in the subway, obviously he would not be facing the camera always. as humans we can figure out that it is the same guy no matter what is orientation is upto a certain point. but it becomes a complicated problem for a computer to solve. so determining the orientation of objects in images could be a challenging problem that can be crowdsourced. given two images with different orientations of the same object, its just a matter of few seconds for a human to recognize the object, but a mammoth task for the image recognition algorithm. this also lends itself as a good game to the players. so proceeding in this lines might be potentially a good idea. 

other ideas that we discussed:
either composition or orchestration of music
  - could be used as a teaching tool for children (educational)
  - Musical associations - colors, images



Monday, January 25, 2010

Idea for the Project session 1

1. Simulation of two countries, awareness of conflict for the player. result: a better citizen in the real world.

Lets model the countries each with input parameter R. This parameter determines the happenings in the world for a person.
R is the resource that is required by both the countries. Each person can vote to share a percentage (beta) of the resource with the other country. 0<=beta<=1. the world changes based on this vote of the person.
W = beta*R. 
We have a happiness meter for person P, which is again a weighted score of how good the overall world is currently as well as how good the person is locally.
H = x * localGood + y*GlobalGood.
localGood is the house that he might have build or the work that he might have done based on the resource R whose share he has got. while globalgood is the sum of happiness of all the people in the world. x and y are proportions that are determined as the game is played. the person's score is determined by how long he was at happiness level H for a time t.. 

Happiness at time t = Sigma ( Hi * ti ).
where Hi is the happiness at time ti.

The objective of the game is to score maximum happiness points. This game teaches conflict mediation for a person. It gives the person a holistic view of the world and how it operates. as the game progresses or as we introduce multiple parameters, we can see higher levels of emergent gameplay.

2. Optimal arrangement of roads for the traffic flow.

consider the following figure
   C
C X C
   C

here X is the player driving his car and C are the other cars that are front, rear, left and right of him. his goal is simple.
1. follow the car in front of him as long as its a straight path
2. avoid collision with cars around him
3. deviate from the path if need be. i.e. if he has to take a left or right turn.

When we see the overall traffic flow, we see the following figure.
 
C   C   C   C   C   C   C   C   C   C   C   C   C   C   C   
   C   C   C   C   C   C   C   C   C   C   C   C   C   C   C   
C   C   C   C   C   C   C   C   C   C   C   C   C   C   C   
   C   C   C   C   C   C   C   C   C   C   C   C   C   C   C   
C   C   C   C   C   C   C   C   C   C   C   C   C   C   C   
   C   C   C   C   C   C   C   C   C   C   C   C   C   C   C   
C   C   C   C   C   C   C   C   C   C   C   C   C   C   C   
   C   C   C   C   C   C   C   C   C   C   C   C   C   C   C   
C   C   C   C   C   C   C   C   C   C   C   C   C   C   C   
   C   C   C   C   C   C   C   C   C   C   C   C   C   C   C   

here each car is just following the simple rule 1 and 2. however we see that the rows of cars or traffic is regulating itself without any central point of focus.
given a route, we can analyse the flow of traffic in the street. we can analyse the bottle necks and rearrange them to see how different the traffic flow would be.
w.r.t a person, he is just playing a game where he sees 4 cars. but on the whole we see a huge difference to the traffic flow system.

3. The Travelling Salesman Problem

TSP is arguably one of the most famous algorithms and the most required one that is known not to have an exact solution in polynomial time. Given a set of points, human eye is very good at determining the visually closest neighbours. we could use this property to solve the TSP. if a large enough problem is broken into chunks and distributed among the people, they would find it interesting to solve.. as demonstrated by 
it would be worth looking for an idea that extends this principles to a vast crowd of people, entertainingly solving the problem.

4. Real parameters
when a user browses the network we become aware of several of his details like the ip address, locale, country, etc. how about using the game as a wrapper to find the details of these parameters. this is some kind of a meta information.

other ideas we talked about were, though not in elaboration
tom tom gps - they calculate the actual speed of a person by using the data from other gps's near by. so these are more reliable than the other gps in the market.
an interesting problem would be to get rid of viruses ( identifying virus signatures? ) by playing the game.

A Few Ideas

Here are a couple of ideas I came up with over the weekend. While these may not be what we are looking for, perhaps they will spur new ideas for everyone else.

The first is a sort of foreign policy model that explores how people would choose to deal with other nations. The game would take place in an imaginary land (think Middle Earth) that is broken up into different regions/nations. The players wouldn't know this, but the different nations' policies in this land would be initially modeled after real policies of different countries. As play progresses, players can modify their nation's policies (perhaps we could find a way to average out the 'votes' for changes so the policies reflect the most popular change). In this way, the game defines itself as people play - eventually the nations of the land would no longer represent the original models at all. It could be used to examine how different foreign policies work and how others fail.

The next idea would be a tool used to teach computers (maybe). Since the human brain is so good at visual calculations and predictions, we could make a game where people predict where a moving object will be in 5 or 10 seconds in the future. The computer could store these movement scenarios as experiences, which could later be used in collision avoidance for vehicles. A computer in a car could identify the movement of different objects and refer to these experiences to predict where the object will be, hopefully helping to prevent collisions.

This next link is a game that is sort of the reverse of the ESP game. In this game, the player is presented a series of images and he must guess the search keyword that would return those images. It's a different spin on the same sort of game.
http://grant.robinson.name/projects/guess-the-google/

I was also looking up National Archive transcriptions, to try to find more information about whether it is even a relevant problem when I found this site. It is in the British National Archives website and it is a series of pages that lead you through a tutorial in how to transcribe old documents. This link leads to the 'Practice' page, but once on that page the links on the left are for the 'tutorial' sections. I thought it was interesting and just wanted to share it with everyone.
http://www.nationalarchives.gov.uk/latinpalaeography/practice-documents.htm

Another idea I had touches on image recognition. People are good at identifying a representation of something as the thing itself (i.e. a stick figure is a person). We could make a game where people identify the object represented in the picture. This could be used to teach computers what rough drawings represent. Here's a link to a 'Similar Search' image search engine that allows you to draw what you want to search and it finds it (for example, try drawing a t-shirt and hit search). It's pretty amazing - I found this website last year when it was still in beta form. Looks like it's live now. It's fun to play with and it's pretty amazing all the stuff it can do!
http://www.gazopa.com/