Yahoo! Labs
Participation and Registration
Q. Can team members come from different organizations?
A. There is no restriction about the affiliations of team members. However, teams must be disjoint, namely each person can only participate in a single team.
Q. Can I compete even if I am not affiliated with an academic or industrial lab?
A. Yes, the competition is open to everyone. If you don't have an affiliation, just enter "self" while registering.
Q. What is the maximum number of team member?
A. Teams may be comprised of up to 10 members.
Data: Download, format, etc.
Q. Is there a sample code that I can use?
A. Yes. We provide a simple sample program that reads the data, produces a simple model and writes a file with predictions in the appropriate format. You can download it from the datasets page
Q. What is the reason for decoding the test predictions of Track1?
A. The test set of Track1 includes 6005940 examples. To reduce the size of the file to be uploaded to the challenge website, we decided to devote a single byte per prediction. Hence, ratings will be expressed as an integer in the range [0,255], encoded as an unsigned byte. The actual rating is given by the encoded one divided by 2.55. Thus, a predicted rating of 100 would be encoded as 255 (0xFF), and a predicted rating of 80 would be encoded as 204 (0xCC).
Q. How were unrated tracks chosen in the Track2 test set?
A. For each user we randomly sampled three tracks not rated by him. The odds of sampling a track is proportional to the number of high (>=80) ratings it got in the complete dataset.
Q. Can we merge the datasets of the Track1 and Track2?
A. The short answer is no. Here's why: The two datasets use a disjoint set of users and employ a different indexing scheme for users and items. Hence, no user appearing in Track1 appears also in Track 2, and vice versa. In addition, two items sharing the same index in the tracks are almost certainly not referring to the same entity.
Evaluation: Tracks, submission format, etc.
Q. Why does Track1 include separate validation and test sets?
A. Although participants could prepare their own training/validation splits for both tracks, we decided to add a validation set for Track 1 to simplify matters for the participants. The train-test split for Track 2 does not require timestamps, and the training dataset for Track 2 is considerably smaller, hence we decided not to further divide it into train and validation sets.
Q. How are the Leaderboard scores evaluated?
A. The test sets for both Track1 and Track2 are randomly divided into two disjoint equal sets: Test1 and Test2. Examples in Test1 are used for calculating the scores shown on the Leaderboard: RMSE for Track 1 and Error rate for Track 2. The examples of Test2 are reserved for choosing the winners of the competition. Hence, the possibility exists that team rankings on the Leaderboard will differ from the final results, which would be calculated on Test2.
Q. If I win in both Track1 and Track2, will I receive two prizes?
A. Yes.
Q. Is there a forum where participants may exchange information?
A. A Yahoo! Groups forum for exchanging ideas, insights etc. was established for this purpose.
Q. Why does the challenge restrict the number of submissions per day?
A. Each team is restricted to one submission per eight hours per contest track for two reasons: First, you should only use the training set for training your model and only moderately use the feedback provided on the Test1 set to enhance the training. The restriction on the number of submissions limits this risk of overfitting the set of examples used to produce the evaluations on the Leaderboard. Second, this limits the load on our servers.
Q. I am new to Recommender Systems. How do I get started?
A.
Recommender Systems have been a hot topic in both research and industry recently. Most prominently, the Netflix Prize solicited a lot of great approaches to the problem. In fact, more than we can mention here.
Thus, we suggest newcomers to start their research with those keywords: "Recommender Systems", "Netflix Prize", "Collaborative Filtering". Also, "recsys" is a commonly used tag / abbreviation on social sites such as delicious and citeulike.
Lastly, you can always join the discussion on our Yahoo! Group.
Q. Which submission is considered for the final winners' assessment?
A. The last submission on each contest track is the one used to measure and rank the team's performance.
Q. Where can I find the Official Rules?
A. To view the full Official Rules of the Contest click here.
Q. Who can I contact for other questions?
A. Please use our discussion forum (http://groups.yahoo.com/group/kddcup2011"). You may also contact the organizers via email at kddcup2011@yahoo-inc.com.