10-701, 15-781: Machine Learning

Hi all,

We've entered the final grades in S3, where you should be able to see them fairly soon.  I also updated the grade statistics (@302) with additional information like the mean and variance for the total of all online HWs (requested in a different thread).  There were a few places where the mean and standard deviation changed significantly due to people dropping or switching to audit; in those cases, we kept the old mean and standard deviation values, since those resulted in more favorable grades.

Here is the algorithm we used:

• put a lower cap on each of HW1..4, OHW1..7, and all project components: if your score was more than 2 sigma below the mean, we raise it to 2 sigma below the mean
• z-score each of HW1..4 and (OHW = sum of OHW1..7)
• compute homework score: HW = (HW1_Z+HW2_Z+HW3_Z+HW4_Z+OHW_Z)/5
• z-score each midterm
• compute midterm score: M = (midterm1_Z+midterm2_Z)/2
• z-score each project component
• compute project score: P = 0.15*proposal_Z + 0.25*interim_report_Z + 0.4*final_report_Z + .2*poster_Z
• compute total score: HW + M + P
• apply grade cutoffs of: 3=A+, 2=A, 1=A-, 0=B+, -1=B, -2=B-, -3=C+, else F (e.g., if you scored 2.7, you would get an A, while -1.53 would result in a B-)
• add in extra credit: up to 0.5 total points if you scribed and scored perfectly on the bonus HW, potentially moving you across one of the grade cutoffs above.

For means and standard deviations, see @302.

Hi all,

Our original algorithm for normalizing project midway grades wound up with some unexpected behavior: basically we didn't regularize the right way, and so some teams wound up getting their score lowered by an unfair amount.  So, we've redone the normalization with hopefully-better regularization (the result of an extensive discussion among TAs and instructors...); the new scores should be posted shortly, and should be fairer to teams whose TA had either high mean or low variance (or both).  Since the original mistake was ours, we're giving everyone $max(\text{old score},\text{new score})$ (although for most teams the max didn't change things much).

We are also planning to do for each project component the "bad day safety net" that we have done for some other assignments: i.e., if you score below -2 z-units, your score is raised to -2.

(EDIT: Abu) The new grades should be up on Autolab now. Again, we fit a hierarchical model to the grades, but with more careful regularization. The newly fitted per-TA means and standard deviations are given below:

• Abu: $24.3\pm 3.2$
• Ada: $27.0\pm 2.5$
• Anthony: $25.3\pm 2.9$
• Ben: $27.5\pm 2.5$
• Nicole: $25.2\pm 2.6$
• Zichao: $27.3\pm 2.5$

To be more robust to outliers, instead of the minimax approach we used last time, we rescaled the scores using the fitted upper-level mean of the hierarchical model: $26.3$, with the constraint that the maximum grade is 30. The class-wide statistics have also been updated @302.

And statistics are @302. The autolab grade you'll see will be out of 30, since two TAs graded your poster, each out of 15 points. 

Cheers,

Ada

Greetings from Canada,

Due to our presence at the NIPS conference, our office hours are cancelled this week. We should be fairly responsive to emails, though. Good luck with your final reports!

Nicole and Ben

Hey all, here's the final report rubric:

Question/Problem Formulation (10 points): Is the scientific question of interest well-formulated as a machine learning problem? Is it clear what the purpose of the project is, and how machine learning fits into the picture?

Motivation/Background (5 points): Is the scientific question of interest well-motivated? Is some review of the problem domain (as well as past attempts at solving the problem) provided?

Approach (10 points): Are the techniques applied appropriate to the problem of interest? Was something attempted beyond simply applying techniques out-of-the-box? Is there evidence of thought and creativity in the approach to the problem?

Experiments (10 points): Were experiments conducted that demonstrate the success (or failure) of the selected techniques? Were experiments conducted that adequately answer the scientific question of interest? Are the experiments statistically rigorous and free of any circularities?

Interpretation of Results (10 points): Are the conclusions drawn borne out by the results? Is there some attempt to relate the results to the bigger picture (e.g. how it relates to previous attempts at answering the question)?

Polish (5 points): Is the document well-written, well-organized, and free of typos and similar errors? Are font sizes and line widths in plots large enough? Are plots clearly labeled and figures clearly captioned?

Note: Your final report should be in the style of a 8-10 pages journal paper, in content, structure, and formatting. A typical journal paper is structured roughly as:

• Introduction - Motivation
• Problem definition
• Proposed method
  • Intuition - why should it be better than the state of the art?
  • Description of its algorithms
• Experiments
  • Description of your testbed; list of questions your experiments are designed to answer
  • Details of the experiments; observations
• Conclusions

Formatting is typically done in latex - though you are free to format your paper in other programs such as Word, we highly encourage you to use Latex as learning it will come in very useful in your future career. You can use the NIPS format as a starting point (latex template here). Please don't forget to include the andrew ids of each student on the project.

Hi all,

I finished graded hw3 problem2. I made a mistake in sending the pdf files, a following correction email was sent with the right pdf file. Sorry for the trouble caused. Any problems, let me know.

best,

zichao

Hi all, 

Sorry for the delay, the midway report grades are finally out. Check out @302 for statistics. 

As before, we normalized the grades to account for variations between TAs (and also to be out of 30pts), so your final grade might be slightly different than what your TA told you. 

Cheers,

Ada

(Abu) I'm adding additional details about the normalization: As in @354, we fit a hierarchical model to the grades. The following are the fitted per-TA means and standard deviations:

• Abu: $22.03\pm 3.51$
• Ada: $27.72\pm 1.61$
• Anthony: $24.56\pm 3.04$
• Ben: $28.80\pm 1.12$
• Nicole: $24.19\pm 1.99$
• Zichao: $28.13\pm 1.63$

The per-TA z-score was computed using the above values. A new mean and standard deviation were selected such that the maximum difference between the original grade and the normalized grade was minimized (and the maximum grade is a 30).

Hi everyone,

Today is the poster session. Please try to arrive early if possible to set up (atrium is reserved starting at 3pm). To facilitate grading, we would like you to put up your posters by TA mentor. There will be signs telling you where to put your poster up depending on who was mentoring you.

Post @592 has all the other information you need.  :)