Learning of classification models often relies on data that are labeled/annotated by a human expert. In general, more expertise and time the labeling process requires, more costly it is to label the data. In addition, there may be constraints on how many data instances one expert can feasibly label. Our goal is to find ways of reducing the number of labels and at the same time preserve or improve the quality of the models based on such labels. In this talk, I present two solutions we have developed to address the above problems. First, I present a learning framework in which the binary class label information that is typically used to learn binary classification models is enriched with soft-label information reflecting a more refined expert's view on the class a labeled instance belongs to. Second, I present a multi-expert learning framework that takes into account information on who labeled the instance in order to learn a better classifier.