PhD Seminar • Artificial Intelligence | Computer Vision • Collision Cross-Entropy for Discriminative Deep ClusteringExport this event to calendar

Thursday, May 4, 2023 — 11:00 AM to 12:00 PM EDT

Please note: This PhD seminar will take place in DC 3317.

Zhongwen Zhang, PhD candidate
David R. Cheriton School of Computer Science

Supervisor: Professor Yuri Boykov

We propose “collision cross-entropy” as a robust alternative to the Shannon’s cross-entropy in the context of self-labeled classification with posterior models. Assuming unlabeled data, self-labeling works by estimating latent pseudo-labels, categorical distributions y, that optimize some discriminative clustering criteria, e.g., “decisiveness” and “fairness”. All existing self-labeled losses incorporate Shannon’s cross-entropy term targeting the model prediction, softmax σ, at the estimated distribution y. In fact, σ is trained to mimic the uncertainty in y exactly. Instead, we propose the negative log-likelihood of “collision” to maximize the probability of equality between two random variables represented by distributions σ and y.

We show that our loss satisfies some properties of a generalized cross-entropy. Interestingly, it agrees with the Shannon’s cross-entropy for one-hot pseudo-labels y, but the training from softer labels weakens. For example, if y is a uniform distribution at some data point, it has zero contribution to the training. Our self-labeling loss combining collision cross entropy with basic clustering criteria is convex w.r.t. pseudo-labels, but non-trivial to optimize over the probability simplex. We derive a practical EM algorithm optimizing pseudo-labels y significantly faster than generic methods, e.g., the projectile gradient descent. The collision cross-entropy consistently improves the results on multiple self-labeled clustering examples using different DNNs.

Location 
DC - William G. Davis Computer Research Centre
DC 3317
200 University Avenue West

Waterloo, ON N2L 3G1
Canada
Event tags 

S M T W T F S
31
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
1
2
3
4
  1. 2024 (120)
    1. May (5)
    2. April (38)
    3. March (27)
    4. February (25)
    5. January (25)
  2. 2023 (296)
    1. December (20)
    2. November (28)
    3. October (15)
    4. September (25)
    5. August (30)
    6. July (30)
    7. June (22)
    8. May (23)
    9. April (32)
    10. March (31)
    11. February (18)
    12. January (22)
  3. 2022 (245)
  4. 2021 (210)
  5. 2020 (217)
  6. 2019 (255)
  7. 2018 (217)
  8. 2017 (36)
  9. 2016 (21)
  10. 2015 (36)
  11. 2014 (33)
  12. 2013 (23)
  13. 2012 (4)
  14. 2011 (1)
  15. 2010 (1)
  16. 2009 (1)
  17. 2008 (1)