Modern language models are trained on data with extremely uneven token distributions. A small number of words appear in almost every sentence, while many rare but meaningful tokens occur only ...

Gradient descent has a fundamental limitation: on most real-world loss surfaces, it is inefficient. When the surface has uneven curvature—steep in one direction and flat in another, which is common in ...

Abstract: In this paper, we address the challenges of asynchronous gradient descent in distributed learning environments, particularly focusing on addressing the challenges of stale gradients and the ...

Abstract: Distributed stochastic gradient descent (SGD) has attracted considerable recent attention due to its potential for scaling computational resources, reducing training time, and helping ...

Gradient descent is like hiking downhill with your eyes closed, following the slope until you hit the bottom (or at least a nice flat spot to rest). Technically, it is a method to minimize an ...

ABSTRACT: The development of artificial intelligence (AI), particularly deep learning, has made it possible to accelerate and improve the processing of data collected in different fields (commerce, ...

A new publication in Opto-Electronic Advances discusses efficient stochastic parallel gradient descent training for on-chip optical processors. With the explosive growth of the global data volume, ...