Utilizing the structure of a probabilistic model can significantly

increase its compression efficiency and learning speed. We consider

these potential improvements under two naturally-omnipresent

structures.

Power-Law: English words and many other natural phenomena are

well-known to follow a power-law distribution. Yet this ubiquitous

structure has never been shown to help compress or predict these

phenomena. It is known that the class of unrestricted distributions

over alphabet of size k and blocks of length n can never be

compressed with diminishing per-symbol redundancy, when k>n. We

show that under power-law structure, in expectation we can compress

with diminishing per-symbol redundancy for k growing as large as

sub-exponential in n.

For learning a power-law distribution, we rigorously explain the

efficacy of the absolute-discount estimator using less pessimistic

notions. We show that (1) it is adaptive to an effective

dimension and (2) it is strongly

related to the Good--Turing estimator and inherits its

competitive properties.

Low-Rank: We study learning low-rank conditional probability

matrices under expected KL-risk. This choice accentuates smoothing,

the careful handling of low-probability elements. We define a loss

function, determine sample-complexity bound for its global minimizer,

and show that this bound is optimal up to logarithmic terms. We

propose an iterative algorithm that extends classical non-negative

matrix factorization to naturally incorporate additive smoothing and

prove that it converges to the stationary points of our loss function.

Power-Law and Low-Rank: We consider learning distributions in

the presence of both low-rank and power-law structures. We study

Kneser-Ney smoothing, a successful estimator for the N-gram language

models through the lens of competitive distribution estimation. We

first establish some competitive properties for the contextual

probability estimation problem. This leads to Partial Low Rank,

a powerful generalization of Kneser-Ney that we conjecture to have

even stronger competitive properties. Empirically, it significantly

improves the performance on language modeling, even matching the

feed-forward neural models, and gives similar gains on the task of

predicting attack types for the Global Terrorism Database.

Pre-2018 CSE ID: CS2004-0790

Modern applications, including natural language processing, sensor networks, collaborative filtering, and federated learning, necessitate data collection from diverse sources. However, these sources may be tainted by untrustworthy, erroneous, or adversarial data. Moreover, even in the absence of corruption, the sources might not conform to a shared underlying distribution. They could be categorized into different groups, with distinct and arbitrarily varying data distributions.

For instance, consider movie recommendation systems where users rate movies. The ratings provided by different users can exhibit variations based on their genre preferences, highlighting the diversity in data distributions among sources.

In this thesis, we consider into a range of issues within the aforementioned contexts:\begin{enumerate} \item Robust estimation of structured distributions, both discrete and continuous. \item Robust classification. \item List-decodable regression \item Mixed linear regression with small batches \item Robust parameter estimation in graph settings. \end{enumerate} Previous approaches to these problems have suffered from limitations in terms of computational complexity, estimation accuracy, and sample complexity due to the presence of corrupted data sources.

This thesis introduces novel methodologies to address the limitations of previous approaches, focusing on robust learning from corrupted data sources. By doing so, it broadens the horizons for achieving precise distribution estimation, regression, classification, and parameter inference across diverse application domains.

PAC maximum selection (maxing) and ranking of $n$ elements via random

pairwise comparisons have diverse applications and have been studied

under many models and assumptions. We consider $(\epsilon,\delta)$-PAC

maxing and ranking using pairwise comparisons for \nobreak{general}

probabilistic models. We present a comprehensive understanding of

three important problems in PAC preference learning: maxing, ranking,

and estimating \emph{all} pairwise preference probabilities, in the

adaptive setting.

{\bf SST + STI:} We consider $(\epsilon,\delta)$-PAC maximum-selection

and ranking using pairwise comparisons for \nobreak{general}

probabilistic models whose comparison probabilities satisfy

\emph{strong stochastic transitivity (SST)} and \emph{stochastic

triangle inequality (STI)}. Modifying the popular knockout

tournament, we propose a simple maximum-selection algorithm that uses

$\mathcal{O}\left(\frac{n}{\epsilon^2} \log

\frac1{\delta}\right)$ comparisons, optimal up to a constant

factor. We then derive a general framework that uses noisy binary

search to speed up many ranking algorithms, and combine it with merge

sort to obtain a ranking algorithm that uses $\mathcal{O}\left(\frac

n{\epsilon^2}\log n(\log \log n)^3\right)$ comparisons for

$\delta=\frac1n$, optimal up to a $(\log \log n)^3$ factor.

{\bf SST +/- STI and Borda:} With just one simple natural assumption:

\emph{strong stochastic transitivity (SST)}, we show that maxing can

be performed with linearly many comparisons yet ranking requires

quadratically many. With no assumptions at all, we show that for the

Borda-score metric, maximum selection can be performed with linearly

many comparisons and ranking can be performed with $\cO(n\log n)$

comparisons.

{\bf General Transitive Models} With just \emph{Weak Stochastic

Transitivity (WST)}, we show that maxing requires $\Omega(n^2)$

comparisons and with slightly more restrictive \emph{Medium Stochastic

Transitivity (MST)}, we present a linear complexity maxing

algorithm. With \emph{Strong Stochastic Transitivity (SST)} and

\emph{Stochastic Triangle Inequality (STI)}, we derive a ranking

algorithm with optimal $\mathcal{O}(n\log n)$ complexity and an

optimal algorithm that estimates all pairwise preference

probabilities.

{\bf Sequential and Competitive} We extend the well-known

\emph{secretary problem} to a probabilistic setting, and apply the

intuition gained to derive the first query-optimal sequential

algorithm for probabilistic-maxing. Furthermore, departing from

previous assumptions, the algorithm and performance guarantees apply

even for infinitely many items, hence in particular do not require

a-priori knowledge of the number of items. The algorithm has linear

complexity, and is optimal also in the streaming setting and for both

traditional- and dueling-bandits. In a non-streaming setting, a

modification of the algorithm is \emph{competitive} in that it

requires essentially the lowest number of queries not just in the

worst case, but for every underlying distribution.

Modern data science calls for statistical inference algorithms that are both data-efficient and computation-efficient. We design and analyze methods that 1) outperform existing algorithms in the high-dimensional setting; 2) are as simple as possible but efficiently computable.

One line of our work aims to bridge different fields and offer unified solutions to multiple questions. Consider the following canonical statistical inference tasks: distribution estimation, functional estimation, and property testing, sharing the model that provides sample access to an unknown discrete distribution. In a recent paper in NeurIPS '19, we showed that a single, simple, and unified estimator – profile maximum likelihood (PML), and its near-linear time computable variant are sample-optimal for estimating multiple distribution attributes. The result covers 1) any appropriately Lipschitz and additively separable functionals; 2) sorted distribution; 3) R enyi entropy; 4) $\ell_2$ distance to the uniform distribution, yielding an optimal tester for distributions' closeness. This work makes PML the first unified sample- and time-optimal method for the learning tasks mentioned above. A single algorithm with such broad applicability is \emph{universal}.

Another line of our work focuses on instance-optimal learning that designs algorithms with near-optimal guarantees for every possible data input. A flagship problem is distribution estimation over discrete or continuous domains, where ordering and geometry play an essential role. Going beyond worst-case guarantees, researchers designed algorithms that compete with a genie estimator that knows the actual distribution but is reasonably restricted. To obtain state-of-the-art algorithms for both tasks, we leveraged the simple but nontrivial idea of ``data binning''. For discrete settings, we group symbols that appear the same number of times. And for continuous settings, we partition the real domain and separate symbols according to pre-designed local quantiles. The respective algorithms run in near-linear-time, achieve the best-known estimation guarantees regarding the genie estimators, and appear in ICML'19 and NeurIPS '20. A genie-like algorithm adaptive to almost every data sample is \emph{competitive}.

We present a comprehensive understanding of universal and competitive algorithms for multiple fundamental learning problems. Our ideas and techniques may shed light on key challenges in modern data science and numerous applications beyond the scope of this dissertation.

Pre-2018 CSE ID: CS2004-0810

Motivated by diverse applications in ecology, genetics, and language modeling, researchers in learning, computer science, and information theory have recently studied several fundamental statistical questions in the large domain regime, where the domain size is large relative to the number of samples.

We study three such basic problems with rich history and wide applications. In the course of analyzing these problems, we also provide provable guarantees for several existing practical estimators and propose estimators with better guarantees.

Competitive distribution estimation and classification:

Existing theory does not explain why absolute-discounting, Good-Turing, and related estimators outperform the asymptotically min-max optimal estimators in practice. We explain their performance by showing that a variant of Good-Turing estimators performs near optimally for all distributions. Specifically, for distributions over $k$ symbols and $n$ samples, we show that a simple variant of Good-Turing estimator is always within KL divergence of $(3+o_n(1))/n^{1/3}$ from a genie-aided estimator that knows the underlying distribution up to a permutation, and that a more involved estimator is within $\tilde{\mathcal{O}_n}(\min(k/n,1/\sqrt n))$. We extend these results to classification, where the goal is to classify a test sample based on two training samples of length $n$ each.

Estimating the number of unseen species:

We study species estimation, where given $n$ independent samples from an unknown species distribution, we would like to estimate the number of new species that will be observed among the next $t \cdot n$ samples. Existing algorithms provide guarantees only for the prediction range $t \leq 1$. We significantly extend the range of predictability and prove that a class of estimators including Efron-Thisted accurately predicts the number of unseen species for $t \propto \log n$. Conversely, we show that no estimator is accurate for $t = \Omega(\log n)$.

Learning Gaussian mixtures:

We derive the first sample-efficient \newline polynomial-time estimator for high-dimensional spherical Gaussian mixtures. It estimates mixtures of $k$ spherical Gaussians in $d$-dimensions to within $\ell_1$ distance $\epsilon$ using $\mathcal{O}({dk^9(\log^2d)}/{\epsilon^4})$ samples and $\mathcal{O}_{k,\epsilon}(d^3\log^5 d)$ computation time. Conversely, we show that any estimator requires $\Omega\bigl({dk}/{\epsilon^2}\bigr)$ samples, hence the algorithm's sample complexity is nearly optimal in the number of dimensions. We also construct a simple estimator for one-dimensional Gaussian mixtures that uses $\tilde{\mathcal{O}}({k }/{\epsilon^2})$ samples and $\tilde{\mathcal{O}}(({k}/{\epsilon})^{3k+1})$ computation time.

Pre-2018 CSE ID: CS2004-0811