Growing ecological studies suggest chronic exposure to air pollution exacerbates risks of COVID-19 incidence and mortality. This study assessed the associations between air pollutants and COVID-19 incidence and mortality in Los Angeles. 2019 annual mean exposure to air pollutants, including PM0.1 total mass, PM2.5 total mass, PM2.5 elemental carbon (EC), PM2.5 tracer from mobile sources, NO2, and ozone in 2019 were estimated at the ZIP code level based solely on residential areas. Negative binomial models and spatial models were used, considering each pollutant individually and adjusting for another pollutant. Exposure to PM0.1 mass, ozone, NO2, and PM2.5 EC were identified as risk factors for COVID-19 incidence and mortality. The results also suggest that air pollution has synergistic effects on COVID-19 outcomes. The study provides localized insights into the spatial and temporal associations between species-specific air pollutants and COVID-19 outcomes, highlighting the need for policy recommendations to mitigate air pollution.

Sequential decision-making problems arise at every occasion that agents repeatedly interact with an unknown environment in an effort to maximize a certain notion of reward gained from interactions with this environment. Examples are abundant in online advertising, online gaming, robotics, deep learning, dynamic pricing, network routing, etc. In particular, multi-armed bandits (MAB) model the interaction between the agent and the unknown environment as follows. The agent repeatedly acts by pulling arms and after an arm is pulled, she receives a stochastic reward; the goal at the end of this process is to select actions that maximize the expected cumulative reward without knowledge of the arms’ distributions. Albeit simple, this model is widely applicable. On the other hand, many sequential decision making occasions deal with more complicated environments modeled through Markov Decision Processes (MDPs) where the environment’s status constantly changes as a result of taking actions and makes learning even more challenging. The field of reinforcement learning (RL) defines a principled foundation for this methodology, based on classical dynamic programming algorithms for solving MDPs.

Our research goal is to expand the applicability of bandit and RL algorithms to new application domains: specifically, safety-critical, lifelong and distributed physical systems, such as robotics, wireless networks, the power grid and medical trials.

One distinguishing feature of many of such “new” potential applications of bandits and RL is their safety-critical nature. Specifically, the algorithm’s chosen policies must satisfy certain system constraints that if violated can lead to catastrophic results for the system. Importantly, the specifics of these constraints often change based on the interactions with the unknown environment; thus, they are often unknown themselves. This leads to the new challenge of balancing the goal of reward maximization with the restriction of playing policies that are safe. We modeled this problem through bandits and RL frameworks with linear reward and constraint structures. It turns out that even this seemingly simple safe linear bandit and RL formulations are more intricate than the original setting without safety constraints. In particular, simple variations of existing algorithms can be shown to be highly suboptimal. Using appropriate tools from high-dimensional probability and exploration-exploitation dilemma, we were able to design novel algorithms and to guarantee that they not only respect the safety constraints, but also have performance comparable to the setting without safety constraints.

Recently, there has been a surging interest in designing lifelong learning agents that can continuously learn to solve multiple sequential decision making problems in their lifetimes. This scenario is in particular motivated by building multi-purpose embodied intelligence, such as robots working in a weakly structured environment. Typically, curating all tasks beforehand for such problems is nearly infeasible, and the problems the agent is tasked with may be adaptively selected based on the agent’s past behaviors. Consider a household robot as an example. Since each household is unique, it is difficult to anticipate upfront all scenarios the robot would encounter. In this direction, we theoretically study lifelong RL in a regret minimization setting, where the agent needs to solve a sequence of tasks using rewards in an unknown environment while balancing exploration and exploitation. Motivated by the embodied intelligence scenario, we suppose that tasks differ in rewards, but share the same state and action spaces and transition dynamics.

Another distinguishing feature of the envisioned applications of bandit algorithms is that interactions involve multiple distributed agents/learners (e.g., wireless/sensor networks). This calls for extensions of the traditional bandit setting to networked systems. In many such systems, it is critical to maintain an efficient communication among the network while achieving a good performance in terms of accumulated reward, usually measured as network’s regret. In view of this, for the problem of distributed contextual linear bandits, we prove a minimax lower bound on the communication cost of any distributed contextual linear bandit algorithm with stochastic contexts that is optimal in terms of regret. We further propose an algorithm whose regret is optimal and communication rate matches this lower bound, and therefore it is provably optimal in terms of both regret and communication rate.

The crashes that cause fatalities and serious injuries typically occur at high speeds. Although the Federal Highway Administration (FHWA) asserts a need to identify the most dangerous locations on a roadway system, FHWA promotes a screening practice that fails to reveal these high-speed (high-risk) locations. In fact, the current practice obscures the most dangerous locations by conflating high-speed crashes with low-speed (low-risk) ones. The current practice supported by FHWA has two problems. Firstly, it analyzes crashes in uncongested and congested conditions together, without distinction. Secondly, it uses Vehicle Miles Traveled (VMT) as the measure of traffic exposure which cannot capture the vehicles’ extra time spent traveling in slow-moving congestion. These problems arise when the current practice is applied to freeways that see their fair share of congestion: the current practice tends to mistakenly identify locations with large numbers of low-speed (low-risk) crashes in congestion as if they were the most dangerous locations. The reason is that crash rates inside congestion are much higher than those occurring in uncongested traffic conditions.

To remedy these problems with current practice, the present study analyzes crashes in a disaggregated fashion based upon the vehicle occupancies measured by the nearest freeway loop detector at the time of the crash. High-speed crashes in uncongested conditions are analyzed separately from low-speed crashes in congested conditions. Crash counts for both traffic conditions were normalized by Vehicle Hours Traveled (VHT), rather than the commonly used Vehicle Miles Traveled (VMT), to capture vehicles’ extra exposure inside congestion. The data used for this research came from a 6-month period in 2016 on a 10-mile stretch of the northbound Interstate 880 freeway in Alameda County, California. Crash records collected by state police and traffic data measured by the site's loop detectors were also used as inputs.

The results show that by using the proposed method, one can better identifies the type of crashes, (uncongested and/or congested), that make some locations stand out as problematic. From there analysts can narrow down the set of these outliers to be the ones driven predominantly by uncongested crashes and are therefore the ones of the greatest safety concern. Additionally, one can rank uncongested outliers based on their magnitudes and focus on the highest ranked ones. For the reasons mentioned above, the proposed approach allows traffic agencies to focus their resources on these most dangerous locations and help improve the overall safety on the transportation system.

The amount of data used for this study is limited. But the findings are very promising. Future research to advance the proposed method and to answer questions that are yet unresolved are discussed.

How to achieve efficient reinforcement learning in various training environments is a central challenge in artificial intelligence. This thesis investigates this question on the spectrum of environments from the most idealized type to a fairly realistic one. We use two characteristics to describe the complexity of an environment: 1. how many observations it contains; 2. how difficult it is to capture high rewards. Based on these two scales, we study four types of environments: 1. finite (a small number of) observations plus a generative model (one of the most idealized sample oracles); 2. finite observations plus an approximate model; 3. rich (possibly infinitely many) but structured observations with an online simulation model; 4. general rich observations with an online simulation model. From the first to the last, the problem becomes more and more difficult and significant to solve. This thesis provides novel algorithms/analyses for each setting to improve both statistical and computational efficiency upon prior work.

Vision-language modeling is a crucial subfield of AI that focuses on jointly learning and representing image and text data, often using one modality to enhance understanding of the other. In cognitive science, humans use their visual system to grasp deep aspects of a concept, such as shape and size, while language helps them understand its semantics. Similarly, a machine can gain a better understanding of the world by utilizing multiple modalities, providing deeper insights compared to learning from a single modality. VL modeling is widely explored in the general domain, thanks to the vast image-text data available online and extensive annotated VL datasets. There are several strong VL models in the general domain, such as CLIP, which perform well on various tasks. However, in low-resource domains with limited data or dense knowledge areas, like the medical field, the data shortage hinders the development of robust multimodal models with reliable performance, especially where model reliability is critical. My research goal is to study the underlying capability of Vision-Language and Language Models and to develop innovative approaches to enhance their usage for low-resource domains such as the medical domain.