This paper presents a statistical method combined with a neural network for efficient traffic prediction from a limited amount of training data. The traffic prediction during a large-scale event is essential to maintain the safety of event participants. The conventional methods for predicting traffic time series, however, cannot be utilized because the rare nature of the large-scale events prevents us from preparing a sufficient amount of training data. To efficiently train traffic prediction from a limited amount of training data, we propose a pattern-aware regression method that reduces the number of model parameters by interpreting traffic data as a weighted sum of latent behavior patterns. The proposed method trains a neural regression model to predict the weights of these patterns from the event information instead of directly predicting the traffic time series. The behavior patterns are jointly estimated during the training in a Bayesian manner to avoid overfitting. We performed experiments with foot traffic data recorded at a real soccer stadium and show that the proposed method outperforms the conventional direct regression methods. We also demonstrate an application of our method for predicting travel time from the stadium to the nearest highway interchange, which outperforms a popular commercial service.