Abstract Background There is a rapid decline in moderate-to-vigorous physical activity (MVPA) during middle childhood and adolescence. Information on the environmental factors implicated in this decline is limited. This study focuses on family factors associated with the rate of decline in objectively measured physical activity during middle childhood and adolescence. Methods Longitudinal analysis of 801 participants from 10 US sites in the NICHD Study of Early Child Care and Youth Development whose data included accelerometer-determined levels of moderate-to-vigorous physical activity (MVPA) between ages 9 and 15 years, as well as family process, BMI and demographic information. The sample included an even split of boys (49%) and girls (51%), was predominantly white (77%), and contained about 26% low income and 19% single parent families. The outcome measure was mean MVPA. It was based on 4 to 7 days of monitored physical activity. Results Boys with lower parental monitoring scores and more days of parental encouragement had significantly more minutes of MVPA at age 9 years. The effect of parental monitoring, however, was moderated by early puberty. High parental monitoring was associated with decreased activity levels for boys experiencing later puberty and increased activity for boy experiencing early puberty. Minutes of MVPA for boys living in the Midwest decreased at significantly faster rates than boys living in any other region; and boys in the South declined faster than boys in the West. Girls in the Midwest and South declined faster than girls in the West and Northeast. Among girls, more days of parental exercise and transportation to activities were associated with more MVPA per day at age 9. However, more parental transportation to activities and less monitoring was associated with faster linear declines in daughters' MVPA between the ages of 9 and 15 years. For girls who experienced puberty early, parental encouragement was associated with more MVPA. Conclusions Parenting processes, such as monitoring and encouragement, as well as the parents' own level of physical activity, showed significant, but small, gender-specific associations with MVPA levels at age nine and the linear rate of decline in MVPA between ages 9 and 15.