The way that parents communicate with their typically developing infants is associated with later infant language development. Here we aim to show that these associations are observed in infants subsequently diagnosed with Autism Spectrum Disorder (ASD). This study had three groups: high-familial-risk infants who did not have ASD (n = 46); high-familial-risk infants who had ASD (n = 14); and low-familial-risk infants who exhibited typical development (n = 36). All-day home language recordings were collected at 9 and 15 months, and language skills were assessed at 24 months. Across all infants in the study, including those with ASD, a richer home language environment (e.g., hearing more adult words and experiencing more conversational turns) at 9 and 15 months was associated with better language skills. Higher parental educational attainment was associated with a richer home language environment. Mediation analyses showed that the effect of education on child language skills was explained by the richness of the home language environment. Exploratory analyses revealed that typically developing infants experience an increase in caregiver–child conversational turns across 9–15 months, a pattern not seen in children with ASD. The current study shows that parent behavior during the earliest stages of life can have a significant impact on later development, highlighting the home language environment as means to support development in infants with ASD.