Brain asymmetry in the sensitivity to spectrotemporal modulation is an established functional feature that underlies the perception of speech and music. Recent studies in humans suggest separable neural mechanisms must underlie these cognitively complex functions. The left Auditory Cortex (ACx) is believed to specialize in processing fast temporal components of speech sounds, and the right ACx slower components. However, the circuit features and neural computations behind these lateralized spectrotemporal processes are poorly understood. To answer these mechanistic questions we use mice, an animal model that captures some relevant features of human communication systems. In this study, we screened for circuit features that could subserve temporal integration differences between the left and right ACx. We mapped excitatory input to principal neurons in all cortical layers and found significantly stronger recurrent connections in the superficial layers of the right ACx compared to the left. We hypothesized that the underlying recurrent neural dynamics would exhibit differential characteristic timescales corresponding to their hemispheric specialization. To investigate, we recorded spike trains from awake mice and estimated the network time constants using a quasi-Bayesian method to combine evidence from multiple weak signal-to-noise ratio neurons. We found longer temporal integration windows in the right ACx compared to the left as predicted by stronger recurrent excitation. Our study shows direct evidence that stronger recurrent synaptic connections lead to longer network time scales. These findings support speech processing theories that purport asymmetric integration time constants is a crucial feature of lateralization in auditory processing.