The electrical activity of neurons has a spatiotemporal footprint that spans three orders of magnitude. Traditional electrophysiology lacks the spatial throughput to image the activity of an entire neural network; besides, labeled optical imaging using voltage-sensitive dyes and tracking Ca2+ ion dynamics lack the versatility and speed to capture fast-spiking activity, respectively. We present a label-free optical imaging technique to image the changes to the optical path length and the local birefringence caused by neural activity, at 4,000 Hz, across a 200 × 200 μm2 region, and with micron-scale spatial resolution and 300-pm displacement sensitivity using Superfast Polarization-sensitive Off-axis Full-field Optical Coherence Microscopy (SPoOF OCM). The undulations in the optical responses from mammalian neuronal activity were matched with field-potential electrophysiology measurements and validated with channel blockers. By directly tracking the widefield neural activity at millisecond timescales and micrometer resolution, SPoOF OCM provides a framework to progress from low-throughput electrophysiology to high-throughput ultra-parallel label-free optophysiology.