Modern man, homo sapiens, and Cro Magnon man, homo whooozis, managed to interbreed, successfully, both ways. I.e. Cro Magnon female with sapiens male, and Cro Magnon male with sapiens female. Neither is ancestor of the other, and both are the same species. Not e.g. homo neanderthalensis, unless it’s OK to say that two populations which manage to interbreed with each other, have the same set of chromosomes, etc. actually constitute two species.
But I digress.
Brain cells can vary in size, number of synapses, how successful and regular their local 3-D structure is, and so on. In utero pan troglodytes and homo sapiens develop along the same basic lines, but homo emerges with a much richer and more extensive 3-D matrix of neurons.
(Edit to remove Auel reference – data point was not correct, i.e. neanderthals had full flexibility of the modern vocal repertoire. Thank You to Mark Suggitt.)
So how does this relate to brain size? Your guess is as good as mine. The clues are that neanderthal’s massive skeleton indicated a need to live much closer to risk and exertion. Individuals without the same strength of body and bone failed to survive, keeping their line at status quo ante. Their physical robustness was critical to their survival.
On the other hand sapiens’ physical robustness faded away. This indicates that, for them/us, the evolutionary cost, the investment in muscle and bone (an the metabolic requirements to produce / sustain them) was too high. Sapiens’ body type economized to spend less on bone and muscle because they could. Neanderthal could not – hence went extinct.
The ‘obvious’ conjecture that neanderthals, in having more cc’s of brain, had better brains, is inconveniently shaded by the fact that they died out.
Oh, and Another Thing 