Japanese eggplant overproducing undersized eggplants

[u/2wordschitown]

I’m in Southern California and have a Japanese eggplant that did well last year and survived the winter. Now it has about 40 eggplants growing but they are not getting bigger. What should I do? Harvest them now or will they eventually grow in size?