There is reason to believe that his father, Charles-François Delacroix, was infertile at the time of Eugène's conception and that his real father was Talleyrand, who was a friend of the family and successor of Charles Delacroix as Minister of Foreign Affairs, and whom the adult Eugène resembled in appearance and character. Throughout his career as a painter, he was protected by Talleyrand, who served successively the Restoration and king Louis-Philippe, and ultimately as ambassador of France in Great Britain, and later by Talleyrand's grandson, Charles Auguste Louis Joseph, duc de Morny, half-brother of Napoleon III and speaker of the French House of Commons.
For example, if you told me that San Francisco 49ers quarterback Colin Kaepernick, a 6'-4" biracial adoptee of tremendous athletic ability, is really the son of, say, Michael Jordan's 6'-8" sidekick Scottie Pippen, I might almost believe you. (Note: I just made that up.) But, there are surprisingly few confirmed examples of today's sports stars being the secret sons of yesterday's sports stars.
One reason for the Scandal Shortage is that there just aren't that many famous people. Talleyrand and Delacroix, for example, are of that tiny number of figures of such historical importance that we only use their surnames.