Adobe Firefly is Way Behind: Is the Commitment to Ethics to Blame?

Adobe Firefly Review

Last week, Adobe made waves by announcing the beta release of its new text-to-image generative artificial intelligence (AI) model, Firefly. Adobe says its new platform wasn’t built using stolen images, but rather, as Adobe boasts, Firefly has been trained using Adobe Stock images, openly licensed content, and public domain content.

Adobe is Building Its AI Model in The Right Way

It’s an admirable way to build an AI platform, especially in the face of competing models that are built using stolen and unauthorized content. Midjourney’s founder David Holz recently admitted that his company didn’t have permission to use the hundreds of millions of images used to train its AI image generator.

Adobe Firefly
“Pastel colored baby dragon with rainbow scales”
Adobe Firefly
“Close-up portrait of a phoenix in a fantasy setting”

Adobe is also committed to countering biases prevalent within AI image generators. Last October, Hugging Face began hosting the “Stable Diffusion Bias Explorer.” This tool lets users see firsthand how AI models struggle with racial and gender stereotypes.

Adobe Firefly
Part of Adobe’s approach seems to involve banning certain words altogether, which isn’t unusual. Midjourney does it too. Firefly wouldn’t let me search for anything involving the word “criminal.” I was tempted to see how the AI visualizes a “criminal.” Firefly had no issue generating “prisoners playing sports,” although it’s fair to say the results are bad.

It’s impressive that Adobe is conscious of these biases and working hard to ensure Firefly is socially responsible. Adobe argues that, “any company building generative AI tools should start with an AI Ethics framework.” Adobe believes its ethical structure will ensure that its AI technologies, including generative AI like Firefly, will respect users and align with the company’s core values.

Adobe Firefly
These “group of friends eating veggie pizza and watching sports” results are pretty good. People’s hands are mostly realistic.

“Mitigating harmful outputs starts with building and training on safe and inclusive datasets. For example, Adobe’s first model in our Firefly family of creative generative AI models is trained on Adobe Stock images, openly licensed content, and public domain content where the copyright has expired. Training on curated, diverse datasets inherently gives your model a competitive edge when it comes to producing commercially safe and ethical results,” Adobe continues.

These goals are not just ambitious, they also deserve respect. I want to give Adobe major kudos for how it has developed and trained Firefly.

But the ‘Right Way’ Delivers Wrong Results

However, early results suggest that perhaps the ethically valuable limits Adobe has placed on itself have negatively affected the performance of its new generative AI model. Adobe Firefly, at this point, needs to catch up with the extremely impressive Midjourney version 5 and it is a long way behind.

Adobe Firefly isn’t bad, but it clearly isn’t as effective as the much more mature Midjourney platform. However, it’s important to consider that not only is Midjourney older, which is a significant advantage concerning the efficacy of generative AI, but it’s also worth thinking about how Midjourney was made.

Unsettled Regulatory Situation Lend Credence to Adobe’s Methods

Not only is building an AI model using stolen content ethically dubious, at best, but it’s also legally murky. Adobe’s ethical framework doesn’t exist solely because the company thinks it’s the right thing to do, it also impacts the commercial viability of its product.

“The law has to catch up to technology,” Mickey H. Osterreicher, the General Counsel for the National Press Photographers Association (NPPA) told PetaPixel.

Adobe Firefly
Firefly has an issue with results being very similar to each other. These results for “macro photo of a red flower with a bee” are repetitive.

Thomas Maddrey, Chief Legal Officer for the American Society of Media Photographers (ASMP), added, “Copyright law is not prepared and is not set up to protect the artists or the users at this moment in time. A lot of it is not going to be determined under copyright statute, but rather, unfortunately, in litigation.”

By building Firefly expressly around fair use and licensed content, Adobe’s new AI model can avoid what feels like an imminent legal catastrophe. Adobe’s primary purpose is, of course, to be financially viable. If operating ethically aligns with this overriding goal and improves Adobe’s business practices, too, all the better.

Adobe Firefly
“Comic book style illustration of lawyers screaming at each other in a courtroom.” This feels like where we’re headed in the ever-changing, tumultuous AI landscape.

However, as I alluded to, after using Firefly, it’s hard not to wonder if its limitations aren’t due in part to the substantially smaller dataset Adobe has allowed itself.

I Only Wanted an Image of a Woman Taking a Photo of a Man, is That Too Much to Ask?

For example, when I typed the text prompt, “Woman taking a portrait of a man,” the results were disappointing. Women are criminally underrepresented in the photography industry, so I wondered if Adobe’s limited dataset could account for what seems like a normal situation to me, but one that is disappointingly unusual to actually see.

Adobe Firefly
“Woman taking a portrait of a man”

The results could have been better. In only one case was a woman taking a photo of a man; in that case, the woman was taking a picture of the back of a man’s head.

I thought I needed to be more specific, so I tried, “A woman taking a photo of a man’s face.” That’s different from how I’d typically describe what I wanted to see: a woman using a camera to take a typical portrait of a man as her subject. This prompt wasn’t much more fruitful, although at least three of the four women had a camera.

Adobe Firefly
“A woman taking a photo of a man’s face”

Well, maybe “A woman using a camera to take a typical portrait of a man as her subject” will work? Definitely not.

My favorite is the image in the top right, with the woman pressing her face against the end of a camera lens. A close second is whatever happened to that poor man in the top left; it looks painful.

Adobe Firefly
“A woman using a camera to take a typical portrait of a man as her subject” is an obtuse way to phrase what I wanted to see. The results are similarly bad.

Firefly Can’t Yet Navigate AI’s Problem with Hands

Firefly could do a better job with hands, which is admittedly a common issue for generative AI models. Midjourney v5 finally delivers mostly realistic human hands, at least for paying customers.

Adobe Firefly
One of the few times Firefly produced a mixed-race couple without specific prompting, it still struggled with hands.

Midjourney does a great job of creating photorealistic portraits of people overall. I wanted to try something similar in Firefly, using specific phrases, which has proved helpful for prior AI models.

Stylized Portraits Aren’t Much Better

Adobe Firefly
“Portrait of a woman, wet plate”
This is not what I meant by “wet plate.” I think this is a case of user error.
Adobe Firefly
However, that said, “Portrait of a man, wet plate” delivered very different results than the query for “woman.” I don’t know why.

I chalk these results up to user error. Using “Wet plate portrait of a woman” delivered significantly better results.

Adobe Firefly
“Wet plate portrait of a woman” performed much better than “portrait of a woman, wet plate.”

Aiming for more specificity, I tried “Full body portrait of a man in the 1800s, wet collodion photography.” I’m not sure what to say about these results, but they aren’t what I expected.

Adobe Firefly
“Full body portrait of a man in the 1800s, wet collodion photography” resulted in some interesting images, to say the least. I’m not sure I’d call them realistic.

Let’s Talk About Race: Diversity in a Vacuum

I decided to try one more attempt at portraits of people. I went for, “Portrait of a happy couple in love 85mm bokeh.” I ran this prompt repeatedly to view different results, and this quartet was the best. Some results had deformed faces, while others featured strange limbs. None of them looked incredibly natural or realistic.

Adobe Firefly
“Portrait of a happy couple in love, 85mm, bokeh”

I also noticed that while the results were constantly generating people of different races and diverse skin tones, every couple included people of the same race.

I thought that maybe a broader prompt, such as “portrait of a couple,” would help. It didn’t. To achieve the expected results, I needed to specify a “mixed-race couple.” On the one hand, I understand why, with only four visible results, I need to be specific to see certain representations of diversity. However, on the other weird AI-generated hand, I don’t think I should need to ask for diversity to see it.

Firefly Requires More Work to Deal with Gender

Something else immediately struck me — not one portrait of a couple, or people in love, showed subjects that appeared to be in a same-sex relationship.

I tried, “wedding photo, outside, daytime.” Now not only were the generated images showing just white heterosexual weddings, one of the photos showed the groom grabbing the bride’s breast. I haven’t attended many weddings, so maybe I attended relatively dull, straightforward celebrations, but that seems very out of place.

Adobe Firefly
“Wedding photo, outside, daytime.”
The bride in the top left appears to be killing her groom.
Adobe Firefly
Meanwhile, this grabby groom is getting a handful.

I searched with terms like “mixed-race couple” and “same-sex wedding,” and the results weren’t great.

Adobe Firefly
“Same-sex wedding” returned rather disappointing results.

Why does a “wedding” require one person to wear a wedding dress? Of course, it’s not just any wedding dress — it’s a very traditional western-style wedding dress.

To get a like-for-like comparison against Midjourney v5, which prohibits the term “sex,” I asked Firefly to generate a “wedding of two men.” Midjourney puts Firefly to shame here, even though both AI models seem fixated on white people with that prompt.

Adobe Firefly
Firefly’s attempt at “wedding of two men” might be slightly better, but much worse than Midjourney’s effort.
Midjourney
Midjourney v5’s take on “wedding of two men.”

Photorealism Often Challenges Firefly

Overall, during my time using Adobe Firefly, my quest for photorealistic AI-generated images of people proved challenging. I’m willing to chalk some of the struggles up to its beta status and that Firefly is a relative infant in the rapidly evolving generative AI space, as well as Firefly’s small, ethically sourced training dataset.

Adobe Firefly
I asked for a “photorealistic portrait of a man with a mustache,” and I suppose that’s what I got. But why do all four men have the same mustache?

My confidence that Firefly will evolve is tempered by its limited dataset. Adobe isn’t going to go the route of Midjourney and start stealing images suddenly and Firefly is already at least six to eight months behind some competitors regarding its results. If the model improves slowly due to a reduced dataset, will it ever catch up?

Specificity Helps in a Big Way

While I find it frustrating that I needed to be specific to see mixed-race couples, and I have no idea what Firefly is doing with same-sex couples, I think that part of the fault lies with me. Vague prompts will rarely meet my expectations.

Adobe Firefly
Firefly did a great job with “Portrait of a young black woman with natural hair wearing fashionable clothes.” As far as the results of people go, this is one of Firefly’s more impressive efforts.

However, precise prompts can. Consider “Portrait of a young black woman with natural hair wearing fashionable clothes.” I’m impressed that Firefly respected “natural hair” and did a nice job with it. It also did an excellent job with “young” and “fashionable clothes,” in my opinion.

Removing People Delivers Better but Monotonous Results

When I fed Firefly different prompts without people, the results improved. However, Firefly is not only limited insofar as it only shows four results at a time, the results also look very similar to one another.

Adobe Firefly
This is an admirable effort at “grizzly bear fishing for salmon, mountain in the background, telephoto lens.” This is in the style of “art” with “golden hour” light and “warm tones.”

This issue persisted when I ditched the “Photo” content type. When I opted for “Art” and added modifiers including “hyper realistic” and “fantasy” with “warm tone” and “golden hour” lighting, the results still lacked diversity. Still, admittedly, they’re quite lovely.

Adobe Firefly
These “polar bears riding a rollercoaster” look pretty good, too.
Midjourney
“Polar bears riding a rollercoaster” as generated by Midjourney v5

Ultimately, Adobe Firefly is a well-intentioned generative AI model that, in pursuing ethical and commercial goals, is currently limited in its performance and practicality. In important, moral ways, Firefly is better than its competitors. When it comes to the results though, it’ is undoubtedly worse.

Adobe Firefly
However, Firefly does a much better job with non-human animals. Consider these “penguins having a barbecue in the snow.” This looks pretty good.
Adobe Firefly
In the spirit of Rosa Diaz from the television show Brooklyn Nine-Nine, “I’ve only had [space sloth] for a day and a half, but if anything happened to him, I would kill everyone in this room and then myself.”

That said, Adobe Firefly just launched, and it will improve over time. Besides, I had a lot of fun generating a “close-up portrait of a sloth astronaut in space eating a taco” in a “cartoon” art style with a blurry background, cool tones, and dramatic lighting. Firefly ticked only some of my requested boxes, but I wonder how much I care — I love my space sloth.

Discussion