Prof Nick Collins, an expert on musical AI at Durham University, admitted it was difficult to evaluate the quality of DeepComposer based on the single audio example provided by Amazon in a blog, external.
"If I had to judge just based on what was on that web page, I wasn't very impressed at all," he told the BBC.
"It's terrible. A musician with a basic sequencing ability could easily mock up better examples."
A spokeswoman for Amazon said the device was intended for developers as a "fun and engaging way" to gain new skills.
Prof Collins said that, among more general users, it might fail to be seen as anything more than "a quick gimmick".
Dr Oded Ben-Tal at Kingston University, who has worked on AI music projects, agreed.
"I see people doing this five times and then saying, 'Yeah that's fine,' then [moving] on to something else," he told the BBC.
Of the audio demo, he said: "I don't think it sounds good."