聊聊Spring AI的Image Model

序

本文主要研究一下Spring AI的Image Model

ImageModel

org/springframework/ai/image/ImageModel.java

@FunctionalInterface
public interface ImageModel extends Model<ImagePrompt, ImageResponse> {

	ImageResponse call(ImagePrompt request);

}

ImageModel继承了Model接口，其call方法入参为ImagePrompt，返回ImageResponse

ImagePrompt

org/springframework/ai/image/ImagePrompt.java

public class ImagePrompt implements ModelRequest<List<ImageMessage>> {

	private final List<ImageMessage> messages;

	private ImageOptions imageModelOptions;

	public ImagePrompt(List<ImageMessage> messages) {
		this.messages = messages;
	}

	public ImagePrompt(List<ImageMessage> messages, ImageOptions imageModelOptions) {
		this.messages = messages;
		this.imageModelOptions = imageModelOptions;
	}

	public ImagePrompt(ImageMessage imageMessage, ImageOptions imageOptions) {
		this(Collections.singletonList(imageMessage), imageOptions);
	}

	public ImagePrompt(String instructions, ImageOptions imageOptions) {
		this(new ImageMessage(instructions), imageOptions);
	}

	public ImagePrompt(String instructions) {
		this(new ImageMessage(instructions), ImageOptionsBuilder.builder().build());
	}

	@Override
	public List<ImageMessage> getInstructions() {
		return this.messages;
	}

	@Override
	public ImageOptions getOptions() {
		return this.imageModelOptions;
	}

	//......
}	

ImagePrompt实现了ModelRequest接口，它定义了messages及imageModelOptions属性，其getInstructions返回的是List<ImageMessage>类型

ImageMessage

org/springframework/ai/image/ImageMessage.java

public class ImageMessage {

	private String text;

	private Float weight;

	public ImageMessage(String text) {
		this.text = text;
	}

	public ImageMessage(String text, Float weight) {
		this.text = text;
		this.weight = weight;
	}

	//......
}	

ImageMessage定义了text及weight属性

ImageOptions

org/springframework/ai/image/ImageOptions.java

public interface ImageOptions extends ModelOptions {

	@Nullable
	Integer getN();

	@Nullable
	String getModel();

	@Nullable
	Integer getWidth();

	@Nullable
	Integer getHeight();

	@Nullable
	String getResponseFormat();

	@Nullable
	String getStyle();

}

ImageOptions继承了ModelOptions，它定义了getN、getModel、getWidth、getHeight、getResponseFormat、getStyle方法

ImageResponse

org/springframework/ai/image/ImageResponse.java

public class ImageResponse implements ModelResponse<ImageGeneration> {

	private final ImageResponseMetadata imageResponseMetadata;

	/**
	 * List of generate images returned by the AI provider.
	 */
	private final List<ImageGeneration> imageGenerations;

	/**
	 * Construct a new {@link ImageResponse} instance without metadata.
	 * @param generations the {@link List} of {@link ImageGeneration} returned by the AI
	 * provider.
	 */
	public ImageResponse(List<ImageGeneration> generations) {
		this(generations, new ImageResponseMetadata());
	}

	/**
	 * Construct a new {@link ImageResponse} instance.
	 * @param generations the {@link List} of {@link ImageGeneration} returned by the AI
	 * provider.
	 * @param imageResponseMetadata {@link ImageResponseMetadata} containing information
	 * about the use of the AI provider's API.
	 */
	public ImageResponse(List<ImageGeneration> generations, ImageResponseMetadata imageResponseMetadata) {
		this.imageResponseMetadata = imageResponseMetadata;
		this.imageGenerations = List.copyOf(generations);
	}

	//......
}	

ImageResponse实现了ModelResponse接口，其getResult返回的是ImageGeneration类型

ImageGeneration

org/springframework/ai/image/ImageGeneration.java

public class ImageGeneration implements ModelResult<Image> {

	private ImageGenerationMetadata imageGenerationMetadata;

	private Image image;

	public ImageGeneration(Image image) {
		this.image = image;
	}

	public ImageGeneration(Image image, ImageGenerationMetadata imageGenerationMetadata) {
		this.image = image;
		this.imageGenerationMetadata = imageGenerationMetadata;
	}

	@Override
	public Image getOutput() {
		return this.image;
	}

	@Override
	public ImageGenerationMetadata getMetadata() {
		return this.imageGenerationMetadata;
	}

	@Override
	public String toString() {
		return "ImageGeneration{" + "imageGenerationMetadata=" + this.imageGenerationMetadata + ", image=" + this.image
				+ '}';
	}

}

ImageGeneration实现了ModelResult接口，其getOutput返回的是Image类型，其getMetadata返回ImageGenerationMetadata

Image

org/springframework/ai/image/Image.java

public class Image {

	/**
	 * The URL where the image can be accessed.
	 */
	private String url;

	/**
	 * Base64 encoded image string.
	 */
	private String b64Json;

	//......
}	

Image定义了url与b64Json属性

ImageGenerationMetadata

org/springframework/ai/image/ImageGenerationMetadata.java

public interface ImageGenerationMetadata extends ResultMetadata {

}

ImageGenerationMetadata目前仅仅是继承了ResultMetadata

示例

pom.xml

		<dependency>
			<groupId>org.springframework.ai</groupId>
			<artifactId>spring-ai-starter-model-zhipuai</artifactId>
		</dependency>

配置

spring:
  ai:
    model:
      image: zhipuai
    zhipuai:
      api-key: xxxx
      base-url: https://open.bigmodel.cn/api/paas
      images:
        options:
          model: cogview-3

代码

    @Test
    public void testGenImage() {
        ImagePrompt imagePrompt = new ImagePrompt("A light cream colored mini golden doodle",
                ZhiPuAiImageOptions.builder()
                        .model("cogview-3").build());
        ImageResponse response = imageModel.call(imagePrompt);
        log.info("resp:{}", JSON.toJSONString(response));
    }

输出示例

resp:{"metadata":{"created":1743578654097,"empty":true,"rawMap":{}},"result":{"output":{"url":"https://sfile.chatglm.cn/testpath/xxx.png"}},"results":[{"$ref":"$.result"}]}

源码

ZhiPuAiImageAutoConfiguration

org/springframework/ai/model/zhipuai/autoconfigure/ZhiPuAiImageAutoConfiguration.java

@AutoConfiguration(after = { RestClientAutoConfiguration.class, SpringAiRetryAutoConfiguration.class })
@ConditionalOnClass(ZhiPuAiApi.class)
@ConditionalOnProperty(name = SpringAIModelProperties.IMAGE_MODEL, havingValue = SpringAIModels.ZHIPUAI,
		matchIfMissing = true)
@EnableConfigurationProperties({ ZhiPuAiConnectionProperties.class, ZhiPuAiImageProperties.class })
public class ZhiPuAiImageAutoConfiguration {

	@Bean
	@ConditionalOnMissingBean
	public ZhiPuAiImageModel zhiPuAiImageModel(ZhiPuAiConnectionProperties commonProperties,
			ZhiPuAiImageProperties imageProperties, RestClient.Builder restClientBuilder, RetryTemplate retryTemplate,
			ResponseErrorHandler responseErrorHandler) {

		String apiKey = StringUtils.hasText(imageProperties.getApiKey()) ? imageProperties.getApiKey()
				: commonProperties.getApiKey();

		String baseUrl = StringUtils.hasText(imageProperties.getBaseUrl()) ? imageProperties.getBaseUrl()
				: commonProperties.getBaseUrl();

		Assert.hasText(apiKey, "ZhiPuAI API key must be set");
		Assert.hasText(baseUrl, "ZhiPuAI base URL must be set");

		var zhiPuAiImageApi = new ZhiPuAiImageApi(baseUrl, apiKey, restClientBuilder, responseErrorHandler);

		return new ZhiPuAiImageModel(zhiPuAiImageApi, imageProperties.getOptions(), retryTemplate);
	}

}

ZhiPuAiImageAutoConfiguration在spring.ai.model.image为zhipuai时启用，它会根据ZhiPuAiConnectionProperties及ZhiPuAiImageProperties自动配置ZhiPuAiImageModel

ZhiPuAiConnectionProperties

@ConfigurationProperties(ZhiPuAiConnectionProperties.CONFIG_PREFIX)
public class ZhiPuAiConnectionProperties extends ZhiPuAiParentProperties {

	public static final String CONFIG_PREFIX = "spring.ai.zhipuai";

	public static final String DEFAULT_BASE_URL = "https://open.bigmodel.cn/api/paas";

	public ZhiPuAiConnectionProperties() {
		super.setBaseUrl(DEFAULT_BASE_URL);
	}

}

class ZhiPuAiParentProperties {

	private String apiKey;

	private String baseUrl;

	public String getApiKey() {
		return this.apiKey;
	}

	public void setApiKey(String apiKey) {
		this.apiKey = apiKey;
	}

	public String getBaseUrl() {
		return this.baseUrl;
	}

	public void setBaseUrl(String baseUrl) {
		this.baseUrl = baseUrl;
	}

}

ZhiPuAiConnectionProperties继承了ZhiPuAiParentProperties属性，ZhiPuAiParentProperties定义了apiKey、baseUrl属性

ZhiPuAiImageProperties

org/springframework/ai/model/zhipuai/autoconfigure/ZhiPuAiImageProperties.java

@ConfigurationProperties(ZhiPuAiImageProperties.CONFIG_PREFIX)
public class ZhiPuAiImageProperties extends ZhiPuAiParentProperties {

	public static final String CONFIG_PREFIX = "spring.ai.zhipuai.image";

	/**
	 * Options for ZhiPuAI Image API.
	 */
	@NestedConfigurationProperty
	private ZhiPuAiImageOptions options = ZhiPuAiImageOptions.builder().build();

	public ZhiPuAiImageOptions getOptions() {
		return this.options;
	}

	public void setOptions(ZhiPuAiImageOptions options) {
		this.options = options;
	}

}

ZhiPuAiImageProperties继承了ZhiPuAiParentProperties，它主要配置spring.ai.zhipuai.image.options

ZhiPuAiImageOptions

org/springframework/ai/zhipuai/ZhiPuAiImageOptions.java

@JsonInclude(JsonInclude.Include.NON_NULL)
public class ZhiPuAiImageOptions implements ImageOptions {

	/**
	 * The model to use for image generation.
	 */
	@JsonProperty("model")
	private String model = ZhiPuAiImageApi.DEFAULT_IMAGE_MODEL;

	/**
	 * A unique identifier representing your end-user, which can help ZhiPuAI to monitor
	 * and detect abuse. User ID length requirement: minimum of 6 characters, maximum of
	 * 128 characters
	 */
	@JsonProperty("user_id")
	private String user;

	public static Builder builder() {
		return new Builder();
	}

	@Override
	@JsonIgnore
	public Integer getN() {
		return null;
	}

	@Override
	public String getModel() {
		return this.model;
	}

	public void setModel(String model) {
		this.model = model;
	}

	@Override
	@JsonIgnore
	public Integer getWidth() {
		return null;
	}

	@Override
	@JsonIgnore
	public Integer getHeight() {
		return null;
	}

	@Override
	@JsonIgnore
	public String getResponseFormat() {
		return null;
	}

	@Override
	@JsonIgnore
	public String getStyle() {
		return null;
	}

	public String getUser() {
		return this.user;
	}

	public void setUser(String user) {
		this.user = user;
	}

	//......
}	

ZhiPuAiImageOptions实现了ImageOptions接口，主要支持了getModel、getUser方法

小结

Spring AI提供了ImageModel接口，它继承了Model接口，其call方法入参为ImagePrompt，返回ImageResponse。spring-ai-starter-model-openai提供了OpenAiImageModel，spring-ai-starter-model-azure-openai提供了AzureOpenAiImageModel，spring-ai-starter-model-qianfan提供了QianFanImageModel，spring-ai-starter-model-stability-ai提供了StabilityAiImageModel，spring-ai-starter-model-zhipuai提供了ZhiPuAiImageModel。

doc

• Image Model API
• OpenAI Image Generation
• Azure OpenAI Image Generation
• QianFan Image Generation
• Stability AI Image Generation
• ZhiPuAI Image Generation