Managing schemas using Schema Registry

Introduction

In the previous Kafka and Avro configuration tutorial, you learned how to add Apache Avro schemas to your AsyncAPI document. Now, you will learn how to save your schema in a central Schema Registry and reuse it.

Background context

The need for schema management has become increasingly prevalent to handle the evolving complexity of modern Event-Driven Architecture. A Schema Registry is a centralized service that stores and maintains schemas for data exchanged between various components of a modern distributed system. Validating exchanged data maintains data consistency and compatibility.

While several Schema Registry implementations exist, you will use the Apicurio Registry for this tutorial. Apicurio Registry is a popular open-source Schema Registry implementation that supports multiple serialization formats and facilitates schema management for diverse data in distributed systems. You will use Apicurio Registry combined with Avro, a language-neutral data serialization system.

Prerequisites

Install Docker from the official website.

AsyncAPI document with Avro Schema

The previous tutorial taught you how to write an AsyncAPI document for Kafka messages using the Avro schema. Here's an example of what an AsyncAPI document fully equipped with Avro schema looks like:

1asyncapi: 3.0.0
2info:
3  title: User Signup API
4  version: 1.0.0
5  description: The API notifies you whenever a new user signs up in the application.
6servers:
7  kafkaServer:
8    host: test.mykafkacluster.org:8092
9    description: Kafka Server
10    protocol: kafka
11operations:
12  onUserSignedUp:
13    action: receive
14    channel:
15      $ref: '#/channels/userSignedUp'
16channels:
17  userSignedUp:
18    description: This channel contains a message per each user who signs up in our application.
19    address: user_signedup
20    messages:
21      userSignedUp:
22        $ref: '#/components/messages/userSignedUp'
23components:
24  messages:
25    userSignedUp:
26      payload:
27        schemaFormat: 'application/vnd.apache.avro;version=1.9.0'
28        schema:
29          type: record
30          name: UserSignedUp
31          namespace: com.company
32          doc: User sign-up information
33          fields:
34            - name: userId
35              type: int
36            - name: userEmail
37              type: string

Start Apicurio Registry

Start the Apicurio Registry locally with the following docker command:

docker run --env CORS_ALLOWED_ORIGINS='*' -it -p 8080:8080 apicurio/apicurio-registry-mem:2.5.8.Final

Upload Avro schema

Once your local instance of Apicurio Registry is running, upload your Avro schema. Open a new terminal window and create an Avro schema artifact with the following command:

1curl \
2http://localhost:8080/apis/registry/v2/groups/my-group/artifacts \
3-X POST  \
4-H "Content-Type: application/json; artifactType=AVRO" \
5-H "X-Registry-ArtifactId: UserSignedUp" \
6--data @- << EOF
7{
8  "type": "record",
9  "name": "UserSignedUp",
10  "namespace": "com.company",
11  "doc": "User sign-up information",
12  "fields": [
13    {
14      "name": "userId",
15      "type": "int"
16    },
17    {
18      "name": "userEmail",
19      "type": "string"
20    }
21  ]
22}
23EOF

Remember

Download your Avro schema by visiting the following URL: http://localhost:8080/apis/registry/v2/groups/my-group/artifacts/UserSignedUp.

Update schema reference

One alternative is to keep your schema in a separate file, as you learned in the previous tutorial, describe Kafka message payload using Avro schema. After uploading your Avro schema, remove the schema from your AsyncAPI document and add a $ref pointing to the previous step's URL.

$ref: http://localhost:8080/apis/registry/v2/groups/my-group/artifacts/UserSignedUp

1asyncapi: 3.0.0
2info:
3  title: User Signup API
4  version: 1.0.0
5  description: The API notifies you whenever a new user signs up in the application.
6servers:
7  kafkaServer:
8    host: test.mykafkacluster.org:8092
9    description: Kafka Server
10    protocol: kafka
11operations:
12  onUserSignedUp:
13    action: receive
14    channel:
15      $ref: '#/channels/userSignedUp'
16channels:
17  userSignedUp:
18    description: This channel contains a message per each user who signs up in our application.
19    address: user_signedup
20    messages:
21      userSignedUp:
22        $ref: '#/components/messages/userSignedUp'
23components:
24  messages:
25    userSignedUp:
26      payload:
27        schemaFormat: 'application/vnd.apache.avro+json;version=1.9.0'
28        schema:
29          $ref: http://localhost:8080/apis/registry/v2/groups/my-group/artifacts/UserSignedUp

Summary

In this tutorial, you managed Avro schemas using a centralized schema registry that enables you to share schemas across multiple applications. The good news is that this approach is valid for various other schema types!

Next steps

Now that you have learned how to manage schemas, check out the bindings with Kafka tutorial to start sending messages between your services.

On this page