protobuf 2 kotlin 插件

proto文件就是一个数据协议的描述文件，基于其中的类型信息会被转化成对应的语言(比如java go OC等等)。

proto的好处就是协议字段非常稳定,而且可被追溯。举个栗子，我们当前是四端共享一个proto仓库，然后只要后端更新了字段内容，另外三端也会同样的更新出新的字段内容。这点是相对于json更好的。

但是我们最近开始尝试kmp了,由于请求有一部分都是proto协议的,但是因为kmp的common层所有的类都必须是kotlin库而不能是jvm的。所以官方proto提供的java类就没办法直接被kmp所引用到。

因为上述原因，所以我们现在急需的是一个proto插件，可以帮助我们把一个proto文件直接转化成kotlin的。当然我们第一目标是最好能在kotlin官方找到这样一个能力，直接支持。其次就是github找一个仓库能转化proto到kt的工程。最最不行只能自己动手了啊。

kotlin serialization

serialization是支持proto转化的，但是这个库并不支持将proto文件转化成data class。不过serialization对于proto的反序列化支持还是非常ok的。而且转化方式也非常的简单。代码如下所示。

    val sample1 = Sample("66666666666")
    val encode = ProtoBuf.Default.encodeToByteArray(sample1)
    val newSample = ProtoBuf.Default.decodeFromByteArray<Sample>(encode)

只要引入kotlinx-serialization插件之后，在添加org.jetbrains.kotlinx:kotlinx-serialization-protobuf:version依赖就可以直接使用了。

但是因为官方库缺少将proto转化成kotlin class的能力，所以我们一开始并没有直接选用它。只能去从github搜索下有没有别的更好支持的库。

pbandk

pbandk 仓库地址

这个库通过protobuf-java编写了一个proto插件。通过对proto的解析，生成了PluginProtos.CodeGeneratorRequest的数据结构，然后读取其中的字段，转义成一个新的data class。

// Below is mostly verbatim from [protobufsrc]/examples/addressbook.proto except
// some surrounding comments are removed and java/csharp options removed

syntax = "proto3";
package tutorial;

import "google/protobuf/timestamp.proto";

message Person {
    string name = 1;
    int32 id = 2;  // Unique ID number for this person.
    string email = 3;

    enum PhoneType {
        MOBILE = 0;
        HOME = 1;
        WORK = 2;
    }

    message PhoneNumber {
        string number = 1;
        PhoneType type = 2;
    }

    repeated PhoneNumber phones = 4;

    google.protobuf.Timestamp last_updated = 5;
}

// Our address book file is just one of these.
message AddressBook {
    repeated Person people = 1;
}

以上述proto文件为例，pbandk会把这个文件翻译成如下的kt文件。

@file:OptIn(pbandk.PublicForGeneratedCode::class)

package pbandk.examples.addressbook.pb

@pbandk.Export
public data class Person(
    val name: String = "",
    val id: Int = 0,
    val email: String = "",
    val phones: List<pbandk.examples.addressbook.pb.Person.PhoneNumber> = emptyList(),
    val lastUpdated: pbandk.wkt.Timestamp? = null,
    override val unknownFields: Map<Int, pbandk.UnknownField> = emptyMap()
) : pbandk.Message {
    override operator fun plus(other: pbandk.Message?): pbandk.examples.addressbook.pb.Person = protoMergeImpl(other)
    override val descriptor: pbandk.MessageDescriptor<pbandk.examples.addressbook.pb.Person> get() = Companion.descriptor
    override val protoSize: Int by lazy { super.protoSize }
    public companion object : pbandk.Message.Companion<pbandk.examples.addressbook.pb.Person> {
        public val defaultInstance: pbandk.examples.addressbook.pb.Person by lazy { pbandk.examples.addressbook.pb.Person() }
        override fun decodeWith(u: pbandk.MessageDecoder): pbandk.examples.addressbook.pb.Person = pbandk.examples.addressbook.pb.Person.decodeWithImpl(u)

        override val descriptor: pbandk.MessageDescriptor<pbandk.examples.addressbook.pb.Person> by lazy {
            val fieldsList = ArrayList<pbandk.FieldDescriptor<pbandk.examples.addressbook.pb.Person, *>>(5)
            fieldsList.apply {
                add(
                    pbandk.FieldDescriptor(
                        messageDescriptor = this@Companion::descriptor,
                        name = "name",
                        number = 1,
                        type = pbandk.FieldDescriptor.Type.Primitive.String(),
                        jsonName = "name",
                        value = pbandk.examples.addressbook.pb.Person::name
                    )
                )
                add(
                    pbandk.FieldDescriptor(
                        messageDescriptor = this@Companion::descriptor,
                        name = "id",
                        number = 2,
                        type = pbandk.FieldDescriptor.Type.Primitive.Int32(),
                        jsonName = "id",
                        value = pbandk.examples.addressbook.pb.Person::id
                    )
                )
                add(
                    pbandk.FieldDescriptor(
                        messageDescriptor = this@Companion::descriptor,
                        name = "email",
                        number = 3,
                        type = pbandk.FieldDescriptor.Type.Primitive.String(),
                        jsonName = "email",
                        value = pbandk.examples.addressbook.pb.Person::email
                    )
                )
                add(
                    pbandk.FieldDescriptor(
                        messageDescriptor = this@Companion::descriptor,
                        name = "phones",
                        number = 4,
                        type = pbandk.FieldDescriptor.Type.Repeated<pbandk.examples.addressbook.pb.Person.PhoneNumber>(valueType = pbandk.FieldDescriptor.Type.Message(messageCompanion = pbandk.examples.addressbook.pb.Person.PhoneNumber.Companion)),
                        jsonName = "phones",
                        value = pbandk.examples.addressbook.pb.Person::phones
                    )
                )
                add(
                    pbandk.FieldDescriptor(
                        messageDescriptor = this@Companion::descriptor,
                        name = "last_updated",
                        number = 5,
                        type = pbandk.FieldDescriptor.Type.Message(messageCompanion = pbandk.wkt.Timestamp.Companion),
                        jsonName = "lastUpdated",
                        value = pbandk.examples.addressbook.pb.Person::lastUpdated
                    )
                )
            }
            pbandk.MessageDescriptor(
                fullName = "tutorial.Person",
                messageClass = pbandk.examples.addressbook.pb.Person::class,
                messageCompanion = this,
                fields = fieldsList
            )
        }
    }

    public sealed class PhoneType(override val value: Int, override val name: String? = null) : pbandk.Message.Enum {
        override fun equals(other: kotlin.Any?): Boolean = other is Person.PhoneType && other.value == value
        override fun hashCode(): Int = value.hashCode()
        override fun toString(): String = "Person.PhoneType.${name ?: "UNRECOGNIZED"}(value=$value)"

        public object MOBILE : PhoneType(0, "MOBILE")
        public object HOME : PhoneType(1, "HOME")
        public object WORK : PhoneType(2, "WORK")
        public class UNRECOGNIZED(value: Int) : PhoneType(value)

        public companion object : pbandk.Message.Enum.Companion<Person.PhoneType> {
            public val values: List<Person.PhoneType> by lazy { listOf(MOBILE, HOME, WORK) }
            override fun fromValue(value: Int): Person.PhoneType = values.firstOrNull { it.value == value } ?: UNRECOGNIZED(value)
            override fun fromName(name: String): Person.PhoneType = values.firstOrNull { it.name == name } ?: throw IllegalArgumentException("No PhoneType with name: $name")
        }
    }

    public data class PhoneNumber(
        val number: String = "",
        val type: pbandk.examples.addressbook.pb.Person.PhoneType = pbandk.examples.addressbook.pb.Person.PhoneType.fromValue(0),
        override val unknownFields: Map<Int, pbandk.UnknownField> = emptyMap()
    ) : pbandk.Message {
        override operator fun plus(other: pbandk.Message?): pbandk.examples.addressbook.pb.Person.PhoneNumber = protoMergeImpl(other)
        override val descriptor: pbandk.MessageDescriptor<pbandk.examples.addressbook.pb.Person.PhoneNumber> get() = Companion.descriptor
        override val protoSize: Int by lazy { super.protoSize }
        public companion object : pbandk.Message.Companion<pbandk.examples.addressbook.pb.Person.PhoneNumber> {
            public val defaultInstance: pbandk.examples.addressbook.pb.Person.PhoneNumber by lazy { pbandk.examples.addressbook.pb.Person.PhoneNumber() }
            override fun decodeWith(u: pbandk.MessageDecoder): pbandk.examples.addressbook.pb.Person.PhoneNumber = pbandk.examples.addressbook.pb.Person.PhoneNumber.decodeWithImpl(u)

            override val descriptor: pbandk.MessageDescriptor<pbandk.examples.addressbook.pb.Person.PhoneNumber> by lazy {
                val fieldsList = ArrayList<pbandk.FieldDescriptor<pbandk.examples.addressbook.pb.Person.PhoneNumber, *>>(2)
                fieldsList.apply {
                    add(
                        pbandk.FieldDescriptor(
                            messageDescriptor = this@Companion::descriptor,
                            name = "number",
                            number = 1,
                            type = pbandk.FieldDescriptor.Type.Primitive.String(),
                            jsonName = "number",
                            value = pbandk.examples.addressbook.pb.Person.PhoneNumber::number
                        )
                    )
                    add(
                        pbandk.FieldDescriptor(
                            messageDescriptor = this@Companion::descriptor,
                            name = "type",
                            number = 2,
                            type = pbandk.FieldDescriptor.Type.Enum(enumCompanion = pbandk.examples.addressbook.pb.Person.PhoneType.Companion),
                            jsonName = "type",
                            value = pbandk.examples.addressbook.pb.Person.PhoneNumber::type
                        )
                    )
                }
                pbandk.MessageDescriptor(
                    fullName = "tutorial.Person.PhoneNumber",
                    messageClass = pbandk.examples.addressbook.pb.Person.PhoneNumber::class,
                    messageCompanion = this,
                    fields = fieldsList
                )
            }
        }
    }
}

@pbandk.Export
public data class AddressBook(
    val people: List<pbandk.examples.addressbook.pb.Person> = emptyList(),
    override val unknownFields: Map<Int, pbandk.UnknownField> = emptyMap()
) : pbandk.Message {
    override operator fun plus(other: pbandk.Message?): pbandk.examples.addressbook.pb.AddressBook = protoMergeImpl(other)
    override val descriptor: pbandk.MessageDescriptor<pbandk.examples.addressbook.pb.AddressBook> get() = Companion.descriptor
    override val protoSize: Int by lazy { super.protoSize }
    public companion object : pbandk.Message.Companion<pbandk.examples.addressbook.pb.AddressBook> {
        public val defaultInstance: pbandk.examples.addressbook.pb.AddressBook by lazy { pbandk.examples.addressbook.pb.AddressBook() }
        override fun decodeWith(u: pbandk.MessageDecoder): pbandk.examples.addressbook.pb.AddressBook = pbandk.examples.addressbook.pb.AddressBook.decodeWithImpl(u)

        override val descriptor: pbandk.MessageDescriptor<pbandk.examples.addressbook.pb.AddressBook> by lazy {
            val fieldsList = ArrayList<pbandk.FieldDescriptor<pbandk.examples.addressbook.pb.AddressBook, *>>(1)
            fieldsList.apply {
                add(
                    pbandk.FieldDescriptor(
                        messageDescriptor = this@Companion::descriptor,
                        name = "people",
                        number = 1,
                        type = pbandk.FieldDescriptor.Type.Repeated<pbandk.examples.addressbook.pb.Person>(valueType = pbandk.FieldDescriptor.Type.Message(messageCompanion = pbandk.examples.addressbook.pb.Person.Companion)),
                        jsonName = "people",
                        value = pbandk.examples.addressbook.pb.AddressBook::people
                    )
                )
            }
            pbandk.MessageDescriptor(
                fullName = "tutorial.AddressBook",
                messageClass = pbandk.examples.addressbook.pb.AddressBook::class,
                messageCompanion = this,
                fields = fieldsList
            )
        }
    }
}

@pbandk.Export
@pbandk.JsName("orDefaultForPerson")
public fun Person?.orDefault(): pbandk.examples.addressbook.pb.Person = this ?: Person.defaultInstance

private fun Person.protoMergeImpl(plus: pbandk.Message?): Person = (plus as? Person)?.let {
    it.copy(
        phones = phones + plus.phones,
        lastUpdated = lastUpdated?.plus(plus.lastUpdated) ?: plus.lastUpdated,
        unknownFields = unknownFields + plus.unknownFields
    )
} ?: this

@Suppress("UNCHECKED_CAST")
private fun Person.Companion.decodeWithImpl(u: pbandk.MessageDecoder): Person {
    var name = ""
    var id = 0
    var email = ""
    var phones: pbandk.ListWithSize.Builder<pbandk.examples.addressbook.pb.Person.PhoneNumber>? = null
    var lastUpdated: pbandk.wkt.Timestamp? = null

    val unknownFields = u.readMessage(this) { _fieldNumber, _fieldValue ->
        when (_fieldNumber) {
            1 -> name = _fieldValue as String
            2 -> id = _fieldValue as Int
            3 -> email = _fieldValue as String
            4 -> phones = (phones ?: pbandk.ListWithSize.Builder()).apply { this += _fieldValue as Sequence<pbandk.examples.addressbook.pb.Person.PhoneNumber> }
            5 -> lastUpdated = _fieldValue as pbandk.wkt.Timestamp
        }
    }
    return Person(name, id, email, pbandk.ListWithSize.Builder.fixed(phones),
        lastUpdated, unknownFields)
}

@pbandk.Export
@pbandk.JsName("orDefaultForPersonPhoneNumber")
public fun Person.PhoneNumber?.orDefault(): pbandk.examples.addressbook.pb.Person.PhoneNumber = this ?: Person.PhoneNumber.defaultInstance

private fun Person.PhoneNumber.protoMergeImpl(plus: pbandk.Message?): Person.PhoneNumber = (plus as? Person.PhoneNumber)?.let {
    it.copy(
        unknownFields = unknownFields + plus.unknownFields
    )
} ?: this

@Suppress("UNCHECKED_CAST")
private fun Person.PhoneNumber.Companion.decodeWithImpl(u: pbandk.MessageDecoder): Person.PhoneNumber {
    var number = ""
    var type: pbandk.examples.addressbook.pb.Person.PhoneType = pbandk.examples.addressbook.pb.Person.PhoneType.fromValue(0)

    val unknownFields = u.readMessage(this) { _fieldNumber, _fieldValue ->
        when (_fieldNumber) {
            1 -> number = _fieldValue as String
            2 -> type = _fieldValue as pbandk.examples.addressbook.pb.Person.PhoneType
        }
    }
    return Person.PhoneNumber(number, type, unknownFields)
}

@pbandk.Export
@pbandk.JsName("orDefaultForAddressBook")
public fun AddressBook?.orDefault(): pbandk.examples.addressbook.pb.AddressBook = this ?: AddressBook.defaultInstance

private fun AddressBook.protoMergeImpl(plus: pbandk.Message?): AddressBook = (plus as? AddressBook)?.let {
    it.copy(
        people = people + plus.people,
        unknownFields = unknownFields + plus.unknownFields
    )
} ?: this

@Suppress("UNCHECKED_CAST")
private fun AddressBook.Companion.decodeWithImpl(u: pbandk.MessageDecoder): AddressBook {
    var people: pbandk.ListWithSize.Builder<pbandk.examples.addressbook.pb.Person>? = null

    val unknownFields = u.readMessage(this) { _fieldNumber, _fieldValue ->
        when (_fieldNumber) {
            1 -> people = (people ?: pbandk.ListWithSize.Builder()).apply { this += _fieldValue as Sequence<pbandk.examples.addressbook.pb.Person> }
        }
    }
    return AddressBook(pbandk.ListWithSize.Builder.fixed(people), unknownFields)
}

已经是完美翻译了proto文件到kotlin了，而且这个库也写了一个protobuf的序列化和反序列化的库。但是由于在类描述文件中使用了java8的语法糖，所以这个库的类数量会有点膨胀。导致了其输出的jvm library的体积会有点大。

我全要？

由于上述的种种原因，我们还是打算自己写一套protoc插件。将serialization和pbandk的优点结合在一起，然后生成一个非常简单的kotlin data class，从而满足kmp工程的需要。

目标也比较简单，就是把上面的proto文件，转化成一个更简单的含有kotlin serialization注解的类，然后把其中的描述文件还有继承关系都删除，只保留最简单的data class。

package tutorial
import kotlinx.serialization.Serializable
import kotlinx.serialization.protobuf.ProtoNumber
import kotlinx.serialization.protobuf.ProtoPacked

@Serializable
public data class KPerson(
    @ProtoNumber  val name: String = "",
    @ProtoNumber  val id: Int = 0,
    @ProtoNumber  val email: String = "",
    @ProtoNumber @ProtoPacked  val phones: List<tutorial.KPerson.KPhoneNumber> = emptyList(),
    @ProtoNumber  val lastUpdated: com.google.protobuf.KTimestamp? = null,
) :  Function0<String> {
        @Serializable
        public enum class KPhoneType(val value: Int){
            MOBILE(0),
            HOME(1),
            WORK(2),
            UNRECOGNIZED(-1);

            public companion object {
                public val values: List<KPerson.KPhoneType> by lazy { listOf(MOBILE, HOME, WORK) }
                fun fromValue(value: Int): KPerson.KPhoneType = values.firstOrNull { it.value == value } ?: UNRECOGNIZED
                fun fromName(name: String): KPerson.KPhoneType = values.firstOrNull { it.name == name } ?: throw IllegalArgumentException("No KPhoneType with name: $name")
            }
        }

        @Serializable
        public data class KPhoneNumber(
            @ProtoNumber  val number: String = "",
            @ProtoNumber  val type: Int = 0,
) :  Function0<String> {
                fun  typeEnum() : tutorial.KPerson.KPhoneType  = tutorial.KPerson.KPhoneType.fromValue(type)


                override fun invoke(): String ="tutorial.Person.PhoneNumber" 
        }

        override fun invoke(): String ="tutorial.Person" 
}

@Serializable
public data class KAddressBook(
    @ProtoNumber @ProtoPacked  val people: List<tutorial.KPerson> = emptyList(),
) :  Function0<String> {

        override fun invoke(): String ="tutorial.AddressBook" 
}

大概生成的类信息如上图所示。另外对于proto3还有proto中的特殊语法比如oneof等等都进行了一系列的支持。

syntax = "proto3";
package foobar;

message Value {
    oneof value {
        int32 int_val = 1;
        string str_val = 2;
    }
}

翻译出来的kotlin内容如下。

package foobar
import kotlinx.serialization.Serializable
import kotlinx.serialization.protobuf.ProtoNumber
import kotlinx.serialization.protobuf.ProtoPacked

@Serializable
public data class KValue(
    @ProtoNumber private val intVal: Int?  = null,
    @ProtoNumber private val strVal: String?  = null,
) :  Function0<String> {

        @delegate:Transient
        private val valueNumber by lazy { 
            if( intVal != null) {
                0 
            } else if( strVal != null){
                1 
            } else {
                -1
            }
        }

        public sealed class KValue(val value:Int)

        public object KIntVal : KValue (0)

        public object KStrVal : KValue (1)


        fun <T> valueValue() : T? {
            if(intVal != null){
                return intVal  as T
            } else if(strVal != null){
                return strVal  as T
            } else { return null }
        }

        fun valueType(): KValue ? = valueValues.firstOrNull { it.value == valueNumber }


        companion object {
            val valueValues : List<KValue> by lazy {
                listOf(KIntVal,KStrVal)
            }
        }



        override fun invoke(): String ="foobar.Value" 
}