Introduction to Protocol Buffer and guidelines on how to use it

 

 

Protocol buffer also known as protobuf was designed by google for internal use to store data and to communicate between processes. Google has now provided support for multiple languages and made it public under open source license. It is an easy to use, flexible, efficient, automated mechanism for serializing and de-serializing structured data. As compared to XML it is compact, simpler and faster. Once you define the structure of data, you can then use it to communicate between client and server or between servers. You just need to pass the stream/byte Array from one end to another, rest will be handled by the source code. It provides you options to validate the data as well. It also provides backward compatibility which means you can change your schema without changing the current implementation.

Here we are giving brief information about how to use Protobuf. All examples use C++ code.

How to use

To use protocol buffer, you first need to download the .exe file which is required to convert the proto file into source code files to be used. Please use the following link to download the file and follow the instructions provided on the same link

https://developers.google.com/protocol-buffers/docs/downloads

Here is an example of a simple schema. Save it into. proto file.

message Student {
  required int32 std_id = 1;
  required string std_name = 2;
  optional string email = 3;
}

In the above example we have created sample schema of student, mentioning its ID and Name as required.

message Class {
  required int32 class_id = 1;
  repeated Student student_list = 2;
}

The above example is about a class, which contains a list of students. We have used the ‘repeated’ tag, this allows us to insert multiple instances of students for this class id.

After creating schema in the proto file and downloading .exe from link given above, execute the following command.

protoc -I=$SRC_DIR --cpp_out=$DST_DIR $SRC_DIR/student.proto

After doing this, we need to include these files into a project. After including source files in server code, we just need to convert data them into byte array using protobuf’s predefined function. i.e.

Class class_obj;
Class_obj. SerializeToArray (&output);

Similarly, on client, we just need to receive data and decode it from byte array i.e.

Class class_obj;
Class_obj.ParseFromArray(&input);

 

Data Validation and schema Extension

Data validation is a very powerful concept in protobuf. It gives you the option to make a field required, optional and repeated. At the time of encoding data, you can verify the structure, and it will throw an error if required fields are not filled.

In the protobuf 3, all the fields are optional and you have to check if the values are filled or not. This feature is introduced to allow extension and to support backward compatibility. For e.g. if you have an optional field and you remove it at the backend, now, if you pass its values from the client, it will not raise an error and will assign default value to the variable.

Schema extension and backward compatibility is a powerful feature of Protobuf. You can add new fields in the schema without hurting or changing other server/client endpoints.

Programming Language Independence

There is no language dependence for protobuf, same schema file can be used for all languages. Every language has almost same methods. You just have to send and receive stream.

Why Use Protobuf

Study reveals that in term of processing/performance, the protobuf is 6 times faster than JSON and XML. But one should not switch to Protobuf only for the performance. Protobuf has many other edges over XML and JSON. As the MetaData is already shared in the form of .Proto file on both side and data is sent in the form of BinaryStream/ByteArray, It has short message size compared to XML/JSON. Another reason to switch over Protobuf should be its rich cross Language schema definitions. Parsing of data received is faster than both XML and JSON.

Disadvantages

With all its features and robustness, protobuf has some minor disadvantages as well.

• Protobuf data is not human readable.
• Not too helpful if the data is used directly in the browser for debugging.
• You can’t decode / de-serialize data if you don’t have schema file. I.e. proto file

Conclusion

Protobuf was designed by google. It beats JSON and XML in performance. It has a strong set of features which contains backward compatibility and Language Independence. It is best suitable where there is any need of communication between two endpoints and user wants to use the same schema, need good performance and less maintenance overhead.

 

Please feel free to comment or reach out if you have any questions. In case you need any help with installation, integration, up-gradation and customization for your enterprise applications, please get in touch with us.