"Fossies" - the Fresh Open Source Software Archive

Member "flatbuffers-23.1.21/docs/source/Schemas.md" (21 Jan 2023, 29414 Bytes) of package /linux/misc/flatbuffers-23.1.21.tar.gz:

As a special service "Fossies" has tried to format the requested source page into HTML format (assuming markdown format). Alternatively you can here view or download the uninterpreted source code file. A member file download can also be achieved by clicking within a package contents listing on the according byte size field. See also the last Fossies "Diffs" side-by-side code changes report for "Schemas.md": 22.9.29_vs_22.10.25.

Writing a schema {#flatbuffers_guide_writing_schema}

The syntax of the schema language (aka IDL, Interface Definition Language) should look quite familiar to users of any of the C family of languages, and also to users of other IDLs. Let's look at an example first:

// example IDL file

namespace MyGame;

attribute "priority";

enum Color : byte { Red = 1, Green, Blue }

union Any { Monster, Weapon, Pickup }

struct Vec3 {
  x:float;
  y:float;
  z:float;
}

table Monster {
  pos:Vec3;
  mana:short = 150;
  hp:short = 100;
  name:string;
  friendly:bool = false (deprecated, priority: 1);
  inventory:[ubyte];
  color:Color = Blue;
  test:Any;
}

root_type Monster;

(Weapon & Pickup not defined as part of this example).

Tables

Tables are the main way of defining objects in FlatBuffers, and consist of a name (here Monster) and a list of fields. Each field has a name, a type, and optionally a default value. If the default value is not specified in the schema, it will be 0 for scalar types, or null for other types. Some languages support setting a scalar's default to null. This makes the scalar optional.

Fields do not have to appear in the wire representation, and you can choose to omit fields when constructing an object. You have the flexibility to add fields without fear of bloating your data. This design is also FlatBuffer's mechanism for forward and backwards compatibility. Note that:

See "Schema evolution examples" below for more on this topic.

Structs

Similar to a table, only now none of the fields are optional (so no defaults either), and fields may not be added or be deprecated. Structs may only contain scalars or other structs. Use this for simple objects where you are very sure no changes will ever be made (as quite clear in the example Vec3). Structs use less memory than tables and are even faster to access (they are always stored in-line in their parent object, and use no virtual table).

Types

Built-in scalar types are

The type names in parentheses are alias names such that for example uint8 can be used in place of ubyte, and int32 can be used in place of int without affecting code generation.

Built-in non-scalar types:

You can't change types of fields once they're used, with the exception of same-size data where a reinterpret_cast would give you a desirable result, e.g. you could change a uint to an int if no values in current data use the high bit yet.

Arrays

Arrays are a convenience short-hand for a fixed-length collection of elements. Arrays can be used to replace the following schema:

struct Vec3 {
    x:float;
    y:float;
    z:float;
}

with the following schema:

struct Vec3 {
    v:[float:3];
}

Both representations are binary equivalent.

Arrays are currently only supported in a struct.

Default, Optional and Required Values

There are three, mutually exclusive, reactions to the non-presence of a table's field in the binary data:

  1. Default valued fields will return the default value (as defined in the schema).
  2. Optional valued fields will return some form of null depending on the local language. (In a sense, null is the default value).
  3. Required fields will cause an error. Flatbuffer verifiers would consider the whole buffer invalid. See the required tag below.

When writing a schema, values are a sequence of digits. Values may be optionally followed by a decimal point (.) and more digits, for float constants, or optionally prefixed by a -. Floats may also be in scientific notation; optionally ending with an e or E, followed by a + or - and more digits. Values can also be the keyword null.

Only scalar values can have defaults, non-scalar (string/vector/table) fields default to null when not present.

You generally do not want to change default values after they're initially defined. Fields that have the default value are not actually stored in the serialized data (see also Gotchas below). Values explicitly written by code generated by the old schema old version, if they happen to be the default, will be read as a different value by code generated with the new schema. This is slightly less bad when converting an optional scalar into a default valued scalar since non-presence would not be overloaded with a previous default value. There are situations, however, where this may be desirable, especially if you can ensure a simultaneous rebuild of all code.

Enums

Define a sequence of named constants, each with a given value, or increasing by one from the previous one. The default first value is 0. As you can see in the enum declaration, you specify the underlying integral type of the enum with : (in this case byte), which then determines the type of any fields declared with this enum type.

Only integer types are allowed, i.e. byte, ubyte, short ushort, int, uint, long and ulong.

Typically, enum values should only ever be added, never removed (there is no deprecation for enums). This requires code to handle forwards compatibility itself, by handling unknown enum values.

Unions

Unions share a lot of properties with enums, but instead of new names for constants, you use names of tables. You can then declare a union field, which can hold a reference to any of those types, and additionally a field with the suffix _type is generated that holds the corresponding enum value, allowing you to know which type to cast to at runtime.

It's possible to give an alias name to a type union. This way a type can even be used to mean different things depending on the name used:

table PointPosition { x:uint; y:uint; }
table MarkerPosition {}
union Position {
  Start:MarkerPosition,
  Point:PointPosition,
  Finish:MarkerPosition
}

Unions contain a special NONE marker to denote that no value is stored so that name cannot be used as an alias.

Unions are a good way to be able to send multiple message types as a FlatBuffer. Note that because a union field is really two fields, it must always be part of a table, it cannot be the root of a FlatBuffer by itself.

If you have a need to distinguish between different FlatBuffers in a more open-ended way, for example for use as files, see the file identification feature below.

There is an experimental support only in C++ for a vector of unions (and types). In the example IDL file above, use [Any] to add a vector of Any to Monster table. There is also experimental support for other types besides tables in unions, in particular structs and strings. There's no direct support for scalars in unions, but they can be wrapped in a struct at no space cost.

Namespaces

These will generate the corresponding namespace in C++ for all helper code, and packages in Java. You can use . to specify nested namespaces / packages.

Includes

You can include other schemas files in your current one, e.g.:

include "mydefinitions.fbs";

This makes it easier to refer to types defined elsewhere. include automatically ensures each file is parsed just once, even when referred to more than once.

When using the flatc compiler to generate code for schema definitions, only definitions in the current file will be generated, not those from the included files (those you still generate separately).

Root type

This declares what you consider to be the root table of the serialized data. This is particularly important for parsing JSON data, which doesn't include object type information.

File identification and extension

Typically, a FlatBuffer binary buffer is not self-describing, i.e. it needs you to know its schema to parse it correctly. But if you want to use a FlatBuffer as a file format, it would be convenient to be able to have a "magic number" in there, like most file formats have, to be able to do a sanity check to see if you're reading the kind of file you're expecting.

Now, you can always prefix a FlatBuffer with your own file header, but FlatBuffers has a built-in way to add an identifier to a FlatBuffer that takes up minimal space, and keeps the buffer compatible with buffers that don't have such an identifier.

You can specify in a schema, similar to root_type, that you intend for this type of FlatBuffer to be used as a file format:

file_identifier "MYFI";

Identifiers must always be exactly 4 characters long. These 4 characters will end up as bytes at offsets 4-7 (inclusive) in the buffer.

For any schema that has such an identifier, flatc will automatically add the identifier to any binaries it generates (with -b), and generated calls like FinishMonsterBuffer also add the identifier. If you have specified an identifier and wish to generate a buffer without one, you can always still do so by calling FlatBufferBuilder::Finish explicitly.

After loading a buffer, you can use a call like MonsterBufferHasIdentifier to check if the identifier is present.

Note that this is best for open-ended uses such as files. If you simply wanted to send one of a set of possible messages over a network for example, you'd be better off with a union.

Additionally, by default flatc will output binary files as .bin. This declaration in the schema will change that to whatever you want:

file_extension "ext";

RPC interface declarations

You can declare RPC calls in a schema, that define a set of functions that take a FlatBuffer as an argument (the request) and return a FlatBuffer as the response (both of which must be table types):

rpc_service MonsterStorage {
  Store(Monster):StoreResponse;
  Retrieve(MonsterId):Monster;
}

What code this produces and how it is used depends on language and RPC system used, there is preliminary support for GRPC through the --grpc code generator, see grpc/tests for an example.

Comments & documentation

May be written as in most C-based languages. Additionally, a triple comment (///) on a line by itself signals that a comment is documentation for whatever is declared on the line after it (table/struct/field/enum/union/element), and the comment is output in the corresponding C++ code. Multiple such lines per item are allowed.

Attributes

Attributes may be attached to a declaration, behind a field/enum value, or after the name of a table/struct/enum/union. These may either have a value or not. Some attributes like deprecated are understood by the compiler; user defined ones need to be declared with the attribute declaration (like priority in the example above), and are available to query if you parse the schema at runtime. This is useful if you write your own code generators/editors etc., and you wish to add additional information specific to your tool (such as a help text).

Current understood attributes:

JSON Parsing

The same parser that parses the schema declarations above is also able to parse JSON objects that conform to this schema. So, unlike other JSON parsers, this parser is strongly typed, and parses directly into a FlatBuffer (see the compiler documentation on how to do this from the command line, or the C++ documentation on how to do this at runtime).

Besides needing a schema, there are a few other changes to how it parses JSON:

When parsing JSON, it recognizes the following escape codes in strings:

It also generates these escape codes back again when generating JSON from a binary representation.

When parsing numbers, the parser is more flexible than JSON. A format of numeric literals is more close to the C/C++. According to the [grammar](@ref flatbuffers_grammar), it accepts the following numerical literals:

Guidelines

Efficiency

FlatBuffers is all about efficiency, but to realize that efficiency you require an efficient schema. There are usually multiple choices on how to represent data that have vastly different size characteristics.

It is very common nowadays to represent any kind of data as dictionaries (as in e.g. JSON), because of its flexibility and extensibility. While it is possible to emulate this in FlatBuffers (as a vector of tables with key and value(s)), this is a bad match for a strongly typed system like FlatBuffers, leading to relatively large binaries. FlatBuffer tables are more flexible than classes/structs in most systems, since having a large number of fields only few of which are actually used is still efficient. You should thus try to organize your data as much as possible such that you can use tables where you might be tempted to use a dictionary.

Similarly, strings as values should only be used when they are truly open-ended. If you can, always use an enum instead.

FlatBuffers doesn't have inheritance, so the way to represent a set of related data structures is a union. Unions do have a cost however, so an alternative to a union is to have a single table that has all the fields of all the data structures you are trying to represent, if they are relatively similar / share many fields. Again, this is efficient because non-present fields are cheap.

FlatBuffers supports the full range of integer sizes, so try to pick the smallest size needed, rather than defaulting to int/long.

Remember that you can share data (refer to the same string/table within a buffer), so factoring out repeating data into its own data structure may be worth it.

Style guide

Identifiers in a schema are meant to translate to many different programming languages, so using the style of your "main" language is generally a bad idea.

For this reason, below is a suggested style guide to adhere to, to keep schemas consistent for interoperation regardless of the target language.

Where possible, the code generators for specific languages will generate identifiers that adhere to the language style, based on the schema identifiers.

Formatting (this is less important, but still worth adhering to):

For an example, see the schema at the top of this file.

Gotchas

Schemas and version control

FlatBuffers relies on new field declarations being added at the end, and earlier declarations to not be removed, but be marked deprecated when needed. We think this is an improvement over the manual number assignment that happens in Protocol Buffers (and which is still an option using the id attribute mentioned above).

One place where this is possibly problematic however is source control. If user A adds a field, generates new binary data with this new schema, then tries to commit both to source control after user B already committed a new field also, and just auto-merges the schema, the binary files are now invalid compared to the new schema.

The solution of course is that you should not be generating binary data before your schema changes have been committed, ensuring consistency with the rest of the world. If this is not practical for you, use explicit field ids, which should always generate a merge conflict if two people try to allocate the same id.

Schema evolution examples (tables)

Some examples to clarify what happens as you change a schema:

If we have the following original schema:

table { a:int; b:int; }

And we extend it:

table { a:int; b:int; c:int; }

This is ok. Code compiled with the old schema reading data generated with the new one will simply ignore the presence of the new field. Code compiled with the new schema reading old data will get the default value for c (which is 0 in this case, since it is not specified).

table { a:int (deprecated); b:int; }

This is also ok. Code compiled with the old schema reading newer data will now always get the default value for a since it is not present. Code compiled with the new schema now cannot read nor write a anymore (any existing code that tries to do so will result in compile errors), but can still read old data (they will ignore the field).

table { c:int; a:int; b:int; }

This is NOT ok, as this makes the schemas incompatible. Old code reading newer data will interpret c as if it was a, and new code reading old data accessing a will instead receive b.

table { c:int (id: 2); a:int (id: 0); b:int (id: 1); }

This is ok. If your intent was to order/group fields in a way that makes sense semantically, you can do so using explicit id assignment. Now we are compatible with the original schema, and the fields can be ordered in any way, as long as we keep the sequence of ids.

table { b:int; }

NOT ok. We can only remove a field by deprecation, regardless of whether we use explicit ids or not.

table { a:uint; b:uint; }

This is MAYBE ok, and only in the case where the type change is the same size, like here. If old data never contained any negative numbers, this will be safe to do.

table { a:int = 1; b:int = 2; }

Generally NOT ok. Any older data written that had 0 values were not written to the buffer, and rely on the default value to be recreated. These will now have those values appear to 1 and 2 instead. There may be cases in which this is ok, but care must be taken.

table { aa:int; bb:int; }

Occasionally ok. You've renamed fields, which will break all code (and JSON files!) that use this schema, but as long as the change is obvious, this is not incompatible with the actual binary buffers, since those only ever address fields by id/offset.

Schema evolution examples (unions)

Suppose we have the following schema:

union Foo { A, B }

We can add another variant at the end.

union Foo { A, B, another_a: A }

and this will be okay. Old code will not recognize another_a. However if we add another_a anywhere but the end, e.g.

union Foo { A, another_a: A, B }

this is not okay. When new code writes another_a, old code will misinterpret it as B (and vice versa). However you can explicitly set the union's "discriminant" value like so:

union Foo { A = 1, another_a: A = 3, B = 2 }

This is okay.

union Foo { original_a: A = 1, another_a: A = 3, B = 2 }

Renaming fields will break code and any saved human readable representations, such as json files, but the binary buffers will be the same.


Testing whether a field is present in a table

Most serialization formats (e.g. JSON or Protocol Buffers) make it very explicit in the format whether a field is present in an object or not, allowing you to use this as "extra" information.

FlatBuffers will not write fields that are equal to their default value, sometimes resulting in significant space savings. However, this also means we cannot disambiguate the meaning of non-presence as "written default value" or "not written at all". This only applies to scalar fields since only they support default values. Unless otherwise specified, their default is 0.

If you care about the presence of scalars, most languages support "optional scalars." You can set null as the default value in the schema. null is a value that's outside of all types, so we will always write if add_field is called. The generated field accessor should use the local language's canonical optional type.

Some FlatBufferBuilder implementations have an option called force_defaults that circumvents this "not writing defaults" behavior you can then use IsFieldPresent to query presence. / Another option that works in all languages is to wrap a scalar field in a struct. This way it will return null if it is not present. This will be slightly less ergonomic but structs don't take up any more space than the scalar they represent.

Writing your own code generator.

See [our intermediate representation](@ref intermediate_representation).