FUSE (Filesystem in Userspace) is an interface for userspace programs to export a filesystem to the Linux kernel. The FUSE project consists of two components: the fuse kernel module (maintained in the regular kernel repositories) and the libfuse userspace library. libfuse provides the reference implementation for communicating with the FUSE kernel module.
A FUSE file system is typically implemented as a standalone application that links with libfuse. libfuse provides functions to mount the file system, unmount it, read requests from the kernel, and send responses back.
libfuse offers two APIs: a "high-level", synchronous API, and a "low-level" asynchronous API. In both cases, incoming requests from the kernel are passed to the main program using callbacks. When using the high-level API, the callbacks may work with file names and paths instead of inodes, and processing of a request finishes when the callback function returns. When using the low-level API, the callbacks must work with inodes and responses must be sent explicitly using a separate set of API functions.
The high-level API that is primarily specified in fuse.h. The low-level API that is primarily documented in fuse_lowlevel.h.
The authoritative source of information about libfuse internals (including the protocol used for communication with the FUSE kernel module) is the source code.
However, some people have kindly documented different aspects of FUSE in a more beginner friendly way. While this information is increasingly out of date, it still provides a good overview: