Our current architecture is to align three components to hardware vsync timers:
The flow of our rendering engine is as follows:
CompositorVsyncDispatcheris specific to one window.
CompositorWidgetVsyncObserverwhen remote compositing, or a
CompositorVsyncScheduler::Observerwhen compositing in-process.
VsyncBridgeChildon the UI process, which sends an IPDL message to the
VsyncBridgeParenton the compositor thread of the GPU process, which then dispatches to
RefreshTimerVsyncDispatchernotifies the Chrome
RefreshTimerthat a vsync has occurred.
RefreshTimerVsyncDispatchersends IPC messages to all content processes to tick their respective active
Compositordispatches input events on the Compositor Thread, then composites. Input events are only dispatched on the Compositor Thread on b2g.
RefreshDriverpaints on the Main Thread.
Hardware vsync events from (1), occur on a specific
Display Object. The
Display object is responsible for enabling / disabling vsync on a per connected display basis. For example, if two monitors are connected, two
Display objects will be created, each listening to vsync events for their respective displays. We require one
Display object per monitor as each monitor may have different vsync rates. As a fallback solution, we have one global
Display object that can synchronize across all connected displays. The global
Display is useful if a window is positioned halfway between the two monitors. Each platform will have to implement a specific
Display object to hook and listen to vsync events. As of this writing, both Firefox OS and OS X create their own hardware specific Hardware Vsync Thread that executes after a vsync has occurred. OS X creates one Hardware Vsync Thread per
CVDisplayLinkRef. We do not currently support multiple displays, so we use one global
CVDisplayLinkRef that works across all active displays. On Windows, we have to create a new platform
thread that waits for DwmFlush(), which works across all active displays. Once the thread wakes up from DwmFlush(), the actual vsync timestamp is retrieved from DwmGetCompositionTimingInfo(), which is the timestamp that is actually passed into the compositor and refresh driver.
When a vsync occurs on a
Display, the Hardware Vsync Thread callback fetches all
CompositorVsyncDispatchers associated with the
CompositorVsyncDispatcher is notified that a vsync has occurred with the vsync’s timestamp. It is the responsibility of the
CompositorVsyncDispatcher to notify the
Compositor that is awaiting vsync notifications. The
Display will then notify the associated
RefreshTimerVsyncDispatcher, which should notify all active
RefreshDrivers to tick.
Display objects are encapsulated in a
VsyncSource object. The
VsyncSource object lives in
gfxPlatform and is instantiated only on the parent process when
gfxPlatform is created. The
VsyncSource is destroyed when
gfxPlatform is destroyed. It can also be destroyed when the layout frame rate pref (or other prefs that influence frame rate) are changed. This may mean we switch from hardware to software vsync (or vice versa) at runtime. During the switch, there may briefly be 2 vsync sources. Otherwise, there is only one
VsyncSource object throughout the entire lifetime of Firefox. Each platform is expected to implement their own
VsyncSource to manage vsync events. On OS X, this is through
CVDisplayLinkRef. On Windows, it should be through
CompositorVsyncDispatcher is notified of the vsync event, the
CompositorVsyncScheduler::Observer associated with the
CompositorVsyncDispatcher begins execution. Since the
CompositorVsyncDispatcher executes on the Hardware Vsync Thread and the
Compositor composites on the
CompositorVsyncScheduler::Observer posts a task to the
CompositorBridgeParent then composites. The model where the
CompositorVsyncDispatcher notifies components on the Hardware Vsync Thread, and the component schedules the task on the appropriate thread is used everywhere.
CompositorVsyncScheduler::Observer listens to vsync events as needed and stops listening to vsync when composites are no longer scheduled or required. Every
CompositorBridgeParent is associated and tied to one
CompositorVsyncScheduler::Observer, which is associated with the
CompositorBridgeParent is associated with one widget and is created when a new platform window or
nsBaseWidget is created. The
nsBaseWidget all have the same lifetimes, which are created and destroyed together.
When compositing out-of-process, this model changes slightly. In this case there are effectively two observers: a UI process observer (
CompositorWidgetVsyncObserver), and the
CompositorVsyncScheduler::Observer in the GPU process. There are also two dispatchers: the widget dispatcher in the UI process (
CompositorVsyncDispatcher), and the IPDL-based dispatcher in the GPU process (
CompositorBridgeParent::NotifyVsync). The UI process observer and the GPU process dispatcher are linked via an IPDL protocol called PVsyncBridge.
PVsyncBridge is a top-level protocol for sending vsync notifications to the compositor thread in the GPU process. The compositor controls vsync observation through a separate actor,
PCompositorWidget, which (as a subactor for
CompositorBridgeChild) links the compositor thread in the GPU process to the main thread in the UI process.
Out-of-process compositors do not go through
CompositorVsyncDispatcher directly. Instead, the
CompositorWidgetDelegate in the UI process creates one, and gives it a
CompositorWidgetVsyncObserver. This observer forwards notifications to a Vsync I/O thread, where
VsyncBridgeChild then forwards the notification again to the compositor thread in the GPU process. The notification is received by a
VsyncBridgeParent. The GPU process uses the layers ID in the notification to find the correct compositor to dispatch the notification to.
CompositorVsyncDispatcher executes on the Hardware Vsync Thread. It contains references to the
nsBaseWidget it is associated with and has a lifetime equal to the
CompositorVsyncDispatcher is responsible for notifying the
CompositorBridgeParent that a vsync event has occurred. There can be multiple
CompositorVsyncDispatcher per window. The only responsibility of the
CompositorVsyncDispatcher is to notify components when a vsync event has occurred, and to stop listening to vsync when no components require vsync events. We require one
CompositorVsyncDispatcher per window so that we can handle multiple
Displays. When compositing in-process, the
CompositorVsyncDispatcher is attached to the CompositorWidget for the window. When out-of-process, it is attached to the CompositorWidgetDelegate, which forwards observer notifications over IPDL. In the latter case, its lifetime is tied to a CompositorSession rather than the nsIWidget.
VsyncSource has an API to switch a
CompositorVsyncDispatcher from one
Display to another
Display. For example, when one window either goes into full screen mode or moves from one connected monitor to another. When one window moves to another monitor, we expect a platform specific notification to occur. The detection of when a window enters full screen mode or moves is not covered by Silk itself, but the framework is built to support this use case. The expected flow is that the OS notification occurs on
nsIWidget, which retrieves the associated
CompositorVsyncDispatcher then notifies the
VsyncSource to switch to the correct
CompositorVsyncDispatcher is connected to. Because the notification works through the
nsIWidget, the actual switching of the
CompositorVsyncDispatcher to the correct
Display should occur on the Main Thread. The current implementation of Silk does not handle this case and needs to be built out.
CompositorVsyncScheduler::Observer handles the vsync notifications and interactions with the
CompositorVsyncDispatcher. When the
Compositor requires a scheduled composite, it notifies the
CompositorVsyncScheduler::Observer that it needs to listen to vsync. The
CompositorVsyncScheduler::Observer then observes / unobserves vsync as needed from the
CompositorVsyncDispatcher to enable composites.
GeckoTouchDispatcher is a singleton that resamples touch events to smooth out jank while tracking a user’s finger. Because input and composite are linked together, the
CompositorVsyncScheduler::Observer has a reference to the
GeckoTouchDispatcher and vice versa.
One large goal of Silk is to align touch events with vsync events. On Firefox OS, touchscreens often have different touch scan rates than the display refreshes. A Flame device has a touch refresh rate of 75 HZ, while a Nexus 4 has a touch refresh rate of 100 HZ, while the device’s display refresh rate is 60HZ. When a vsync event occurs, we resample touch events, and then dispatch the resampled touch event to APZ. Touch events on Firefox OS occur on a Touch Input Thread whereas they are processed by APZ on the APZ Controller Thread. We use Google Android’s touch resampling algorithm to resample touch events.
Currently, we have a strict ordering between Composites and touch events. When a touch event occurs on the Touch Input Thread, we store the touch event in a queue. When a vsync event occurs, the
CompositorVsyncDispatcher notifies the
Compositor of a vsync event, which notifies the
GeckoTouchDispatcher processes the touch event first on the APZ Controller Thread, which is the same as the Compositor Thread on b2g, then the
Compositor finishes compositing. We require this strict ordering because if a vsync notification is dispatched to both the
GeckoTouchDispatcher at the same time, a race condition occurs between processing the touch event and therefore position versus compositing. In practice, this creates very janky scrolling. As of this writing, we have not analyzed input events on desktop platforms.
One slight quirk is that input events can start a composite, for example during a scroll and after the
Compositor is no longer listening to vsync events. In these cases, we notify the
Compositor to observe vsync so that it dispatches touch events. If touch events were not dispatched, and since the
Compositor is not listening to vsync events, the touch events would never be dispatched. The
GeckoTouchDispatcher handles this case by always forcing the
Compositor to listen to vsync events while touch events are occurring.
When the nsBaseWidget shuts down - It calls nsBaseWidget::DestroyCompositor on the Gecko Main Thread. During nsBaseWidget::DestroyCompositor, it first destroys the CompositorBridgeChild. CompositorBridgeChild sends a sync IPC call to CompositorBridgeParent::RecvStop, which calls CompositorBridgeParent::Destroy. During this time, the main thread is blocked on the parent process. CompositorBridgeParent::RecvStop runs on the Compositor thread and cleans up some resources, including setting the
CompositorVsyncScheduler::Observer to nullptr. CompositorBridgeParent::RecvStop also explicitly keeps the CompositorBridgeParent alive and posts another task to run CompositorBridgeParent::DeferredDestroy on the Compositor loop so that all ipdl code can finish executing. The
CompositorVsyncScheduler::Observer also unobserves from vsync and cancels any pending composite tasks. Once CompositorBridgeParent::RecvStop finishes, the main thread in the parent process continues shutting down the nsBaseWidget.
At the same time, the Compositor thread is executing tasks until CompositorBridgeParent::DeferredDestroy runs, which flushes the compositor message loop. Now we have two tasks as both the nsBaseWidget releases a reference to the Compositor on the main thread during destruction and the CompositorBridgeParent::DeferredDestroy releases a reference to the CompositorBridgeParent on the Compositor Thread. Finally, the CompositorBridgeParent itself is destroyed on the main thread once both references are gone due to explicit main thread destruction.
CompositorVsyncScheduler::Observer, any accesses to the widget after nsBaseWidget::DestroyCompositor executes are invalid. Any accesses to the compositor between the time the nsBaseWidget::DestroyCompositor runs and the CompositorVsyncScheduler::Observer’s destructor runs aren’t safe yet a hardware vsync event could occur between these times. Since any tasks posted on the Compositor loop after CompositorBridgeParent::DeferredDestroy is posted are invalid, we make sure that no vsync tasks can be posted once CompositorBridgeParent::RecvStop executes and DeferredDestroy is posted on the Compositor thread. When the sync call to CompositorBridgeParent::RecvStop executes, we explicitly set the CompositorVsyncScheduler::Observer to null to prevent vsync notifications from occurring. If vsync notifications were allowed to occur, since the
CompositorVsyncScheduler::Observer’s vsync notification executes on the hardware vsync thread, it would post a task to the Compositor loop and may execute after CompositorBridgeParent::DeferredDestroy. Thus, we explicitly shut down vsync events in the
CompositorVsyncScheduler::Observer during nsBaseWidget::Shutdown to prevent any vsync tasks from executing after CompositorBridgeParent::DeferredDestroy.
CompositorVsyncDispatcher may be destroyed on either the main thread or Compositor Thread, since both the nsBaseWidget and
CompositorVsyncScheduler::Observer race to destroy on different threads. nsBaseWidget is destroyed on the main thread and releases a reference to the
CompositorVsyncDispatcher during destruction. The
CompositorVsyncScheduler::Observer has a race to be destroyed either during CompositorBridgeParent shutdown or from the
GeckoTouchDispatcher which is destroyed on the main thread with ClearOnShutdown. Whichever object, the CompositorBridgeParent or the
GeckoTouchDispatcher is destroyed last will hold the last reference to the
CompositorVsyncDispatcher, which destroys the object.
The Refresh Driver is ticked from a single active timer. The assumption is that there are multiple
RefreshDrivers connected to a single
RefreshTimer. There are two
RefreshTimers: an active and an inactive
RefreshTimer. Each Tab has its own
RefreshDriver, which connects to one of the global
RefreshTimers execute on the Main Thread and tick their connected
RefreshDrivers. We do not want to break this model of multiple
RefreshDrivers per a set of two global
RefreshDriver switches between the active and inactive
Instead, we create a new
VsyncRefreshTimer which ticks based on vsync messages. We replace the current active timer with a
VsyncRefreshTimer. All tabs will then tick based on this new active timer. Since the
RefreshTimer has a lifetime of the process, we only need to create a single
Display when Firefox starts. Even if we do not have any content processes, the Chrome process will still need a
VsyncRefreshTimer, thus we can associate the
RefreshTimerVsyncDispatcher with each
When Firefox starts, we initially create a new
VsyncRefreshTimer in the Chrome process. The
VsyncRefreshTimer will listen to vsync notifications from
RefreshTimerVsyncDispatcher on the global
Display. When nsRefreshDriver::Shutdown executes, it will delete the
VsyncRefreshTimer. This creates a problem as all the
RefreshTimers are currently manually memory managed whereas
VsyncObservers are ref counted. To work around this problem, we create a new
RefreshDriverVsyncObserver as an inner class to
VsyncRefreshTimer, which actually receives vsync notifications. It then ticks the
With Content processes, the start up process is more complicated. We send vsync IPC messages via the use of the PBackground thread on the parent process, which allows us to send messages from the Parent process’ without waiting on the main thread. This sends messages from the Parent::PBackground Thread to the Child::Main Thread. The main thread receiving IPC messages on the content process is acceptable because
RefreshDrivers must execute on the main thread. However, there is some amount of time required to setup the IPC connection upon process creation and during this time, the
RefreshDrivers must tick to set up the process. To get around this, we initially use software
RefreshTimers that already exist during content process startup and swap in the
VsyncRefreshTimer once the IPC connection is created.
During nsRefreshDriver::ChooseTimer, we create an async PBackground IPC open request to create a
VsyncChild. At the same time, we create a software
RefreshTimer and tick the
RefreshDrivers as normal. Once the PBackground callback is executed and an IPC connection exists, we swap all
RefreshDrivers currently associated with the active
RefreshTimer and swap the
RefreshDrivers to use the
VsyncRefreshTimer. Since all interactions on the content process occur on the main thread, there are no need for locks. The
VsyncParent listens to vsync events through the
VsyncRefreshTimerDispatcher on the parent side and sends vsync IPC messages to the
VsyncChild notifies the
VsyncRefreshTimer on the content process.
During the shutdown process of the content process, ActorDestroy is called on the
VsyncParent due to the normal PBackground shutdown process. Once ActorDestroy is called, no IPC messages should be sent across the channel. After ActorDestroy is called, the IPDL machinery will delete the VsyncParent/Child pair. The
VsyncParent, due to being a
VsyncObserver, is ref counted. After
VsyncParent::ActorDestroy is called, it unregisters itself from the
RefreshTimerVsyncDispatcher, which holds the last reference to the
VsyncParent, and the object will be deleted.
Thus the overall flow during normal execution is:
On the parent process, newer vsync messages update a vsync timestamp but do not actually queue any tasks on the main thread. Once the parent process’ main thread executes the refresh driver tick, it uses the most updated vsync timestamp to tick the refresh driver. After the refresh driver has ticked, one single vsync message is queued for another refresh driver tick task. On the content process, the IPDL
compress keyword automatically compresses IPC messages.
In order to have multiple monitor support for the
RefreshDrivers, we have multiple active
RefreshTimer is associated with a specific
Display via an id and tick when it’s respective
Display vsync occurs. We have N RefreshTimers, where N is the number of connected displays. Each
RefreshTimer still has multiple
When a tab or window changes monitors, the
nsIWidget receives a display changed notification. Based on which display the window is on, the window switches to the correct
CompositorVsyncDispatcher on the parent process based on the display id. Each
TabParent should also send a notification to their child. Each
TabChild, given the display ID, switches to the correct
RefreshTimer associated with the display ID. When each display vsync occurs, it sends one IPC message to notify vsync. The vsync message contains a display ID, to tick the appropriate
RefreshTimer on the content process. There is still only one VsyncParent/VsyncChild pair, just each vsync notification will include a display ID, which maps to the correct
VsyncObservers are notified on the Hardware Vsync Thread. It is the responsibility of the
VsyncObservers to post tasks to their respective correct thread. For example, the
CompositorVsyncScheduler::Observer will be notified on the Hardware Vsync Thread, and post a task to the Compositor Thread to do the actual composition.