SDK Delegate Integration

Delegate backends are a prominent component of on-device models due to their flexibility in defining behavior. A side effect of this flexibility is that it operates as an opaque transformation. This obfuscates rich associations and mutations that are valuable in post-processing.

• For example, if two different operator fusions were to occur within a delegate, post processing wouldn’t be able to separate the two transformations.

Specifically, it makes associating runtime information (such as profiling results) through delegated graphs difficult. Delegate Debug Identifiers provides a framework through which delegate authors can propagate this information and utilize it for post run analysis.

The preparation is broken down into three stages:

• Ahead-of-time (AOT): Delegate authors generate a Debug Handle Map.

• Runtime: Delegate authors log using the Delegate Debug Identifiers registered AOT in the Debug Handle Map.

• Deserialization: Delegate authors provide a parser for custom metadata in delegate events.

Ahead-of-Time

Delegate authors propagate what transformations occur in a lowered backend by returning a Debug Handle Map from the backend implementation.

Generating a Debug Handle Map

Debug Handle Maps communicate what transformations occurred in a backend by mapping Delegate Debug Identifiers to debug handles.

Delegate Debug Identifiers are generated or user-provided identifiers for representing points of interest during runtime. Recall that debug handles are unique identifiers to operator instances in the model graph.

For example:

• { 0: (10, 11), 1: (11, 12) }: Identifiers 0 and 1 in the runtime correspond to operators with the debug handles (10, 11) and (11, 12) respectively.

• { “fused_op_1_2_3”: (11, 12, 15) }: Identifier “fused_op_1_2_3” in the runtime corresponds to operators with debug handles (11, 12, 15), and 11, 12, 15 corresponds to the op 1, op 2 and op 3.

Note

Identifiers are a means of connecting runtime results to the model graph; the interpretation of the identifiers is defined by the delegate author.

Debug Handle Maps are constructed through the use of DelegateMappingBuilder and returned as a part of PreprocessResult.

class PreprocessResult:
    processed_bytes: bytes = bytes()

    debug_handle_map: Optional[
        Union[Dict[int, Tuple[int]], Dict[str, Tuple[int]]]
    ] = None

PreprocessResult is defined here.

DelegateMappingBuilder

DelegateMappingBuilder is a helper class for managing and constructing Debug Handle Maps. The result of the builder should be passed in when constructing PreprocessResult.

DelegateMappingBuilder is defined here

A DelegateMappingBuilder instance can be constructed in one of 2 modes: manual identifiers or generated identifiers.

# Manual Identifiers, Default
builder = DelegateMappingBuilder(generated_identifiers=False)

# Generated Identifiers
builder = DelegateMappingBuilder(generated_identifiers=True)

With manual identifiers, users pass in a Delegate Debug Identifier when creating entries. With generated identifiers, the builder will auto-assign a Delegate Debug Identifier.

To add an entry to the Debug Handle Map, use insert_delegate_mapping_entry. It associates one of fx.Node(s) or debug handles(s) (sourced from node.meta[“debug_handle”]) to an optional Delegate Debug Identifier (used for the manual identifiers). The identifier recorded is returned from the call.

def insert_delegate_mapping_entry(
    self,
    nodes: Optional[Union[Node, List[Node]]] = None,
    handles: Optional[Union[int, List[int]]] = None,
    identifier: Optional[Union[int, str]] = None,
) -> Union[int, str]:

To retrieve the Debug Handle Map, use get_delegate_mapping.

def get_delegate_mapping(
    self,
) -> Union[Dict[int, Tuple[int]], Dict[str, Tuple[int]]]

A demo of the AOT mapping can be found here

Runtime

Corresponding to the AOT map, the runtime then defines the functionality through which these events are logged.

Real-Time Logging

ExecuTorch allows you to log in real time. Real time Logging is useful when timestamps are available as the execution occurs. It provides minimal overhead and is intuitive for authors to call.

To log events in real-time (for example, explicitly denoting the profiling start and stop), event_tracer_start_profiling_delegate is used to create an EventEntry and event_tracer_end_profiling_delegate is used to conclude the EventEntry for the provided EventTracer.

To start an EventTracerEntry using event_tracer_start_profiling_delegate, the Delegate Debug Identifier (provided AOT to the debug_handle_map) is passed as either the name or delegate_debug_id argument depending on the Delegate Debug Identifier type (str and int respectively)

EventTracerEntry event_tracer_start_profiling_delegate(
    EventTracer* event_tracer,
    const char* name,
    DebugHandle delegate_debug_id)

To conclude an EventTracerEntry, event_tracer_end_profiling_delegate is simply provided the original EventTracerEntry.

Optionally, additional runtime metadata can also be logged at this point.

void event_tracer_end_profiling_delegate(
    EventTracer* event_tracer,
    EventTracerEntry event_tracer_entry,
    const void* metadata = nullptr,
    size_t metadata_len = 0)

Post-Time Logging

ExecuTorch also allows you to log in post time. Some runtime settings don’t have access to timestamps while it is executing. Post-Time Logging enables authors to still be able to log these events.

To log events in post (for example, logging start and end time simultaneously) event_tracer_log_profiling_delegate is called with a combination of the arguments used in the real-time logging API’s and timestamps.

void event_tracer_log_profiling_delegate(
    EventTracer* event_tracer,
    const char* name,
    DebugHandle delegate_debug_id,
    et_timestamp_t start_time,
    et_timestamp_t end_time,
    const void* metadata = nullptr,
    size_t metadata_len = 0)

A demo of the runtime code can be found here.

Surfacing custom metadata from delegate events

As seen in the runtime logging API’s above, users can log an array of bytes along with their delegate profiling event. We make this data available for users in post processing via the Inspector API.

Users can pass a metadata parser when creating an instance of the Inspector. The parser is a callable that deserializes the data and returns a list of strings or a dictionary containing key-value pairs. The deserialized data is then added back to the corresponding event in the event block for user consumption. Here’s an example of how to write this parser:

NOTE: The input to the deserializer is a list where each entry is a series of bytes (essentially each entry is an immutable bytearray). Users are expected to iterate over this list, deserialize each entry and then return it in the expected format which is either a list of strings, or a dict.

Inspector(
    etdump_path=etdump_path,
    # Optional
    etrecord=etrecord_path,
    # Optional, only needed if debugging was enabled.
    buffer_path=buffer_path,
    delegate_metadata_parser=parse_delegate_metadata
)


def parse_delegate_metadata(delegate_metadatas: List[bytes]) -> Union[List[str], Dict[str, Any]]:
    metadata_str = []
    for metadata_bytes in delegate_metadatas:
        metadata_str += str(metadata_bytes)
    return metadata_str