Skip to content

[PyTorch] End-to-end MoE grouped tensor support in grouped MLP #2466

@ptrendx

Description

@ptrendx

Create TransformerLayer implementation to enable full end-to-end support for grouped tensor flows as required by MoE training and inference workflows. This could be achieved by either extending the existing TransformerLayer or creating a separate layer.
This is both for the ease of use and easier end-to-end testing and benchmarking MoE workloads.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions