Sort object listing by relation Many-to-Many

Question

OBJECTIVE: To order a list of eloquent entities based on a column of a Many-to-Many relationship.

To create a system of favorite posts, I created a favorite_posts table that would be the link between the posts table and the users table. This favorite table_posts has a column id, post_id, user_id, created_at and updated_at.

This relationship was created as follows:

//POSTS MODEL
public function favoritePosts()
{
    return $this->belongsToMany(User::class, 'favorite_posts', 'post_id', 'user_id')->withTimestamps();
}

To create this listing, I am making the following query in the repository that serves the model posts:

//POST REPOSITORY
    $posts = $this->model->with([
             'favorite_posts' => function ($return) {
                $return->where('user_id', '=', auth()->user()->id);
          }])->orderBy('favorite_posts.created_at', 'desc')->get();

However, in doing so, the eloquent assembles the following query:

select * from `posts` order by `favorite_posts`.`created_at` desc limit 10 offset 0

That is, it is not done Join in the table favorite_posts, which generates the following error:

Column not found: 1054 Unknown column 'favorite_posts.created_at' in 'order clause'

The goal would be to mount a query like:

    SELECT
    posts.* FROM posts
    INNER JOIN favorite_posts
        ON posts.id=favorite_posts.post_id
WHERE favorite_posts.user_id = 4
ORDER BY favorite_posts.created_at DESC;

I even managed to make it work with query Builder, but since I have many relations in this table using query Builder, it does not come with all mounted objects, so it is not interesting to do by query Builder.

What I need is to return the posts ordered by the creation date of the last record inserted in favorite_posts.

Can you help me, please?