Smooth height change

Question

I’m making a Navbar where it will decrease the height and change the color in the scroll event. My problem is in making this transition smooth. I managed to do for the color, but not for the height. How can I make the height change smooth?

$(window).on('scroll', function(event) {
  if ($(window).scrollTop()) {
    $('.navbar').addClass('bg-dark');
    $('.navbar').removeClass('bg-primary teste');
  } else {
    $('.navbar').removeClass('bg-dark');
    $('.navbar').addClass('bg-primary teste');
  }
  if ($(window).scrollTop() > 100) {
    $('.navbar').removeClass('teste');
  } else {
    $('.navbar').addClass('teste');
  }
});

nav {
  transition: 2s all ease-in-out;
}

.navbar.teste>.container {
  min-height: 54px;
}

.navbar.teste>.container>.navbar-brand {
  padding: 0 15px;
  height: 54px;
  line-height: 54px;
}

.navbar.teste>.container>.navbar-brand>.navbar-toggle {
  margin-top: 18px;
  padding: 9px 10px !important;
}

@media (min-width: 768px) {
  .navbar.teste>.container>div>.navbar-nav>li>a {
    padding-top: 16.5px;
    padding-bottom: 16.5px;
    line-height: 16px;
  }
}

<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css">
<script src="https://code.jquery.com/jquery-3.2.1.slim.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/js/bootstrap.min.js"></script>

<nav class="navbar teste fixed-top navbar-expand-sm navbar-dark bg-primary px-0">
  <div class="container">
    <a class="navbar-brand" href="#">Marca</a>
    <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarContent" aria-controls="navbarContent" aria-expanded="false" aria-label="Toggle navigation">
        <span class="navbar-toggler-icon"></span>
    </button>

    <div class="collapse navbar-collapse" id="navbarContent">
      <ul class="navbar-nav ml-auto">
        <li class="nav-item active"><a class="nav-link" href="#">Projetos</a></li>
        <li class="nav-item"><a class="nav-link" href="#">Notícias</a></li>
        <li class="nav-item"><a class="nav-link" href="#">Eventos</a></li>
        <li class="nav-item"><a class="nav-link" href="#">Biblioteca</a></li>
      </ul>
    </div>
  </div>
</nav>
<h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1>

Answer 1

The height does not decrease smooth because only the navbar was with 2s Transition. Class .test was not changing the Nav style, was changing other tags within the Nav (like the container and the logo) and they did not have the atrabuto Transition, which was exactly what made the Nav decrease.

What I did was set the Nav size and with the class .test, it decreases the size of it. Look how it turned out:

$(window).on('scroll', function(event) {
  if ($(window).scrollTop() > 100) {
    $('.navbar').addClass('bg-dark');
    $('.navbar').removeClass('bg-primary');
    $('.navbar').addClass('teste');
  } else {
    $('.navbar').removeClass('bg-dark');
    $('.navbar').addClass('bg-primary');
    $('.navbar').removeClass('teste');
  }
});

nav {
  transition: 1s all ease-in-out;
}

.navbar {
  height: 65px;
}
.navbar.teste {
  height: 50px;
}

.navbar>.container>.navbar-brand {
  padding: 0 15px;
  line-height: 54px;
}

.navbar>.container>.navbar-brand>.navbar-toggle {
  margin-top: 18px;
  padding: 9px 10px !important;
}

@media (min-width: 768px) {
  .navbar>.container>div>.navbar-nav>li>a {
    padding-top: 16.5px;
    padding-bottom: 16.5px;
    line-height: 16px;
  }
}

<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css">
<script src="https://code.jquery.com/jquery-3.2.1.slim.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/js/bootstrap.min.js"></script>

<nav class="navbar fixed-top navbar-expand-sm navbar-dark bg-primary px-0">
  <div class="container">
    <a class="navbar-brand" href="#">Marca</a>
    <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarContent" aria-controls="navbarContent" aria-expanded="false" aria-label="Toggle navigation">
        <span class="navbar-toggler-icon"></span>
    </button>

    <div class="collapse navbar-collapse" id="navbarContent">
      <ul class="navbar-nav ml-auto">
        <li class="nav-item active"><a class="nav-link" href="#">Projetos</a></li>
        <li class="nav-item"><a class="nav-link" href="#">Notícias</a></li>
        <li class="nav-item"><a class="nav-link" href="#">Eventos</a></li>
        <li class="nav-item"><a class="nav-link" href="#">Biblioteca</a></li>
      </ul>
    </div>
  </div>
</nav>
<h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1>

Answer 2

Young transition with color happens because you are trading one class for another. But with class .teste that you put the height does not occur pq does not have a value reference.

Includes a new class .testex just stop you to see that the problem is the transition between one value and another. Apparently the script is ok...

I made the example with Navbar increasing and with the transition, but you can do as you want by customizing the .textex, I left a comment in CSS

$(window).on('scroll', function(event) {
  if ($(window).scrollTop()) {
$('.navbar').addClass('bg-dark testex');
$('.navbar').removeClass('bg-primary teste');
  } else {
$('.navbar').removeClass('bg-dark testex');
$('.navbar').addClass('bg-primary teste');
  }
  if ($(window).scrollTop() > 100) {
$('.navbar').removeClass('teste');
  } else {
$('.navbar').addClass('teste');
  }
});

nav {
  transition: 2s all ease-in-out;
}

.navbar.testex>.container {
  transition: 2s all ease-in-out;
  min-height: 74px; /* valor de referência pro valor da altura da NavBar */
}

.navbar.teste>.container {
  transition: 2s all ease-in-out;
  min-height: 54px;
}

.navbar.teste>.container>.navbar-brand {
  padding: 0 15px;
  height: 54px;
  line-height: 54px;
}

.navbar.teste>.container>.navbar-brand>.navbar-toggle {
  margin-top: 18px;
  padding: 9px 10px !important;
}

@media (min-width: 768px) {
  .navbar.teste>.container>div>.navbar-nav>li>a {
    padding-top: 16.5px;
    padding-bottom: 16.5px;
    line-height: 16px;
  }
}

<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css">
<script src="https://code.jquery.com/jquery-3.2.1.slim.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/js/bootstrap.min.js"></script>

<nav class="navbar teste fixed-top navbar-expand-sm navbar-dark bg-primary px-0">
  <div class="container">
    <a class="navbar-brand" href="#">Marca</a>
    <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarContent" aria-controls="navbarContent" aria-expanded="false" aria-label="Toggle navigation">
        <span class="navbar-toggler-icon"></span>
    </button>

    <div class="collapse navbar-collapse" id="navbarContent">
      <ul class="navbar-nav ml-auto">
        <li class="nav-item active"><a class="nav-link" href="#">Projetos</a></li>
        <li class="nav-item"><a class="nav-link" href="#">Notícias</a></li>
        <li class="nav-item"><a class="nav-link" href="#">Eventos</a></li>
        <li class="nav-item"><a class="nav-link" href="#">Biblioteca</a></li>
      </ul>
    </div>
  </div>
</nav>
<h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1>
<h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1>
<h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1><h1>Teste</h1>