Performance issues with Asp.Net Core 2.1?

Question

I have an API written in Asp.Net Core 2.1 and I have some performance problem, I am not able to understand if the problem is in the code or the server.

The problem boils down to the following, I have a simple update that just disables a record and saves a log. But when the API is running at a time on the production server, it is called starts to get slower and slower until it starts to Failure. Server memory and processor usage are always below 30%. I set some recycles on IIS and activated Responsecompression but the problem persists.

Follow the Startup codes and one of the methods that have the problem.

Startup.Cs :

using System;
using SPTC.SISPLAN.API.WebSockets;
using SPTC.WebSockets;
using SPTC.Data;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using SPTC.SISPLAN.Service.Interfaces;
using SPTC.SISPLAN.Service;
using SPTC.SISPLAN.Service.Listas;
using SPTC.SISPLAN.Service.Interfaces.Listas;
using Swashbuckle.AspNetCore.Swagger;
using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.IdentityModel.Tokens;
using System.Text;
using Newtonsoft.Json;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc.Authorization;
using Microsoft.AspNetCore.Http;
using SPTC.SISPLAN.Service.Validation;
using Newtonsoft.Json.Serialization;
using AutoMapper;
using Microsoft.Extensions.Hosting;
using IHostingEnvironment = Microsoft.AspNetCore.Hosting.IHostingEnvironment;
using SPTC.SISPLAN.API.HostedServices;

namespace SPTC.SISPLAN.API
{
    public class Startup
    {
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        public void ConfigureServices(IServiceCollection services)
        {
            services.AddDbContextPool<SPTCContext>(options =>
            {
                options.UseSqlServer(Configuration.GetConnectionString("DefaultConnection"));
            });

            services.AddSwaggerGen(s =>
            {
                s.SwaggerDoc("v2", new Info
                {
                    Version = "v2",
                    Title = "SISPLAN API",
                    Description = "Documentação de API do sistema SPTC-SISPLAN",
                });
                s.EnableAnnotations();
            });

            services.AddResponseCompression(options =>
            {
                options.EnableForHttps = true;
            });


            services.AddCors();
            services.AddAutoMapper();

            services.AddTransient<IAcaoService, AcaoService>();
            services.AddTransient<IAcaoLogRevisaoAndamentoService, AcaoLogRevisaoAndamentoService>();
            services.AddTransient<IInstitutoService, InstitutoService>();
            services.AddTransient<IPerfilService, PerfilService>();
            services.AddTransient<IPreservacaoService, PreservacaoService>();
            services.AddTransient<IStatusService, StatusService>();
            services.AddTransient<ITipoLocalService, TipoLocalService>();
            services.AddTransient<ITipoOperacaoService, TipoOperacaoService>();
            services.AddTransient<ITipoPendenciaService, TipoPendenciaService>();

            services.AddTransient<IAndamentoService, AndamentoService>();
            services.AddTransient<IDesignacaoService, DesignacaoService>();
            services.AddTransient<ILocalService, LocalService>();
            services.AddTransient<ILogAcessoUsuarioService, LogAcessoUsuarioService>();
            services.AddTransient<ILogRevisaoAndamentoService, LogRevisaoAndamentoService>();
            services.AddTransient<IOperacaoService, OperacaoService>();
            services.AddTransient<IPendenciaService, PendenciaService>();
            services.AddTransient<IPermissaoService, PermissaoService>();
            services.AddTransient<IPlantaoService, PlantaoService>();
            services.AddTransient<IProtocoloService, ProtocoloService>();
            services.AddTransient<IRegiaoService, RegiaoService>();
            services.AddTransient<ITimeService, TimeService>();
            services.AddTransient<IUnidadeService, UnidadeService>();
            services.AddTransient<IUsuarioService, UsuarioService>();
            services.AddTransient<IUsuarioPlantaoService, UsuarioPlantaoService>();
            services.AddTransient<IVitimaService, VitimaService>();
            services.AddTransient<IConfiguracaoService, ConfiguracaoService>();
            services.AddTransient<IEmergenciaService, EmergenciaService>();
            services.AddTransient<ITelefonesUteisService, TelefonesUteisService>();
            services.AddTransient<IHostedService, ReloadProtocolosTask>();
            services.AddTransient<IHttpContextAccessor, HttpContextAccessor>();
            services.AddTransient<AuthenticatedUser>();

            services.AddWebSocketManager();

            services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
                .AddJwtBearer(options =>
                {
                    options.TokenValidationParameters = new Microsoft.IdentityModel.Tokens.TokenValidationParameters
                    {
                        ValidateIssuer = true,
                        ValidateAudience = true,
                        ValidateLifetime = true,
                        ClockSkew = new TimeSpan(0),
                        ValidateIssuerSigningKey = true,
                        ValidIssuer = "testeIssuer",
                        ValidAudience = "testeAudience",
                        IssuerSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(Configuration["SecurityKey"]))
                    };
                });

            services.AddMvc(config =>
            {
                var policy = new AuthorizationPolicyBuilder()
                    .AddAuthenticationSchemes(JwtBearerDefaults.AuthenticationScheme‌​)
                    .RequireAuthenticatedUser().Build();
                config.Filters.Add(new AuthorizeFilter(policy));
            }).AddJsonOptions(opts =>
            {
                opts.SerializerSettings.ContractResolver = new LowercaseContractResolver();
                opts.SerializerSettings.ReferenceLoopHandling = ReferenceLoopHandling.Ignore;
                opts.SerializerSettings.NullValueHandling = NullValueHandling.Ignore;
            });

            services.AddDistributedRedisCache(options =>
            {
                options.Configuration =
                  Configuration.GetConnectionString("ConexaoRedis");
                options.InstanceName = "Sisplan";
            });

        }

        public void Configure(IApplicationBuilder app, IHostingEnvironment env, IServiceProvider serviceProvider)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
                app.UseSwagger();
                app.UseSwaggerUI(s =>
                {
                    s.SwaggerEndpoint("v2/swagger.json", "SISPLAN API");
                });
            }
            else
            {
                app.UseHttpsRedirection();
                app.UseHsts();
            }

            var wsOptions = new WebSocketOptions()
            {
                KeepAliveInterval = TimeSpan.FromSeconds(60),
                ReceiveBufferSize = 4 * 1024
            };

            app.UseWebSockets(wsOptions);
            app.MapWebSocketManager("/protocolos", serviceProvider.GetService<ProtocolosHandler>());
            app.MapWebSocketManager("/mapa", serviceProvider.GetService<MapaHandler>());
            app.MapWebSocketManager("/unidades", serviceProvider.GetService<UnidadesHandler>());

            app.UseCors(builder => { 
                builder.AllowAnyOrigin(); 
                builder.AllowAnyMethod(); 
                builder.AllowAnyHeader(); 
            });

            app.UseResponseCompression();

            app.UseMvc();

        }

    }

    public class LowercaseContractResolver : DefaultContractResolver
    {
        protected override string ResolvePropertyName(string propertyName)
        {
            return propertyName.ToLowerInvariant();
        }
    }
}

Method :

[Route("Cancelar/{motivo}")]
[HttpPost]
public async Task<ActionResult> Cancelar(Protocolo protocolo, string motivo)
{
    try
    {
        Protocolo _protocolo = _protocoloService
            .Search(p => p.ID == protocolo.ID, p => p.Include(d => d.Designacoes))
            .FirstOrDefault();

        _protocolo.Active = false;

        foreach (Designacao designacao in _protocolo.Designacoes)
        {
            designacao.Active = false;
            designacao.ID_Status = (int)ListaStatus.Cancelado;
        }

        Andamento andamento = new Andamento
        {
            ID_Acao = (int)Acoes.Cancelado,
            ID_Usuario = int.Parse(_user.ID),
            ID_Protocolo = protocolo.ID,
            Detalhes = motivo
        };

        await _protocoloService.Update(_protocolo);
        await _andamentoService.Create(andamento);

        await this.TriggerWebSocketAsync("web", _protocolo.ID);

        return Ok();

    }
    catch (Exception e)
    {
        new TemplateEmailLog().EnviarEmailLog(
                    _configuration.GetValue<String>("EmailLog"),
                    "Protocolos",
                    "Cancelar",
                    _user.ID,
                    protocolo,
                    e);
        return BadRequest("Ocorreu um erro ao cancelar o protocolo.");
    }

}

It is worth mentioning that I used a Stopwatch to measure the running time of the search, update and create, and all do not reach even 1 second even when the answer starts to take longer.

When the system stops responding at once the following exception occurs:

Message (Exception): The Operation was canceled.

Details (Inner Exception): Cannot write to the Response body, the Answer has completed. Object name: 'Httpresponsestream'.

Stacktrace: at System.Net.Websockets.ManagedWebSocket.Sendframefallbackasync(Messageopcode opcode, Boolean endOfMessage, Readonlymemory`1 payloadBuffer, Cancellationtoken cancellationToken) at SPTC.WebSockets.Websockethandler.Sendmessageasync(Websocket socket, String message) in C: Users William.wos Documents Projectsisplan trunk SPTC.Websocket SPTC.Websocket Websocket Websockethandler.Cs:line 34 at SPTC.WebSockets.Websockethandler.Sendmessageasync(String socketId, String message) in C: Users William.wos Documents Projectsisplan trunk SPTC.Websocket SPTC.Websocket Websocket Websockethandler.Cs:line 46 at SPTC.WebSockets.Websockethandler.Sendmessagetoallasync(String message) in C: Users William.wos Documents Projectsisplan trunk SPTC.Websocket SPTC.Websocket Websocket Websockethandler.Cs:line 64 at SPTC.SISPLAN.API.WebSockets.Unidadeshandler.Sendmessagetoallasync(String Sender, Object message, Action) in Z: Renner Systems Projects SISPLAN branches SPTC.SISPLAN.API Fase02 SPTC.SISPLAN.API Websockets Unidadeshandler.Cs:line 25 at SPTC.SISPLAN.API.Controllers.Protocoloscontroller.Triggerwebsocketasync(String origin, Int32 id_protocol, Boolean refused Z: Renner Systems Projects SISPLAN branches SPTC.SISPLAN.API Fase02 SPTC.SISPLAN.API Controllers Protocoloscontroller.Cs:line 153 at SPTC.SISPLAN.API.Controllers.Protocoloscontroller.Cancel(Protocol protocol, String motif) in Z: Renner Systems Projects SISPLAN branches SPTC.SISPLAN.API Fase02 SPTC.SISPLAN.API Controllers Protocoloscontroller.Cs:line 769

Thanks for your help!

Answer 1

Apparently the exception described in stack trace seems to be in timeout error in the method call TriggerWebSocketAsync caused by an attempt to write data to a Websocket whose connection was terminated abruptly by the client without notifying the server. To reproduce this problem, simply pull the network cable from a client that is connected in this Websocket.

This could be solved as follows:

Task.Run(async () =>
{
    await this.TriggerWebSocketAsync("web", _protocolo.ID);
});

In addition, another point of slowness to be observed is in the method EnviarEmailLog. Ideally, your application does not depend on a service as unstable and relatively slow as sending emails in order to generate logs. Which in turn may generate other exceptions.