Redis caching suddenly becomes slow from mid July’22 without any changes in the application – Urgent help required please

Issue

This Content is from Stack Overflow. Question asked by Pooh005

Urgent help required please
We are struggling very much with a performance issue in Redis. Our application is running on Redis version 3.2.100 and it was running fine for few months but looks to have slow down since July 11th very gradually. Now its very unpredictable and is taking twice the time almost than before making the application very slow during the first hit of the page when it is trying to cache the data in Redis.
We have tried multiple options but to no benefit:
1. Create a new server for Redis and make it as the new Redis server so that it has enough memory available
2. Uninstall and install the Redis service
3. Upgrade to the latest version.
The performance is very unpredictable and the application is really struggling when the cache gets refreshed and it needs to build the cache in the first bite.
Note: Nothing changed in the application code or way Redis is used in the application or even the volume of data being cached but still we had started seeing this performance issue suddenly from 11th July.
The errors we generally see in the Redis server log is as below:

1. 2022-09-03 03:30:42.2650|INFO|RedisRefreshScheduler.BusinessLogic.RedisRefresh|Total keys available before clear: 518329

2022-09-03 03:32:39.6417|INFO|RedisRefreshScheduler.BusinessLogic.RedisRefresh|Error occuredMISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

2. [9488] 03 Sep 06:05:06.058 * 10 changes in 300 seconds. Saving...
[9488] 03 Sep 06:05:06.245 * Background saving started by pid 10388


=== REDIS BUG REPORT START: Cut & paste starting from here ===
Redis version: 3.2.100
[10388] 03 Sep 06:05:12.230 # --- ABORT
[10388] 03 Sep 06:05:12.230 # --- STACK TRACE
redis-server.exe!LogStackTrace(c:releaseredissrcwin32_interopwin32_stacktrace.cpp:95)(0x00000016, 0x49006009, 0x00000000, 0x00000001)
redis-server.exe!AbortHandler(c:releaseredissrcwin32_interopwin32_stacktrace.cpp:208)(0x00000016, 0x00000000, 0x401671E0, 0x401671E0)
redis-server.exe!raise(f:ddvctoolscrtcrtw32miscwinsig.c:587)(0x00000001, 0x00000000, 0x6E6F6971, 0x6E6F6974)
redis-server.exe!abort(f:ddvctoolscrtcrtw32miscabort.c:82)(0x401671E0, 0x6E6F6971, 0x6E6F6971, 0x0000005D)
redis-server.exe!zmalloc_default_oom(c:releaseredissrczmalloc.c:136)(0x6E800000, 0x6E6F6971, 0x00000001, 0x4004A472)
redis-server.exe!zmalloc(c:releaseredissrczmalloc.c:146)(0x6E6F6974, 0x0000006A, 0x0014F1F0, 0x400BFA57)
redis-server.exe!rdbSaveLzfStringObject(c:releaseredissrcrdb.c:281)(0x6E6F6974, 0x0014F2B0, 0x184D3760, 0x49006009)
redis-server.exe!rdbSaveRawString(c:releaseredissrcrdb.c:349)(0x00000001, 0x184D3760, 0x44329FEC, 0x400ACC91)
redis-server.exe!rdbSaveObject(c:releaseredissrcrdb.c:580)(0x4EFB3473, 0x0014F2B0, 0x0014F2B0, 0x00000001)
redis-server.exe!rdbSaveKeyValuePair(c:releaseredissrcrdb.c:721)(0x0014F2B0, 0x00000001, 0x0014F2B0, 0x03C070FC)
redis-server.exe!rdbSaveRio(c:releaseredissrcrdb.c:814)(0x40167210, 0x00E10000, 0x00000005, 0x01212754)
redis-server.exe!rdbSave(c:releaseredissrcrdb.c:884)(0x00E10000, 0x00E10000, 0x6303BD74, 0x00000005)
redis-server.exe!QForkChildInit(c:releaseredissrcwin32_interopwin32_qfork.cpp:337)(0x00000005, 0x00000000, 0x004DEF10, 0x00000005)
redis-server.exe!QForkStartup(c:releaseredissrcwin32_interopwin32_qfork.cpp:515)(0x00000006, 0x00000000, 0x00000000, 0x004C5720)
redis-server.exe!main(c:releaseredissrcwin32_interopwin32_qfork.cpp:1240)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
redis-server.exe!__tmainCRTStartup(f:ddvctoolscrtcrtw32startupcrt0.c:255)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
KERNEL32.DLL!BaseThreadInitThunk(f:ddvctoolscrtcrtw32startupcrt0.c:255)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(f:ddvctoolscrtcrtw32startupcrt0.c:255)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(f:ddvctoolscrtcrtw32startupcrt0.c:255)(0x00000000, 0x00000000, 0x00000000, 0x00000000)

3. 9488] 01 Sep 20:15:06.152 # fork operation complete
[9488] 01 Sep 20:15:06.792 * Background saving terminated with success
[9488] 01 Sep 20:16:52.040 * 10000 changes in 60 seconds. Saving...
[9488] 01 Sep 20:16:52.118 * Background saving started by pid 16328


=== REDIS BUG REPORT START: Cut & paste starting from here ===
Redis version: 3.2.100
[16328] 01 Sep 20:16:52.603 # --- EXCEPTION_ACCESS_VIOLATION
[16328] 01 Sep 20:16:52.618 # --- STACK TRACE
redis-server.exe!LogStackTrace(c:releaseredissrcwin32_interopwin32_stacktrace.cpp:95)(0x0014E260, 0x0014FF60, 0x0014E260, 0x4013A7F8)
redis-server.exe!UnhandledExceptiontHandler(c:releaseredissrcwin32_interopwin32_stacktrace.cpp:185)(0x40028F20, 0x40028F20, 0x0014E260, 0x00639FE8)
KERNELBASE.dll!UnhandledExceptionFilter(c:releaseredissrcwin32_interopwin32_stacktrace.cpp:185)(0x00000000, 0xAF0A3090, 0x00000000, 0x00000000)
ntdll.dll!memset(c:releaseredissrcwin32_interopwin32_stacktrace.cpp:185)(0x0014EEF0, 0x9F99D2F0, 0x0014EEF0, 0x0014E828)
ntdll.dll!_C_specific_handler(c:releaseredissrcwin32_interopwin32_stacktrace.cpp:185)(0x00000000, 0x0014E810, 0x00000000, 0x40000000)
ntdll.dll!_chkstk(c:releaseredissrcwin32_interopwin32_stacktrace.cpp:185)(0x0014E810, 0x00000000, 0xADF1BC3C, 0xADE70000)
ntdll.dll!RtlWalkFrameChain(c:releaseredissrcwin32_interopwin32_stacktrace.cpp:185)(0x00610820, 0x00000104, 0x0014F1F0, 0x016FB210)
ntdll.dll!KiUserExceptionDispatcher(c:releaseredissrcwin32_interopwin32_stacktrace.cpp:185)(0x40094F01, 0x553D60C9, 0x400ACC91, 0x0014F262)
redis-server.exe!dictSdsHash(c:releaseredissrcserver.c:496)(0x553D60C9, 0x400ACC91, 0x0014F262, 0x45942174)
redis-server.exe!dictFind(c:releaseredissrcdict.c:517)(0x0014F1F0, 0x00000001, 0x0006BDB3, 0x00000001)
redis-server.exe!getExpire(c:releaseredissrcdb.c:871)(0x18BFB7A0, 0x0014F2B0, 0x0014F2B0, 0x03C07040)
redis-server.exe!rdbSaveRio(c:releaseredissrcrdb.c:814)(0x40167210, 0x00DD0000, 0x00000005, 0x011D2754)
redis-server.exe!rdbSave(c:releaseredissrcrdb.c:884)(0x00DD0000, 0x00DD0000, 0x6303BD74, 0x00000005)
redis-server.exe!QForkChildInit(c:releaseredissrcwin32_interopwin32_qfork.cpp:337)(0x00000005, 0x00000000, 0x00601140, 0x00000005)
redis-server.exe!QForkStartup(c:releaseredissrcwin32_interopwin32_qfork.cpp:515)(0x00000006, 0x00000000, 0x00000000, 0x005E5720)
redis-server.exe!main(c:releaseredissrcwin32_interopwin32_qfork.cpp:1240)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
redis-server.exe!__tmainCRTStartup(f:ddvctoolscrtcrtw32startupcrt0.c:255)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
KERNEL32.DLL!BaseThreadInitThunk(f:ddvctoolscrtcrtw32startupcrt0.c:255)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(f:ddvctoolscrtcrtw32startupcrt0.c:255)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(f:ddvctoolscrtcrtw32startupcrt0.c:255)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
[16328] 01 Sep 20:16:52.649 # --- INFO OUTPUT
[9488] 01 Sep 20:16:55.728 # fork operation failed
[9488] 01 Sep 20:16:56.337 # Background saving terminated by signal 1
[9488] 01 Sep 20:16:58.087 * 10000 changes in 60 seconds. Saving...
[9488] 01 Sep 20:16:58.243 * Background saving started by pid 11172
[9488] 01 Sep 20:17:33.135 # fork operation complete



Solution

Check the Answers

This Question and Answer are collected from stackoverflow and tested by JTuto community, is licensed under the terms of CC BY-SA 2.5. - CC BY-SA 3.0. - CC BY-SA 4.0.

people found this article helpful. What about you?