It might be normal, really it depends on what the fan model and what is normal for that fan in terms of idle and maximum RPMs.
While that RPM does not seem particularly high for a laptop fan, if it WAS high then this means that either there is a problem with the thermal sensing capability on the laptop, whether due to the fan controller or the thermal sensor itself, has failed, and is defaulting to full speed operation, or the CPU itself is actually overheating because it has been thermally degraded, or there is dust and dirt built up inside the unit, or the thermal paste has become dried up enough that it is not longer working effectively, OR the fan itself has some kind of issue but it would seem unlikely to be the fan itself since the RPMs actually seem pretty normal in terms of being pretty middle of the road for a laptop fan.
My guess is that either in the BIOS, or in the Windows advanced power plan settings, there is a setting which is set to active cooling policy rather than passive cooling policy, and to be honest this is probably preferred. If the fan level is DIFFERENT than it used to be and you've made no changes, then I'd recommend getting a can of compressed air and blowing out the dust from each vent in the sides or bottom of the unit. Alternatively you might consider taking it to a repair facility and having them thoroughly clean any dust out from inside the unit and then remove the old paste between the CPU and heatsink and replace it with fresh paste. This MIGHT be something you can do yourself if you are very handy, and highly technically inclined, but it is not usually an endeavor for those lacking experience or are faint of heart. It's easy to completely ruin a laptop by disassembling it. There are many ways it can go wrong for the uninitiated.
Knowing the EXACT model would certainly be helpful.
I am using a Dell Latitude E6440.
I upgraded the CPU from i5-4310M(4 threads) which is 37 watts to i7-4800MQ(8 threads) which is 47 watts.
The fan that came with the i5-4310 was 0.32A with a CFM of 2.3.
The fan that I installed for the i7-4800MQ is 0.4A with a CFM of 4.7
This is what I get: https://ibb.co/Tw7GDfN The fan rarely stays at 0 RPM when idle.
I eventually want to install Linux, under Linux I can use tlp to lower MAX CPU freq or something...
Did you change the BIOS to a version that included programmed support for a different fan and CPU? Laptop fans should NEVER stay at 0 RPM, whether idle or not. They should always be running to some degree. There is no way for anything to be cooled passively EXCEPT for the system throttling the GPU or GPU otherwise. Fan should always run. Period.
Whether it should always run at THAT RPM, is anybody's guess, because you have a CPU and a fan that are not the same as what the BIOS was programmed to include data for, and that changes everything. If the fan is not loud enough to be annoying, I'd forget it as long as it is not overheating. THAT is the primary concern. If the fan behavior annoys you and the temperature is low enough at idle or low speeds to accomodate dropping it lower, then you might want to ask around or look over at https://www.bios-mods.com/ to see if anybody has or is willing to make a modified BIOS for that unit. You might even be able to simply GET a BIOS version from the same model of laptop that you have now, but from a submodel that DID come with that CPU and fan.
No, I'm saying that the hardware tables for what actual amount of airflow exists at WHATEVER RPM it is running at, won't be the same because the BIOS you have was not meant for use with any hardware other than what was installed from the factory.
It's like putting a different size of tires on your vehicle. It instantly makes the speedometer wrong. You have to either have the PCM recalibrated to account for the different size of tires or on older vehicles, change the speedo gear in the transmission to account for it.
But if you don't have a problem, then it's probably not worth bothering to worry about. If the idle and off idle noise levels are tolerable and if the maximum temperature is within specifications, then I would probably not worry about it.