Go to the router's manufacturer's website and lookup firmware updates. It should have instructions.
I'll preface this by saying that firmware upgrades are worthwhile because they often fix security vulnerabilities and performance issues.
They are not a beginner's thing to do, but with a little research to boost confidence they're not a big deal. The potential is there for something to go wrong and then need to factory reset the router and then set it up again. The most likely cause of this would be if the device loses power during the actual process of writing the new firmware image to memory. This is not something that happens very often at all.
Some routers let you check for and upgrade directly from the web interface after logging in to it. Otherwise, yes, a trip to the manufacturer's site to search for support downloads of a firmware image would be next on the list to do. Either way, you'll almost certainly need to be able to log in to that web interface.
To do that you'll need to find your router's IP address. A fast way to do that, assuming a Windows machine is used, is to open command prompt (click Start, start typing "command prompt" and you should see it), in the black window that opens type "ipconfig" without the quotes and press enter, look for the "Default Gateway" value.
Most home routers will have a value such as 192.168.0.1 or 192.168.1.1, but it could be lots of other potential addresses.
Open your web browser of choice and type in your default gateway in the address bar.
Usually you'll see a login prompt of some sort right away. You can guess admin and admin or admin and password (very common credentials to log in to home routers), check the back of your router for the login info, or Google your router info and "default password" to hopefully find what you need to successfully login.
Note - this is for the situation where you have your internet provider's device handing off the connection to your own router. Some folks will be in the situation where the ISP's device is the router as well and you will not have access without a support call to your ISP.
Once you're in the web interface you'll just have to click around to find where to check for or upgrade the firmware. It is sometimes in "Advanced" settings, "System", or somewhere seemingly random.