Why am I having elevated lymphocytes?

Hello doctor,

For the past five years, my absolute lymphocytes have been 4.2 to 4.9 range, never going over 5 (I have had multiple CBC and every single one of my absolute lymphocytes is elevated). However, they have been continually elevated to the point that I was recently sent to a hematologist who initially thought I had CLL, but the flow cytometry test came back negative, then he did a T-cell gamma rearrangement test, which also came negative. At that point, he decided it must be autoimmune and sent me back to my PCP.

Several years earlier when all this started, I woke up with joint pain, and swelling and was sent to a Rheumatologist who decided I had RA despite every RA test being negative, and the joint swelling not following what is clinically seen in RA (my joints were not bilaterally swollen, red, or hot. They were randomly swelling in my hands, feet and were really painful). So, I refused to take Methotrexate because I had no positive RA tests, and after about a year, the swelling went down, the pain faded away and I have had no other flares since that time.

So, my question is, should I try to get to the bottom of my chronically elevated absolute lymphocyte and if so, are there any other tests I should have my PCP pursue? I do not want to blame it on the RA when there is no proof I ever had RA but at this point, five years of elevated absolute lymphocytes are concerning to me and I feel like it is indicative of something, but I do not know what.