The primary determining factor is driver skill, but comparing two driver's skill with such a huge %-difference in total kart+driver weight is impossible, due largely to major kart setup differences required. Your seat position and fluid weight distribution (body weight is fluid and moves around a lot) will greatly affect the balance of weight distribution under varying loads such as cornering. Differences in driver height also affect the lateral load on your tyres (overall tilt of the kart and axle), etc.
A general guide at the sharp end of the driver skill-stick (when you're approaching optimal performance with your setup), a difference of 5-10kg can/will affect approx 1 tenth of lap time.
You have to realise however that it's not a linear curve - there is an optimal weight zone.
Heavier weight will almost always lower acceleration, but can improve handling if correctly distributed.
That is to say that a super light driver+kart (significantly below race-regulation weight) can gain handling-performance by adding small amounts of weight correctly. Eventually the lap-time gains from adding weight fall off, whereby continued weight gain will begin lowering performance again. Basically, this is to say that you need a certain amount of weight (correctly distributed under load) to cause the tyres to perform optimally.
At the end of the day, it always comes down to how those tyres are being used, and what condition they're in.
I couldn't answer your question conclusively or succinctly, but I hope I've tickled your thoughts far beyond the weight question.