I get where this is coming from, but as someone who recently did an extensive systematic literature review: you benefit from doing the work, not from getting an automatic summary. It's the little details you keep stumbling upon, that make you think "Wait a second!", that are really important. You miss them the first 100 times you come across them, but by the 101st time, you have learned something.
You’re right. I did a singular literature review myself during my masters degree, and much of what I learned in that period has been really beneficial, especislly the nuances.
But having an extensive summary or table of contents generated for you to begin your review? Priceless and would have saved me so much time especially on the junk papers. There was a demo recently at work where they built a pipeline to do literature reviews (topic was not scientific, more data analysis) and generate a report. It was genuinely incredible.
How about if you have set aside four hours for a literature review, and you use LLM assistance in the first hour to narrow down the best options for things to spend the next three hours diving deeper into?
Regardless of how much time you have for your literature review, you likely are doing multiple passes through each paper. On the first pass you'll likely just read the summary and skim through the sections to see what kind of argument is being made. After that, many people do another read through to identify the main points of each argument. It's generally the 3rd or later read through that's a deep reading (and likely a final one where you read critically and look to see if you were going to make the argument would it be the same or are there potential logical issues).
LLMs can help with those early reads and save some time and get you into the deep reading sooner with more context. If during the deep reading you would normally jump back to a previous section to check something, it's likely faster and easier to just have a conversation with the paper (enabled with an LLM). The same would be true for that final read where you're doing logical checks.
If you use an LLM to give you a summary and leave it at that, you'll have done the equivalent of the first pass through a paper. That could be enough for you to know you want to filter it out and not do a deep reading, but you'll lose the benefits of the deeper reading. It seems like there are clear benefits and areas where LLMs can help improve that current paper reading process but if you skip (instead of just replacing with a more efficient LLM alternative) major parts of that process you'll get less out of it than you would without skipping those steps.
I get where this is coming from, but as someone who recently did an extensive systematic literature review: you benefit from doing the work, not from getting an automatic summary. It's the little details you keep stumbling upon, that make you think "Wait a second!", that are really important. You miss them the first 100 times you come across them, but by the 101st time, you have learned something.