I've always gotten into go on a high level and would love to delve in deeper. Coming from a java/js background, my pointers are a bit rusty (and I try to avoid them as much as possible). Would it be possible to use go without the use of pointers at all?
No, but pointers in Go are much gentler than pointers in C (for instance: there's no pointer math, and no pointer casting). If you understand the difference between passing by value and passing by reference (something you already have to understand to use Javascript), you've got all the background you need.
You will probably need to acknowledge their existence, but the good news is they're quite a bit easier to use than C and there are far fewer foot guns (i.e. no pointer arithmetic or any such thing). They also deference automatically so they're more ergonomic to write (i.e. you can call functions on a pointer as if it was a normal struct value, and there is no "." vs. "->" distinction like in C-family languages).
I'd suggest just getting in and trying to write some. It'll take a little bit of playing around, but the language is quite intuitive and I'd expect that you'd figure it out pretty quickly.
Most meaningful applications will require the use of pointers. For example, unmarshalling json comes to mind.
I wouldn't "avoid the use of pointers" if I were you. Rather, use them, run into problems, fix the problems, and you'll learn what you need to know about pointers in Go. Given your background, you code. Figuring out pointers in Go will be trivial for you, especially when you learn in context (e.g., writing something you think is cool). You got this.
AFAIK Go pointers are closer to Java references than they are to C or C++ pointers. They mostly make the difference between "references" and "values" clear and flexible (though they do have pitfalls, value receivers versus pointer receivers are a big one) rather than move that to type-level semantics (as C# does).
No, but also you use pointers for a far narrower set of semantics in Go than you do in languages like C. I think you'll find that if you have a grasp of reference semantics in java and javascript, then Go's pointers won't cause you much grief.