I guess some drummers can get away with it. I can read notes, though I'm not an on-the-spot, fast reader. But I'm glad I've learned how to read drum charts, even if I'm more of an audio-visual guy. Whenever I encounter written lessons, whether on books or my magazines, I would be able to decode it. I also know a few drummer friends who are dependent on written learning resources, because it can somehow give them a clearer picture of what's happening on the music, compared to just watching or listening to the drum parts played.
Kanya-kanya rin, I guess. A lot of drummers can play very well even without expertise in note reading. In fact, I've witnessed a number of schooled drummers who can read really well, but when it's time to play music with other musicians and make it groove and feel good, they're unable to deliver.
Regarding rudiments, as long as you know the three basic foundations of rudiments - single strokes, double strokes, and flams - you'll do fine. If you want to take your rudiments to the next level, that would be great. And being able to read the rudiment charts correctly and effectively can be an asset.