The most important thing to understand is that Karma is a reflection of your recent match history, and as that history changes so does your Karma score. This means that each match you play removes an older match (along with its karma) from your history, for better or worse.
With this in mind you could play a match where you get no positive or negative votes, but you push a match where you dodged or AFKed out of your history, resulting in your Karma score improving. The same with pushing out a match with a high number of up votes, your Karma score would decrease.
It's also important to be aware that all votes are not counted the same. Many votes have a lower value if used gratuitously, others need confirmation from other players to have full weight; this is to prevent votes from being abused to push another player's Karma up or down artificially. Dodging or AFKing from a match will result in a Karma penalty for that match from the system at full weight.
When Karma gets low a warning will be sent, and if it continues to fall it will result in being placed in the Low Priority Queue.
If your karma gets very high it will become Great Karma and you will get bonus Glory after your matches!