Forums: Index > Rating Committee > Proxies?

After stumbling across a 4e article on the list of rated pages, it occurred to me that it will probably never get more than a few favors; most of us are not terribly familiar with 4e. What does everyone think of a RC member asking for a volunteer to serve as a proxy (subject to the Rater's approval) on a given article when the Rater feel they lack the knowledge to rate it? --DanielDraco 14:35, January 22, 2010 (UTC)

We really need a more comprehensive overhaul of the RC. Surgo 15:22, January 22, 2010 (UTC)
We definately need a solution to the whole 4e rating issue... So far, the Arachonomicon has only had one rating (is that the 4e article you mean?), and I am fairly sure that it will be the same for all other 4e articles. The only thing is, I can't think of a solution; there certainly aren't enough regular 4e contributors for a second comitee, and the proxy idea could lead to one or two people rating the sam article several times for a lack of enough proxies. -- SamAutosig Sam Kay   talk    contribs    email   12:02, January 31, 2010 (UTC)
The problem with the 4e articles is pretty bad, but I think it's bigger than that even. The raters that we have tend to be distracted with writing their own stuff or working on site background or just not present as often as they thought they would be back when this was first discussed. I'm actually in favor of just dumping the RC entirely and letting any registered user favor stuff, we have regular contributors who are not on the committee and don't get to help boost material on the wiki as a result. I don't see what having the ratings restricted to a small group of increasingly busy people does to help us remain credible that tagging articles with raters and balance doesn't. So, since I think the issue mentioned here is bigger than discussed and potentially intractable, let's just open it up and let our active users rate stuff. The Rated By property on the article page will still let other users find material rated by and supported by particular individuals, and that in combination with balance levels should let people find material that they can trust to be reviewed / balanced in a consistent way. - TarkisFlux 20:13, January 31, 2010 (UTC)
That's definitely worth a try; this RC thing isn't working as well as we expected, because we're not all rating the same articles and they therefore rarely get higher than bronze. If we were to start casting our two cents to what others have rated (I, for example, have thus far only rated things that others have already rated), then it might work. Failing that, opening it to all users may work better, perhaps reassigning the RC to the duty of making sure that ratings are well-thought-out and not simply "ur class sux lol". --DanielDraco 21:09, January 31, 2010 (UTC)
I am strongly against this, I dunno, this jury duty setup has on flaw, what if the people selected for rating don't have a good enough understanding of the rules to spot a fatal flaw in the design of a page, or what if they simply can't put aside a personal bias against a certain type of material. I for one despise tome material and I might rate a tome feat badly despite its great design simply because I can't see past it being a Tome feat. I think instead that RC members should get messages suggesting that they rate page so and so, maybe stopping after they rate three or more pages, that way they would remember to get things done. Or maybe we should just find new raters if the current ones are too busy.--ThirdEmperor 02:26, February 1, 2010 (UTC)
My idea was to just let anyone with a gold-rated article rate shit. Surgo 02:35, February 1, 2010 (UTC)
@TE: The thing is that people are lazy, and without an incentive to do something, they rarely do it. There's little-to-no incentive in favoring articles, so people don't really do it much, whether they're busy or not. --Ghostwheel 02:43, February 1, 2010 (UTC)
Don't kill me for this but if you need incentive I'm sure we could rig something up to stop you guys from editing anything until you rate something each day. Oh, and problem with Surgo's idea is that pretty everyone who has a gold rated article is already on the committee.--ThirdEmperor 03:04, February 1, 2010 (UTC)
No, no, no. A reward for rating is not the same thing as a punishment for not rating. Surgo 03:33, February 1, 2010 (UTC)
All right, that was partly a joke, although we should think of a way to get people to rate stuff, I for one work best when I've got no other choice, maybe you have to rate X articles or you lose RC status? But that's pretty close to my last one, maybe alternating rating duty among people who are agreed upon as reliable and knowledgeable?--ThirdEmperor 04:23, February 1, 2010 (UTC)
@3E - You can't actually rate things badly in our setup. The worst you can do is rate it a 0, which is only an official stand on not rating it at all, which is basically what's happening right now anyway. So even if you hate Tome stuff you can't actually hurt it's ratings. And Surgo's idea isn't bad because it keeps the same committee members (I'd be out, but I'm thinking about leaving anyway so I can do site crap / article cleanup and my own stuff without guilt) since it also opens the door for basically infinity more. So even if half of those people aren't moving things forward, the other half can bring an article up to gold to bring more active people onboard. Which doesn't sound like a bad way to go at all.
I've also tweaked the RC Favor template to accept ratings without reasons, which should take some of the work out of giving approval to an article. It should probably be avoided early in the rating process, but it makes parroting what everyone has already said more convenient (and thus we might get some more ratings out of people). Also, we should absolutely stop pulling articles from the favor list after they hit 3 ratings. I don't remember who started doing that, but it was a terrible plan. - TarkisFlux 05:09, February 1, 2010 (UTC)
Fair enuff, but I officially reserve the right to say I told you so if this goofs up.--ThirdEmperor 05:17, February 1, 2010 (UTC)
Wow, lots of discussion here overnight. There are two things said which I want to vehemently say are terrible ideas.
Firstly: if you cannot get past your own predisposed notions of what the best balance point is (e.g. by being unwilling to give tome material a good rating, even though it hits its high wizard target), then you should not be on the rating committee. Our job is to judge on how well-written it is and how well it hits its target, not how well it hits our own preferred target.
Secondly: restricting rating rights to only users with gold articles excludes those of us who are better at examining work than creating it; some are meant to be critics, not writers. --DanielDraco 23:22, February 1, 2010 (UTC)

→Reverted indentation to one colon

DD, if we eliminate the RC or open it up to anyone with a gold rated article, then it doesn't make any sense to talk about people who shouldn't be in it. They're either in it, by meeting whatever selection criteria we have, or they're not and at no time does suitability or non-bias requirements enter into that. I don't think that we would suffer from having raters who just don't rate certain levels of material though, since they can't actually do any harm to other ratings even if they hate it. TK's Variant Weapon Focus is a great example of what happens when you think an article is OP for it's balance point. You put in your 2 bits and don't give it any points. You can't actually hurt things in our current setup, you can just say why you think it's junk, argue your position if you need to, and then move on. And that's the worst that you can do to an article, even if you think it's complete junk and you want the minutes you spent reading it back. Which actualy does leave us open to a group of people rating something inappropriately and boosting it to a stupid place (and thus could lead to some bizarre things in the Featured Article queue), but that's about the worst of it. - TarkisFlux 01:18, February 2, 2010 (UTC)

The current system does not work with an unlimited number of raters. It would need to be changed back to a system similar to youtube star ratings or what we had on the paleowiki (except probably without the categorization), because that is really the only practical way to handle an unlimited number of voters. In such a system as that, biased ratings certainly are an issue. --DanielDraco 01:30, February 2, 2010 (UTC)
Any system change must be forward-compatible with existing rates. Surgo 01:54, February 2, 2010 (UTC)
I don't see why the current setup fails in a larger raters world. The numbers maybe need to change, but what is unworkable about it in general? And I really really don't want to move to an averaged rating system a la you tube. I'd rather scrap the ratings entirely and just have a couple of properties to track the names of people who like / dislike the material in question, and then not actually do anything with those lists aside from allow people to find material approved of / disproved of by a particular user, than do what the old wiki tried to do.
Also, what does forward compatible mean? Compatible with itself? Compatible with the thing we would replace it with? I think you mean backwards compatible, but I'm not actually sure. - TarkisFlux 02:13, February 2, 2010 (UTC)
I like the idea of openning up the rating to everyone, for sure, but the idea of restricting it to people who have gold articles, runs the risk of this turning into a popularity contest instead of an actual attempt to balance material. As for the RC, I think it would still have reason to exist, even if just to make sure that ratings that are given are legible, logically valid, or are removed. If that purpose for the RC was added in, then numbers wouldn't matter, though, for each 9 points higher than FA an article got, perhaps would increases it's likelyness of appearing in the FA qeue? → Rith (talk) 04:07, February 2, 2010 (UTC)
The current setup fails with infinite raters because the thresholds are finite. We would need to be constantly adjusting the thresholds. Even if we do that, it puts an unfair advantage on older articles; if the thresholds are always increasing, it makes it more and more unlikely for a newer article to ever get the high rating that it might deserve. If the number of possible ratings is infinite and the total favor on a single article can range anywhere from zero to infinity, then the thresholds need to be adjusted to unique levels for each article to remain fair; this means a proportional scale of thresholds, which is exactly what youtube has with its 11 stages of rating (in increments of a half star). We need to choose between continuing to fiddle with our current system (finite votes and constant thresholds) or a youtube-esque system (infinite votes and scaling thresholds), though the latter can be modified by only allowing intelligent votes. People seem to have significant issues with both of these possibilities. I personally am in favor of a youtube-esque system with vote moderation, so that anyone can cast their vote as long as they can justify it. --DanielDraco 20:45, February 2, 2010 (UTC)
I don't see why the current setup would fail with lots of raters. Infinite raters sure, but we don't have infinite raters. Surgo 22:41, February 2, 2010 (UTC)
Right. I think we disagree that infinite voters will lead to infinite votes that would require some sort of scaling. I think it more likely that voters will leave, become disinterested, or just generally not exercise their voting ability to any large degree (ya know, like now) and that this would leave us with a fairly reasonable average number of votes per voter. My guess is that the number of articles will grow in proportion to the number of voters and that coupled with newer articles (or at least more active articles) getting more attention (and thus favor) than older ones will spread the favor around fairly evenly and in a way that doesn't destroy the benchmarks we have set or unfairly boost old over new. It's not idiot proof, but I think your assumptions about user behavior are wrong and lead you to see this as worse an option than it actually is.
Anyway, we already explicitly rejected the youtube-esque averaged systems for problems that it brought with it. Moderation ameliorates a number of those concerns (and setting a minimum number of votes before it can get any rating at all might cover the rest) but requires a crap ton more admin work to keep those things clean. I don't think we have the dedication or manpower to pull it off. - TarkisFlux 22:52, February 2, 2010 (UTC)
"Infinite" was the wrong term to use. "Boundless" would have been better. The problem isn't that there is an impossibly large number of voters, it's that the number of voters will always be increasing.
You assume that I have assumed. Take another look at youtube for example; looking solely at the age of the video and the number of votes on any random sample of videos, you will clearly see that older videos will generally have more votes.
Video A is five years old and has 100,000 total votes. For simplicity, it has an even distribution of vote values: 20,000 for each star value 1 to 5. Video B is two years old and has 40,000 total votes. It has an even distribution: 8,000 for each star value 1 to 5. Because youtube uses proportional thresholds, both videos have equal ratings despite the head start that Video A has. If the thresholds were static, Video A would be of a vastly greater rating than Video B (despite being generally received exactly as well as Video B), and Video B would be unlikely to ever catch up.
Put simply, I don't see how we can possibly make boundless votes work with static thresholds. --DanielDraco 23:46, February 2, 2010 (UTC)
I think that the extra work required to vote on something in this setup, the smaller user base, and the differing demographics combined with a setup where you're wasting your vote if you vote on something that is already voted up to a level you agree with will sort out the age issue, but I don't have any actual data to point at. So I'll give you that it's potentially problematic or even unworkable, and table it for now. That still leaves unanswered criticisms for the youtube model you're proposing that make it look like a non-starter. I'd like to see arguments that it would actually work in a wiki format lacking vote counting / placing automation, that the small voting group problems it invites can be dealt with satisfactorily, and that the moderation will actually occur with any frequency or reliability. - TarkisFlux 00:38, February 3, 2010 (UTC)
It was actually working quite well on the paleowiki. The only part that wasn't working was the idiotic categorization of it, which forced all aspects to be put on equal priority. Beyond that example, I'm afraid I can't provide much argument. --DanielDraco 00:51, February 3, 2010 (UTC)
Okay, I'll try to approach this sensibly and without pissing everybody off (fat chance), first off I totally agree with DD on everything but that it worked well on the old wiki, particularly about how if you can't put aside your bias against of for a particular type of material you shouldn't be a rater. This is the exact point I was trying to explain, some people shouldn't be raters even if they somehow manage to get a gold star, me being the the main example. What's worse, although restricting it to people with gold stars will prevent it from getting out of hand at first, eventually as more and more people rate stuff more and more stuff will get rated gold and it'll hit critical mass before you can say d% damage to common sense. Averages would work except you have the problem that bad votes do nigh irreparable damage to somethings ratings and when everyone can vote you'll get a lot of ill thought out ratings. I think the best solution is to get different raters, but fact is we've already got the best folks for the job so I don't really now how to fix this. Anyway there's my two copper pieces.--ThirdEmperor 10:27, February 3, 2010 (UTC)
Here is my two cents about the discussion. Firstly: Going back to the old wiki's rating system might not work as I found a couple articles where one user rated 5 on everything but by like 3 others was rated 2s and 3s, yet it showed up as 5 for everything on the main article and one had to look at the discussion to find the more realistic ratings, this is not a particulary good argument and I know that , but do we want the same thing to happen here on this wiki? even to one article. Secondly: Why not have a Rating Election page or some such were possible candidates are selected based on how balanced they rate some pages and if they do well enough they are given rating capabilities, if not there ratings are stricken from the record. Thirdly: You could elect a certain day of the week Ratings Day where you rate like 5 or ten articles and then can do whatever you want. Fourth and Final: why not instead of arguing over the ratings you could actually RATE something, just for a change, might be an idea. With many smiley faces. --Stryker-Fyre 12:10, February 3, 2010 (UTC)
I might be new here in this wiki but I'll still express my opinion (the worst that can happen is everyone ignore it, so... free expression ftw). I've recently came back to activity in the D&D comunity and I found out what happened to the old wiki. Even though that was horrible, one thing that didn't went wrong (in my opinion) was the rating system. As for what I've seen so far, there are a few raters over here. If everyone could rate in the same system that there was in the paleowiki (I've found this term very funny) (and I don't mean the SAME system, just the giving a numeric note with a justification, open for anyone), we would have a good number of well thought ratings and a few "ill thought" ones. If a rating was unfair (as a maximum rating for a crappy article or a crappy rating for FA material), the RC would have the power to nullify it. Other than that, uncoherent or unjustified ratings would automatically be nullified. The authors could update the ratings to the pages, keeping the RC from getting overloaded.
Then the ratings would be averaged, say a score from 6~8 would be bronze, 8~9 would be silver and 9~10 would be gold. For the FA, there could be like a voting amongst the best rated gold ones and the articles would slowly being "promoted" to FA. Let's be real, we have a quite mature community from what I've seen and giving a justified coherent rating take sometime, so that people wouldn't spend that time just to give a hideous rating and get it negated. --ElfsMaster 23:12, February 9, 2010 (UTC)
I agree. I never really saw anything wrong with the paleowiki's system other than the but-heads using it (I mean Celestial Dire Lions as a FA? Really?).--ThirdEmperor 05:21, February 10, 2010 (UTC)
Indeed, the only issues were the equal-priority categories and the unintelligent ratings -- and the latter issue wasn't much of a problem, because we were actually doing a fairly good job at weeding those out. --DanielDraco 23:53, February 11, 2010 (UTC)
Community content is available under CC-BY-SA unless otherwise noted.