During yesterday evening's D&D game, I tried out a test-implementation of a variation on the AD&D friendly-fire rule, with an extra bit of complication whereby ranged attackers could either aim for specific targets at a penalty, or just shoot into the group and hope for the best. It worked pretty much as expected from a simulationist point of view; but it was indeed on the complicated side from a running-the-game perspective. Overall, I liked it—it worked—but it was clunky enough that it could use some serious refinement.
In researching the subject of missile-fire in D&D, I've oft considered implementing some of Delta's tweaks to the rules, or a variation on the same; I agree, for example, that the chance to hit a target at medium or long range ought to drop off precipitously. In my opinion, though, the best way to model this isn't hefty penalties on the to-hit roll: it's shorter effective ranges with missile weapons. To that end, I'm probably going to implement a new range table next game, along the lines of—
daggers, hatchets, clubs, grenades: 10'/20'/30'*
spears, tridents, darts, harquebuses: 20'/40'/60'*
javelins, slings, short bows, crossbows: 30'/60'/90*
long bows, arbalests, atlatls, staff-slings: 40'/80'/120'*
—with the range modifiers as AD&D (nil at close, −2 at medium, −5 at long), and long-range missile-fire only possible where the ceiling is taller than the close range increment (meaning that in most dungeon corridors and rooms, only daggers, hatchets, etc. can be thrown out to long range, and in rooms with high-vaulted ceilings, so can spears and suchlike; but bows and crossbows will nearly always be limited to medium range indoors). Beyond long range for a given weapon, single shots are effectively so inaccurate that they always miss; a weapon's extreme range (which could be upwards of ten times the weapon's listed "long" range in feet) only matters where massed volleys come into play in large battles.
Now this provides a good excuse to do away with the largest source of complexity that I introduced with yesterdays' test-rule. High-arcing volley fire doesn't happen in D&D's sort of small-scale, man-to-man skirmish. Missile attacks are thus always assumed to be direct rather than indirect fire, with the shooter aiming to hit a specific target. But that doesn't change the fact that a mêlée is a dynamic situation, that any participant might dart into the path of a missile at any time, and that the chance of friendly fire should always be at least significant.
One good way to model this, of course, is Basic D&D's own rule for soft cover, which I mentioned yesterday. This is a penalty that ranges from −1 for quarter cover to −4 for full cover, very simple and easy to implement. Now if we treat other creatures in a mêlée (friend and foe to the shooter alike) as potential "cover" for the shooter's intended target, we can make the reasonable assumption that because everyone is moving, the target will never have full cover. Also, shooting farther means more flight-time for the missile, which increases the chance that someone will inadvertently dart into its path after the shooter has already loosed the missile.
That said, the armor class of the intended target can have no bearing on whether friendly fire happens, so any rule that says "friendly fire happens if the initial shot was a miss" is inherently flawed. The target wearing plate armor does increase the likelihood of a miss; but it should not increase the likelihood of a stray arrow hitting an adjacent friendly.
So now I'm thinking that the penalty for shooting into a mêlée should range from −1 to −5, with a penalty of −1 to −3 coming directly from the cover of creatures surrounding the target (say, −1 for one or a few other participants in the mêlée, −2 for four or more, and −3 if it's eight or more), worsened by −1 at medium range and −2 at long range (both in addition to the usual range penalties). If the shot on the intended target misses because of this net cover/range penalty (so anywhere from 5% to 25% of the time, depending on the circumstances), friendly fire (or a shot on some different enemy, it's the same difference where this notion is concerned) is possible.
Ordinarily, I really don't like mechanics which turn pass–fail task-resolution rolls into gradations of success or failure. I find them far more cumbersome than just rolling a different die. But in this case, I have to admit that it does fit the circumstances. It correctly models that the intended target's AC isn't actually affecting the chance of friendly fire happening (outside of rare corner-cases that result in a 95% chance to hit or miss because it's only possible on natural 20 or natural 1, but I'm ignoring those sorts of extremities for the moment). And it will make logical sense to the players: they missed their target because someone else got in the way—cover, just as if they were shooting at an enemy peeking around a corner or ducking behind a table.
As for actual resolution of the friendly fire, here I'm inclined to go with rolling a pair of dice, one to pick the target at random (again possibly accounting for size differences, but most of the time it won't matter too much beyond saying that big monsters are twice as likely or more to be hit than man-sized creatures, and halflings are half as likely), and a d10 alongside it to see if the arrow penetrates the unfortunate's armor (d10 rather than d20 because it's quick, it's easy, and a d20 roll has already been made in the first place by the shooting player to hit the alternate target).
For example: an archer with adjusted (for Dex, magic, etc.) THAC0 12 is shooting at an ogre (AC 4) which is in a mêlée with two hobgoblin buddies (AC 6) and two of the archer's friends, a fighter and a dwarf (both AC 3 when ignoring their shields). Say the archer is shooting at medium range, which bumps his THAC0 circumstantially up to 14. Ordinarily, then, the archer would have to roll 10+ (THACO 14 – AC 4) to hit the ogre; but he's shooting into a mêlée with four other creatures at medium range, and so the DM rules that the ogre has one-half cover and imposes a −3 penalty on the shot. Now the archer will hit something if he rolls 10+, but he needs 13+ to nail the ogre. Say that he rolls a 10; welp, the DM rolls 1d8 and 1d10 together (considering the d8 to be 1–4 hobgoblins, 5–6 fighter, 7 dwarf, 8 arrow goes over dwarf's head). I've rolled some dice just now and gotten 1 on the d8 and 4 on the d10, which means that the stray arrow hits a hobgoblin, and, 4 being lower than its AC 6, causes damage.
This is certainly simpler than the rule I'd discussed yesterday, and I'm hard-pressed to find a situation where I couldn't apply it (again, beyond "only hits on a 20" or "only misses on a 1, even with the penalties" sorts of rolls; ad hoc rulings would have to apply in either case). That said, I won't get to test it out in actual gameplay until next Saturday—so we'll see how it fares in a week's time.