Earl Brooks is a bespectacled, bowtie-wearing businessman with a modern-architecture mansion in Portland, Oregon. Still, he clings to the Serenity Prayer like a kid would a security blanket.

But is that God to whom Earl speaks with talk of surrendering to His will? Or is it Marshall, a hallucinogenic manifestation of Earl’s inner demon that makes him enjoy one murder at a time?

Mr. Brooks is filled with bravery: casting unfailingly likeable Kevin Costner in a radically different role as a presumably conservative, overtly pro-life closeted serial killer; the straightforward conceit of Earl conversing with a backseat-riding homicidal id (William Hurt); trusting Dane Cook to flex more than his super-finger, playing a scumbag photographer.

Apart from commingling religion, mental illness and pervy violence, writers Raynold Gideon and Bruce A. Evans (who also directs) unapologetically explore clinical, ritual, sexual, prideful and shameful natures of killing. You know, the same approach they took with Jungle 2 Jungle.

Costner wisely never overplays Earl as a psychologically unhinged man, and there are plenty of moral thorns here on which to prick a finger. It’s balanced with black comedy, courtesy of how devilishly Costner and Hurt play off each other. Synchronizing movements (and facial scrunches, at times), the two find chillingly casual humor in applying control and precision to ending a life.

There’s unsettling exhilaration in the way Mr. Brooks seems prepared to plow forth through similarly dark waters as American Psycho. Much like Christian Bale’s Patrick Bateman, it’s fascinating, captivating nastiness that you neither root for nor against.

It damns every torpedo — except for the one that sinks it.

Demi Moore plays Tracy Atwood, a top cop with issues as laughably boilerplate as Earl’s are complicated. Independently wealthy, Atwood is an ambitious policewoman just to shake her daddy-wanted-a-boy past, and she’s divorcing a catty chef who wants a big piece of her financial pie. (Reiko Aylesworth wastes her talent on the nothing role of her hubby’s attorney.) There’s also an escaped convict (Matt Schulze) who would like to see Atwood dead, with whom every encounter is accompanied by deafening gunfire and ridiculous techno music.

If this sounds grafted from a dumber movie, it feels even more while watching. Rather than make her the lawman yin to Earl’s lawless yang, the movie dedicates too much time to her singular story, and Moore’s stiffness stops the movie dead.

Meanwhile, potentially fascinating subplots involving Earl die on the vine.

Sloppiness at a crime scene causes Cook’s character to capture Earl’s latest evil deed on film. In exchange for his silence, he wants in on Earl’s next kill, a step-up in despicable behavior for this slimy voyeur. (“Even if that guy was charming and funny, I wouldn’t like him,” quips Hurt.)

Also, Earl’s daughter, Jane, (Danielle Panabaker) has interesting reasons for dropping out of college. Where Evans and Gideon should crank up home-front tension (Knocked Up isn’t the only movie this weekend with an awkward breakfast conversation), they instead bog it down with a plot turn that makes Earl seem more like a Hannibal Lecter-esque superhero, less like a quiet killer.

There’s the germ of a great movie in Mr. Brooks but the disease of several stupid ones. Ultimately, Evans and Gideon lack the wisdom to know the difference.