In an earlier column, I discussed the legal ramifications of shrink-wrap agreements on software packaging. But, there is another kind of software “shrink” being used in schools around the country that has far more serious legal and ethical implications.

Based on a written questionnaire developed with federal mental health funds almost two decades ago, the program uses a sophisticated question and answer format to diagnose potential suicide tendencies in students, as well as other psychological problems. The questions are typically read off the computer screen by an interviewer, the responses are entered into the software, and (at the end of the process), the computer provides an assessment. The program is based on considerable research into symptoms and recognizes indicators that are useful in predicting teenage suicide. It is being used in public and private school venues as well as in juvenile detention facilities and facilities such as Boys Town.

Because teenage suicide is a growing and tragic phenomenon, it is hard to criticize a process for discovering the recognized danger signs. Many of the proponents of the software say it is more accurate than unassisted personal (human to human) interviews. It also is supposed to be useful in detecting other mental problems, such as phobias. In addition, if the computer program flags a participant as having suicidal tendencies or other symptoms, a referral is made to a qualified psychologist who makes a final diagnosis and recommendation for intervention or treatment.

Now all this doesn’t seem like such as bad thing, but it raises some substantial legal and ethical red flags.

As we become more “short-cut” obsessed and dependent on computer-aided decision-making in everything from management to buying lottery tickets, every program of this type has latent, but very real, “Big Brother” potential. In the ’60s and ’70s, schools became enamored of “ability testing,” and grouped students in classes according to test results gleaned at an early age. When this “tracking” system was challenged in the U.S. District Court in Washington, D.C., Judge Joseph Waddy handed down a decree in Mills v. Board of Education banning the practice. That case threw out the baby with the bath water, forcing the D.C. Public Schools to end the beneficial use of flexible ability grouping along with the more pernicious and “tagged for life” rigidity of “tracking.” The potential for computer-aided psychological diagnosis is similar.

If the beneficial components of the program are overshadowed by the ease of use, it would be easy to use this software to attach labels rather than provide a means to provide needed help.

The software rarely misses a problem. But it’s designed to be sensitive to potential (rather than obvious) symptoms. As a result, the program is more likely to give “false positive” readings. Careful follow-up can catch these mistakes. But in the hands of lazy, under-trained, or just uncaring practitioners, overdiagnosed students could be “tracked” as potential problems.

There is often a thin line between a well-meaning diagnosis and a stigma, and this line must be carefully guarded. As with the Mills case, too many referrals of students for marginal mental problems can be worse than no screening at all. In the hands of an eager lawyer, mistakes can be magnified and the beneficial aspects of the program overshadowed by Luddite cries of “invasion of privacy” and violation of real or imagined constitutional rights.

So the guidelines for use of these computer-aided diagnostics in the public schools should be clear and strict. Extreme caution must be exercised to avoid labeling, grouping, or any other form of “cutting the weak out of the herd.” No student should ever be put in the position of having to “pass” such a test. Finally, if your school system is not prepared to spend the funds necessary to provide follow-up treatment, or at least referrals to readily accessible treatment, then you are not ready to use the diagnostic tools.

Remember, it wasn’t the use of computers and automation Orwell warned us against, but the mis-use of them.